Take screenshot when a test fails

[gregberge]

As suggested in #43, we would like to have a screenshot when a test fail, it is the same idea as #130.

Technical detail

• The screenshot would be created in a .jest-puppeteer/screenshots folder.
• The name of the screenshot would be ${fileName}-${describe}-${it}-${timestamp}.png

We could redefine it and test but I think this is not the cleanest method. Any help from Jest team to do it properly is welcome @SimenB.

[SimenB]

This is exactly the same as #130 (from Jest's perspective), right? Whether you take a screenshot or not shouldn't matter, as long as you can intercept the error reporting. Unless I'm misunderstanding?

[testerez]

Here is how I did it:

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  jasmine.getEnv().addReporter({
    specDone: async result => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

[elmolicious]

+1 for a much needed feature

[kevlened]

The jest-screenshot-reporter handles screenshots the way @testerez shows. That reporter currently assumes browser uses the webdriver api, but the logic is simple, so you could write something similar for puppeteer or add a contribution there.

[jacobweber]

@testerez Can you tell me where you're calling that code from? I was trying to do it the way jest-screenshot-reporter does, but it seems like the page gets closed before I can take the screenshot.

[testerez]

@jacobweber I just call registerScreenshotReporter() in setupTestFrameworkScriptFile (I'm not using jest-puppeteer anymore)

[jacobweber]

@testerez Thanks! Got it working.

[ricwal-richa]

@testerez On using your custom reporter, I get an error that takeScreenshot is not a valid method. I am using jest-puppeteer so I searched for this method in jasmine2 (as jest-jasmine2 used in jest-puppeteer) but could not find a reference. Can you please tell me if jasmine reporter would also need browser/page handle explicitly for this method to work ?

eg. browser.takeScreeshot() or page.takeScreenshot

I am able to make this work with puppeteer screenshot method (using that in custom defined method); but would prefer to use the in-built support for screenshot available with jasmine reporter, if any.

[testerez]

@ricwal-richa yes, takeScreenshot was not part of my code sample so it's normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

[petrogad]

Did you ever see blank screenshots and get around that issue?

Thank you for sharing!

@testerez ^

[gregberge]

It looks like a lot of people are close to the solution. Could someone try to submit a PR to add screenshot? 🙏

[helloguille]

@testerez Thanks for the code! However, I cannot I get undefined error for the pageInstance variable.

I just put all the code (In javascript) in my setupTestFrameworkScriptFile file + a call to registerScreenshotReporter().

I modified the declaration for Vainilla as such:

export const takeScreenshot = (testName, pageInstance = page) => { ... }
And page is indeed the name of the page instance in my tests.

[RoiEXLab]

To come back at this issue, before I found this issue I had several attempts at this myself.
My goal had higher ambitions though: I wanted jest to automatically generate a screenshot before and after each individual test.
Disclaimer: I haven't really used the jest API before, so this is probably the best thing I came up with:

const path = require('path');
const fs = require('fs').promises;

const hash = (str) => {
    if (str.length === 0) {
        return 0;
    }
    let hash = 0;
    for (let i = 0; i < str.length; i++) {
        hash  = ((hash << 5) - hash) + str.charCodeAt(i);
        hash |= 0; // Convert to 32bit integer
    }
    return hash;
};

const origIt = global.it;
global.it = (description, test, timeout) => {
    const result = origIt(description, async () => {
        const basePath = path.join(result.result.testPath, '..', '__screenshots__');
        try {
            await fs.mkdir(basePath, { recursive: true });
            await page.screenshot({ path: path.join(basePath, `before-${hash(result.result.fullName)}.png`) });
        } finally {
            try {
                return await test();
            } finally {
                await page.screenshot({ path: path.join(basePath, `after-${hash(result.result.fullName)}.png`) });
            }
        }
    }, timeout);
    return result;
};

Re-defining it is kinda ugly but it works better than all my other versions. Also I ignored test for now, because in my case I don't need it.
Now with Jest 24 the file can be specified in setupFilesAfterEnv as part of an array.
I'd really like to see such a feature soon in this project, and I would create a PR as well, it's just that I'm not confident enough in my code. So if someone could nudge me into the right direction that would be great.

[xiaoyuhen]

@RoiEXLab

Great solution！welcome a PR to solve this problem.

Now, jest-puppeteer has a bug with jest@24 and we will fix it this week as soon as possible. see #193

[celador]

// jest.setup.js

global.it = async function(name, func) {
  return await test(name, async () => {
    try {
      await func();
    } catch (e) {
      await fs.ensureDir('e2e/screenshots');
      await page.screenshot({ path: `e2e/screenshots/${name}.png` });
      throw e;
    }
  });
};

[redrockzee]

Use this package:
https://github.com/jeeyah/jestscreenshot

[gregberge]

It is amazing to see that everyone has a solution and nobody wants to make a PR, welcome in Wordpress world 🤗

[lukaszfiszer]

@petrogad check if you're not calling jestPuppeteer.resetPage() or doing something else with global page object in afterEach/afterAll. This was the reason I was getting blank screenshots.

@neoziro I should be able to submit a PR with adapted solution of @testerez in following days

[ajs139]

Incase it's helpful, It seemed for me like the page was closed by the time jasmine reporter was called when using one of the above solutions, and overriding it prevents things like it.only from working. Ended up with this riff on @testerez's code, which takes a screenshot after every test (on a CI env), but only saves the screenshot if the test fails:
https://gist.github.com/ajs139/3ddc10e807ee9b94b581c80a762de587

[rostgoat]

@testerez where would u even call registerScreenshotReporter ?

[testerez]

@rostgoat you can call it in your setupFilesAfterEnv jest script.

[rostgoat]

is this how your jest.config.js file looks @testerez?

setupFilesAfterEnv: [
    "<rootDir>/test/e2e/methods/common/screenshot.js"
  ]

how would this call the registerScreenshotReporter though?

[testerez]

registerScreenshotReporter should be called in the file you set as setupFilesAfterEnv

[AlexMarchini]

Thought I'd share my solution to this, using jest-circus and extending the jest-puppeteer environment. It's a little messy with the sleeps, I want to find a better way to handle that, but it works for now. Maybe jest-puppeteer can integrate with jest-circus out of the box once circus has more adoption and make this better.

const fs = require('fs');
const PuppeteerEnvironment = require('jest-environment-puppeteer');
require('jest-circus');

const retryAttempts = process.env.RETRY_ATTEMPTS || 1;

class CustomEnvironment extends PuppeteerEnvironment {

  async setup() {
    await super.setup();
  }

  async teardown() {
    // Wait a few seconds before tearing down the page so we
    // have time to take screenshots and handle other events
    await this.global.page.waitFor(2000);
    await super.teardown()
  }

  async handleTestEvent(event, state) {
    if (event.name == 'test_fn_failure') {
      if (state.currentlyRunningTest.invocations > retryAttempts) {
        const testName = state.currentlyRunningTest.name;
        // Take a screenshot at the point of failure
        await this.global.page.screenshot({ path: `test-report/screenshots/${testName}.png` });
      }
    }
  }

}

module.exports = CustomEnvironment

[allimuu]

@AlexMarchini thanks for that! Perfect, I wanted to use jest-circus as well. If anyone else is wondering (or anyone has suggestions) like I was how to actually use this this is what my jest-config.js looks like:

{
    ...
    globalSetup: 'jest-environment-puppeteer/setup',
    globalTeardown: 'jest-environment-puppeteer/teardown',
    testEnvironment: './jest-environment.js',
    testRunner: 'jest-circus/runner',
    setupFilesAfterEnv: ['./jest-setup.js'],
    ...
}

So I removed preset and jest-environment.js is the above snippet. And jest-setup.js has nothing but require('expect-puppeteer'); in it.

And that seems to be working.. with jest-circus as the runner. (Note just run yarn add --dev jest-circus)

Hope this helps someone.

[favna]

Previously I commented here that we had a solution for work. The short version is that it was based on @testerez work. Sadly for whatever reason that was a moment of glory yesterday evening as when I started to get it into our CI/CD pipeline this morning it all crashed and burned and even refused to work locally so I deleted that comment.

That said, I ended up completely rewriting the library I made now relying on a solution very similar to @AlexMarchini / @allimuu thus relying on Jest Circus and at least it seems to work properly now.

For those interested you can find it here (bumped to v2.0.0 as it broke previous releases). Instructions and example on how to use the library are included in the README you can read on the yarnpkg / npm site. This includes a maybe more detailed (due to including code sample) version of @allimuu 's contribution.

And as before here is the actual source code behind it

[dennismphil]

This was not as straightforward as I thought.

I found in the above methods it.only stops working. What worked for me was the below solution.

Monkey patching the Jest.it method.

// jest.config.js

module.exports = {
    ...
    setupFilesAfterEnv: [
        '<rootDir>/setupFilesAfterEnv.js',
    ]
};

// setupFilesAfterEnv.js

// Wrapper function that wraps the test function
// to take a screenshot on failure
const wrappedTest = (test, description) => {
    return Promise.resolve()
        .then(test)
        .catch(async (err) => {
            // Assuming you have a method to take a screenshot
            await takeScreenshot('some file name');

            throw err;
        });
};

// Make a copy of the original function
const originalIt = global.it;

// Modify `it` to use the wrapped test method
global.it = function it(description, test, timeout) {
    // Pass on the context by using `call` instead of directly invoking the method.
    return originalIt.call(this, description, wrappedTest.bind(this, test, description), timeout);
};

// Copy other function properties like `skip`, `only`...
for (const prop in originalIt) {
    if (Object.prototype.hasOwnProperty.call(originalIt, prop)) {
        global.it[prop] = originalIt[prop];
    }
}

// Monkey patch the `only` method also to use the wrapper method
global.it.only = function only(description, test, timeout) {
    return originalIt.only.call(this, description, wrappedTest.bind(this, test, description), timeout);
};

[noomorph]

I think this pull request jestjs/jest#9397 may come to the rescue if you are struggling with asynchronicity.

[tihuan]

For future readers, if you wanna get #131 (comment) to work, please make sure BOTH of your jest and jest-circus are on versions that include jestjs/jest#9397, which means both jest and jest-circus need to be v25.3.0+

Our project updated jest to v25.5.0, but jest-circus stayed at v25.2.7, resulting in Circus runner not waiting for test_fn_failure test event handler:

https://github.com/facebook/jest/pull/9397/files#diff-e77e110fa517918f09bcd38e70996fafR164

Note to self that jest and jest-circus have version parity, so always update both libraries together!

[stefanteixeira]

In case someone wants to know how to get the test describe name when using jest-circus (#131 (comment) saves a screenshot only with the test method name), you can get it like:

const testDescription = state.currentlyRunningTest.parent.name 

I had to spend some time searching in the source code how State is defined inside jest-circus to find that information, hope it helps someone 😄

[sovoid]

I have somehow got video recording of test runs and screenshots of tests before and after each test working using Jest and Puppeteer. Please have a look if you guys find it helpful...

#361

[ajaykhobragade]

This is all quite helpful, it would be even better if somehow log would be able to generate with information and saved in directory on test failure, along with screenshot. Any help\suggestions around that?

[ajaykhobragade]

Hi Stefan – Thanks for quick reply! No, I meant some kind of information that you see on terminal when test fails (ran by Jest>> npm test), just like in code below where screenshot is being captured and saved in the path directory. I am trying to build a code that will grab fail test results and put it in directory that can be shared with DevOps during CI. async handleTestEvent(event) { if (event.name === 'test_done' && event.test.errors.length > 0) { const parentName = event.test.parent.name.replace(/\W/g, '-') const specName = event.test.name.replace(/\W/g, '-') await this.global.page.screenshot({ path: `screenshots/${parentName}_${specName}.png`, fullPage: true, }) } Hope this helps. From: Stefan Teixeira <notifications@github.com> Reply-To: smooth-code/jest-puppeteer <reply@reply.github.com> Date: Wednesday, August 12, 2020 at 4:50 PM To: smooth-code/jest-puppeteer <jest-puppeteer@noreply.github.com> Cc: "Khobragade, Ajay" <ajay.khobragade@startribune.com>, Mention <mention@noreply.github.com> Subject: Re: [smooth-code/jest-puppeteer] Take screenshot when a test fails (#131) @ajaykhobragade<https://github.com/ajaykhobragade> you mean saving the console log output in a file? If so, you can get console output from Puppeteer like this: https://stackoverflow.com/a/46245945 Then you can save that output to a file in a directory alongside screenshots. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#131 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOSVEZJ66XGYYPNXBXA527LSAMFCNANCNFSM4FWW7K6Q>.

[ShivamJoker]

Whats the status no PR's for taking screenshot ?

[jpourdanis]

@ricwal-richa yes, takeScreenshot was not part of my code sample so it's normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

Hey @testerez where we can call that? on protractor.conf.js ?

[andredesousa]

Hi everyone.
Based on these documents, I created a setup for React with jest-puppeteer and jest-html-reporters.
For more details:

https://github.com/andredesousa/essential-react-scaffold

Basically, I cretead a new Test Environment:

class E2EEnvironment extends PuppeteerEnvironment {
  async handleTestEvent(event, state) {
    if (event.name === 'test_fn_failure') {
      const data = await this.global.page.screenshot();

      await addAttach(data, 'Full Page Screenshot', this.global);
    }
  }
}

...and I imported it into my jest.config.js file:

module.exports = {
  preset: 'jest-puppeteer',
  testRunner: 'jest-circus/runner',
  testEnvironment: '<rootDir>/environment.js',
  testMatch: ['<rootDir>/specs/**/*.spec.js'],
  reporters: [
    'default',
    [
      'jest-html-reporters',
      {
        publicPath: './reports/e2e',
        filename: 'index.html',
      },
    ],
  ],
};

Thus, when the test fails, a screenshot is attached to the report.

[madhusudhan-patha]

@ricwal-richa yes, takeScreenshot was not part of my code sample so it's normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

@jpourdanis I am trying to use this code (typescript) along with jest-cucumber, but it gives me error saying.

 ReferenceError: jasmine is not defined

      21 |      * have access to the test result, test name and page instance at the same time.
      22 |      */
    > 23 |     (jasmine as any).getEnv().addReporter({

[karlhorky]

Maybe the PR to WordPress Gutenberg by @kevin940726 :

WordPress/gutenberg#28449

...would provide a basis for a PR to the jest-environment-puppeteer package in this repo. Specifically, it seems to be this file that needs to be updated:

https://github.com/smooth-code/jest-puppeteer/blob/master/packages/jest-environment-puppeteer/src/PuppeteerEnvironment.js

[lucassardois]

In my use case, I would like to be able to take screenshots for all open pages of puppeteer when a test fail. Because my test needs to have some background pages running.

[gregberge]

@lucassardois PR was done but I had to migrate to TypeScript before. Will be done in the next weeks.