Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expect test to fail (xfail style) #10030

Closed
willstott101 opened this issue May 12, 2020 · 5 comments
Closed

Expect test to fail (xfail style) #10030

willstott101 opened this issue May 12, 2020 · 5 comments

Comments

@willstott101
Copy link

🚀 Feature Proposal

The ability to mark a test as expected to fail probably with the following syntax:

describe('Test isCool function works', {
  test('jeff is the best', () => {
    expect(isCool("jeff")).toBe(true);
  });

  test.xfail("for some reason I'm not coming out as cool...", () => {
    expect(isCool("will")).toBe(true);
  });
});

And suggested output:

Test Suites: 1 failed, 5 passed 6 total
Tests:       1 failed, 16 passed, 2 expected failures, 2 unexpected passes, 21 total
Snapshots:   0 total
Time:        2.627s, estimated 4s
Ran all test suites.

Motivation

Many testing frameworks allow you to specify a test as a expected failure. This can be very useful for long-term TDD and for testing known-bugs are still bugs without implying that it's intended behaviour.

Example

In the above output, an engineer should be surprised (and pleased) by the 2 unexpected passes and change those tests to test from test.xfail if their commit caused the fixes. Otherwise they can leave as is for an engineer who understands those tests to mark as resolved.

Pitch

I feel this is a core feature as it's a fundamentally new result status for tests.

See some great comments from an older issue #8317

@snoozbuster
Copy link

snoozbuster commented Jun 9, 2021

+1 on this. Sometimes we have frontend functionality which needs to be hard-disabled due to a backend/API bug. Often times this is code which is tested and works, but the data isn't reliable enough to show it to customers. After stubbing the code we can use it.skip we can disable these tests, but there is nothing which forces them to be re-enabled after the upstream services are working and the code is re-enabled (someone unaware of the test may leave the it.skip in - after all, "the tests are passing"). An xfail like interface would start failing after the feature was re-enabled in this case, ensuring that the test is also turned on in conjunction.

@dgkimpton
Copy link

dgkimpton commented Nov 30, 2021

I just came here to make a similar request, I was going to call it test.quarantine but the functionality is the same.
I have a test that is currently broken and I accept that but would like to know if something happens to make it pass. So ideally this would make a failing test pass until it would actually pass, at which point it should fail.

updated: I previously added some workarounds, none of which were very good. I've since come up with a "better" one (although it's pretty unpleasant code).

test.quarantine = function (description, func) {
  describe("in quarantine", () => {
    try {
      func()
      test(description, () => {
        const e = new Error("[" + description + "] was expected to fail, but it passed")
        e.name = 'Quarantine Escape'
        throw e
      })
    } catch (e) {
      test.todo('fix ' + description)
    }
  })
}

used as

  test.quarantine('testA', () => {
    expect(false).toBe(true)
  })
  test.quarantine('testB', () => {
    expect(false).toBe(false)
  })

which shows

in quarantine                                                                                                                                                                                                                                                                                             
      × testB (1 ms)                                                                                                                                                                                                                                                                                          
      ✎ todo fix testA      

I'll leave this here in case it's useful to anyone.

@vlovich
Copy link

vlovich commented May 19, 2022

The example doesn't work if you have any afterAll/afterEach/beforeEach/beforeAll which screws up the state. Haven't figured out a solution for that. I tried:

Object.defineProperty(test, 'bug', {
  value: (description: string, fn: () => void, timeout?: number) => {
    test(
      description,
      () => {
        try {
          fn()
        } catch (e) {
          test.todo(`[${description}] needs to be fixed`)
          return
        }

        throw new Error(
          `[${description}] was expected to fail, but it passed. Did you fix the bug?`,
        )
      },
      timeout,
    )
  },
  writable: false,
})

but fn doesn't throw and I don't know how to check jest whether or not the current test failed...

@SimenB
Copy link
Member

SimenB commented May 20, 2022

Duplicate of #4627, which shipped in jest@28.1.0

@SimenB SimenB closed this as completed May 20, 2022
@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants