Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: match jest json output by reporting location of error in test file #4826

Conversation

bard
Copy link
Contributor

@bard bard commented Dec 29, 2023

Description

This change improves compatibility of the JSON reporter with Jest's --json output by pointing to the last stack frame (where the error surfaced in the test) instead of the first.

Repro

$ git clone https://github.com/bard/vitest-json-reporter-bugs-repros && cd vitest-json-reporter-bugs-repros && pnpm i
$ pnpm run repro-lineindex-bug

$ pnpm repro-location-bug

> repro@1.0.0 repro-location-bug /tmp/vitest-json-reporter-bugs-repros
> echo '### WITH JEST:'; pnpm run location-bug-jest; echo; echo '### WITH VITEST:'; pnpm run location-bug-vitest

### WITH JEST:

> repro@1.0.0 location-bug-jest /tmp/vitest-json-reporter-bugs-repros
> jest --json --testLocationInResults src/location-bug/jest.test.ts 2>/dev/null | jq .testResults[0].assertionResults[0].location

{
  "column": 5,
  "line": 4
}

### WITH VITEST:

> repro@1.0.0 location-bug-vitest /tmp/vitest-json-reporter-bugs-repros
> vitest run --reporter=json src/location-bug/vitest.test.ts | jq .testResults[0].assertionResults[0].location

{
  "line": 1,
  "column": 8
}

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
  • Ideally, include a test that fails without this PR but passes with it.
  • Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

  • Run the tests with pnpm test:ci.

Documentation

  • If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

  • Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

Copy link

netlify bot commented Dec 29, 2023

Deploy Preview for fastidious-cascaron-4ded94 canceled.

Name Link
🔨 Latest commit 6b5b7dc
🔍 Latest deploy log https://app.netlify.com/sites/fastidious-cascaron-4ded94/deploys/658f02df5bee9000083088d3

@bard bard marked this pull request as ready for review December 29, 2023 18:19
@bard bard changed the title fix: match jest output by reporting location of error in test file fix: match jest json output by reporting location of error in test file Dec 29, 2023
@sheremet-va
Copy link
Member

I don't think we can make this change. It was already reverted in #3645

@bard
Copy link
Contributor Author

bard commented Dec 31, 2023

I don't think we can make this change. It was already reverted in #3645

@sheremet-va apologies for not searching previous issues enough.

I'm a little perplexed. Consider the following output relating to a test run involving index.ts and index.test.ts, where an error was thrown inside index.ts:

  "testResults": [
    {
      "assertionResults": [
        {
          "ancestorTitles": [
            ""
          ],
          "fullName": " addition",
          "status": "failed",
          "title": "addition",
          "duration": 8,
          "failureMessages": [
            "error thrown in source file"
          ],
          "location": {
            "line": 4,
            "column": 9
          }
        }
      ],
      "startTime": 1704030241630,
      "endTime": 1704030241638,
      "status": "failed",
      "message": "",
      "name": "/tmp/repro/src/index.test.ts"
    }
  ]

There's a line and column number, but no way of telling in which file that line and column are, which makes the location field... not very useful.

Perhaps the actual bug here (if alignment with jest is not desired) is that there should be a file property in location?

@sheremet-va
Copy link
Member

There's a line and column number, but no way of telling in which file that line and column are, which makes the location field... not very useful.

What even is the location in json result? Is it the test function location? Or error location? I can also see that jest prints the whole stack trace in the failure message - maybe we should also do that?

@bard
Copy link
Contributor Author

bard commented Dec 31, 2023

There's a line and column number, but no way of telling in which file that line and column are, which makes the location field... not very useful.

What even is the location in json result? Is it the test function location? Or error location? I can also see that jest prints the whole stack trace in the failure message - maybe we should also do that?

Jest seems to define location as "location of the test() call for the test that failed".

A stack trace would be great IMHO. Sure, it requires parsing, but it has all the information and none of the ambiguity. If it's an acceptable addition, I'd be happy to look into adding it to the JSON reporter.

@sheremet-va
Copy link
Member

Jest seems to define location as "location of the test() call for the test that failed".

Maybe we should try to actually locate it then? Both the first and the last frames are not the test call. On top of that, I think we can add a stack trace and/or parsed stack trace (we always parse it, so it's always available).

@bard
Copy link
Contributor Author

bard commented Jan 1, 2024

Jest seems to define location as "location of the test() call for the test that failed".

Maybe we should try to actually locate it then? Both the first and the last frames are not the test call. On top of that, I think we can add a stack trace and/or parsed stack trace (we always parse it, so it's always available).

I've opened #4843 for the stack trace part (it's minimal). Will close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants