Add custom reporter to produce Problem Matcher output #2

stefanbuck · 2020-04-24T10:53:31Z

stefanbuck · 2020-04-24T10:58:59Z

json-reporter.js

+      const col = item.location.column;
+      const message = item.failureMessages[0].replace(/\n/g, newLine);
+
+      console.log(`::error file=${name},line=${line},col=${col}::${message}`);


The GitHub runner has some build in matchers and therefore all we have to do is producing output in this format actions/toolkit#260 (comment)

json-reporter.js

stefanbuck · 2020-04-24T11:02:20Z

json-reporter.js

+      if (item.status !== "failed") {
+        return;
+      }
+      const newLine = "%0A";


We have to url encode new lines in order to produce a matcher message that can spawn over multiple lines

stefanbuck · 2020-04-24T11:02:50Z

json-reporter.js

+
+      const line = item.location.line + 1;
+      const col = item.location.column;
+      const message = item.failureMessages[0].replace(/\n/g, newLine);


We might need to iterate over failureMessages, but for this MVP its should be enough

SimenB · 2020-04-24T15:28:57Z

json-reporter.js

+        );
+      }
+
+      const line = item.location.line + 1;


+ 1? testLocation is for the it, not the expect. You're probably better of looking at the stack trace in the error

I misread the documentation

Note that column is 0-indexed while line is not.
https://jestjs.io/docs/en/cli.html#--testlocationinresults

Anyway, thanks for the hint about the it this wasn't clear from the documentation. Will parse the error message as you suggested.

I updated the implementation according to your comment. Should I try to get a Jest PR up? If you could point me to a location / package where I should make changes, that would be amazing. From a quick look it feels like it should be part of https://github.com/facebook/jest/tree/master/packages/jest-reporters/

Part of reporters, yeah.

Auto-adding it would go somewhere here: https://github.com/facebook/jest/blob/faa52673f4d02ee1d273103ada9382a0805b54a4/packages/jest-core/src/TestScheduler.ts#L267-L298

We should probably wait a bit with that until we're happy, though

I'm already super happy with the result 😉

I want to get this feature in so I can deprecate a Danger.js plugin I wrote to present failing tests on a Pull Request. Also I think the Jest community will love this integration which works out of the box with every GitHub Action setup.

The image below shows a comment on a Pull Request generated by a custom Danger.js](https://github.com/danger/danger-js) plugin

SimenB · 2020-04-27T09:40:23Z

Would be interesting to get some colors in the posted message. Is that supported? We could probably find some way to translate ANSI codes to html/css

stefanbuck · 2020-04-29T22:57:39Z

I tried a few different approaches to color the output, but none of them worked. I raised a feature request in the toolkit repo actions/toolkit#435.

SimenB · 2020-04-29T23:02:34Z

I guess we could generate some markdown and post that? That should take care of the diff. Would be nice to have some ansi->html translation though as coloring after the fact is painful

stefanbuck · 2020-04-29T23:07:09Z

I guess we could generate some markdown and post that?

As a comment on the Pull Request? This would require a dedicated Jest Action. The current solution works out of the box if jest runs on GitHub Action, no setup, no config.

SimenB · 2020-04-29T23:08:22Z

Ah, we can't dump markdown via a problem matcher?

stefanbuck · 2020-04-29T23:11:20Z

I would be surprised if it does. The Problem Matcher is meant to forward "terminal" output to GitHub's interface.

SimenB · 2020-04-29T23:16:12Z

Ah, ok. Too bad, although I guess that could be abused somewhat to post images and what not

SimenB · 2020-04-29T23:17:34Z

My only remaining concern adding it as a default then is that there would be no easy way to turn it off if people don't want it. Except defining all reporters yourself. Maybe not a huge issue, though

stefanbuck · 2020-04-29T23:20:04Z

Markdown is not supported

stefanbuck · 2020-04-29T23:21:01Z

My only remaining concern adding it as a default then is that there would be no easy way to turn it off if people don't want it

What if we add a flag to Jest config like

{
  githubActionEnabled: true // (default: true)
}

or we could listen for an env var JEST_GITHUB_ACTION_REPORTER

SimenB · 2020-04-29T23:24:08Z

can it be made to dismiss earlier entries? I'm guessing no...

stefanbuck · 2020-04-29T23:28:29Z

Usually it does. Before a new run starts it removes all annotations. Looks like a bug caused by me force pushing

stefanbuck · 2020-04-29T23:29:41Z

I'll raise a new PR to ensure its working as expected

stefanbuck · 2020-04-29T23:33:19Z

On this PR three actions are running where it should be only one

stefanbuck · 2020-04-29T23:35:29Z

Fixed by recreating my last commit. I've noticed quite a few GitHub Actions problems when force pushing commits.

jsg2021 · 2020-07-06T19:36:07Z

This is super exciting. I hope the momentum hasn't trailed off. :)

stefanbuck · 2020-07-11T12:15:47Z

I had a closer look at the Jest repository, but I have to admit that I was quite overwhelmed by the architecture given that I'm not a TypeScript developer. @SimenB what's the best way to get this thing moving? Should I raise an issue in the Jest repo and outline the main motivation for this feature and summarise everything I explored so far?

stefanbuck · 2020-07-23T20:54:54Z

json-reporter.js

+            }
+
+            const [, line, col] = captureGroup;
+            console.log(`::error file=${testFilePath},line=${line},col=${col}::${message}`);


This is the magic which turns a console.log into a GitHub annotation.

xt0rted · 2020-08-23T06:41:46Z

I just started using a slightly modified version of this reporter for one of my projects. I'm using the issueCommand function from @actions/core which also handles the encoding, and I stripped the call stack since it's extra noise in this context.

The tests are running twice in my workflow, once in a build job and again in a code coverage job, so I set an env var on one of them and conditionally add the reporter so I don't get duplicate annotations.

I can post my changes if anyone's interested.

mariusGundersen · 2020-10-23T09:10:33Z

This is a fantastic simple solution to the problem. I see that it has been suggested in the jest repo but maybe it could be suggested in jest-community as well? And if they won't add it then maybe make an unofficial npm package. I might consider doing that myself, with probably more documentation than code as this seems to be a very common problem with lots of bad solutions based on some google searching.

SimenB · 2020-10-23T10:37:50Z

I'm happy to move the code for this over into the Jest repo (as long as it gets some tests), then we can figure out whether or not it should be enabled by default later

stefanbuck · 2020-10-27T08:15:43Z

@SimenB as mentioned earlier, I had a closer look at the Jest repository, but I have to admit that I was quite overwhelmed by the architecture. The actual code is quite simple, but I wonder how to integrate this into Jest best. @SimenB do you mind providing an outline which files are involved? That would be really helpful.

SimenB · 2020-10-27T08:28:00Z

Put it into https://github.com/facebook/jest/tree/master/packages/jest-reporters/src and export it from index.ts.

It needs to be ported to TypeScript and have some tests written for it - both unit and integration. As a start just open a PR and we can work together to land it 🙂

After it has landed we can figure how to document it, expose it to the user and whether or not it should be enabled by default when we think we are in a GH Actions run

mariusGundersen · 2020-10-30T14:23:23Z

I created an initial PR here: jestjs/jest#10741

stefanbuck · 2020-10-30T22:18:00Z

Hey @mariusGundersen, what is the purpose of this placeholder PR? Maybe I'm just confused because you called it placeholder. Anyway, I started working on this as well. I'm more than happy to collaborate on this. I used the SummaryReporter as a base which doesn't look much different from your solution stefanbuck/jest@9d5f65a

How should we proceed? Are you familiar with the integration part?

SimenB · 2022-04-26T08:27:53Z

shipped in jest 28, fwiw: https://jestjs.io/blog/2022/04/25/jest-28#github-actions-reporter

Add custom reporter to produce Problem Matcher ouput

8245003

stefanbuck commented Apr 24, 2020

View reviewed changes

json-reporter.js Outdated Show resolved Hide resolved

stefanbuck commented Apr 24, 2020

View reviewed changes

update test

6cb8f1e

SimenB reviewed Apr 24, 2020

View reviewed changes

stefanbuck added 2 commits April 25, 2020 22:22

Do not rely on test location to get line number

56a75b4

Loop all the arrays

42c3323

stefanbuck force-pushed the add-custom-reporter branch from e14b89a to daf2e9e Compare April 25, 2020 20:35

stefanbuck force-pushed the add-custom-reporter branch 2 times, most recently from 2d66ef7 to daf2e9e Compare April 28, 2020 22:00

stefanbuck force-pushed the add-custom-reporter branch from aa84fac to ca062e8 Compare April 29, 2020 23:15

stefanbuck force-pushed the add-custom-reporter branch from ca062e8 to daf2e9e Compare April 29, 2020 23:21

more tests

82d50fd

stefanbuck force-pushed the add-custom-reporter branch from daf2e9e to 82d50fd Compare April 29, 2020 23:33

stefanbuck mentioned this pull request Jul 23, 2020

Show failing tests in the GitHub UI using Problem Matchers jestjs/jest#10309

Closed

stefanbuck commented Jul 23, 2020

View reviewed changes

mariusGundersen mentioned this pull request Oct 30, 2020

feat: Added GithubActionsReporter jestjs/jest#10741

Closed

Qrox mentioned this pull request Dec 27, 2020

Display multi-line json error messages with github action CleverRaven/Cataclysm-DDA#46353

Merged

ockham mentioned this pull request Apr 20, 2021

feat: Add GitHub Actions Reporter jestjs/jest#11320

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add custom reporter to produce Problem Matcher output #2

Add custom reporter to produce Problem Matcher output #2

stefanbuck commented Apr 24, 2020 •

edited

stefanbuck Apr 24, 2020

stefanbuck Apr 24, 2020

stefanbuck Apr 24, 2020

SimenB Apr 24, 2020

stefanbuck Apr 24, 2020

stefanbuck Apr 26, 2020

SimenB Apr 27, 2020

stefanbuck Apr 29, 2020 •

edited

SimenB commented Apr 27, 2020

stefanbuck commented Apr 29, 2020

SimenB commented Apr 29, 2020

stefanbuck commented Apr 29, 2020

SimenB commented Apr 29, 2020

stefanbuck commented Apr 29, 2020

SimenB commented Apr 29, 2020

SimenB commented Apr 29, 2020

stefanbuck commented Apr 29, 2020 •

edited

stefanbuck commented Apr 29, 2020 •

edited

SimenB commented Apr 29, 2020 •

edited by stefanbuck

stefanbuck commented Apr 29, 2020

stefanbuck commented Apr 29, 2020

stefanbuck commented Apr 29, 2020 •

edited

stefanbuck commented Apr 29, 2020

jsg2021 commented Jul 6, 2020

stefanbuck commented Jul 11, 2020

stefanbuck Jul 23, 2020

xt0rted commented Aug 23, 2020

mariusGundersen commented Oct 23, 2020

SimenB commented Oct 23, 2020

stefanbuck commented Oct 27, 2020

SimenB commented Oct 27, 2020

mariusGundersen commented Oct 30, 2020

stefanbuck commented Oct 30, 2020 •

edited

SimenB commented Apr 26, 2022

Add custom reporter to produce Problem Matcher output #2

Are you sure you want to change the base?

Add custom reporter to produce Problem Matcher output #2

Conversation

stefanbuck commented Apr 24, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanbuck Apr 29, 2020 • edited

Choose a reason for hiding this comment

SimenB commented Apr 27, 2020

stefanbuck commented Apr 29, 2020

SimenB commented Apr 29, 2020

stefanbuck commented Apr 29, 2020

SimenB commented Apr 29, 2020

stefanbuck commented Apr 29, 2020

SimenB commented Apr 29, 2020

SimenB commented Apr 29, 2020

stefanbuck commented Apr 29, 2020 • edited

stefanbuck commented Apr 29, 2020 • edited

SimenB commented Apr 29, 2020 • edited by stefanbuck

stefanbuck commented Apr 29, 2020

stefanbuck commented Apr 29, 2020

stefanbuck commented Apr 29, 2020 • edited

stefanbuck commented Apr 29, 2020

jsg2021 commented Jul 6, 2020

stefanbuck commented Jul 11, 2020

Choose a reason for hiding this comment

xt0rted commented Aug 23, 2020

mariusGundersen commented Oct 23, 2020

SimenB commented Oct 23, 2020

stefanbuck commented Oct 27, 2020

SimenB commented Oct 27, 2020

mariusGundersen commented Oct 30, 2020

stefanbuck commented Oct 30, 2020 • edited

SimenB commented Apr 26, 2022

stefanbuck commented Apr 24, 2020 •

edited

stefanbuck Apr 29, 2020 •

edited

stefanbuck commented Apr 29, 2020 •

edited

stefanbuck commented Apr 29, 2020 •

edited

SimenB commented Apr 29, 2020 •

edited by stefanbuck

stefanbuck commented Apr 29, 2020 •

edited

stefanbuck commented Oct 30, 2020 •

edited