Custom reporters need the way to provide metadata for tests

[vovsemenv]

Clear and concise description of the problem

In allure team want to create custom reporter for vitest but there is no way to provide metadata to each test such as linked issue on another important data.
Also we need the way to consume that metadata from reporter API.

Suggested solution

for example playwright provide custom annotation API

test('user profile', async ({ page }) => {
  test.info().annotations.push({ type: 'issue', description: 'https://github.com/microsoft/playwright/issues/<some-issue>' });
  // ...
});

Alternative

Maybe we can provide way to send messages to reporter.

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[sheremet-va]

Can you be more specific about usecase, and what exactly do you want? I do not know what annotations are.

Can you use context?

[vovsemenv]

@sheremet-va Hello. We just need something to put metadata during tests and then consume this metadata in our custom reporter.
For example with this feature we can add rich markdown descriptions for each test or link Jira or GH tasks to specific tests.
This feature also will be useful for vitest own ui.

How i think it would look.

import { it, addMeta } from 'vitest'
it('should work', ({ foo }) => {
  addMeta("metaString")
  addMeta({type:"metaObj"})
})

export class CustomReporter implements Reporter {
  onFinished(files?: File[]) {
    console.log(files[0].tasks[0].meta) // ["metaString", {type:"metaObj"}]
  }
}

[sheremet-va]

How i think it would look.

We cannot do this with a global function, since we don't now what test is running with test.concurrent.

Why can't you use context?

it('some test', ({ context }) => {
  context.type = 'meta_obj'
})

We can technically add setContext function, if you prefer this style:

it('some test', ({ setContext }) => {
  setContext({ type: 'myObj' })
})

Right now context is not transferred to reporter, of course. But it can be implemented.

[vovsemenv]

Why can't you use context?

I definitely can. I thought this solution was limited by the fact that context is not accessible in the reporter 😅

But it can be implemented.

🔥🔥🔥

[sheremet-va]

Although I think it might be a breaking change, because you wouldn't be able to add a function on context (functions cannot be transferred between worker and main thread). So maybe indeed we need a new concept for this.

I like the name metadata, how would you like to interact with it? Is it an array, as I see in your examples?

[vovsemenv]

how would you like to interact with it

i like the way it implemented in playwright. there you can do thing like

test("basic test", async (testArgs, testInfo) => {
  testInfo.attachments.push({
    name: "screenshot",
    body: await testArgs.page.screenshot(),
    contentType: "image/png",
  });

  testInfo.annotations.push({
    type: "issue-link",
    description: "https://github.com/vitest-dev/vitest/issues/1951",
  });
});

Wonder if we can do same thing like

it("basic test", (context, testInfo) => {
  testInfo.metadata.push("something")
});

[vovsemenv]

testInfo in TestFunction will be really handy when we want to customize our test report from inside of the test

it("basic test", (context, testInfo) => {
  if(somCondition){
    testInfo.fail()
  }
  if(somCondition){
    testInfo.attach(someAttachemnt)
  }
});

This also allow us to access vite and vitest config from test.

it("basic test", (context, testInfo) => {
  testInfo.viteConfg
  testInfo.vitestConfg
});

[sheremet-va]

i like the way it implemented in playwright. there you can do thing like

We already provide meta:

test('some test', ({ meta }) => {
  console.log('metadata', meta)
})

This also allow us to access vite and vitest config from test.

Vitest config is part of Vite config. It's also not fully possible, because it needs to be serializable, but some properties, like plugins, are not.

[vovsemenv]

We already provide meta:

Is it not internal-only thing? It's ok to modify this meta from inside of test?
If so we really can provide something like meta.attachments to attach anything you want to this specific test.

[vovsemenv]

ofc we also need to expose the meta to reporter

[sheremet-va]

We already provide meta:

Is it not internal-only thing? It's ok to modify this meta from inside of test? If so we really can provide something like meta.attachments to attach anything you want to this specific test.

Well, it's not used internally. Vitest has direct access to it without needing context, but I guess the contents should be readonly (although it's not imposed).

[stefnotch]

We have a similar use-case for "sending test information to a custom reporter".

We're using Vitest for a university course, and all unit tests have a lot of helpful information for the students. We basically have an array of "steps", and then the reporter prints both the succeeded steps, and the (frequently customized) error message.

Currently, the only way of getting such information to the reporter is by doing horrible hacks. We've resorted to

1. disabling the default reporter
2. abusing the error message and packing an entire object into it, as in new Error(JSON.stringify({ steps, message }), { cause: e })

In an ideal world, we could use the context to send over that information.

[sheremet-va]

We need to expose meta to the reporter. Currently only the "result" is transported to the reporter. We cannot reuse context for this because we cannot serialize it.

Also, it's only possible to pass down clonable data. I'm not sure if we can impose it with typescript.

[NKJe]

Is this resolved by #3449?

[sheremet-va]

Is this resolved by #3449?

Yes, you are right!

[vovsemenv]

Wow!!! That's awesome will check later