feat: benchmark

[poyoho]

fix #917

native support benchmark.

• Create a new bench command to start vitest and distinguish the current running mode of vitest (unit test or benchmark).
• Reuse the Vitest suite collection benchmark, add the bench API to distinguish between unit test and benchmark and collect the benchmark execution function.
• When running the test, unit-test normally enters the unit-test branch and executes the unit-test function once. And benchmark will use Tinybench@2.1.3 as engine asynchronous multiple execution function.
• provide verbose and json reporter to show the benchmark result.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	facaf89
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/629f5bc82c6ad70007dd3853
😎 Deploy Preview	https://deploy-preview-1029--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[antfu]

Let's discuss the design in #917 first.

About benchmark.js, if we really end up using it, I could try to contact the author and see if our team could grant the access and help on maintaining it.

[poyoho]

It seems need to make benchmark(xxxx.bench.js) whit diff files and add a new command to run benchmark 👏

[sheremet-va]

I think we can now use tinybench by @Uzlopak?

[poyoho]

yes, It seems to had the same api with benchmark

[Uzlopak]

Published version 1.0.2 of tinybench.

[antfu]

We already support the reporter interface to access the full test result.

[poyoho]

In fact, we had the results, but this style can't fill too much data 😁

[poyoho]

@poyoho the ops/sec are identical, there might be something wrong

two decimal places will be rounded off. The real result is: 0.006637481844709122, 0.005769233822295151...

[poyoho]

updated

[poyoho]

I think we can native support deno style benchmark result.

[antfu]

For ops/sec <1, should we display the duration of one op instead?

[poyoho]

maybe we can change ops/sec -> ops/ms

[Uzlopak]

I dont think, that ops/ms is that relevant. I personally run benchmarks multiple times because it also depends on which core you run the benchmark and if there is currently a heavier process running on it. So basically benchmarks only indicate if you have some heavy performance degredation. So I would think it is totally OK to have the ops/s. Also you have the percentage on how they differ, so you can actually see which one is faster.

Is it possible to pass options to the benchmark like

describe('sort', { minSample: 100 }, () => {
  bench('normal', () => {
    const x = [1, 5, 4, 2, 3]
    x.sort((a, b) => {
      return a - b
    })
  })
})

[poyoho]

feat json format report to show all the result

[sheremet-va]

What do we need to release this?

[antfu]

I would like to improve the default output to a table, but not blocking as we could improve it later. Since this is experimental, I think we could ship it in the next minor.