feat: make sequencer public, add option to run tests in random order

[sheremet-va]

Closes #1570

TODO

• Docs
• Tests

This PR adds a few new options to config:

1. --shuffle to CLI flags
2. sequence to options

• sequencer allows users to overwrite builtin logic for sharding and sorting
• shuffle allows tests to be run in random order
• seed makes random order more deterministic

Also added shuffle flag to describe block to make only part of suite random. I've noticed we have some tests rely on running in the same order, so I though this option might be helpful.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	8cade60
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/62c55c9bb99f0d0008df8a57
😎 Deploy Preview	https://deploy-preview-1582--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[antfu]

If so, should we also support repeat for describe? For example:

describe.repeat(10).random('repect 10 times randomly', () => {
  // ...
})

[sheremet-va]

How is this has anything to do with random? 👀

[antfu]

No necessary in this PR, just an idea. Using random feels more unstable to identify the ordering bug (like sometimes the change passes the CI because the order does not hit the bug, but will later). Running random tests multiple times could reduce such scenarios and make the tests more stable.

[sheremet-va]

No necessary in this PR, just an idea. Using random feels more unstable to identify the ordering bug (like sometimes the change passes the CI because the order does not hit the bug, but will later). Running random tests multiple times could reduce such scenarios and make the tests more stable.

Sure, but this also should be done on the global level, yes? This PR adds vitest --random, so there should also be vitest --random --repeat 10.

It's also not clear what should be done on error. Do we fail the suite when it hits the first error? Do we show summary of all 10 runs? I think if we want to add this, it's better be done in other PR, and if/when people will ask for it.

I really want to switch to @vitest/browser, but all this enhancements PRs stop me 😄

[antfu]

Sure we could do that later 👍

[antfu]

The feature looks good to me. But do you think shuffle would be better than random?

[sheremet-va]

The feature looks good to me. But do you think shuffle would be better than random?

Well, it's sequence.random, so I think it's straightforward enough, or do you mean --random?

[antfu]

I mean the term random -> shuffle in general. From my understanding:

• random is a more general term, it could combine with many things, like random in order, or decide if a test should run randomly, or pick a random test, or both.
• shuffle specifically means random the order of an array, in which the items of that array are still the same.

Refs:

• https://wikidiff.com/random/shuffle
• https://www.italki.com/post/question-297430

[sheremet-va]

I mean the term random -> shuffle in general. From my understanding:

• random is a more general term, it could combine with many things, like random in order, or decide if a test should run randomly, or pick a random test, or both.
• shuffle specifically means random the order of an array, in which the items of that array are still the same.

Refs:

• https://wikidiff.com/random/shuffle
• https://www.italki.com/post/question-297430

Sure, let's name it shuffle instead.

[sheremet-va]

@antfu renamed :)