Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for cargo nextest && Fixed support for deno #3920

Open
wants to merge 223 commits into
base: main
Choose a base branch
from

Conversation

spigaz
Copy link

@spigaz spigaz commented Apr 10, 2024

This is still work in progress:
https://nexte.st/book/custom-test-harnesses.html

Progress

  • Added support for -h and --help w/tests
  • Added support for -V and --version w/tests
  • Added support for --list --format terse w/tests
  • Added support for --list --format --ignored w/tests
  • Add support for <test-name> --nocapture --exact w/tests
  • Reinstated support for --include-ignored w/tests
  • Reinstated support for --skip name w/tests
  • Reinstated support for --skip=name w/tests
  • Reinstated support for filter w/tests
  • Solved the temporary directory conflict caused by the concurrent invocation
  • Update --include-ignored to use docopt
  • Update --skip name to use docopt
  • Added tests to execute tests on nodejs
  • Fixed support and added tests to execute tests on deno
  • Add tests to execute tests on chrome
  • Added tests to execute tests on firefox
  • Add tests to execute tests on safari
  • Add tests to execute tests on edge
  • Add tests with cargo nextest invoking wasm-bindgen-test-runner
  • Add feature! macro to simplify testing
  • Add feature_template_runtime! to execute the specification over all the runtimes

Updated to use docopt

  • It was used already in wasm-bindgen
  • Not sure if you want to use something more modern, its easy to change
  • Just checked its unmaintained https://github.com/docopt/docopt.rs and recommends clap or structopt, but structopt's docs state "As clap v3 is now out, and the structopt features are integrated into (almost as-is), structopt is now in maintenance mode"

Updated the macro wasm_bindgen_test

  • To export and additional method to allow listing handling as required by nextext
  • Not sure if it might make sense to override current export

Testing

  • I wasn't sure what was the best place to put the tests in, because of the custom test runner in the cli crate, so I placed them on main tests folder.

  • I used a variation of BDD that I use for many years now.
    -- Although it seems a bit more verbose, it makes writing tests a lot simpler and faster, anyone can understand them and add new ones.
    -- When something breaks its very easy to understand why.
    -- Although conter-intuitive, in practice I have found that they are a lot easier to update on refactors.

Overhead

  • When used from cargo nextest the overhead is pretty real, as it has to load the runtime and the wasm into it, so it can be a lot, but it runs them in parallel, so it makes up for it on tests that take a long to execute.

Architecture:

  • I would prefer to organize the code into:
    -- a folder with the CLI and environment stuff
    -- a folder with the wasm handling
    -- a folder with the runtimes

@daxpedda
Copy link
Collaborator

Apologies for the delay, I was on vacation, still catching up.
Planning to take a look tomorrow!

@spigaz
Copy link
Author

spigaz commented Apr 18, 2024

That's okay!
I still have one missing piece ahead to get it working with cargo nextest, so its fine.

@spigaz
Copy link
Author

spigaz commented Apr 22, 2024

Well its working, actually its my second version working, the fist one used locks.

But according to the the library documentation it shouldn't work (at least cross-platform):

See the tests in lib.rs for cross-platform lock behavior that may be relied upon
// Concurrent shared access is OK, but not shared and exclusive.
// Once all shared file locks are dropped, an exclusive lock may be created;
// No other access is possible once an exclusive lock is created.
// Once the exclusive lock is dropped, the second file is able to create a lock.
https://github.com/danburkert/fs2-rs/blob/master/src/lib.rs

Anyway, the overhead was as bad as I expected, as each test execution now requires the same overhead as a complete assembly, I only tested on firefox so far, about 12-13s each, thermal throttled.

But when tests take longer than the overhead, the extra cores start to payoff.

I'm having thermal throttling issues, but my speed boost might be around 3x when I have the tests configured for very heavy parameters, its a property based testing variation.

@spigaz spigaz changed the title WIP: Add Support for cargo nextest Add Support for cargo nextest Apr 23, 2024
…warning to invocation with_an_empty_assembly.
…gen-cli because of dependency of wasm-bindgen-test-runner. Updated assembly builder to avoid runner runtime conflicts.
…ation with_an_assembly with_one_successful_test.
…ructure to allow multiple instances of the runner using the same assembly to make the tests faster.
…t generate random suffix for assembly directory name.
…Execution to invocation with_an_assembly with_one_successful_test.
…ation with_an_assembly with_one_failing_test.
…to invocation with_an_assembly with_one_failing_test.
…ution to invocation with_an_assembly with_one_failing_test.
…invocation with_an_assembly with_one_failing_test.
…ation with_an_assembly_with_one_successful_and_one_failing_test.
…execution with_an_assembly_with_one_successful_and_one_failing_test.
…invocation with_an_assembly with_arguments TESTNAME --nocapture --exact with successful_test_match_1_of_2.
…ts of the CLI handling of TESTNAME --nocapture --exact.
…st to the invocation with_an_assembly with_arguments TESTNAME --nocapture --exact with successful_test_match_1_of_2.
…d --exact arguments, although not yet used, but clippy forced it.
…t summary to the invocation with_an_assembly with_arguments TESTNAME --no-capture --exact with_successful_test_match_1_of_2.
…bout test 2 to the invocation with_an_assembly with_arguments TESTNAME --no-capture --exact with_successful_test_match_1_of_2.
…an_assembly with_arguments TESTNAME --no-capture --exact with_successful_test_match_2_of_2.
…an_assembly with_arguments TESTNAME --no-capture --exact with_partial_test_match.
…an_assembly with_arguments TESTNAME --no-capture --exact without_test_match.
…est 1 standard output to the invocation with_an_assembly with_arguments TESTNAME --nocapture --exact with_successful_test_match_1_of_2.
…est 1 standard output to the invocation with_an_assembly with_arguments TESTNAME --nocapture --exact with_successful_test_match_2_of_2.
@spigaz
Copy link
Author

spigaz commented Apr 27, 2024

I was waiting for your review, because as I do more tests, I end up finding more stuff to fix, making the review harder on you. Sorry.

I ended up fixing support for deno, not sure why, but it was for sure broken.
I added some tests and updated the github action, so now its under control.
You can cherry pick that if you want.

@spigaz spigaz changed the title Add Support for cargo nextest Add Support for cargo nextest && Fixed support for deno Apr 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants