Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] WIP: run tests concurrently; see #2839 #4198

Closed
wants to merge 92 commits into from

Commits on Apr 20, 2020

  1. Copy the full SHA
    ef40c9e View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    0a62f3f View commit details
    Browse the repository at this point in the history
  3. revert refactor of setting test.speed in runner

    increase timeout in `mocha.spec.js` "unit" test
    boneskull committed Apr 20, 2020
    Copy the full SHA
    db4b9a9 View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    7eac21a View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    d4a649e View commit details
    Browse the repository at this point in the history
  6. Copy the full SHA
    61ab96e View commit details
    Browse the repository at this point in the history
  7. browserify fixes

    boneskull committed Apr 20, 2020
    Copy the full SHA
    fbf7d8a View commit details
    Browse the repository at this point in the history
  8. add some tests

    - 100% coverage for `lib/serializer.js`
    - add a trivial integration test for `--parallel`
    - fix typo in integration test helper; output spawned command in a copy/pastable format
    - remove some unused code
    - rename `deserializeMessage` => `deserialize`
    - rename `serializeObject` => `serialize`
    - docstrings for `lib/serializer.js`
    - rewrite `SerializableEvent.serialize` as a loop instead of recursive function due to possibility of exceeding max stack trace; other refactors
    - do not freeze objects returned from various `Runnable`'s `serialize()` method, because `SerializableEvent#serialize` needs to mutate them.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    485d6ab View commit details
    Browse the repository at this point in the history
  9. add a lot of unit test coverage around serialization/deserialization

    - add more comments/docstrings
    - some refactors
    boneskull committed Apr 20, 2020
    Copy the full SHA
    e604fa0 View commit details
    Browse the repository at this point in the history
  10. increase timeout for options tests

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    d154141 View commit details
    Browse the repository at this point in the history
  11. upgrade unexpected-eventemitter

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    76105e7 View commit details
    Browse the repository at this point in the history
  12. add some tests for BufferedRunner

    - refactor `BufferedRunner#run` to be a void fn
    - tweak some "bail" handling
    
    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    def4815 View commit details
    Browse the repository at this point in the history
  13. add more buffered-runner tests

    - remove `promise.allsettled` shim as may not be needed
    - update `unexpected` to latest, which gives us a new assertion
    - tweak a unit test for `Runner` to leverage `unexpected-eventemitter`
    - tweak wallaby settings; increase test timeout
    boneskull committed Apr 20, 2020
    Copy the full SHA
    20c6604 View commit details
    Browse the repository at this point in the history
  14. Copy the full SHA
    5092f08 View commit details
    Browse the repository at this point in the history
  15. Copy the full SHA
    da7e206 View commit details
    Browse the repository at this point in the history
  16. Copy the full SHA
    ce015ee View commit details
    Browse the repository at this point in the history
  17. add unit-ish tests for worker

    - refactoring for `lib/worker.js`
    - fixed some docstrings
    boneskull committed Apr 20, 2020
    Copy the full SHA
    e3a38a9 View commit details
    Browse the repository at this point in the history
  18. add some assertions for RawRunResult to match JSONRunResult

    you can now use "to have passed test count", "to have failed test count" and "to have pending test count" with a `RawRunResult` (the result of calling `helpers.runMocha()`)
    
    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    947a00c View commit details
    Browse the repository at this point in the history
  19. Copy the full SHA
    b161fc0 View commit details
    Browse the repository at this point in the history
  20. Copy the full SHA
    45233d2 View commit details
    Browse the repository at this point in the history
  21. fix issue with serialization of cyclic objects

    this was discovered when testing `--parallel` with retries.
    
    also add some debug statements
    boneskull committed Apr 20, 2020
    Copy the full SHA
    8aaf72f View commit details
    Browse the repository at this point in the history
  22. Copy the full SHA
    9090d41 View commit details
    Browse the repository at this point in the history
  23. Copy the full SHA
    552d92b View commit details
    Browse the repository at this point in the history
  24. refactor/reorganize some of the integration test helpers

    - adds `runMochaAsync` and `runMochaJSONAsync` for convenience
    boneskull committed Apr 20, 2020
    Copy the full SHA
    269bccc View commit details
    Browse the repository at this point in the history
  25. Copy the full SHA
    9a2a816 View commit details
    Browse the repository at this point in the history
  26. Copy the full SHA
    b4c7b7f View commit details
    Browse the repository at this point in the history
  27. fix some flaky tests when run in wallaby

    - `config.spec.js` was misusing `rewiremock`
    - `mocha.spec.js` was loading files it shouldn't
    boneskull committed Apr 20, 2020
    Copy the full SHA
    2cb5768 View commit details
    Browse the repository at this point in the history
  28. Copy the full SHA
    4cfafda View commit details
    Browse the repository at this point in the history
  29. Copy the full SHA
    1b05f6f View commit details
    Browse the repository at this point in the history
  30. serialization: add missing functions

    `Runnable#isPending()` and `Suite#isPending()` needed by reporters, as was `Test#state`
    boneskull committed Apr 20, 2020
    Copy the full SHA
    c6e560d View commit details
    Browse the repository at this point in the history
  31. Copy the full SHA
    2c3b564 View commit details
    Browse the repository at this point in the history
  32. Copy the full SHA
    10cd2c4 View commit details
    Browse the repository at this point in the history
  33. Copy the full SHA
    4cfe37c View commit details
    Browse the repository at this point in the history
  34. add --parallel test for ESM

    boneskull committed Apr 20, 2020
    Copy the full SHA
    d5d6e89 View commit details
    Browse the repository at this point in the history
  35. fix serialization of circular data structures

    - more useful debug info
    - renamed a couple debug namespaces; doing `DEBUG=mocha:parallel*` now provides good general-purpose info
    boneskull committed Apr 20, 2020
    Copy the full SHA
    291b40a View commit details
    Browse the repository at this point in the history
  36. Copy the full SHA
    42e466c View commit details
    Browse the repository at this point in the history
  37. Copy the full SHA
    c9c3779 View commit details
    Browse the repository at this point in the history
  38. remove unnecessary bail handling

    - add more realistic tests around bail usage
    - serializer: rename `failures` to `failureCount` to be more obvious
    - warn if a single file is executed with `--parallel`
    - update some comments / debug statements
    boneskull committed Apr 20, 2020
    Copy the full SHA
    fc912e8 View commit details
    Browse the repository at this point in the history
  39. Copy the full SHA
    c8e8938 View commit details
    Browse the repository at this point in the history
  40. fix "sort" issue

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    aaef65f View commit details
    Browse the repository at this point in the history
  41. Copy the full SHA
    1a37e18 View commit details
    Browse the repository at this point in the history
  42. cleanup .mocharc.yml

    - quote strings because YAML is YAML
    - remove comment cruft
    boneskull committed Apr 20, 2020
    Copy the full SHA
    1c0971c View commit details
    Browse the repository at this point in the history
  43. Copy the full SHA
    9d75852 View commit details
    Browse the repository at this point in the history
  44. Copy the full SHA
    622d334 View commit details
    Browse the repository at this point in the history
  45. fix return value of BufferedRunner#run

    - also rename `opts` to `options`for optional param to `BufferedRunner#run`
    - tweak warning
    boneskull committed Apr 20, 2020
    Copy the full SHA
    e52fea8 View commit details
    Browse the repository at this point in the history
  46. Copy the full SHA
    938ce25 View commit details
    Browse the repository at this point in the history
  47. fix debug statements to use plain strings

    using template strings makes them eagerly-loaded, which is not needed
    boneskull committed Apr 20, 2020
    Copy the full SHA
    9a8adf6 View commit details
    Browse the repository at this point in the history
  48. reporter fixes for --parallel

    - `tap` reporter must report the plan at the END, which is spec-compliant.
    - `progress` reporter is incompatible because we do not know the total number of tests up-front.
    - `markdown` reporter _could_ be fixed, but it's too much effort atm.  the reason is we don't have a single root suite; we'd need to build the TOC based on every `EVENT_SUITE_BEGIN` begin where the parameter (a deserialized `Suite`) has a `root` prop of `true`.  `Suite#serialize` would need to recursively call `serialize()` on each item in its `suites` prop.
    - added a `SIGINT` handler to the `landing` reporter to restore the cursor and not befoul terminals
    boneskull committed Apr 20, 2020
    Copy the full SHA
    525cb7c View commit details
    Browse the repository at this point in the history
  49. implement --parallel --watch

    - rename `loadAsync` prop of `Mocha` to `lazyLoadFiles`, which is more descriptive of its intent
    - refactor a lot of `watch-run.js` to support these changes
    - remove unneeded `runOptions` parameter of `Mocha#run`
    - add a docstring or two
    - refactor `--watch` tests to avoid assigning to the context object
    boneskull committed Apr 20, 2020
    Copy the full SHA
    c9bfa67 View commit details
    Browse the repository at this point in the history
  50. integration test helper improvements

    - the major change is that `STDERR` will, by default, display in the terminal.  this can be used to catch problems that may otherwise have been overlooked, e.g., `test/integration/fixtures/uncaught/listeners.fixture.js`
    - `DEBUG` is explicitly removed from the env, so we can use it without a bunch of noise, and it will not affect inspection of child process output when `pipe` is used to grab `STDERR` from the child.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    a775327 View commit details
    Browse the repository at this point in the history
  51. implement an alternative way to define root hooks; works w/ --parallel

    When a module loaded by `--require` exports an object property `mochaHooks`, it can contain four properties corresponding to the four different types of hooks.
    
    Each prop can be a function or array of functions.  Here's an example using just functions:
    
    ```
    // hooks.js
    exports.mochaHooks = {
      beforeAll() {
        console.log('beforeAll');
      },
      beforeEach() {
        console.log('beforeEach');
      },
      afterAll() {
        console.log('afterAll');
      },
      afterEach() {
        console.log('afterEach');
      }
    };
    ```
    
    Use:
    
    ```
    mocha --require hooks.js test/**/*.spec.js
    ```
    
    `--file` does not work as expected before using `--parallel`, because order is non-deterministic.
    
    The `hooks.js` file (and anything else `--require`'d) will be loaded _once per child process_.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    9ffe03f View commit details
    Browse the repository at this point in the history
  52. remove warning about --parallel run with a single test file

    this gets annoying too quickly
    boneskull committed Apr 20, 2020
    Copy the full SHA
    926a934 View commit details
    Browse the repository at this point in the history
  53. rootHooks fix

    boneskull committed Apr 20, 2020
    Copy the full SHA
    7210ca0 View commit details
    Browse the repository at this point in the history
  54. another rootHooks fix

    boneskull committed Apr 20, 2020
    Copy the full SHA
    c79c359 View commit details
    Browse the repository at this point in the history
  55. Copy the full SHA
    fb6300b View commit details
    Browse the repository at this point in the history
  56. try to force color in CI

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    41dcbc9 View commit details
    Browse the repository at this point in the history
  57. json-stream reporter is incompatible with --parallel

    this could be fixed (but probably not without breaking changes), but again, `runner.total` is the issue.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    0415bb7 View commit details
    Browse the repository at this point in the history
  58. Copy the full SHA
    8d1a173 View commit details
    Browse the repository at this point in the history
  59. better failure modes

    - add `SIGINT` handling for clean worker death
    - don't sent `SIGTERM` in parallel mode; otherwise it can leave processes hanging around
    - make `--jobs=0` or `--jobs=1` cause Mocha to run in serial even if `--parallel` is true
    - reorganize some yargs checks
    - handle uncaught exceptions from worker processes, recovering where possible
    - handle `--allow-uncaught` to the best of our ability
    - ensure `uncaught.spec.js` doesn't flirt with max listeners
    - make `exit.spec.js` handle `SIGINT` and send `SIGTERM` as fixture should be run in serial, but also with `_mocha` (so there is no global `SIGINT` handler; unsure how the hell this worked before); renamed a misnamed test
    - skip ESM tests for `--parallel` in unsupported environments
    - fix `DEBUG` handling in integration tests
    - fix `bail`-related test for `--parallel`
    - adds module `promise.allsettled` as a polyfill, which allows us to recover from exceptions in worker processes
    - upgrades `unexpected-eventemitter` to v2.1.0, which allows for making event assertions in async functions
    boneskull committed Apr 20, 2020
    Copy the full SHA
    7f35825 View commit details
    Browse the repository at this point in the history
  60. utils: add a test for async functions in type()

    an async function (`async () => {}`) will return `asyncfunction` instead of `function`.
    who knew!?
    
    limited to node tests for now.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    da006c7 View commit details
    Browse the repository at this point in the history
  61. all for async definition of mocha root hooks via --require

    a module can now do:
    
    `exports.mochaHooks = async () => ({/* mocha hooks obj */})`
    
    in addition to
    
    `exports.mochaHooks = { /* mocha hooks obj */ }`
    
    (the function style can be sync or async)
    
    to do this, we use a `middleware()` function in yargs, which (by default)
    runs _after_ the checks, but before the handler, _and_ can be async, which it is.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    43ae67f View commit details
    Browse the repository at this point in the history
  62. Copy the full SHA
    e289e40 View commit details
    Browse the repository at this point in the history
  63. remove some done TODOs

    boneskull committed Apr 20, 2020
    Copy the full SHA
    28158ef View commit details
    Browse the repository at this point in the history
  64. Copy the full SHA
    63d703a View commit details
    Browse the repository at this point in the history
  65. refactor validatePlugin() to throw nicer errors

    created `createInvalidPlugin` method in `lib/errors.js`
    boneskull committed Apr 20, 2020
    Copy the full SHA
    680240f View commit details
    Browse the repository at this point in the history
  66. exit fixes

    - issue `SIGKILL` in `exit.spec.js` to avoid a potential exception when quitting
    - main mocha executable: use `SIGKILL` instead of unsupported `SIGTERM` on main subprocess in win32.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    4b96461 View commit details
    Browse the repository at this point in the history
  67. fix jsdoc types

    boneskull committed Apr 20, 2020
    Copy the full SHA
    97561ad View commit details
    Browse the repository at this point in the history
  68. Copy the full SHA
    a0423a0 View commit details
    Browse the repository at this point in the history
  69. improve another regexp in --require test

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    118c137 View commit details
    Browse the repository at this point in the history
  70. Copy the full SHA
    6afce46 View commit details
    Browse the repository at this point in the history
  71. Copy the full SHA
    813016b View commit details
    Browse the repository at this point in the history
  72. fix integration test helper to not completely ignore SIGINT

    also allows override of `DEBUG` in environment if we really need it.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    bcc0e4e View commit details
    Browse the repository at this point in the history
  73. BufferedRunner fixes

    - do not force-terminate worker pool when bailing (unnecessary)
    - use a finite-state-machine-ish thing to manage state
    - "correctly" resend `SIGINT` in cleanup listener
    - differentiate between "bailing" and "aborting", which have diff't behaviors:
        - "bailing" is the `bail` flag
        - "aborting" is either an uncaught exception (with `--allow-uncaught`) _or_ a `SIGINT` signal
    boneskull committed Apr 20, 2020
    Copy the full SHA
    88887b6 View commit details
    Browse the repository at this point in the history
  74. try to fix ESM test

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    6545108 View commit details
    Browse the repository at this point in the history
  75. tweak nyc

    boneskull committed Apr 20, 2020
    Copy the full SHA
    e10b5e6 View commit details
    Browse the repository at this point in the history
  76. add warning if no multiple cores

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    fdf9916 View commit details
    Browse the repository at this point in the history
  77. get some debug info

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    d8c7fbc View commit details
    Browse the repository at this point in the history
  78. Copy the full SHA
    e23b4e2 View commit details
    Browse the repository at this point in the history
  79. fix test timeouts in slow parallel tests

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    52368a9 View commit details
    Browse the repository at this point in the history
  80. reduce debug noise from worker

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    a509d1c View commit details
    Browse the repository at this point in the history
  81. fix ESM parallel tests

    exposes `getSummary()` from integration test helpers
    boneskull committed Apr 20, 2020
    Copy the full SHA
    18f7030 View commit details
    Browse the repository at this point in the history
  82. fix node arg passing to worker processes

    warnings about too many jobs requested; add test
    boneskull committed Apr 20, 2020
    Copy the full SHA
    cba2d7e View commit details
    Browse the repository at this point in the history
  83. Copy the full SHA
    4ad404f View commit details
    Browse the repository at this point in the history
  84. fix --require integration test

    mainly fiddling with regex's.  `[^]` is apparently deprecated (?), so use `[\s\S]` instead.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    7e2717b View commit details
    Browse the repository at this point in the history
  85. Copy the full SHA
    25d57ec View commit details
    Browse the repository at this point in the history
  86. add a serial-mode run (and job names?)

    Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
    boneskull committed Apr 20, 2020
    Copy the full SHA
    e934030 View commit details
    Browse the repository at this point in the history
  87. Copy the full SHA
    4328619 View commit details
    Browse the repository at this point in the history
  88. Copy the full SHA
    80d5c06 View commit details
    Browse the repository at this point in the history
  89. squelch warning about missing language in a fenced code block

    markdownlint wants a language or fails, but prism doesn't know what to
    do with `text` or `plain`, so let's just forget it.
    boneskull committed Apr 20, 2020
    Copy the full SHA
    da44ea1 View commit details
    Browse the repository at this point in the history
  90. Copy the full SHA
    f5a8a76 View commit details
    Browse the repository at this point in the history
  91. Copy the full SHA
    c8236d5 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2020

  1. fix bad anchors

    boneskull committed Apr 21, 2020
    Copy the full SHA
    0b13c92 View commit details
    Browse the repository at this point in the history