Plugin system for extending cucumber-js

[davidjgoss]

🤔 What's the problem you're trying to solve?

If you want to augment the core functionality of Cucumber - for something that's not a formatter - you have two choices:

1. Push it into the core (i.e. what has been done for publishing to the reports service)
2. Wrap Cucumber, influencing its inputs and acting on its outputs (i.e. what has been started for TCP-based ordering)

Neither of these options are great for the average user with a good idea, and both of these specific use cases would be better served by an official plugin mechanism so you can hook into Cucumber's test run lifecycle, change things, and react to things.

✨ What's your proposed solution?

TBA!

At a basic level we need two things:

A way to write a plugin

The bare minimum might be something like:

export default {
  async init() {}
  onMessage(envelope) {}
}

We'll also likely need:

• A way to provide configuration for/to the plugin
• A way for the plugin to output to stderr (e.g. the publish banner)
• "Lifecycle hooks" so plugins can provide a function which can influence core logic - so for the TCP case something like async inferOrder(pickles: Pickle[]): Promise<Pickle[]>

A way to reference a plugin

Something like:

npx cucumber-js --plugin @cucumber/publish

---

I think a good start would be to create some plumbing for plugins internally, and reimplement the publish functionality in terms of that. I'm happy to give that a go.

[aurelien-reeves]

I like the idea, and I agree with your analysis and choices 👌

[mattwynne]

I think an approach where we try to identify and refactor existing/internal code towards using an API boundary that we could stabilise and make public sounds great. Happy to pair with you on this @davidjgoss if you want to bounce some ideas around.

Not necessarily solving the same problem but worth adding for context:

When @tooky and I re-wrote the guts of cucumber-ruby, we ended up with a "plugin" mechanism based on the chain-of-responsibility design pattern. Express's request handlers are another example of the same pattern. Each plugin (or "filter" as we called them) receives the stream of test cases (like a pickle, but with extra info about how to invoke it) and is given a receiver (the next filter in the chain) to whom it should pass on whichever test cases it chooses. It can filter out test cases by choosing not to pass them on, and it can also enrich or otherwise mutate test cases before passing them on.

A lot of code fell into place when we adopted this model, allowing us to assemble most of Cucumber's behavour from lots of small pieces, e.g:

• filtering test cases by name
• filtering and sorting test cases by location
• glueing step definition functions to the test case's steps
• randomizing test cases
• initializing the world

I'm not sure if you already have something like this inside cucumber-js, but it might be worth thinking about as part of this. I'm pretty sure it would provide what the TCP extension needs, for example.

[mattwynne]

https://github.com/cucumber/cucumber-electron is another piece worth considering in all this, currently implemented as a wrapper but we agreed (in theory) that it should really be part of this codebase.

[mattwynne]

More prior art is the https://github.com/cucumber/cucumber-ruby-wire/blob/main/lib/cucumber/wire.rb extension for cucumber-ruby which does use an officially-supported (though poorly documented / publicised) plugin API.

[charlierudolph]

Formatters could also be a special form of a plugin. Maybe have someway for the plugin to declare its a formatter and thus to get passed a function to log (which will be either a file stream or stdout)

In general, like the idea. Very curious what exactly will be the scope of what get supported

[davidjgoss]

Formatters could also be a special form of a plugin. Maybe have someway for the plugin to declare its a formatter and thus to get passed a function to log (which will be either a file stream or stdout)

I really like this idea. We could definitely make it easier than the current class-based formatter system. And providing a function to write without direct access to the stream would be an improvement too.

[davidjgoss]

I just raised a PR that refactors the publish functionality to be a plugin (only an internal concept for now). So far this is just a plugin that passively listens for messages and does stuff outside of Cucumber off the back of that, but it's a good draft to drive some more discussion.

[davidjgoss]

As I start to experiment with other use cases for this, it occurs to me that a plugin, much like a formatter, can only really operate at the "coordinator" level, since test execution often happens in a worker process (via the parallel runtime). This is kind of similar to Cypress, where a plugin operates within the Node.js process, and anything you want to happen in the browser is done via support code.

I don't think this is a problem per se, or that we should change direction. I think most of the interesting plugin use cases are coordinator-level. But I think long term we'll want to come up with a nice solution for:

• Causing extra support code to be loaded from a plugin (maybe via an extension point to modify the configuration before code is loaded?)
• Communicating with a plugin from support code

[davidjgoss]

I've started tagging issues with plugins-usecase where it's not something that'd be a core feature but we could potentially support with plugins: https://github.com/cucumber/cucumber-js/issues?q=is%3Aopen+is%3Aissue+label%3Aplugins-usecase