Support custom watchers

[alexeagle]

Feature Use Case

My build system (Bazel) knows when rollup inputs are changed and when to trigger rollup to re-bundle. For example, it might have a number of independent typescript compilations, following the tsc --build model, where .js outputs are written over a period of time. Normally chokidar will spuriously trigger multiple times during the typescript build as it sees newly written .js files on disk. Instead under my build system, I would trigger rollup exactly once, after all its prerequisite steps have run.

Feature Proposal

Allow a custom watcher implementation to be passed to the Rollup watch API.
This might be constrained to match the API surface of chokidar for convenience.

Webpack allows this via a custom filesystem https://webpack.js.org/api/node/#custom-file-systems

(Note, @jbedard or I might contribute this if we agree on a design for it.)

[lukastaegert]

I think this is in general a good idea, but the question is how to sweat the details, maybe you have some opinions? Personally, I would love to have this entirely via a plugin API, but the open question is how this API should work.

Option 1: There can be only one plugin to implement this, and this plugin takes over the entire watch system, i.e. Rollup watching is disabled unless the plugin triggers rebuilds. This would probably be good enough here but would have its limitations.

Option 2: There can be multiple watcher plugins. ids to be watched are passed to each plugin in order until one plugin signals that it handles watching this id. Then it will not be passed to other plugins. If no plugin watches an id, Rollup takes over. Makes more sense to me as it would basically also allow plugins to just handle watching individual files, e.g. to invalidate "virtual" files when there is an external reason for them to change. Question is though how files are passed to each plugin. Passing them individually would be easiest to implement. Alternatively we could also just pass all files to the first plugin, but then the plugin would need to take care of passing on the rest. I think passing them individually should be good enough performance wise.

[lukastaegert]

This would also be nice for TypeScript workflows where we could re-use TypeScripts watch mode.

[tivac]

Option 1 seems totally fine to me, but I think if you're going to support Option 2 and the waterfall behavior it should definitely be one file at a time. Dealing with an array of inputs and outputs is gonna be very easy to make mistakes in the filtering code, whereas a single file is already the common pattern in plugins and very idiomatic to filter via pluginutila.

[tivac]

@alexeagle does that build even need to use the rollup watcher? If you know exactly when the updated files are written it seems like you could invoke rollup at that point, using the cache, and get exactly what you're looking for with regard to fine grained control over when it runs?

[jbedard]

If you know exactly when the updated files are written it seems like you could invoke rollup at that point, using the cache, and get exactly what you're looking for with regard to fine grained control over when it runs?

That's exactly what we've done, but that means we can't use the rollup CLI and need to essentially re-invent the CLI, merge CLI + config files etc (unless that was made available as an API, converting CLI args to a single rollup config that you can pass to the API).

(Sort of related... we also discovered #3465 while using this API so it's not very usable atm)

[tivac]

There's an API for loading config files the exact way the CLI does it as of 2.3.0, funnily enough.

Does a watcher-powered rebuild not exhibit the same behavior from #3465?

[jbedard]

There's an API for loading config files the exact way the CLI does it as of 2.3.0, funnily enough.

Does it also support parsing CLI arguments though? Or only the config? I assume this is essentially exposing what I copied?

Does a watcher-powered rebuild not exhibit the same behavior from #3465?

I'll have to double check, but I was wondering if the CLI does something special that avoids it because the consequences of the bug are pretty bad and noticable in large projects. I'll post back here after I get a chance to try it out.

[jbedard]

Does a watcher-powered rebuild not exhibit the same behavior from #3465?

Looks like it does: #3465 (comment)

[lukastaegert]

Does it also support parsing CLI arguments though? Or only the config?

It includes CLI parsing. CLI arguments need to be provided as an object, though. The only CLI arguments that are ignored are those that are not reflected as Rollup options, e.g. env.

[lukastaegert]

does that build even need to use the rollup watcher? If you know exactly when the updated files are written it seems like you could invoke rollup at that point, using the cache, and get exactly what you're looking for with regard to fine grained control over when it runs?

The advantage is that Rollup watcher knows which files need to be watched, including those registered by plugins via this.addWatchFile. Via this API, we could even add basic watching without file system access to the browser build.

[jbedard]

It includes CLI parsing.

Thanks for the info 😁

[Pauan]

I have a similar problem: I created a Rollup plugin for Rust. This plugin converts Rust code into .js + .wasm files and then imports the .js files.

Everything works fine, except whenever the .js files change it triggers a recompilation (which causes some other problems).

I would like to prevent Rollup from watching the .js files, because the plugin knows best when to recompile. So I guess in my case I would actually want a this.removeWatchFile(id) method.