New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build and plugin hooks refactoring for code splitting and asset emission workflows #2208
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a tiny beginning of a review, looks really promising, great work so far!
src/rollup/types.d.ts
Outdated
@@ -265,10 +322,10 @@ export interface RollupCache { | |||
modules: ModuleJSON[]; | |||
} | |||
|
|||
export interface Bundle { | |||
export interface RollupFileBuild { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This name confused me at first as we are always building files. Maybe RollupSingleFileBuild
as the "single" part is the important one ?
this.property instanceof Identifier && this.property.name, | ||
options.format | ||
); | ||
if (importMetaMechanism) code.overwrite(this.start, this.end, importMetaMechanism); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we already touched this in one of the previous reviews but great this could be moved to MetaProperty
!
src/finalisers/amd.ts
Outdated
@@ -16,7 +16,7 @@ export default function amd( | |||
intro, | |||
outro, | |||
dynamicImport, | |||
importMeta, | |||
hasImportMetaUrl, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
9ec757d
to
fa62c2e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this is the next round of reviews. This time, I focused on trying out as much as possible though I did not manage to test everything yet.
Overall this is really impressive and seems really solid from an API perspective, well done 👍! Now what we need is more tests to get this ready to be merged. Quite a few things did not work properly or not at all for me and tests would also help to document how certain things are meant to be used and provide an easy way to check out edge case work flows.
Not sure if I can manage to continue my review tomorrow, it will probably be Monday. Thanks for the hard work!
src/Graph.ts
Outdated
}); | ||
asset.source = source; | ||
}, | ||
resolveId: undefined, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know it is only a very minor thing but I'd like to have a test that uses this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure will add one. There is a circular risk here too in calling this from the resolve hook itself, but I think that's just common sense to avoid.
src/Graph.ts
Outdated
}; | ||
|
||
this.pluginContext = { | ||
isExternal: this.isExternal, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should also have a test for this hook. In fact during the definition of the context, this.isExternal
is still undefined. The simplest approach could be to wrap it in a function, otherwise you probably need to reorder things.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
src/Graph.ts
Outdated
|
||
this.pluginContext = { | ||
isExternal: this.isExternal, | ||
emitAsset: (name: string, source?: string | Buffer) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder what is the use case of defining an assets in a two-step process, i.e. first we register
it and then at a later stage we set the source? And we can only set the source once?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason it's a two-step process is due to the order of information:
- You may want to construct an asset (say a CSS file) to match the list of modules defined by a chunk, so you will only know the asset source after generate is called.
- The chunk source itself might want to reference the asset URL via
import.meta.ROLLUP_ASSET_URL_sd9f8fg0s
, even though the asset source isn't yet known at this point. - Similarly, assets themselves might need to reference other assets, and only be generated late.
So the separation of assetId through the three methods allows us to dodge all these issues!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I see. import.meta.ROLLUP_ASSET_URL was one of the few things I had not tested yet. That makes sense!
src/utils/getAssetFileName.ts
Outdated
if (source === undefined) | ||
error({ | ||
code: 'MISSING_ASSET_SOURCE', | ||
message: `Plugin error creating asset ${assetName} The asset source must always be provided in order to support hash file names for assets.` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There should probably be a "." after the asset name.
src/utils/transform.ts
Outdated
code: 'PLUGIN_WARNING', | ||
id, | ||
source, | ||
pluginName: plugin.name |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If a plugin does not have a name (which happens often for ad-hoc plugins) the warning will read: "(!) undefined plugin: ..." which is somewhat confusing. One approach could be to set the pluginName
to "unnamed" which makes for less confusing warnings.
The same would apply to errors though in that case, the error message is less confusing (the message seems to assume this is not a plugin error). Still, I think if the error reads "[!] (unnamed plugin) Error: ...", this would provide the helpful information that the error did not originate from rollup itself but probably from some code the user wrote themselves.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another issue I stumbled upon: When I generate warnings or errors from the transformBundle
hook, I get a maximum callstack error instead.
return promise.then(code => { | ||
return Promise.resolve() | ||
.then(() => | ||
(plugin.transformChunk || plugin.transformBundle).call( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I must admit I could not get transformChunk
to be called at all in my experiments. A test to take some inspiration from would be really helpful.
src/utils/transformChunk.ts
Outdated
options: OutputOptions | ||
) { | ||
return graph.plugins.reduce((promise, plugin) => { | ||
if (!plugin.transformBundle) return promise; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, this might be the reason why transformChunk
does not work at all...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh, thanks.
src/rollup/index.ts
Outdated
try { | ||
const inputOptions = getInputOptions(rawInputOptions); | ||
initialiseTimers(inputOptions); | ||
const graph = new Graph(inputOptions); | ||
|
||
timeStart('BUILD', 1); | ||
for (let plugin of graph.plugins) { | ||
if (plugin.buildStart) plugin.buildStart.call(graph.pluginContext, inputOptions); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using this.warn
in the buildStart
and buildEnd
hooks also caused maximum callstack errors. I think we should have a few more simple tests for the plugin context API. There are already some tests in the function
category for this.warn
that could probably be easily adapted for the other hooks.
fa62c2e
to
1dcec6d
Compare
I'll integrate the feedback and some further changes I'd like to make on this tomorrow, so no need to review further before then. |
1dcec6d
to
53805de
Compare
Thanks for testing out those cases - I've fixed and included tests for these now. I also added support for even later asset emissions at the generateBundle phase, along with a Ready for further review! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, works much better! Did not manage to do as much testing today as I would have liked, just two comments for now.
src/rollup/index.ts
Outdated
|
||
if (!codeSplitting) { | ||
if (!inputOptions.experimentalCodeSplitting) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice and clean to have all these validations grouped together. I wonder if it would be possible to move these option validations out of the rollup function and into getInputOptions
? @
src/rollup/index.ts
Outdated
|
||
timeStart('GENERATE', 1); | ||
|
||
const generated: { [chunkName: string]: OutputChunk } = {}; | ||
// populate asset files into output | ||
const assetFileNames = outputOptions.assetFileNames || 'assets/[name]-[hash][ext]'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be really nice if "[ext]" would not contain the "." as in this case, I could use a pattern such as "[ext]/[name].[ext]" i.e. grouping different types of assets into different folders.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue with this is that in theory we support assets without an extension, so the [ext]
pattern can be the empty string. Perhaps we could introduce a new variable - [extname]
or something like that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note we can always move to supporting a function for these patterns as well -
assetFileNames: (vars) => {
if (vars.ext === '.css')
return `assets/style/${vars.name}`;
return `assets/${vars.name}`;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Supporting a function would definitely be great though I would not strictly require it for the initial version. But of course the default value should support that plugins emit extensionless files which would not be possible with my approach.
I am still not 100% happy with this as I think it could be confusing for non-Linux-natives that the dot is part of the extension (it was for me). Having a second placeholder could actually solve this.
Also there does not seem to be any prior art from the likes of Webpack.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that this is the same output as the NodeJS path.extname
, by inluding the dot.
Perhaps instead of [ext]
it should be called [extname]
with [ext]
being the version without the dot? That might make it more obvious to Node users too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good
src/rollup/index.ts
Outdated
|
||
timeStart('GENERATE', 1); | ||
|
||
const generated: { [chunkName: string]: OutputChunk } = {}; | ||
// populate asset files into output | ||
const assetFileNames = outputOptions.assetFileNames || 'assets/[name]-[hash][ext]'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Supporting a function would definitely be great though I would not strictly require it for the initial version. But of course the default value should support that plugins emit extensionless files which would not be possible with my approach.
I am still not 100% happy with this as I think it could be confusing for non-Linux-natives that the dot is part of the extension (it was for me). Having a second placeholder could actually solve this.
Also there does not seem to be any prior art from the likes of Webpack.
test/hooks/index.js
Outdated
}); | ||
}); | ||
|
||
it('supports warnings in buildStart and buildEnd hooks', () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This appears to be a duplicate
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks
test/hooks/index.js
Outdated
.rollup({ | ||
input: 'input', | ||
experimentalCodeSplitting: true, | ||
experimentalDynamicImport: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
acorn will only parse import.meta
if experimentalDynamicImport is true, otherwise it will produce an "unexpected token" error. Since we want this asset emission API to be used by plugins today, I guess this needs to be the PR where we make this feature no longer experimental and always add the acorn plugin.
Alternatively, we could just make sure the plugin for import.meta
is always added independent of the flag.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can unflag experimentalDynamicImport
, but I'm still weary of the dynamic import hook being documented.
test/hooks/index.js
Outdated
}); | ||
}); | ||
|
||
it('supports CommonJS asset urls', () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried this for amd
and got something like
define(function () { 'use strict';
console.log(new URL((typeof process !== 'undefined' && process.versions && process.versions.node ? 'file:' : '') + module.uri + '/../assets/asset.ext').href);
});
I would have expected module
to be imported by the amd loader similar to how it is done for import.meta.url
.
Also, system
is not supported at all as a format but throws in a rather non-pleasing way with a TypeError.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, will fix.
c15837d
to
429b55e
Compare
I've updated these now. In adding tests for the System output I caught a magic string bug, which this now depends on at Rich-Harris/magic-string#144. |
I've tweaks the hooks slightly with the following additional changes (updated in the PR summary as well):
Note that the I also created Rich-Harris/magic-string#145 for this to try optimize getting the length a bit. |
I love this very much. This could be the first step to enable a whole now class of analysis and visualization plugins. |
@@ -159,7 +159,7 @@ export default class ExportDefaultDeclaration extends NodeBase { | |||
); | |||
const hasTrailingSemicolon = code.original.charCodeAt(this.end - 1) === 59; /*";"*/ | |||
if (systemBinding) { | |||
code.appendLeft( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have thought a long time about this. So the reason you were having troubles is because the code directly to the left your insertion was overwritten which signals to magic-string to remove everything appended to it.
So the appendRight now either associates the inserted code with the semicolon or with the next white-space. I think it is not possible to have an export default declaration without some trailing white-space or semi-colon. So your fix should always work and I do not have a better idea.
But you should change the appendLeft
in the alternate case of the if statement as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's particularly the system binding case that suffers from the double write issue, where we want to add an appendage first, then later on rewrite the inner content. It might be more obvious think in terms of overwriteInner
and overwriteOuter
here, see Rich-Harris/magic-string#144 (comment) for further comment on this.
src/rollup/index.ts
Outdated
); | ||
}); | ||
// TODO: deprecate legacy single chunk return | ||
let singleInputChunk: Chunk; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be an optional type
src/rollup/index.ts
Outdated
) { | ||
singleInputChunk = chunks.find(chunk => chunk.entryModule !== undefined); | ||
imports = singleInputChunk.getImportIds(); | ||
exports = singleInputChunk.getExportNames(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe instead of defining imports and exports here and then use them the next time more than 200 lines later (with other variables of the same name in between) we can define them around line 365 and only add them specifically in the single input chunk case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At one point I was sharing this to avoid duplicate work, but now that it isn't that makes sense, thanks.
message: | ||
'UMD and IIFE output formats are not supported with the experimentalCodeSplitting option.' | ||
}); | ||
if (inputOptions.experimentalCodeSplitting) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So as far as I can see, it should be possible to do single file builds with experimentalCodeSplitting
turned on. So could we make it the default soon or are there still major things missing?
Really great work refactoring this though I am not sure I will catch everything here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly, I've turned it into a flag in this PR that we can then just disable when we are ready.
In terms of a checklist for launch, I think it would just be nice to have a little bit more feedback on usage patterns, to give us a bit more space to not worry about any unexpected API churn for a little while.
But I do think this gets us to almost stable... think of it as a release candidate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The other thing to note is that adding the flag is a breaking change in that imports, exports and modules as properties of the build bundle before generate are no longer present.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I see, yes that would be a breaking change. So maybe we keep it at least for the 0.60.0 and then remove it for the 1.0.0. One thing I keep wondering about is if it is a wise idea to directly return a map of the chunks from generate. This means we will never be able to add any other meta-information that is not chunk-specific here. Maybe instead we could return an object
{chunks: {[chunk]: <chunk-data>}}
here? That would leave the door open for extensions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing we could do for the 0.60.0, however, is to change all our tests to use experimentalCodeSplitting
and thus test the new code paths. Should be a great way to find lingering bugs and inconsistencies. I fear I will not be able to finish the review this week as I had planned as I will be on a hiking trip until Sunday but it looks really great!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks really good. There are still a few things I want to check which is playing around with inlined imports and having a look at tests which will unfortunately have to wait until Monday.
@@ -0,0 +1,20 @@ | |||
const assert = require('assert'); | |||
const path = require('path'); | |||
const URL = require('url-parse'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
url-parse needs to be added to rollup's dev dependencies. This is why tests on CI have been failing for some time.
message: | ||
'UMD and IIFE output formats are not supported with the experimentalCodeSplitting option.' | ||
}); | ||
if (inputOptions.experimentalCodeSplitting) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I see, yes that would be a breaking change. So maybe we keep it at least for the 0.60.0 and then remove it for the 1.0.0. One thing I keep wondering about is if it is a wise idea to directly return a map of the chunks from generate. This means we will never be able to add any other meta-information that is not chunk-specific here. Maybe instead we could return an object
{chunks: {[chunk]: <chunk-data>}}
here? That would leave the door open for extensions.
message: | ||
'UMD and IIFE output formats are not supported with the experimentalCodeSplitting option.' | ||
}); | ||
if (inputOptions.experimentalCodeSplitting) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing we could do for the 0.60.0, however, is to change all our tests to use experimentalCodeSplitting
and thus test the new code paths. Should be a great way to find lingering bugs and inconsistencies. I fear I will not be able to finish the review this week as I had planned as I will be on a hiking trip until Sunday but it looks really great!
I've adjusted the return value of generate and write to be Good idea on switching the tests to Added url-parse as a dependency now, hopefully that does fix the tests, thanks was wondering what was up with that. |
While playing around with the dynamic import inlining, I discovered the following bug (which actually seems to have existed for some time): // foo.js
export const foo = 1;
// main.js
import { foo } from './foo.js';
export const bar = 2
import('./foo.js')
// output when bundling main.js
const foo = 1;
var foo$1 = /*#__PURE__*/Object.freeze({
foo: foo
});
const bar = 2;
Promise.resolve().then(function () { return foo$1; });
export { foo }; Note that the bundle is actually exporting |
Considering that I would love to have this PR in 0.60.0 and that there are now some important fixes in master that should be released soon, I would actually postpone this. Once this is released, we could just do the actual switch to remove the flag and would not have to edit all config files twice. If you could have a look at the bug I outlined above, that would be really cool, though. Seems to be more like your area of expertise but I could have a look later myself as well. |
0418fbb
to
4160d25
Compare
4160d25
to
b9f085a
Compare
ea21780
to
649a046
Compare
Thanks for spotting the |
eabff85
to
a7058a6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! If there are no objections form your side, I will merge it tomorrow morning and release 0.60.0.
This is the rework of #2126.
Resolves #2163, #2092, #1611, #1629, #1282, #803.
Refactoring Summary (for the third time...):
rollup.rollup
lifecycle, regardless of generate or write calls.buildEnd
takes an error parameter if there was a build error.this.resolveId
andthis.isExternal
, as well as the existingthis.warn
andthis.error
and now applies equally for all plugin hooksentryNames
andchunkNames
are nowentryFileNames
andchunkFileNames
.inlineDynamicImports
input option, which by default is set to try for a single string input and false otherwiseAsset Emission Features
The goal here is to support workflows of building assets like CSS alongside chunks.
Use cases such as:
I've really gone back and fourth on a lot of these quite a bit to try and have Rollup do the minimal work necessary to keep maintenance down and to just open the right doors.
This PR provides:
assetFileNames
option is added, which by default is set toassets/[name]-[hash][ext]
.emitAsset(name: string, source?: string | Buffer) => assetId: string
,setAssetSource(assetId: string, source: string | Buffer)
andgetAssetFileName(assetId: string)
. This way an asset can be defined by a plugin at any time, and assets can also have their sources deferred until after the chunking and treeshaking so that they can perform optimizations based on matching the chunking and treeshaking behaviours.getAssetFileName
can only be called after both the asset source is set and generate has been called, throwing an error otherwise.import.meta.ROLLUP_ASSET_URL_[assetid]
is supported in code which is replaced in the build just likeimport.meta.url
but instead pointing to the URL of the given asset. Getting the AssetID early here is important to be able to emit these tokens in saytranslate
, while the asset source itself might only be populated afterwards.generateBundle(outputOptions, outputBundle, isWrite)
hook is provided ideally to replaceonwrite
andongenerate
in the deprecation path. TheoutputBundle
takes the form of{ [fileName: string]: Chunk | string | Buffer }
, where a string or buffer indicates an asset.{ [moduleId: string]: { renderedExports: string[], removedExports: string[], renderedLength: number, originalLength: number } }
. This way it is possible to determine what exports have been tree-shaken, as well as workout out how much source reduction that corresponds to.rollup.generate
androllup.write
still return the direct chunk in the single-file build case (being deprecated), and return the outputBundle object now by default when experimentalCodeSplitting is enabled.I think the above finally gets us these use cases in a relatively simple and straightforward way for Rollup, allowing plugins to explore the space further and drawing a clear line on some of these hard problems so that we don't fall too deep into the rabbit hole on these things.
@lukastaegert ready for review finally :)