[v3.0] New hashing algorithm that "fixes (nearly) everything"

[lukastaegert]

This PR contains:

• bugfix
• feature
• refactor
• documentation
• other

Are tests included?

• yes (bugfixes and features will not be merged without tests)
• no

Breaking Changes?

• yes (breaking changes will not be merged unless absolutely necessary)
• no

List any relevant issue numbers:

• Resolves Chunk hashes should be based on the output #2839
• Resolves Parametrize output.banner/output.footer #3398
• Resolves JS hash fails to change when asset URL changes #3415
• Resolves JS hash changes when src folder moved #3416
• Resolves Ability to collect chunk info before or during renderchunk #3519
• Resolves Inconsistent (JS) Output Hashes with emitted CSS files #3697
• Resolves A method of adding an import based on dependency graph #3790
• Resolves I need a longer hash with customizable length. #4317
• Resolves How to keep [hash] unchanged when in different path #4426

Description

This is the one feature I always wanted to have ready before releasing Rollup v3, which ultimately made me delay that release far too much.

The previous hashing algorithm

This is how the previous chunk hashing algorithm worked:

• render all modules except dynamic imports and import.meta chunk references
• based on this, calculate the content hash of all modules in a chunk
• extend those hashes by taking all known dependencies into account and everything dynamic that may be added in the chunk wrapper
• update the dynamic imports and import.meta chunk references
• render the chunk wrappers that include all static imports and exports
• run the result through the renderChunk plugin hook

This approach has several problems:

• any transformation in renderChunk is completely ignored.
• taking every possible change in the chunk wrappers into account is a fragile business
• hashes are chronically unstable as they may take things into account that are actually not relevant for the result.

Possible solution: Sorted rendering

One way to solve this would be to first render all chunks that do not have any dependencies, then iteratively render all chunks that only have rendered chunks as dependencies until all chunks are rendered.
While this would work in some cases, this approach has several drawbacks:

• It does not support circular dependencies between chunks. This is a very important feature because in that context, a circular dependencies would also be two chunks that dynamically import each other. Moreover, Rollup heavily relies on this for dynamic imports by moving all dependencies that are shared between the importing chunk and the import target into the importing chunk, causing a static import of the importing chunk to appear in the imported chunk.
• It means we need to know all dependencies of a chunk before rendering the chunk even though the renderChunk hook might introduce additional dependencies.

Implemented solution: Hash placeholders

Therefore I decided upon a different solution that works like this:

• Assign a preliminary chunk name to each chunk. If there is no hash in the file name this will be the final file name, but if it would contain a hash it will instead contain a placeholder of equal length.
• Render all modules in the chunk. As we already have the preliminary file names, we can directly render all dynamic imports and import.meta chunk references.
• Render the chunk wrapper. We also use the preliminary file names for the chunk imports.
• Run the chunk through the renderChunk hook.
• Calculate the content-only hash of the chunk by replacing all placeholders in the chunk with a default placeholder and then generating a hash.
• Once this is done for all chunks, calculate the final hashes by searching which placeholders are contained in each chunk and updating the chunk hash, taking the content-only hashes of all transitive dependencies into account.
• Replace the placeholders with the final hashes. As we made sure the placeholders have the same length as the hashes, we do not need to update sourcemaps.

In order to avoid accidentally replacing non-placeholders, the placeholders make use of the fact that JavaScript supports unicode characters. I used four random characters from the reserved plane, currently \uf7f9\ue4d3 for the start of the placeholder and \ue3cc\uf1fe for the end of the placeholder.

This approach also allows us, to give full access to the chunk graph in renderChunk, though the names are preliminary at this point. But as Rollup does not make assumptions about the output of renderChunk, you can freely inject chunk names in that hook now.

All in all, this changes the plugin hook graph in the following way:

flowchart TB
    classDef default fill:transparent, color:#000;
    classDef hook-parallel fill:#ffb3b3,stroke:#000;
    classDef hook-sequential fill:#ffd2b3,stroke:#000;
    classDef hook-first fill:#fff2b3,stroke:#000;
    classDef hook-sequential-sync fill:#ffd2b3,stroke:#f00;
    classDef hook-first-sync fill:#fff2b3,stroke:#f00;

    augmentchunkhash("augmentChunkHash"):::hook-sequential-sync
    click augmentchunkhash "/guide/en/#augmentchunkhash" _parent

	banner("banner"):::hook-sequential
	closebundle("closeBundle"):::hook-parallel
	footer("footer"):::hook-sequential
	generatebundle("generateBundle"):::hook-sequential
	intro("intro"):::hook-sequential
	outputoptions("outputOptions"):::hook-sequential-sync
	outro("outro"):::hook-sequential
	renderchunk("renderChunk"):::hook-sequential
	renderdynamicimport("renderDynamicImport"):::hook-first-sync
	rendererror("renderError"):::hook-parallel
	renderstart("renderStart"):::hook-parallel
	resolvefileurl("resolveFileUrl"):::hook-first-sync
	resolveimportmeta("resolveImportMeta"):::hook-first-sync
	writebundle("writeBundle"):::hook-parallel

	outputoptions
	--> renderstart
	--> |each chunk|beforerenderdynamicimport

	afteraddons
	--> |each chunk|renderchunk

	augmentchunkhash
	--> generatebundle
	--> writebundle
	.-> closebundle

	subgraph generateChunks [" "]
	    direction TB
	    beforerenderdynamicimport(( ))
        ---> beforeresolveimportmeta(( ))
        ----> beforereaddons(( ))
        --> banner & footer & intro & outro
        --> afteraddons(( ))
        .-> |next chunk|beforerenderdynamicimport

	    beforerenderdynamicimport
        --> |"each import()"|renderdynamicimport
        --> beforerenderdynamicimport

    	beforeresolveimportmeta
    	--> |each import.meta.*|beforeimportmeta(( ))
    	--> |import.meta.url|resolvefileurl
    	--> afterresolveimportmeta(( ))

	    beforeimportmeta
	    --> |other|resolveimportmeta
	    --> afterresolveimportmeta

	    afterresolveimportmeta
	    --> beforeresolveimportmeta
	end

	renderchunk
    --> augmentchunkhash
    .-> |next chunk|renderchunk

	style generateChunks stroke-width:0px;

	rendererror
	.-> closebundle

Chunk information available to hooks and options

Some hooks now receive additional information. To go into details, let's first define some types:

• A PrerenderedChunk contains very basic chunk information before any rendering takes place and before the chunk name has been generated. After this update, this reduced set of chunk information is only passed to the entryFileNames and chunkFileNames options. Note that for obvious reasons, we no longer have the rendered module information available. Instead, this type now gets a list of included moduleIds so that one can at least know what is roughly contained in the chunk.

  interface PreRenderedChunk {
  	exports: string[];
  	facadeModuleId: string | null;
  	isDynamicEntry: boolean;
  	isEntry: boolean;
  	isImplicitEntry: boolean;
  	moduleIds: string[];
  	name: string;
  	type: 'chunk';
  }

• A RenderedChunk contains the full rendering information of the chunk. File names in imports and rendered modules will contain placeholders instead of file hashes, though. This information is available in the renderChunk hook, the augmentChunkHash hook, and in the banner/footer/intro/outro hooks and options.

  Also, the signature of renderChunk is extended to receive a fourth parameter meta: { chunks: { [fileName: string]: RenderedChunk } } that gives access to the entire chunk graph.

Note also that when adding or removing imports or exports in renderChunk, plugins should now take care to update the RenderedChunk object they receive to reflect the updated information. This will provide correct information to subsequent plugins and to the final bundle as Rollup will also replace hashes in imports, importedBindings and dynamicImports (besides implicitlyLoadedBefore and fileName).

interface RenderedChunk {
	dynamicImports: string[];
	exports: string[];
	facadeModuleId: string | null;
	fileName: string;
	implicitlyLoadedBefore: string[];
	importedBindings: {
		[imported: string]: string[];
	};
	imports: string[];
	isDynamicEntry: boolean;
	isEntry: boolean;
	isImplicitEntry: boolean;
	moduleIds: string[];
	modules: {
		[id: string]: RenderedModule;
	};
	name: string;
	referencedFiles: string[];
	type: 'chunk';
}

Additional new features

• If intro, outro, banner, footer are functions, they are now called per chunk. While they do not get access to the rendered modules in the chunk, they get a list of all included module ids.
• Hash length can be changed in the file name pattern, e.g. [name]-[hash:12].js will create 12 character hashes

Breaking changes

• entryFileNames and chunkFileNames no longer get access to a modules object with the content of rendered modules. Instead, the get a list of included moduleIds.
• The order of plugin hooks has changed, compare the graph above with the graph at https://rollupjs.org/guide/en/#output-generation-hooks
• The 'fileNameand referencedimportsin therenderChunk` hook will get file names with placeholders instead of hashes. These file names can be safely used in the return value of the hook, though, as any hash placeholders will be replaced with the actual hashes in the end.

Also, some deprecated features have been removed, and the Node compatibility target has been raised, but I will create separate PRs for those to make them explicit.

[github-actions]

Thank you for your contribution! ❤️

You can try out this pull request locally by installing Rollup via

npm install rollup/rollup#new-hashing

or load it into the REPL:
https://rollupjs.org/repl/?pr=4543

[codecov]

Codecov Report

Merging #4543 (4f1f6af) into release-3.0.0 (490b49b) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@                Coverage Diff                @@
##           release-3.0.0    #4543      +/-   ##
=================================================
+ Coverage          98.88%   98.90%   +0.01%     
=================================================
  Files                208      210       +2     
  Lines               7282     7377      +95     
  Branches            2081     2101      +20     
=================================================
+ Hits                7201     7296      +95     
  Misses                26       26              
  Partials              55       55              

Impacted Files	Coverage Δ
src/finalisers/index.ts	100.00% <ø> (ø)
src/finalisers/shared/getExportBlock.ts	100.00% <ø> (ø)
src/finalisers/shared/trimEmptyImports.ts	100.00% <ø> (ø)
src/utils/renderHelpers.ts	100.00% <ø> (ø)
src/Bundle.ts	100.00% <100.00%> (ø)
src/Chunk.ts	100.00% <100.00%> (ø)
src/ExternalChunk.ts	100.00% <100.00%> (ø)
src/ExternalModule.ts	100.00% <100.00%> (ø)
src/Module.ts	100.00% <100.00%> (ø)
src/ast/nodes/ImportExpression.ts	100.00% <100.00%> (ø)
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 490b49b...4f1f6af. Read the comment docs.

[lukastaegert]

@yyx990803, @patak-dev, @antfu, @marvinhagemeister while it will take some time before I release this with Rollup 3 as I want to get some other PRs ready, this may be quite relevant for you, and supporting this will likely be a breaking change for you as well.
I will try to create a first pre-release version soon.

[patak-dev]

@lukastaegert this is huge! It will allow some heavy simplifications in Vite core, we may be able to remove most of our custom asset path handling in favor of a solution that plays well with other plugins in the rollup ecosystem.

It is also quite timely. We were about to release Vite v3, in around 10 days. But we may need to evaluate delaying it to already include the breaking to rollup v3. We may probably release v3, and then do another major in a few months. Thanks for the heads up, and a pre-release version will be quite helpful. Do you have some ETA in mind for releasing Rollup 3 stable?

[marvinhagemeister]

This is awesome!! Thank you so much for tackling this problem 💯

[lukastaegert]

Thank you! @patak-dev Rollup 3 will definitely not be ready in 10 days, but a solid pre-release should be no problem. I hope we can pull off v3 in a month or two at most, I will try not to put too much else in it, but there are some other smaller breaking changes that have been held back for some time that I want to tackle.

[patak-dev]

Thanks for the info @lukastaegert, after some discussions with the team and ecosystem, we are thinking of keeping our current plans and releasing Vite v3 using Rollup v2 in around ~10 days. Once Rollup v3 is stable, and plugins in the rollup ecosystem have been updated to support it, we can release Vite v4. So no rush from our side. Thanks again for all your work on improving rollup.

[frank-dspeed]

I see one edge case that needs extended info and testing when we use html as input it could get processed by parse5 and i am not sure if parse5 handels this correct parse5 gets used in many SSR stuff and other places its the defacto most often used HTMLParser at present in the NodeJS Ecosystem

as far as i am aware we could use hashes and also file names via references to build up html

Does Parse5 handle that correct?

the placeholders make use of the fact that JavaScript supports unicode characters. I used four random characters from the reserved plane, currently \uf7f9\ue4d3 for the start of the placeholder and \ue3cc\uf1fe for the end of the placeholder.

the relevant code to test is in: https://github.com/inikulin/parse5/blob/master/packages/parse5/lib/tokenizer/preprocessor.ts

when this emits the unicode chars untouched rollup is fine and can release that.

Note

I guess this can only happen when you create some plugins that do some how use this placeholder hash like hot swap maybe
but i am not sure if that case gets triggered early i only want to document that concerns so later when errors do come up we directly know what the root cause is.

[lukastaegert]

Currently, placeholders are only used inside JavaScript files while the asset handling logic has not changed. Therefore I do not see why you would use a placeholder in an HTML file, considering you would need to replace the placeholder yourself. That being said, if you would support us by testing this, it definitely does not hurt, maybe there will be a reason to extend the logic to asset files eventually.
In any case, you can use UTF-8 characters like emojis in HTML already, it would feel more like a bug of parse5 if it is not supported. There is a good reason I stayed away from characters with "meaning" like \0.
If you need to reference chunks with hashes in an HTML file, you emit it in generateBundle where the "real" names are known, but this is probably no different from how it usually works today.

[frank-dspeed]

@lukastaegert your maybe correct but i only saw that rollup uses unicode chars an i remember that there are a lot of problems thats why i tought it would be clever to document that if it later becomes relevant i will also now work on a test inside parse5 with that chars and when it works it will not hurt and we are sure that there are no upcoming magic errors that we can not track.

It would be clever to support HTML compatible tokens because there could be plugins that depend on that. I for my self often code loaders with rollup and do not even reach the generate bundle point in that cases.

[frank-dspeed]

also i guess we should add @jakearchibald here and invite him to test that as he is deep into hashing and has already tests for some cases i guess he creates the input for web.dev

[lukastaegert]

I already added comments to the issues he created over time 😜

[rollup-bot]

This PR has been released as part of rollup@3.0.0-7. Note that this is a pre-release, so to test it, you need to install Rollup via npm install rollup@3.0.0-7 or npm install rollup@beta. It will likely become part of a regular release later.

[rollup-bot]

This PR has been released as part of rollup@3.0.0-8. Note that this is a pre-release, so to test it, you need to install Rollup via npm install rollup@3.0.0-8 or npm install rollup@beta. It will likely become part of a regular release later.

[rollup-bot]

This PR has been released as part of rollup@3.0.0. You can test it via npm install rollup.

[develohpanda]

Thank you for this wonderful write up and the work that went into this PR! 🎉

The order of plugin hooks has changed, compare the graph above with the graph at rollupjs.org/guide/en/#output-generation-hooks

If anybody else was curious about the previous graph to compare against, now that this PR has been merged, I dug it out of Wayback Machine here.