--incremental Build, only process files that have changed.

[zachleat]

This is the step that requires knowledge of each template’s dependencies:

• What collections the changed template is a part of and what other templates consume those collections.
• Keep in mind JavaScript dependencies (for 11ty.js)
• If the file is a layout, rebuild other templates use the layout.
• If the file is a data file, rebuild what templates use the data (global data will require some extra work but directory and template data are straightforward)
  • Stretch goal here for huge pagination templates is that it would be super nice to know what changed in the data file to have a bit more precision about what pagination templates we need to write. (Selfishly I want this for my twitter archive project) (Filed separately at Incremental Builds stretch goal: incremental pagination data sets #1087)
• Passthrough copy files filed separately as Incremental passthrough copy #977

Punt into separate issues, later:

• If the file is a config file, package.json, rebuild everything for now.
• If the file is an include or an extend or an import or other internal-to-template-language specific feature, how do we discover what other files consume this? Maybe no way to tell for relative includes that don’t live in the includes folder?

[zachleat]

The really difficult part about this task is that doing it correctly would require a detailed dependency map of the content, specifically around what uses what collections.

[dillonbheadley]

I believe the pug cli watch option does this. Even if included css file changes it only rebuilds templates that reference it.
It’s very smart. Not sure how it’s done tho.

[zachleat]

This repository is now using lodash style issue management for enhancements (see https://twitter.com/samselikoff/status/991395669016436736)

This means enhancement issues will now be closed instead of leaving them open. The enhancement backlog can be found here: https://github.com/11ty/eleventy/issues?utf8=%E2%9C%93&q=label%3Aneeds-votes+sort%3Areactions-%2B1-desc+

[zachleat]

Use Object.defineProperties to add getters to track dependencies between templates and data.

[zachleat]

Proxies and Sorted Directed Acyclic Graphs, y’all.

https://www.npmjs.com/package/toposort
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy

[zellwk]

Just adding another thought: I wonder how would incremental watch mode work with a task runner like Gulp and browser sync?

[jakearchibald]

If/when this happens, it'd be great if the optimisations were available outside of watch mode.

TypeScript does this via .tsbuildinfo files.

This means that users that are using different watchers would benefit from the incremental build too.

[Ryuno-Ki]

Hm, some people referred different existing implementations. Could we gather their source code here so to lay out a plan?
(If we get lucky, we find shared libraries among them!)

[zachleat]

Did a really dumb/rough start of this here 6e294bd

[zachleat]

After working on https://www.zachleat.com/web/own-my-tweets/ I’m also thinking a nice feature might be some smarts about pagination data too, specifically adding entries into a huge paginated array would only write files for the new/changed entries.

[zachleat]

@matthewp had an interesting idea about reading files in the output directory to compare contents to avoid re-writing and save on rsync tasks https://twitter.com/matthewcp/status/1174013502215852032

[zachleat]

I did a fair bit of work to experiment playing around with @matthewp’s idea to compare output before writing is in the incremental-file-hash-cache branch but it wasn’t quite good enough. Will require more work but might get scrapped. https://github.com/11ty/eleventy/tree/incremental-file-hash-cache

Hashing the content seemed to be fast enough (using farmhash https://www.npmjs.com/package/farmhash) but writing an extra file for the cache seemed to negate the gains here.

Some super quick numbers (farmhash versus node’s built-in crypto module)

Yeah, not writing any files did provide a small performance improvement for large projects (but at the cost of up-front hashing for the first run to create the cache file)

I dunno, just not quite sold here on this yet.

[matthewp]

@zachleat Thanks for giving it a try! For clarity, I wasn't expecting the comparison to speed up builds, I was expecting it to speed up rsync.

Out of curiosity, was this hashing implementation a separate hash file for every page, or did you concat these some how into 1 hashed file so you only need to do 1 read.

From an implementation simplicity perspective you could skip the hashing and just read in the current version of the file. This would indeed be slower, but how much slower?

[piperhaywood]

Another thought about this:
If --incremental or something similar is too difficult, a different approach that might help is to exclude files via the config file instead of .eleventyignore. This way, we can use the preexisting --config flag to swap configs when developing locally.

My specific use case:
We’re pulling in data from an API on each build. Right now there are only 20 pages on the whole site, but the build time is taking anywhere from 9 to 50 seconds each time due to the API. Ideally we wouldn’t hit that API every time, it’s only really necessary when the site is deployed to Netlify. If I could exclude that particular file within an Eleventy config file such as eleventy.dev.js, then I could run npx @11ty/eleventy --serve --config=eleventy.dev.js for much faster development.

[dceddia]

This feature would be awesome to have. I was thinking about how to do the dependency mapping and how it seems like each template language would need to be able to do its own map.

Poking around Nunjucks docs I found the load event (here). It sounds potentially useful! They say:

The 'load' event gets emitted whenever a Loader retrieves the source of a template. It can be listened to in order to determine template dependencies at runtime.

Still gotta build up that DAG I guess, but this might be a step in the right direction, for Nunjucks anyway.

[AleksandrHovhannisyan]

@Ryuno-Ki Not necessarily. Jekyll does this very intelligently by only rebuilding a file under one of two conditions:

1. If the file itself was modified. For example, if you modify a blog post or a page, it will get rebuilt.
2. If the file is included by other files/pages, or if it's a layout for other pages, then all those dependencies will get rebuilt.

Right now, I'm seeing build times of only 4-5 seconds with Jekyll 4 whenever a single file changes. With 11ty, that increases to 20 seconds because everything is getting rebuilt every single time. Eventually, if I save a file frequently enough, it just runs out of memory and crashes. I'm dumb, ignore. I wasn't using the incremental flag, lol. It actually does work and is faster than Jekyll from what I can tell.

[uxdiscovery]

Really looking forward to this epic feature. I'm working with an i18n project which eventually generates 5000+ pages. That takes 15 mins to finish a build

[Takazudo]

Isn't eleventy interested in incremental build?
The fact that eleventy does not have this feature is the huge disadvantage against other SSG solutions, I guess.

[Ryuno-Ki]

https://github.com/orgs/11ty/projects/3 lists the feature as in-progress.

[Takazudo]

@Ryuno-Ki Oh I understood. Thank you for the guidance!

As a person who develops web sites and web applications for client works, I want you to know that we can't choose eleventy for the big projects because of this.

[Ryuno-Ki]

What a bummer!
Perhaps you can find another great tool at https://jamstack.org/generators/ that suits your requirements better.

[AleksandrHovhannisyan]

Actually, now that I've revisited working on migrating my site to 11ty, I'm curious: Isn't this how the --incremental flag already behaves? I've noticed that it doesn't actually rebuild the entire site, which is a good thing. I must've not been using it before.

[pbuzdin]

https://www.11ty.dev/docs/usage/incremental/

[j-f1]

A neat addition to this would be to have --serve mode only build/copy files that are requested. This would mean that changing a file would immediately tell Browsersync to reload, which would then tell Eleventy to trigger a build of everything needed by that file (optionally skipping if none of the dependencies have changed).

[zachleat]

@j-f1 related to #108 (comment) #2456

[rr923]

@zachleat Can I use this feature for production build? I noticed that the document says:
Incremental builds perform a partial build operating only on files that have changed to improve build times when doing local development.

[zachleat]

@rr923 Work on higher fidelity incremental builds is happening and I will likely add more issues to reflect more of the nuances but for this to offer improvements to production builds on your CI server we’ll need cold start incremental first:

https://www.11ty.dev/docs/usage/incremental/#cold-start

[zachleat]

Docs updated: https://www.11ty.dev/docs/usage/incremental/

As of 2.0.0-canary.21 we’re now shipping:

• Incremental builds for layout file changes
• Incremental builds for collections consume/publish
• an API for dependency mapping that is used for incremental builds (for custom templates), already in use by the WebC plugin
• For-free passthrough copy

Further issues have been filed for future improvements:

• Cold start
• Data files
• Automated collections dependencies

This is finally in good enough shape that I’m ready to say that Eleventy now has incremental builds support. I’m going to milestone --incremental as complete in 2.0 🏆