2.0.0-canary.21 in dev mode: permalinks and collections

[brycewray]

Operating system

macOS Ventura v.13.1 (22C65)

Eleventy

2.0.0-canary.21

Describe the bug

With Eleventy 2.0.0-canary.20, the following file...

https://github.com/brycewray/eleventy_site/blob/main/_data/eleventyComputed.js

... controls whether files have permalinks and appear in collections based on their timestamps and draft status, as I described in https://www.brycewray.com/posts/2022/12/drafts-timestamp-based-publishing-eleventy/ (earlier today). This doesn’t work fully in dev mode when --incremental is in use; it adjusts some pages but not others. I had hoped that 2.0.0-canary.21 would correct this, but I’ve found that, instead, it doesn’t work in 2.0.0-canary.21 at all, and that’s with or without --incremental. Reverting to 2.0.0-canary.20 solved the issue, but still only if --incremental is not part of the Eleventy command.

Reproduction steps

1. Install Eleventy 2.0.0-canary.20.
2. Add a JavaScript file to the site data directory which uses eleventyComputed (as described above) to control whether files are built and appear in collections, based on their draft status and timestamps within their respective front matter.
3. In dev mode, run ELEVENTY_ENV=development npx @11ty/eleventy --serve. In 2.0.0-canary.20, the site respects the item from step 2. (For example: change an existing Markdown file's timestamp to a future date and/or its draft status to true, and this should hide it by killing its permalink and removing it from collections; then revert these changes and the file should reappear.)
4. Still in dev mode, run ELEVENTY_ENV=development npx @11ty/eleventy --serve --incremental. In 2.0.0-canary.20, the site doesn’t uniformly respect the item from step 2.
5. Repeat step 3 and confirm that all again works as it should in 2.0.0-canary.20.
6. Install Eleventy 2.0.0-canary.21.
7. Repeat step 3. In 2.0.0-canary.21, the site doesn't respect the item from step 2. (The parenthetical test example in step 3 doesn’t work.)
8. Repeat step 4. In 2.0.0-canary.21, the site doesn't respect the item from step 2. (The parenthetical test example in step 3 doesn’t work.)

Expected behavior

With or without --incremental in place, files should get permalinks and be in collections — or not — based on how the eleventyComputed evaluation goes. (But, even if --incremental remains problematic for the 2.0.0 canaries, the desired behavior absolutely should occur when that switch is not used.)

Reproduction URL

https://github.com/brycewray/eleventy_site

Screenshots

No response

[zachleat]

Ah, yeah I found two regressions here, one in non --incremental builds using a template cache that they weren’t supposed to. Fixed with e4501e8

And more controversially, during --incremental: whether or not the cached data cascade should be used for non-matching templates. I did a commit to reset the cache for the data cascade in this scenario, I think it should likely fix your problem. 01c1cf2

Shipping with v2.0.0-canary.22

[brycewray]

@zachleat Thanks as always, sir!

[zachleat]

v2.0.0-canary.22 just shipped, can you retest please?? 🙌🏻

[brycewray]

v2.0.0-canary.22 just shipped, can you retest please?? 🙌🏻

Desired result: draft: true or future timestamp in a post’s front matter causes no permalink and excludes from collections.

Without --incremental, works like 2.0.0-canary.20 did: achieves desired result. 👍

With --incremental, works like 2.0.0-canary.20 did: some pages (e.g., paginated lists of posts) show resulting changes to the “posts” collection, but some (e.g., home page and HTML sitemap) do not show these changes — even though all are referencing the same “posts” collection. 😞

I have pushed changes to my repo without invoking a new build in case it would be helpful (i.e., seeing if at this point it’s a PEBKAC issue on my part):
https://github.com/brycewray/eleventy_site

[zachleat]

Yeah, we do require a tiny bit of configuration in order to map collection dependencies for incremental builds (for now? #975)

Docs for this: https://www.11ty.dev/docs/data-configuration/#advanced

In your case it seems like you’ll need to add: eleventyImport.collections: ["post"] to your home page, /posts/, and sitemap in order to rebuild those files when a post changes.

[brycewray]

@zachleat Had missed seeing that. Thank you!

[zachleat]

(with the caveat that I just spotted one more caching edge case from this discussion, look for that in the next release) hides

[brycewray]

(with the caveat that I just spotted one more caching edge case from this discussion, look for that in the next release) hides

Re my what-if-it’s-PEBKAC comment, I did find some inconsistencies with the calls to the collection itself (some called it posts and some called it post), all of which I have now fixed (and pushed) — but still getting same behavior as noted in that test shortly ago even with eleventyImport.collections applied; so will wait for that next release. 👍

[brycewray]

I also see that, with --incremental used in 2.0.0-canary.22, changes to (e.g.) a post’s title follow the same pattern as noted earlier re showing collections or slices thereof: the paginated posts list does reflect the change, but the home page and HTML sitemap do not. With --incremental off, the change appears in all places as it should. So this does indeed sound like a caching issue re --incremental.

[brycewray]

@zachleat In case you wanted me to test 2.0.0-canary.23 as well: be advised that I just loaded it and saw the same behavior as in canary.20 and canary.22 re --incremental. Let me know if you want me to try anything else. In any event, certainly just a small concern and absolutely not worth losing any of your holidays-related time worrying about IMHO. 🎄 👪

[zachleat]

Spent a little time on this this morning and ran into a few things!

For some reason eleventy-sass broke with canary.24. I’m not sure what’s happening there but I did a little spelunking into the repo and there is quite a bit of TemplateContent monkey patching going on there which is a bit worrying! 🙁 I did have to make some changes to TemplateContent to make duplicate read calls share the same promises and make it more async friendly. cc @kentaroi

On that same note I see https://github.com/brycewray/eleventy_site/blob/09bb306fa753b5a0ca17019a730f681cf7795909/eleventy.config.js#L274-L281 also does frontMatter.content monkey patching. I will add some code to accommodate that one in the next canary but that seems risky in the same way as the previous point. We should get this logged as a separate issue, officially.

All of that said, I think the only issue here is that you used eleventyImport.collections in your front matter when it needed to be:

---
eleventyImport:
  collections: ["post"]
---

Incremental builds are applied correctly to index pages when this is used!

[brycewray]

@zachleat Yes, tried 2.0.0-canary.24 last night and encountered the eleventy-sass glitch. Thought maybe it was something in my config but it was late and I was/am old and tired, so I let it go until another time. Guess I will wait to see what happens re the issues you mentioned to @kentaroi.

And thanks very much 🙏 for the info re eleventyImport!

[zachleat]

I think this one is resolved! I also improved the docs because the confusion here definitely would have been repeated!

Improvements here:

https://www.11ty.dev/docs/collections/#declare-your-collections-for-incremental-builds
https://www.11ty.dev/docs/data-configuration/#advanced

[kentaroi]

Thanks for notifying me, @zachleat, @brycewray.

Thanks for all the work you have done for the recent releases, @zachleat.
I am really excited about it.

I'm sorry that eleventy-sass bothers you.
I will do my best to make eleventy-sass catch up with the latest canary of Eleventy hopefully in the coming weeks.

[zachleat]

No worries @kentaroi I would just love to know what APIs you need to make it work in a more stable way moving forward! Keep those lines of communication open! Appreciate you!

[kentaroi]

I've just released eleventy-sass@2.1.6 that should fix all of the issues with @11ty/eleventy-2.0.0-canary.24 and above.
It started to use addDependencies function, added in 2.0.0-canary.19, which works like a charm.
Thanks a lot, @zachleat!!

And, thank you very much for the opportunity to communicate with you.

I don't understand things fully, and let me explain my situation, instead of proposing APIs.
I monkeypatched Eleventy in the plugin for revision hashes and for working around an issue caused by eleventyComputed.

Revision hash
When rev: true, eleventy-sass does all of the heavy lifting in the permalink function to create CSS files with revision hashes in their filenames.
Eleventy calls permalink functions before calling compile functions, which is great since the names of the result CSS files should have been determined when generating <link rel="stylesheet" href="your-style-xxxxx.css" /> tags in layout files.
However, Eleventy does not pass inputContent to permalink functions,
which makes generating revision hashes pretty difficult.
eleventy-sass monkeypatched TemplateContent to get this (Template object) from within the permalink function as the third argument, and uses its inputContent and config.dir properties (and config.uses.addDependency function) to compile inputContent (Sass/SCSS) and to generate a revision hash from the result CSS.
https://github.com/kentaroi/eleventy-sass/blob/d1d755c797820e29d4873d2230e0b9af93226d53/lib/eleventy-sass.js#L211-L212

Issue caused by eleventyComputed, which was filed by @DannyJoris.
kentaroi/eleventy-sass#13
(I rashly patched Eleventy in the plugin to work around it, although I should have filed it in this repo.)

Actually, this also happens with stub-2258 config.
If you have a property with a string value in an eleventyComputed in any data files that affect a Sass/SCSS file, it will break.
Here is a minimal project.

I am not sure what eleventyComputed actually does in Eleventy, but extensions' compile functions will be called with a property value (string) as inputContent, when an eleventyComputed has a property with a string value.
There is no way for extensions to distinguish between a real inputContent and a string value of a property in an eleventyComputed, and it is a big problem when an extension expects a certain format in an inputContent.

To work around it, eleventy-sass patched TemplateRender to pass a !useMarkdown value to the Custom engine and patched Custom engine to pass the value to the plugin's compile function as the third argument. (Actually, I don't understand eleventyComputed very much, and the plugin's compile function just returns () => inputContent if the third argument is true (useMarkdown is false).)
kentaroi/eleventy-sass@93e3123

[brycewray]

Can confirm that eleventy-sass and the Eleventy canary (incl. the newly issued 2.0.0-canary.27) play nicely together. Thanks, @kentaroi!

However, FWIW @zachleat , going back to my original comment opening this issue, changing a post’s status to draft: true or its date to a future date now causes the following in 2.0.0-canary.27, regardless of whether --incremental is part of the build command:

Problem writing Eleventy templates: (more in DEBUG output)
[11ty] 1. Having trouble rendering njk template ./src/posts/2023/01/another-move-mastodon.md (via TemplateContentRenderError)
[11ty] 2. (./src/posts/2023/01/another-move-mastodon.md)
[11ty]   EleventyShortcodeError: Error with Nunjucks shortcode `stoot` (via Template render error)
[11ty] 3. Unable to call `macro["liteYT"]`, which is undefined or falsey (via Template render error)
[11ty] 
[11ty] Original error stack trace: Error: Unable to call `macro["liteYT"]`, which is undefined or falsey
[11ty]     at Object.callWrap (/Users/thewrays/eleventy_site/node_modules/nunjucks/src/runtime.js:266:11)
[11ty]     at eval (eval at _compile (/Users/thewrays/eleventy_site/node_modules/nunjucks/src/environment.js:633:18), <anonymous>:14:67)
[11ty]     at /Users/thewrays/eleventy_site/node_modules/@11ty/eleventy/src/Engines/Nunjucks.js:211:15
[11ty]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[11ty] Wrote 0 files (skipped 309) in 0.14 seconds (v2.0.0-canary.27)

. . . although, of course, it’s entirely likely that this is due to PEBKAC on my part. Still, I have pushed the latest to my Eleventy repo in case you want to see whether there’s anything that would help you in the ongoing 2.0.0 dev process.

Edit, 2023-01-10: This remains true on 2.0.0-canary.28 and .29, too — BUT I have now seen that it occurs on only a file using the referenced shortcode and macro. If I change the draft status or timestamp on a post without either of those, it works as it should; so this is probably far too much of an edge case for you to worry about. My apologies for the false alarm.