Certain pages are not built when data changes

[nboliver]

Using gatsby-source-sanity some document types are not getting built by Netlify when the data changes. The build is triggered, but these specific pages don't show the new data. Using the same config & dataset while building locally does build the documents in question, so it's looking like this is a cache issue. This is also happening when some documents are unpublished in Sanity - a build is triggered, but the page is retained, even though the page-data.json is not.

Is there a way to clean the gatsby cache on Netlify? Other suggestions on how to remedy this?

[jaobrown]

@nboliver I have had some issues with changes in documents (content changes on a collection, for example) not pulling in to the builds while using this plugin.

One temp fix for this is to go to your "deploys" tab in the Netlify dashboard, "trigger deploy" > "Clear cache and deploy site". This will scrap the cache saved by this plugin, and initiate a fresh build for you.

Has worked for me in the interim until I can put together a decent issue with more context/examples of this occurring.

[nboliver]

Thanks @jaobrown. Clearing the cache via the UI works for that specific build but then the next one triggered by a data change / deploy hook does not publish the changed document. The --log-pages option clearly shows that the pages are not being built.

@jlengstorf Any update / input here? Would love to keep using this plugin, but right now it's not usable, since we can't expect clients to monitor Netlify to see if a page built or not, and clear the cache if it didn't.

[MaltheBlichfeldt]

We are having sort of the same problem. Using Forestry, the change in content starts a build, but the build is just a mirror of the cache, so no changes gets published and we are stuck ind the same cached version forever. We just migrated to Gatsby 3.0.4 (from 2.0) to see if this helped, but same story... 🤔

[nboliver]

Small update here in hopes of some resolution -
@jlengstorf We've noticed that this issue seems specifically related to Gatsby > 2.3.x (including 3.x as @MaltheBlichfeldt described above)
We upgraded a site from 2.24.67 to 2.32.11 yesterday and the issue immediately came up. Do you have any insight into what could be causing this? I don't see anything in the plugin source that looks responsible. Netlify support has said that you'll be supporting development of this plugin, but there hasn't been any activity here for months. Really hoping for a resolution! Thanks.

[MaltheBlichfeldt]

Second that, @nboliver, since Gatsby 3.0 has incremental builds as default, a cache plugin is even more important. In our case this plugin can save our site from hours of build time if @jlengstorf could get it to work or hint us in a direction, so we can contribute ❤️

[jlengstorf]

hey, everyone! all this plugin does is copy the .cache and public folders from Gatsby, so I'm not sure how it could be affecting this unless Gatsby moved something outside of those folders?

@sidharthachatterjee any chance the caching mechanisms changed in >=2.3.x?

[jlengstorf]

I did a little digging here and deployed the "Blog with Gatsby" project from https://www.sanity.io/create

I installed the cache plugin, added the --log-pages flag, and made some changes. as far as I can tell it's working properly

• repo: https://github.com/jlengstorf/netlify-plugin-gatsby-cache-50
• log for an incremental build (note that only three pages updated): https://app.netlify.com/sites/netlify-plugin-gatsby-cache-50/deploys/60886863dfabad00d4a52b5f

does anyone have a reproduction of this issue for debugging?

[MaltheBlichfeldt]

@nboliver we found an easy “fix” to this described here:

gatsbyjs/gatsby#26520 (comment)

Now, using a shell script doesn’t entirely take advantage of the incremental build feature, but does persists image handling and other time consuming stuff, so via this little “solution” we brought our build time down to 5 min from 18 min. And it took 10 min, so worth trying while we figure out a real solution.

Good luck!
Malthe

[MaltheBlichfeldt]

I did a little digging here and deployed the "Blog with Gatsby" project from https://www.sanity.io/create

I installed the cache plugin, added the --log-pages flag, and made some changes. as far as I can tell it's working properly

• repo: https://github.com/jlengstorf/netlify-plugin-gatsby-cache-50
• log for an incremental build (note that only three pages updated): https://app.netlify.com/sites/netlify-plugin-gatsby-cache-50/deploys/60886863dfabad00d4a52b5f

does anyone have a reproduction of this issue for debugging?

@jlengstorf I’m afraid we cannot reference our site, since it’s a live costumer site. The plug-in also does what you replicated, but the problem is that the site doesn’t update the content. So the log seams fine but the site is no more in sync with what’s actually being build. Did you check to see, if the content on those three sites were being updated? Thanks for getting back btw 👍🏻

[nboliver]

@jlengstorf Both the sites this is affecting are production sites so we can't share access publicly. Are there logs or perhaps a screen recording that we could create to help isolate the issue?
The workflow is:

• Upgrade Gatsby to > v2.24.67
• Deploy to Netlify
• Make changes in Sanity that trigger a build hook when published
• Review build log and see that documents which triggered build hook are not logged in the list created by --log-pages flag.

[jlengstorf]

@MaltheBlichfeldt

Did you check to see, if the content on those three sites were being updated?

I did check to make sure it was updating the content.

is the content that's not updating a custom field? is it all content, or only certain fields?

without a reproduction it's really hard for me to determine where the issue is

@nboliver

Review build log and see that documents which triggered build hook are not logged in the list created by --log-pages flag.

if the pages are not logged in the list, then this plugin isn't the culprit. that's either Sanity not telling Gatsby that the pages have changed, or Gatsby not handling the change properly. this plugin can't touch the data coming in from Sanity or Gatsby's code that determines what pages to rebuild

[nboliver]

@jlengstorf That's great info, appreciate the response.
Sounds like a possible solution could be to upgrade to gatsby-source-sanity v7.x / Gatsby 3 and hope for some good luck.