cdnjs updates - still working?

[adulau]

Hi All,

We are regularly ingesting data from cdnjs but it seems the libraries were not update for multiple weeks even if the last-sync is updated.

Is this a known issue?

Thank you very much.

[MattIPv4]

👋 It looks like updates are making their way into KV, which is what backs the CDN (based on logs), but are not being mirrored back to this repo.

@xtuc when you're back in office, could you take a look at why this is? Wondering if we're running into git limitations again w/ size?

[adulau]

Thanks for the quick feedback. As a side note, we are using the repository to ingest all the hashes into hashlookup and to make queries when analysing website from a security perspective with LookyLoo another open source software.

[MattIPv4]

Just as an update here, we've identified the issue -- the version of Git in the default GitHub Actions Ubuntu image had been updated to 2.34 which includes some changes to how spare checkouts work. Historically you could configure a sparse checkout and then just use git add to add files outside the sparse config, but now you need to explicitly pass an extra flag, --sparse.

We've updated our tooling to pass this flag and have rolled back the sync timestamp in the repo, so GitHub Actions is now correctly running and attempting to push updates to this repo, but as this has been broken for a while we now have more updates in the queue than Actions can handle.

Our next step is going to be to further update our tooling to push in chunks as it goes, so each run of GitHub Actions will get as far through the queue as it can, with the next run then picking up from where it left off. Hopefully we'll get that change made soon, and this repo will then be up-to-date once again :)

[adulau]

Thanks a lot for looking into it. I just did a pull but it seems that JS files are not updated (just the workflow file was).

[MattIPv4]

👋 Just as a final update here to close the loop, we got the bot up and running again, so new assets that are published on the CDN are being correctly copied to this repo.

[adulau]

Thanks a lot. It works like a charm.