Consider pinning down version of Reffy #849
Comments
tidoust
added a commit
that referenced
this issue
Aug 3, 2023
Hopefully, that will fix the transient Puppeteer installation errors that we seem to get nowadays when jobs run. Note this rewrites slightly the logic of the update jobs, but that should make things slightly more straightforward (job could be further simplified to replace the double checkout with a simple checkout, followed by a `git pull` after the crawl). The use of `npm ci` throughout the jobs means that we're now de facto pinning down the version of Reffy that the crawler uses, as envisioned in #849. This requires a manual action to bump the version in Webref when we release a new version of Reffy but that's a cleaner approach.
tidoust
added a commit
that referenced
this issue
Aug 3, 2023
Hopefully, that will fix the transient Puppeteer installation errors that we seem to get nowadays when jobs run. Note this rewrites slightly the logic of the update jobs, but that should make things slightly more straightforward (job could be further simplified to replace the double checkout with a simple checkout, followed by a `git pull` after the crawl). The use of `npm ci` throughout the jobs means that we're now de facto pinning down the version of Reffy that the crawler uses, as envisioned in #849. This requires a manual action to bump the version in Webref when we release a new version of Reffy but that's a cleaner approach.
|
The update jobs now install the version of Reffy specified in |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Crawl currently fails because of a hiccup in version 2.6.8 of node-fetch released a few days ago. Hiccup should get fixed by node-fetch/node-fetch#1699 (Note I don't know why we end up with a response without body. That seems to be triggered by css-display-4, but only when running a full crawl).
The
package-lock.jsonfile in Webref does have previous version 2.6.7 for node-fetch. However, the update job runsnpm install reffyon purpose and notnpm cito always use the latest version of the crawler, so that version is not used. In turn, Reffy does not specify a version of node-fetch because the dependency typically comes from fetch-filecache-for-crawling, which merely sets the version as^2.6.0, also on purpose not to have to issue a new version whenever a patch is released.It could make sense to make the update job use the version of Reffy and node-fetch in Webref's
package-lock.jsonfile instead. Note we already somewhat need to update the version of Reffy in Webref whenever Reffy gets updated in any case ("somewhat" because that dependency is only really used for testing purpose)The text was updated successfully, but these errors were encountered: