Videos missing

[rgaudin]

Investigating openzim/zimit#71 I realized I can't seem to be able to scrape videos reliably with the current version.

Even a very simple tests doesn't work:

• https://wulu.yeleman.ml/yt.html (YT share with privacy mode)
• https://wulu.yeleman.ml/yt2.html (default YT share)
• https://wulu.yeleman.ml/vim.html (vimeo)

The actual video content is not fetched.

I think this is the root cause behind fondamentaux missing most (if not all) of videos. See openzim/zimit#78

That may indicate that the problem is not new (those November runs were using zimit:dev which at that time used webrecorder/browsertrix-crawler:0.1.0). Maybe something has changed on Youtube.com at that time?

[ikreymer]

Thanks for including specific examples! I found an issue with Youtube, both for capture and replay, which will require a fix to both pywb and wabac.js.

I can confirm from other reports that Youtube changed something about their player recently, which requires updating the rules.

You can follow this issue for the pywb fix: webrecorder/pywb#607
Will add one for wabac.js as well soon.

For Vimeo, I was not able to find an issue with manual capture.
Will try running the crawler on these to see if any issues can be reproed.

However, lesfondamentaux is not using Youtube or Vimeo, so if there is an issue with those videos, it would actually be a separate issue.

It's possible that there are multiple issues involved, youtube and vimeo each require custom rules, and the default fondamentaux player I believe is more standard html5. Will test more soon.

[ikreymer]

After further investigation, here are some findings:

• It appears the autoplay script doesn't run correctly in all frames unless headless mode is used, with --headless. This is a bit odd, and maybe something with puppeteer.. I don't recall seeing this before, but with the headless flag, the embedded videos are detected, while without it, it seems the script does not run in those iframes.

• An additional replay fix was needed in pywb for vimeo.

• Seems to be no issues with lesfondamenatux. It's just using default html5 video, so should not have been affected. The embedded youtube video on the home page was archived as well..

So it seems the solution, in addition to the youtube + vimeo-specific rule updates, is probably to default zimit to run with --headless flag set.. Still not sure why it doesn't work in regular mode, though.

[rgaudin]

OK, can't really think of what could have happen to get different results. Maybe that working zimit:dev was a manual push with a different version of chrome?

Thanks for the progress.

[ikreymer]

Found the cause and fix. It turns out the chrome command-line args need an additional flag when not in headless mode, which I thought was the default already but isn't: --disable-features=IsolateOrigins,site-per-process

Background: Without it, iframes for youtube and vimeo are not accessible, and therefore can not be autoplayed: see related issue and work on that from: puppeteer/puppeteer#4960

Will add that to default options in next image release.

[ikreymer]

With latest pywb and wabac.js, it should be possible to capture and replay these videos, both in pywb and replayweb.page.

However, for the zimit use case, it is unfortunately a bit more complicated, because additional custom fuzzy matching rules, specific to zimit are needed.. Youtube now requires a POST request to load some data for the video, which was not required before. Both wabac.js and pywb support a prefix based query to find the 'best' match, while zimit currently requires an exact match only, which is less flexible. Perhaps can discuss in more detail on a zimit issue..

[rgaudin]

OK, please open a ticket there with the details. My understanding is that the problem will be in the replaying as we need a fixed prefix so that we have a unique URL to query data from the ZIM right ?

[ikreymer]

The crawler is now capable of capturing these videos, and they replay at least in pywb and replayweb.page.
ZIM replay will require some additional work in warc2zim, to be tracked in the linked issue.