Can't run on macs with M1 processors

[kazazor]

This has been discussed here: puppeteer/puppeteer#6622 and here: puppeteer/puppeteer#6634

There are a few things that need to happen here:

1. Change puppeteer to puppeteer-core
2. Be able to pass executablePath to the package

[esakal]

@kazazor Hi, we already support it. You can use the https://www.npmjs.com/package/israeli-bank-scrapers-core

Does it helps?

[kazazor]

Awesome, I'll give it a try

[kazazor]

@esakal maybe I'm missing something here.
In the Getting chromium version used by puppeteer-core section of israeli-bank-scrapers-core it says to:

1. To get the required chromium revision use the following code

import { getPuppeteerConfig } from 'israeli-bank-scrapers-core';

const chromiumVersion = getPuppeteerConfig().chromiumRevision;

2. Once you have the chromium revision, you can either download it manually or use other liraries like [download-chromium](https://www.npmjs.com/package/download-chromium) to fetch that version

3. provide the path to chromium to the library using the option key executablePath

I did just that here:

  const options: ScaperOptions = {
    companyId: source,
    startDate,
    combineInstallments: false,
    showBrowser: false,
    executablePath: await getChromiumExecutablePath(), // Requires in order to support also ARM based CPUs (like Apple's M1 CPUs)
  };

  const scraper = createScraper(options);

And here is the getChromiumExecutablePath function:

async function getChromiumExecutablePath(): Promise<string> {
  const browserFetcher = puppeteer.createBrowserFetcher();
  const chromiumRevision = getPuppeteerConfig().chromiumRevision;
  let revisionInfo;

  if ((await browserFetcher.localRevisions()).includes(chromiumRevision)) {
    revisionInfo = browserFetcher.revisionInfo(chromiumRevision);
  } else {
    revisionInfo = await browserFetcher.download(chromiumRevision, (downloadBytes: number, totalBytes: number) =>
      console.log(`Downloading Chromium: ${downloadBytes}/${totalBytes} (${Math.round((downloadBytes / totalBytes) * 100)}%)`)
    );
  }

  return revisionInfo.executablePath;
}

I'm using israeli-bank-scrapers-core and puppeteer-core to make that work.

The problem here is that for M1 CPUs (arm64) I'm getting: The chromium binary is not available for arm64.

What am I missing here?

[esakal]

when do you get this message, whwen trying to download? when executing scrape?

[kazazor]

During the download. I found this thread online: https://stackoverflow.com/a/66044814

Which points to this fix here: puppeteer/puppeteer@9a8479a

In this release (6.0.0) : https://github.com/puppeteer/puppeteer/releases/tag/v6.0.0

support fetching and launching on Apple M1

Seems to me like updating puppeteer-core to use 6.0.0 may fix that.

---

Another indication this is the case is this issue: puppeteer/puppeteer#6614

Seems like executablePath is not being passed correctly using version 5.5.0.
I did the steps he suggested here: puppeteer/puppeteer#6614 (comment) and it worked!
So definitely something is wrong with puppeteer-core in version 5.5.0 in the manner of M1.

Looking at the breaking changes of version 6.0.0 seems to me like probably you can just update the version and it will work, right?

[kazazor]

By the way, also updating up until version 7.1.0 seems easy enough. The only breaking change there is around page.screenshot which I don't believe is actually breaking anything for you here:

israeli-bank-scrapers/src/scrapers/base-scraper-with-browser.ts

Line 282 in 72f5e13

await this.page.screenshot({

Even more, looking at the breaking changes all the way to the latest version 14.x.x seems simple enough and mainly renaming, maybe worth considering updating it

[esakal]

@kazazor thank you for suggestion this fix. I ended up upgrading to v6 as more up-dated versions although works are breaking the typescript in many places.

As you mentioned the newer versions don't seems to have complicated breaking changes fixes but for a reason the @types/puppeteer does introduce many changes in versions later then v6 so as you stated that it is good enough I ended up upgrading to v6.

Before I merge & deploy, can you please pull and confirm the fix for the core version?

[kazazor]

@esakal It seems to be doing the trick :)

[kazazor]

By the way, would be great if in your merge and deploy you could already take this PR: #695

Which will solve #694

[github-actions]

🎉 This issue has been resolved in version 1.14.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[vitalybe]

Is the latest commit should just work on M1?

When I executed yarn I am getting:

[2/3] ⠁ husky
error /Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer: Command failed.
Exit code: 1
Command: node install.js
Arguments:
Directory: /Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer
Output:
The chromium binary is not available for arm64.
If you are on Ubuntu, you can install with:

 sudo apt install chromium


 sudo apt install chromium-browser

/Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:115
                    throw new Error();
                    ^

Error
    at /Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:115:27