(feat) Add option to fetch Firefox Nightly (#5467)

* (feat) Add option to fetch Firefox Nightly

Add Firefox support to BrowserFetcher and the install script.
By default, the latest Firefox Nightly is downloaded
directly from archive.mozilla.org (dmg, tar.bz2 and zip)

This also required changes that impact `puppeteer.launch()`
and `puppeteer.executablePath()`

Fixes #5151

* Update docs/api.md

Co-Authored-By: Mathias Bynens <mathias@qiwi.be>

* Clean up revision promise

* Improve error handling in revision check

* Remove matchAll

* Use explicit octal mode

* Update .gitignore

Co-authored-by: Mathias Bynens <mathias@qiwi.be>

.gitignore

@@ -3,6 +3,7 @@
 /test/output-firefox
 /test/test-user-data-dir*
 /.local-chromium/
+/.local-firefox/
 /.dev_profile*
 .DS_Store
 *.swp

.npmignore

@@ -14,6 +14,7 @@ utils/node6-transform
 # repeats from .gitignore
 node_modules
 .local-chromium
+.local-firefox
 .dev_profile*
 .DS_Store
 *.swp

README.md

@@ -37,13 +37,13 @@ npm i puppeteer
 # or "yarn add puppeteer"
 ```
 
-Note: When you install Puppeteer, it downloads a recent version of Chromium (~170MB Mac, ~282MB Linux, ~280MB Win) that is guaranteed to work with the API. To skip the download, see [Environment variables](https://github.com/puppeteer/puppeteer/blob/v2.1.1/docs/api.md#environment-variables).
+Note: When you install Puppeteer, it downloads a recent version of Chromium (~170MB Mac, ~282MB Linux, ~280MB Win) that is guaranteed to work with the API. To skip the download, or to download a different browser, see [Environment variables](https://github.com/puppeteer/puppeteer/blob/v2.1.1/docs/api.md#environment-variables).
 
 
 ### puppeteer-core
 
 Since version 1.7.0 we publish the [`puppeteer-core`](https://www.npmjs.com/package/puppeteer-core) package,
-a version of Puppeteer that doesn't download Chromium by default.
+a version of Puppeteer that doesn't download any browser by default.
 
 ```bash
 npm i puppeteer-core
@@ -173,13 +173,13 @@ pass in the executable's path when creating a `Browser` instance:
 const browser = await puppeteer.launch({executablePath: '/path/to/Chrome'});
 ```
 
-See [`Puppeteer.launch()`](https://github.com/puppeteer/puppeteer/blob/v2.1.1/docs/api.md#puppeteerlaunchoptions) for more information.
+You can also use Puppeteer with Firefox Nightly (experimental support). See [`Puppeteer.launch()`](https://github.com/puppeteer/puppeteer/blob/v2.1.1/docs/api.md#puppeteerlaunchoptions) for more information.
 
 See [`this article`](https://www.howtogeek.com/202825/what%E2%80%99s-the-difference-between-chromium-and-chrome/) for a description of the differences between Chromium and Chrome. [`This article`](https://chromium.googlesource.com/chromium/src/+/master/docs/chromium_browser_vs_google_chrome.md) describes some differences for Linux users.
 
 **3. Creates a fresh user profile**
 
-Puppeteer creates its own Chromium user profile which it **cleans up on every run**.
+Puppeteer creates its own browser user profile which it **cleans up on every run**.
 
 <!-- [END runtimesettings] -->
 
@@ -301,7 +301,7 @@ See [Contributing](https://github.com/puppeteer/puppeteer/blob/master/CONTRIBUTI
 Historically, Puppeteer supported Firefox indirectly through puppeteer-firefox, which relied on a custom, patched version of Firefox. This approach was also known as “Juggler”.
 After discussions with Mozilla, we collectively concluded that relying on custom patches was infeasible.
 Since then, we have been collaborating with Mozilla on supporting Puppeteer on “stock” Firefox.
-From Puppeteer v2.1.0 onwards, as an experimental feature, you can specify [`puppeteer.launch({product: 'firefox'})`](https://github.com/puppeteer/puppeteer/blob/v2.1.1/docs/api.md#puppeteerlaunchoptions) to run your Puppeteer scripts in Firefox, without any additional custom patches.
+From Puppeteer v2.1.0 onwards, as an experimental feature, you can specify [`puppeteer.launch({product: 'firefox'})`](https://github.com/puppeteer/puppeteer/blob/v2.1.1/docs/api.md#puppeteerlaunchoptions) to run your Puppeteer scripts in Firefox Nightly, without any additional custom patches.
 
 We will continue collaborating with other browser vendors to bring Puppeteer support to browsers such as Safari.
 This effort includes exploration of a standard for executing cross-browser commands (instead of relying on the non-standard DevTools Protocol used by Chrome).
@@ -356,6 +356,18 @@ npm install puppeteer-core@chrome-71
 
 Look for `chromium_revision` in [package.json](https://github.com/puppeteer/puppeteer/blob/master/package.json). To find the corresponding Chromium commit and version number, search for the revision prefixed by an `r` in [OmahaProxy](https://omahaproxy.appspot.com/)'s "Find Releases" section.
 
+
+#### Q: Which Firefox version does Puppeteer use?
+
+Since Firefox support is experimental, Puppeteer downloads the latest [Firefox Nightly](https://wiki.mozilla.org/Nightly) when the `PUPPETEER_PRODUCT` environment variable is set to `firefox`. That's also why the value of `firefox_revision` in [package.json](https://github.com/puppeteer/puppeteer/blob/master/package.json) is `latest` -- Puppeteer isn't tied to a particular Firefox version.
+
+To fetch Firefox Nightly as part of Puppeteer installation:
+
+```bash
+PUPPETEER_PRODUCT=firefox npm i puppeteer
+# or "yarn add puppeteer"
+```
+
 #### Q: What’s considered a “Navigation”?
 
 From Puppeteer’s standpoint, **“navigation” is anything that changes a page’s URL**.

docs/api.md

@@ -36,8 +36,10 @@
 - [class: BrowserFetcher](#class-browserfetcher)
   * [browserFetcher.canDownload(revision)](#browserfetchercandownloadrevision)
   * [browserFetcher.download(revision[, progressCallback])](#browserfetcherdownloadrevision-progresscallback)
+  * [browserFetcher.host()](#browserfetcherhost)
   * [browserFetcher.localRevisions()](#browserfetcherlocalrevisions)
   * [browserFetcher.platform()](#browserfetcherplatform)
+  * [browserFetcher.product()](#browserfetcherproduct)
   * [browserFetcher.remove(revision)](#browserfetcherremoverevision)
   * [browserFetcher.revisionInfo(revision)](#browserfetcherrevisioninforevision)
 - [class: Browser](#class-browser)
@@ -391,7 +393,7 @@ If Puppeteer doesn't find them in the environment during the installation step,
 - `PUPPETEER_DOWNLOAD_HOST` - overwrite URL prefix that is used to download Chromium. Note: this includes protocol and might even include path prefix. Defaults to `https://storage.googleapis.com`.
 - `PUPPETEER_CHROMIUM_REVISION` - specify a certain version of Chromium you'd like Puppeteer to use. See [puppeteer.launch([options])](#puppeteerlaunchoptions) on how executable path is inferred. **BEWARE**: Puppeteer is only [guaranteed to work](https://github.com/puppeteer/puppeteer/#q-why-doesnt-puppeteer-vxxx-work-with-chromium-vyyy) with the bundled Chromium, use at your own risk.
 - `PUPPETEER_EXECUTABLE_PATH` - specify an executable path to be used in `puppeteer.launch`. See [puppeteer.launch([options])](#puppeteerlaunchoptions) on how the executable path is inferred. **BEWARE**: Puppeteer is only [guaranteed to work](https://github.com/puppeteer/puppeteer/#q-why-doesnt-puppeteer-vxxx-work-with-chromium-vyyy) with the bundled Chromium, use at your own risk.
-- `PUPPETEER_PRODUCT` - specify which browser you'd like Puppeteer to use. Must be one of `chrome` or `firefox`. Setting `product` programmatically in [puppeteer.launch([options])](#puppeteerlaunchoptions) supercedes this environment variable. The product is exposed in [`puppeteer.product`](#puppeteerproduct)
+- `PUPPETEER_PRODUCT` - specify which browser you'd like Puppeteer to use. Must be one of `chrome` or `firefox`. This can also be used during installation to fetch the recommended browser binary. Setting `product` programmatically in [puppeteer.launch([options])](#puppeteerlaunchoptions) supersedes this environment variable. The product is exposed in [`puppeteer.product`](#puppeteerproduct)
 
 > **NOTE** PUPPETEER_* env variables are not accounted for in the [`puppeteer-core`](https://www.npmjs.com/package/puppeteer-core) package.
 
@@ -461,9 +463,10 @@ This methods attaches Puppeteer to an existing Chromium instance.
 
 #### puppeteer.createBrowserFetcher([options])
 - `options` <[Object]>
-  - `host` <[string]> A download host to be used. Defaults to `https://storage.googleapis.com`.
-  - `path` <[string]> A path for the downloads folder. Defaults to `<root>/.local-chromium`, where `<root>` is puppeteer's package root.
+  - `host` <[string]> A download host to be used. Defaults to `https://storage.googleapis.com`. If the `product` is `firefox`, this defaults to `https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central`.
+  - `path` <[string]> A path for the downloads folder. Defaults to `<root>/.local-chromium`, where `<root>` is puppeteer's package root. If the `product` is `firefox`, this defaults to `<root>/.local-firefox`.
   - `platform` <[string]> Possible values are: `mac`, `win32`, `win64`, `linux`. Defaults to the current platform.
+  - `product` <[string]> Possible values are: `chrome`, `firefox`. Defaults to `chrome`.
 - returns: <[BrowserFetcher]>
 
 #### puppeteer.defaultArgs([options])
@@ -522,7 +525,7 @@ try {
 > **NOTE** The old way (Puppeteer versions <= v1.14.0) errors can be obtained with `require('puppeteer/Errors')`.
 
 #### puppeteer.executablePath()
-- returns: <[string]> A path where Puppeteer expects to find bundled Chromium. Chromium might not exist there if the download was skipped with [`PUPPETEER_SKIP_CHROMIUM_DOWNLOAD`](#environment-variables).
+- returns: <[string]> A path where Puppeteer expects to find the bundled browser. The browser binary might not be there if the download was skipped with [`PUPPETEER_SKIP_DOWNLOAD`](#environment-variables).
 
 > **NOTE** `puppeteer.executablePath()` is affected by the `PUPPETEER_EXECUTABLE_PATH` and `PUPPETEER_CHROMIUM_REVISION` env variables. See [Environment Variables](#environment-variables) for details.
 
@@ -572,17 +575,19 @@ const browser = await puppeteer.launch({
 > See [`this article`](https://www.howtogeek.com/202825/what%E2%80%99s-the-difference-between-chromium-and-chrome/) for a description of the differences between Chromium and Chrome. [`This article`](https://chromium.googlesource.com/chromium/src/+/lkgr/docs/chromium_browser_vs_google_chrome.md) describes some differences for Linux users.
 
 #### puppeteer.product
-- returns: <[string]> returns the name of the browser that is under automation ("chrome" or "firefox")
+- returns: <[string]> returns the name of the browser that is under automation (`"chrome"` or `"firefox"`)
 
 The product is set by the `PUPPETEER_PRODUCT` environment variable or the `product` option in [puppeteer.launch([options])](#puppeteerlaunchoptions) and defaults to `chrome`. Firefox support is experimental.
 
 
 ### class: BrowserFetcher
 
-BrowserFetcher can download and manage different versions of Chromium.
+BrowserFetcher can download and manage different versions of Chromium and Firefox.
 
 BrowserFetcher operates on revision strings that specify a precise version of Chromium, e.g. `"533271"`. Revision strings can be obtained from [omahaproxy.appspot.com](http://omahaproxy.appspot.com/).
 
+In the Firefox case, BrowserFetcher downloads Firefox Nightly and operates on version numbers such as `"75"`.
+
 An example of using BrowserFetcher to download a specific version of Chromium and running
 Puppeteer against it:
 
@@ -615,14 +620,20 @@ The method initiates a HEAD request to check if the revision is available.
 
 The method initiates a GET request to download the revision from the host.
 
+#### browserFetcher.host()
+- returns: <[string]> The download host being used.
+
 #### browserFetcher.localRevisions()
-- returns: <[Promise]<[Array]<[string]>>> A list of all revisions available locally on disk.
+- returns: <[Promise]<[Array]<[string]>>> A list of all revisions (for the current `product`) available locally on disk.
 
 #### browserFetcher.platform()
 - returns: <[string]> One of `mac`, `linux`, `win32` or `win64`.
 
+#### browserFetcher.product()
+- returns: <[string]> One of `chrome` or `firefox`.
+
 #### browserFetcher.remove(revision)
-- `revision` <[string]> a revision to remove. The method will throw if the revision has not been downloaded.
+- `revision` <[string]> a revision to remove for the current `product`. The method will throw if the revision has not been downloaded.
 - returns: <[Promise]> Resolves when the revision has been removed.
 
 #### browserFetcher.revisionInfo(revision)
@@ -633,6 +644,10 @@ The method initiates a GET request to download the revision from the host.
   - `executablePath` <[string]> path to the revision executable
   - `url` <[string]> URL this revision can be downloaded from
   - `local` <[boolean]> whether the revision is locally available on disk
+  - `product` <[string]> one of `chrome` or `firefox`
+
+> **NOTE** Many BrowserFetcher methods, like `remove` and `revisionInfo`
+> are affected by the choice of `product`. See [puppeteer.createBrowserFetcher([options])](#puppeteercreatebrowserfetcheroptions).
 
 ### class: Browser