Image Credit

In this Web Automation with Pyppeteer repo, we have covered the following scenarios:

• Handling dynamic web content
• Web Scraping using Pyppeteer
• Web automation using Pyppeter with Pyunit (or unittest)
• Web automation using Pyppeter with Pytest
• Handling Button Clicks
• Custom Chromium Configurations for automation
• Capturing full-page & element screenshots
• Generating pdf's with Pyppeteer
• Handling dialog-boxes
• Handling iFrames
• Customizing view ports
• Setting user-agents

Pre-requisites for test execution

Step 1

Create a virtual environment by triggering the virtualenv venv command on the terminal

virtualenv venv

Step 2

Navigate the newly created virtual environment by triggering the source venv/bin/activate command on the terminal

source venv/bin/activate

Step 3

Procure the LambdaTest User Name and Access Key by navigating to LambdaTest Account Page. You might need to create an an account on LambdaTest since it is used for running tests (or scraping) on the cloud Grid.

Step 4

Add the LambdaTest User Name and Access Key in the Makefile that is located in the parent directory. Once done, save the Makefile.

Dependency/Package Installation

All the configuration settings are located in pypoject.toml

Run the make install command on the terminal to install the desired packages (or dependencies) - pytest, pytest-asyncio, flake8, and more.

make install

Once the dependencies are installed, set the EXEC_PLATFORM to either of the following:

• local : Execution on local grid (or machine) by triggering export EXEC_PLATFORM=local on the terminal
• cloud : Execution on LambdaTest cloud grid by triggering export EXEC_PLATFORM=cloud on the terminal

Custom Chromium Versions

As mentioned in the Official Documentation of Pyppeteer, there is a way to specify a particular Chromium version for web automation with Pyppeteer.

For the Custom Configuration scenario, we have downloaded Chromium versions (for macOS Intel) - latest, 121, and 113 from AppSpot.

Please copy the above mentioned Chromium versions and paste it in /mac-chrome folder. You can find the different Chromium versions here

Execution

By default, all the tests using Pyppeteer are executed with the non-headless Chromium version. However, involving custom Chromium version(s) and headless browser testing can be executed in headless and non-headless mode.

• non-headless (default) : Execution on Chromium browser in headed mode. export BROWSER_MODE=non-headless on the terminal
• headless : Execution on Chromium browser in headless mode. export BROWSER_MODE=headless on the terminal

Irrespective of whether the tests are run on local machine or LambdaTest grid, simply trigger the respective command along with make on the terminal for test execution. Run the command make help to get a list of the available test operations that can be performed with Pyppeteer:

Let's take the case of Web Scraping with Pyppeteer on local machine & LambdaTest Grid.

Scraping using Pyppeteer on local machine

Run the command export EXEC_PLATFORM=local on the terminal. Since you would have triggered make install earlier, downloaded Chromium version will already be present in the /Users//Library/Application Support/pyppeteer folder (on macOS)

In case you are using some other OS, please refer to Environment variables section in Pyppeteer official documentation to know more about the download location of Chromium browser.

Run the command make pyppeteer-web-scraping for scraping content from the test website using local Chromium browser.

The same test can also be executed using the browserWSEndpoint set to "wss://cdp.lambdatest.com/puppeteer". For running Pyppeteer tests on LambdaTest, set the environment variable EXEC_PLATFORM to cloud by running the command export EXEC_PLATFORM=cloud on the terminal

Run the command make pyppeteer-web-scraping for scraping content from the test website with Pyppeteer on LambdaTest. Shown below are the test execution snapshots on LambdaTest:

On similar lines, you can run web automation tests (or operations like handling dialog boxes, iFrames, buttons, etc.) using Pyppeteer on local Chromium as well as LambdaTest cloud grid. For more information on the commands, simply trigger make help on the terminal and run the relevant make option after setting the EXEC_PLATFORM and/or BROWSER_MODE & CHROMIUM_VERSION environment variables on the terminal.

Have feedback or need assistance?

Feel free to fork the repo and contribute to make it better! Email to himanshu[dot]sheth[at]gmail[dot]com for any queries or ping me on the following social media sites:

LinkedIn: @hjsblogger
Twitter: @hjsblogger