feat: E2E test suite with puppeteer+ipfsd-ctl #1353

lidel · 2019-12-05T19:23:57Z

Parent: #1164
cc @hugomrdias @hacdias @autonome for visibility/early feedback

This PR adds E2E tests for WebUI that run against HTTP API from real IPFS node

Try with:

$ npm run test:e2e # headless against go-ipfs

Implementation details

The end-to-end tests (e2e) test the full app in a headless Chromium browser. They spawn real IPFS node for HTTP API and a static HTTP server to serve the app.
The purpose of those tests is not being comprehensible, but act as a quick regression and integration suite.

Make sure npm run build is run before starting E2E tests:

$ npm run build
$ npm run test:e2e # end-to-end smoke tests (fast, headless, use go-ipfs)

Customizing E2E Tests

Default behavior can be tweaked via env variables below.

`E2E_IPFSD_TYPE`

Variable named E2E_IPFSD_TYPE defines which IPFS backend should be used for end-to-end tests.

CI setup of ipfs-webui repo runs tests against both JS and GO implementations:

$ E2E_IPFSD_TYPE=go npm run test:e2e # 'go' is the default (if missing)
$ E2E_IPFSD_TYPE=js npm run test:e2e

It is possible to test against arbitrary versions by tweaking ipfs (js-ipfs)
and go-ipfs-dep (go-ipfs) in devDependencies section of package.json and applying the change via npm i

`E2E_API_URL`

Instead of spawning a disposable node and repo for tests, one can point the E2E test suite at arbitrary HTTP API running on localhost:

$ E2E_API_URL=http://127.0.0.1:5001 npm run test:e2e

Caveat 1: HTTP API used in tests needs to run on the local machine for Peers screen to pass (they test manual swarm connect to ephemeral /ip4/120.0.0.1/.. multiaddr)

Caveat 2: CORS requests from http://localhost:3001 (static server hosting dev version of webui) need to be added to Access-Control-Allow-Origin whitelist array in node's config:

"API": {
  "HTTPHeaders": {
    "Access-Control-Allow-Origin": [
      "http://localhost:3001"
    ]
  }
}

Can be done ad-hoc via command line:

$ ipfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["http://localhost:3001"]'
$ ipfs config --json API.HTTPHeaders.Access-Control-Allow-Methods '["PUT", "GET", "POST"]'

Debugging E2E tests

Show the browser

By default, the test run headless, so you won't see the browser. To debug test errors, it can be helpful to see the robot clicking around the site.
To disable headless mode and see the browser, set the environment variable DEBUG=true:

$ DEBUG=true npm run test:e2e # e2e in slowed down mode in a browser window

Breakpoints

It is possible to set a "breakpoint" via await jestPuppeteer.debug() to stop tests at a specific line:

jest.setTimeout(600000) // increase test suite timeout
await jestPuppeteer.debug() // puppeteer will pause here

In a continuous integration environment we lint the code, run the unit tests, build the app, start an http server and run the unit e2e tests:

> npm run lint
> npm test
> npm run build
> npm run test:e2e

TODO

This is initial version that uses jest-puppeteer for end to end integration/regression tests against real HTTP API. For now it runs headless chromium against go-ipfs, but can be extended to run test matrix against js-ipfs as well. Single file upload acts as a POC test case.

(we don't want to use go-ipfs-dep from ipfsd-ctl

autonome

A few minor changes, but LGTM otherwise!

test/e2e/e2e.test.js

hacdias

So far looks good! Please let me know when you want a deep review of this!

lidel · 2019-12-21T15:35:44Z

Ok, I've split tests into separate files but kept them small to test as much they can while still easy to maintain.
When merged, I'll PR js-ipfs repo to run external webui E2E with js-ipfs backend.

Updated README with details about E2E.

@hugomrdias I used new ipfsd-ctl in test/e2e/peers.test.js and test/e2e/setup/global-init.js – would appreciate sanity check, thanks!

hacdias

LGTM! Thank you @lidel for taking over this. This is awesome!

hugomrdias · 2019-12-22T19:54:37Z

test/e2e/peers.test.js

+  beforeAll(async () => {
+    // spawn an ephemeral local node for manual swarm connect test
+    const factory = Ctl.createFactory({ type: 'proc', test: true, disposable: true })
+    ipfsd = await factory.spawn()


if you dont need to handle multiple nodes you can use createController

hugomrdias · 2019-12-22T20:09:17Z

test/e2e/setup/global-init.js

+  if (endpoint) {
+    // create http client for endpoint passed via E2E_API_URL=
+    ipfs = ipfsClient({ apiAddr: endpoint })
+  } else {
+    // use ipfds-ctl to spawn daemon to expose http api used for e2e tests
+    const type = process.env.E2E_IPFSD_TYPE || 'go'
+    const factory = Ctl.createFactory({
+      type,
+      test: true, // sets up all CORS headers required for accessing HTTP API port of ipfsd node
+      overrides: { // call findBin here to ensure we use version from devDependencies, and not from ipfsd-ctl
+        js: { ipfsBin: findBin('js') },
+        go: { ipfsBin: findBin('go') }
+      }
+    })
+    ipfsd = await factory.spawn()
+    ipfs = ipfsd.api
+  }


damn this is totally wrong we need to talk so i can improve the ctl docs :)

you only need to control one node type per test run ? if yes just use createController()

overrides are the in the second param https://github.com/ipfs/js-ipfsd-ctl#createfactoryoptions-overrides

you can do the endpoint === true with ctl also using createController({remote: true, endpoint: endpoint})

@hugomrdias: thanks for the tips, I fixed first two points in bac8428 and opened PR to improve ipfsd-ctl docs in ipfs/js-ipfsd-ctl#421

As for the third point about remote endpoint, I don't think the createController({remote: true, endpoint: process.env.E2E_API_URL }) will work, because E2E_API_URL is the IPFS API, not the one provided by Ctl.createServer.

Running createController({remote: true, endpoint: 'http://127.0.0.1:5001' }) produces error (probably due to 404 for /api/v0/..):

HTTPError: Not Found at fn (/home/lidel/project/ipfs-webui/node_modules/ky/umd.js:337:12) at process._tickCallback (internal/process/next_tick.js:68:7)

Let me know if I got this wrong.

if aren't using createServer you shouldn't need ctl only http-client right ?

Correct. We don't use createServer in e2e tests.
If E2E_API_URL is defined it means HTTP API got spawned by a third party, and we just use regular client for connecting to it.

ecstatic is deprecated

lidel · 2020-01-08T11:04:59Z

@hugomrdias applied your suggestions (details in #1353 (comment)). Should be good enough.
I'm going to merge this to unblock integration with js-ipfs and go-ipfs CI processes, we can improve things later in separate PRs.

lidel added 8 commits December 5, 2019 20:13

test: e2e for Explore screen

7a6d719

test: e2e Peers and Settings screens

5c34cfb

refactor: ipfsd-ctl v1

3850829

feat: E2E with js-ipfs backend: E2E_IPFSD_TYPE=js

fa13608

fix(ci): run both versions of E2E_IPFSD_TYPE

6117446

fix: use devDependencies versions in e2e

687d083

(we don't want to use go-ipfs-dep from ipfsd-ctl

feat: support E2E_API_URL

d807244

autonome approved these changes Dec 19, 2019

View reviewed changes

test/e2e/e2e.test.js Outdated Show resolved Hide resolved

test/e2e/e2e.test.js Outdated Show resolved Hide resolved

test/e2e/e2e.test.js Outdated Show resolved Hide resolved

test/e2e/e2e.test.js Outdated Show resolved Hide resolved

lidel added 5 commits December 19, 2019 20:03

chore: cleanup

d93e799

refactor: one e2e test file per screen

7b65a2e

fix: lint

49b0245

refactor: switch Files tests to static fixtures

8fd6224

chore: fix lint

f4dfb89

hacdias approved these changes Dec 20, 2019

View reviewed changes

lidel added 5 commits December 20, 2019 22:23

tests: e2e for import of two files

6a068f0

test: e2e Peers with real swarm connect

b9e544d

chore: fix lint

f66843f

docs: README for E2E tests

793850c

Merge remote-tracking branch 'origin/master' into feat/end-to-end-tests

58c90af

lidel marked this pull request as ready for review December 21, 2019 15:28

lidel requested review from autonome and hacdias December 21, 2019 15:45

hacdias approved these changes Dec 22, 2019

View reviewed changes

hugomrdias suggested changes Dec 22, 2019

View reviewed changes

lidel mentioned this pull request Jan 6, 2020

docs: improve usage examples ipfs/js-ipfsd-ctl#421

Merged

lidel added 3 commits January 6, 2020 13:18

refactor: switch to createController

bac8428

refactor: switch to http-server

421a5c4

ecstatic is deprecated

Merge remote-tracking branch 'origin/master' into feat/end-to-end-tests

89a2491

lidel changed the title ~~feat: e2e test suite against real go-ipfs~~ feat: E2E test suite with puppeteer+ipfsd-ctl Jan 8, 2020

lidel merged commit 59b2d67 into master Jan 8, 2020

lidel deleted the feat/end-to-end-tests branch January 8, 2020 12:19

lidel mentioned this pull request Jan 8, 2020

refactor: run all unit and e2e tests by default #1373

Merged

lidel mentioned this pull request Jan 24, 2020

Improve Test Suite #1164

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: E2E test suite with puppeteer+ipfsd-ctl #1353

feat: E2E test suite with puppeteer+ipfsd-ctl #1353

lidel commented Dec 5, 2019 •

edited

autonome left a comment

hacdias left a comment

lidel commented Dec 21, 2019 •

edited

hacdias left a comment

hugomrdias Dec 22, 2019

hugomrdias Dec 22, 2019 •

edited

lidel Jan 6, 2020 •

edited

hugomrdias Jan 8, 2020

lidel Jan 8, 2020 •

edited

lidel commented Jan 8, 2020

feat: E2E test suite with puppeteer+ipfsd-ctl #1353

feat: E2E test suite with puppeteer+ipfsd-ctl #1353

Conversation

lidel commented Dec 5, 2019 • edited

Implementation details

Customizing E2E Tests

E2E_IPFSD_TYPE

E2E_API_URL

Debugging E2E tests

Show the browser

Breakpoints

TODO

autonome left a comment

Choose a reason for hiding this comment

hacdias left a comment

Choose a reason for hiding this comment

lidel commented Dec 21, 2019 • edited

hacdias left a comment

Choose a reason for hiding this comment

hugomrdias Dec 22, 2019

Choose a reason for hiding this comment

hugomrdias Dec 22, 2019 • edited

Choose a reason for hiding this comment

lidel Jan 6, 2020 • edited

Choose a reason for hiding this comment

hugomrdias Jan 8, 2020

Choose a reason for hiding this comment

lidel Jan 8, 2020 • edited

Choose a reason for hiding this comment

lidel commented Jan 8, 2020

lidel commented Dec 5, 2019 •

edited

`E2E_IPFSD_TYPE`

`E2E_API_URL`

lidel commented Dec 21, 2019 •

edited

hugomrdias Dec 22, 2019 •

edited

lidel Jan 6, 2020 •

edited

lidel Jan 8, 2020 •

edited