Improve Test Suite

[hacdias]

We should aim to improve the test suite and make it more throughout. There should be E2E tests with both go-ipfs and js-ipfs implementations to catch eventual differences in both implementations.

This is a draft issue.

[lidel]

I believe at minimum we should run end-to-end tests for adding files and directories via Web UI in specific environments.

The test matrix would be to test adding files in each runtime (browsers + ipfs-desktop) against each implementation:

	go-ipfs	js-ipfs
chromium	?	?
firefox	?	?
electron	?	?

Sidenotes:

• In past we had issues in go-ipfs that surfaced with specific file sizes, so we should test 1/10/96/256/512MB file sizes
  • if it does not take forever, we could also add 1/2/4GB, but for now regular users won't do that as its too slow
• Directory test should use a tree at least three levels deep to ensure there are no regression in custom logic present in webui
• I would also like to have E2E for companion+js-ipfs+chrome.sockets
  (JS IPFS in Brave: Embedded JS-IPFS in Brave (experiment) ipfs-companion#716)

[lidel]

I am taking a look at this today.

[hacdias]

@lidel cool, let me know of any progress you made. I was also thinking of tackling E2E tests on this

[lidel]

I've tried to see how things look like when we add aegir (for running tests in different runtimes).
It costs about +200MB in node_modules (1.1G without vs 1.3G with),
and does not solve the problem of testing against different runtimes. See below.

Current State

Things get complicated fast, because aegir requires tests written in Mocha and Karma for browser tests (Firefox, Chrome, Electron), while Web UI uses Jest for tests and Puppeteer for end-to-end tests in Chrome.

End-to-end options these days seem to be limited to puppeteer-compatible APIs, which means Chromium, Firefox (80% complete) and (maybe) Electron (experimental puppeteer-electron, puppeteer-in-electron).

(A) Move to Mocha/Karma

IMO does not make much sense, refactoring cost may be high and projects usually go in the opposite direction: Jest is believed a successor to Mocha-based tests.
Karma does not provide us puppeteer capabilities, just runs mocha tests in browser with orchestration for spawning ipfs/http servers (the focus being on libraries, not end-user apps)

(B) Stay with Jest, improve Puppeteer setup

Stay with Jest & Puppeteer. We could replace manual puppeteer orchestration with jest-puppeteer which supports pre/post hooks (for spawning IPFS API via go/js etc)

• having that, we could give it a try with puppeteer-firefox and possibly electron one (submit patches if needed).

(C) Something else?

Something heavier, like Cypress?

Next Steps

• evaluate (A),(B),(C)
  • I am leaning towards giving (B) a try, as it seems to be leanest
• consider reusing aegir for non-test things, like:
  • bundle size check
  • code coverage (probably not possible without switching to mocha, but perhaps we could extend/use compatible format from jest)
  • switching lint to aegir lint (ensures webui follows common code practices)
    (but needs to be extended with react-app rules from .eslintrc.js))
• add ipfs-webui to CI tests somewhere in js-ipfs/.travis.yml#L107-L109

Sidenotes:

• Firefox version of Puppeteer is wip, but should be good enough for testing most of the things:
  • https://aslushnikov.github.io/ispuppeteerfirefoxready
  • https://wiki.mozilla.org/Remote/Milestones#Alpha

[lidel]

#1353 got merged, we use puppeteer with Chromium.

[lidel]

Running WebUI E2E tests on every PR in go-ipfs and js-ipfs repos

End to end tests (#1353) are now run as a part of test suites of go-ipfs (ipfs/kubo#6825) and js-ipfs (ipfs/kubo#6825) that run on every PR.

The goal of this is to detect ipfs-webui regressions between known stable version of IPFS daemon and a developer version as soon as possible.

The way it works:

1. ipfs-webui repo is fetched, npm ci installs deps and npm test is run against known stable version of ipfs daemon
2. IF upstream tests fail, finish early without breaking the build (this is a failsafe, so we don't break other project's builds if webui repo is in messy state for some reason)
3. IF upstream tests pass, we swap IPFS daemon to developer version built from go-ipfs or js-ipfs repo's HEAD and run npm test again

This way, we detect changes to go-ipfs or js-ipfs that could impact ipfs-webui before they land in respective master.

Whats next?

E2E tests could be improved by running them against Firefox and Webkit.

Tests are simple, so switching to https://github.com/microsoft/playwright may be worth evaluating:

We are the same team that built Puppeteer. Puppeteer proved that there is a lot of interest in the new generation of ever-green, capable and reliable automation drivers. With Playwright, we'd like to take it one step further and offer the same functionality for all the popular rendering engines. We'd like to see Playwright vendor-neutral and shared governe

[lidel]

Running E2E against Firefox should be possible even without switching to Playwright,
support was added to Puppeteer at the end of Feb:

• (feat) Add option to fetch Firefox Nightly puppeteer/puppeteer#5467
• PUPPETEER_PRODUCT at https://github.com/puppeteer/puppeteer/blob/v2.1.1/docs/api.md#environment-variables

[lidel]

Continued in #1802, #1737 & #1802