Implement unit testing and continuous integration

[SebastianZimmeck]

As we seem to break functionality frequently upon code changes, we should implement (1) unit testing and (2) continuous integration (CI).

The basic idea is to come up with a fairly comprehensive list of features to test (e.g., third party domain display, USP API properly recognized before sending GPC signal, after sending GPC signal), write unit test(s) for each of these features, and run CI every time new code is merged into the main branch.

Here is a tutorial on Jest and GitHub Actions, which may be a good choice for the job.

This is a long-term effort that we can do over the winter break (or earlier if someone has time).

[SebastianZimmeck]

As discussed, @sophieeng will take the lead on unit testing and continuous integration. This is a more of an experimental phase at the moment. What technologies do we use, how do they work, how do they integrate with GitHub, ...? So, @sophieeng, feel free to practically explore and write one unit test, for example, to see how it works.

[sophieeng]

Some notes about Unit Testing.

I think Jest is still the best framework to use from what I've read online. The second recommended Javascript testing framework is Mocha, which is an older program. Jest seems to be the most recommended and popular to use.

I am going to start playing around with Jest and if I can run a test.

[SebastianZimmeck]

Great!

Jest goes back to a suggestion by @lr-brown. Both @lr-brown and @jjeancharles are working on unit testing and CI for Privacy Pioneer. So, if it is helpful, please sync expertise, discuss challenges, etc.

[SebastianZimmeck]

As discussed, @sophieeng will start a test folder, start compiling a list of features to test, and implement a few tests.

[sophieeng]

An ongoing list of features to test, starting using Oliver's list from issue #289.

Below is a rudimentary checklist of things to verify concerning the functionality of optmeowt.
Chrome:
From settings page:

• Domain List button works
• Domain list build properly (no visual issues, visit a new site and see if it pops up next time you visit domainlist)
• Domains can be toggled on/off
• About button works
• Enable setting works (reflected in popup, visit site and see if headers and dom are sent)
• Domain list setting works (turning a site off from domain list stops the signals - turn a site off and check)
• Disable settings works (reflected in popup, visit site and see if the signal is sent)
• For the different settings, choose a setting then navigate off of the page and reopen it, has the setting stayed the same?
• Export domain list works
• Import domain list works
  From popup:
• Disable, enable button works visually
• Disable, enable button effectively sends or doesn't send signals
• More button brings you back to settings page
• Help button opens tutorial
• Help button while options page is open opens tutorial
• Toggle send signal on and off, check if it is reflected on the site when you reload it
• Toggle send signal on and off, check if it is reflected in the domainlist
• Visit big site and check for 3rd party domains
• Test toggling on and off these 3rd party domains, check domainlist
• Check site that has the well-known property (https://global-privacy-control.glitch.me/, for example), make sure the well-known tab works
• Domain list popup button works
• Toggle darkmode, make sure it is consistent on the settings page

Firefox:
From settings page:

• Domain List button works
• Domain list build properly (no visual issues, visit a new site and see if it pops up next time you visit domainlist)
• Domains can be toggled on/off
• About button works
• Enable setting works (reflected in popup, visit site and see if headers and dom are sent)
• Domain list setting works (turning a site off from domain list stops the signals - turn a site off and check)
• Disable settings works (reflected in popup, visit site and see if the signal is sent)
• For the different settings, choose a setting then navigate off of the page and reopen it, has the setting stayed the same?
• Export domain list works
• Import domain list works
  From popup:
• Disable, enable button works visually
• Disable, enable button effectively sends or doesn't send signals
• More button brings you back to settings page
• Help button opens tutorial
• Help button while options page is open opens tutorial
• Toggle send signal on and off, check if it is reflected on the site when you reload it
• Toggle send signal on and off, check if it is reflected in the domainlist
• Visit big site and check for 3rd party domains
• Test toggling on and off these 3rd party domains, check domainlist
• Check site that has the well-known property (https://global-privacy-control.glitch.me/, for example), make sure the well-known tab works
• Domain list popup button works
• Toggle darkmode, make sure it is consistent on the settings page
  Analysis:
• Change to analysis mode, clear cookies
• Navigate to site (preferably one that respects GPC, like nba.com)
• Collect ground truth based on methodology outlined in our data spreadsheet
• Run analysis based on methodology outlined in our data spreadsheet
• Compare the two, they should be the same

[SebastianZimmeck]

One important limitation (of any/most unit testing frameworks) is that we cannot test the UI fully. For example, if there is a pure display issue, e.g., a string is correctly put into the UI but for whatever reason is displayed incorrectly, we can test the correctness of the string but not how it is actually displayed. Just as a note.

[sophieeng]

An update before the weekly meeting:

I have a working test that is giving what I expect, but there are still bugs I'm trying to smooth out before I push the changes as an example unit test. The bug is about importing and exporting between modules, currently messing with Jest configuration and am planning on messaging Logan.

Things I'm thinking about: does browser matter? Like do firefox and chrome need different test, especially for Analysis mode which is only for ff.

[SebastianZimmeck]

does browser matter? Like do firefox and chrome need different test, especially for Analysis mode which is only for ff.

If there are differences in implementation, we could think about different tests. However, at this point let's keep it simple and just add some tests for the analysis mode for Firefox. Other than that Chromium browsers and Firefox can have the same test set.

[SebastianZimmeck]

@sophieeng, here is part of @lr-brown's implementation.

[SebastianZimmeck]

@sophieeng, let's also add a shield in the readme to give the status on whether the tests are passing (as in Privacy Pioneer). That way, we immediately can spot a problem. @lr-brown has info on how to do it.

[sophieeng]

The next steps for unit testing are adding the shield and then creating a list of issues for the test suite.

[SebastianZimmeck]

Nice work, @sophieeng!

[SebastianZimmeck]

As discussed, once @sophieeng has finalized the documentation in the readme, @katehausladen will write a new test. This will help us to make sure that the documentation is clear. The test can come from the test suite above.

[sophieeng]

I am closing this issue and separating it into smaller ones. The ReadMe changes will be published in #380. Kate's test following the ReadMe is in #391. The testing architecture is in #390. The core tests are in #392.

All the code in this issue is referring to Jest, a testing framework we are not using, so I will not be pushing anything to main.