Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Karma is deprecated #5547

Open
joeyparrish opened this issue Aug 28, 2023 · 2 comments
Open

Karma is deprecated #5547

joeyparrish opened this issue Aug 28, 2023 · 2 comments
Labels
priority: P3 Useful but not urgent type: CI An issue with our continuous integration tests
Milestone
Backlog

Comments

@joeyparrish
Copy link
Member

I just found out that Karma was officially deprecated 4 months ago. https://dev.to/this-is-angular/angular-testing-in-2023-past-present-and-future-j5m

This isn't urgent, but we should decide what our test runner will look like in the future.

Things we need:

  1. Real browser testing through Selenium grid so we can test video playback on real devices (not using jsdom or phantomjs)
  2. Testing in-browser through a URL, instead of through WebDriver commands that manipulate the browser (this gets us the exotic devices in our lab through generic-webdriver-server)
  3. Serve tests over HTTPS with a custom certificate (without which we lose EME coverage and encrypted playback tests in the lab)
  4. A way for the in-browser tests to invoke the WebDriver screenshot command in the test runner and have the runner compare that screenshot to the reference screenshots (today we have a custom Karma middleware that the tests can call into)
  5. Machine-readable test coverage reports

Things that would be make the transition easier:

  1. Support for our current test framework, Jasmine (though we could rewrite our tests if necessary)
  2. Test coverage output in the same JSON format we parse today in our workflows (though we could hypothetically convert to that or rewrite our workflows)
  3. A way to run local browser tests over WebDriver, maybe with a plugin (this is how we get screenshot tests without using the lab today, via karma-local-wd-launcher)

Input from the community is very welcome. I've done very little research so far.
@joeyparrish joeyparrish added priority: P3 Useful but not urgent type: CI An issue with our continuous integration tests labels Aug 28, 2023
@joeyparrish joeyparrish pinned this issue Aug 28, 2023
@github-actions github-actions bot added this to the Backlog milestone Aug 28, 2023
@joeyparrish
Copy link
Member Author

I believe jasmine-browser-runner is the way forward. It will let us keep our existing Jasmine tests, it will connect to a remote Selenium grid, and we can reimplement our custom middlewares in express via configuration.

I'm not yet sure about:

  1. TLS configuration for testing over HTTPS
  2. Launching local WebDriver instances to support local screenshot testing (probably could do it by starting with something like this)
  3. Coverage reporting

joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
@joeyparrish
Add TLS and hostname support
c66c5bc 
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
@joeyparrish
Copy link
Member Author

jasmine-browser-runner also seems to only run a single browser. That will also need to be addressed.

joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
@joeyparrish
Add TLS and hostname support
aa09f1e 
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
@joeyparrish
Add TLS and hostname support
58f22e0 
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
joeyparrish added a commit to joeyparrish/jasmine-browser-runner that referenced this issue Mar 8, 2024
@joeyparrish
Add TLS and hostname support
b64915b 
This adds both TLS and hostname support.

TLS support is activated by specifying paths to a key and a cert.
There are caveats about self-signed certs in the README.

This also adds hostname support. This impacts both the URL sent to
browsers and the IP the server socket listens to.  There are caveats
about valid names, valid IPs, and HSTS preload in the README.
HSTS-type errors in both Chrome and Firefox are detected automatically
and translated into friendlier errors with a short link to the
documentation.

Closes jasmine#42
Replaces PR jasmine#43
One part of solving shaka-project/shaka-player#5547
@joeyparrish joeyparrish mentioned this issue Mar 8, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority: P3 Useful but not urgent type: CI An issue with our continuous integration tests
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
1 participant
@joeyparrish