Skip to content

Commit

Permalink
Add documentation for unit and functional tests
Browse files Browse the repository at this point in the history
  • Loading branch information
@dsilhavy
dsilhavy committed Apr 15, 2024
1 parent 74b7e00 commit ab70b8b
Show file tree
Hide file tree
Showing 4 changed files with 181 additions and 0 deletions.
Binary file added assets/images/unit-tests-result.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
142 changes: 142 additions & 0 deletions pages/testing/functional-test.md
View file
@@ -0,0 +1,142 @@
---
layout: default
title: Functional Tests
parent: Testing
---

<details open markdown="block">
<summary>
Table of contents
</summary>
{: .text-delta }
1. TOC
{:toc}
</details>

# Functional Tests

Functional testing in the context of media players refers to the process of testing the functionality and behavior of a
media player. It involves verifying that the media player performs its intended tasks correctly, such as playing various
types of media files (e.g. DASH manifests and segments), managing playback (e.g. play, pause, seek) and
controlling external events (e.g. a user induced switch of the audio or subtitle language). Ideally, the functionality
of a media player can be completely tested in an automated fashion obsoleting the need to perform time-consuming and
resource-consuming manual tests.

## Functional Tests in dash.js

A single functional test in dash.js typically consists of multiple steps and test-assertions:

````js
import Constants from '../../src/Constants.js';
import Utils from '../../src/Utils.js';

import {
checkIsPlaying,
checkIsProgressing,
checkNoCriticalErrors,
initializeDashJsAdapter
} from '../common/common.js';

const TESTCASE = Constants.TESTCASES.PLAYBACK.PLAY;

Utils.getTestvectorsForTestcase(TESTCASE).forEach((item) => {
const mpd = item.url;

describe(`${TESTCASE} - ${item.name} - ${mpd}`, () => {
let playerAdapter;

before(() => {
playerAdapter = initializeDashJsAdapter(item, mpd);
})

after(() => {
playerAdapter.destroy();
})

it(`Checking playing state`, async () => {
await checkIsPlaying(playerAdapter, true);
})

it(`Checking progressing state`, async () => {
await checkIsProgressing(playerAdapter);
});

it(`Expect no critical errors to be thrown`, () => {
checkNoCriticalErrors(playerAdapter);
})

})
})
````

In the example above the player is initialized and the playback is triggered. The test passes if the player successfully
transitions to the playing state and the playback time is progressing. Moreover, no critical errors shall be thrown.

## Structure

The functional tests are located in `test/functional` and are divided into different folders:

- `adapter`: The tests are implemented in a generic fashion enabling different media players to be
plugged in. The concrete implementation of the player interfaces is realized through adapter classes located in this
folder. As an example, the `DashJsAdapter` class implements the required functions to control dash.js.
- `config`: This folder contains the test configuration files. The functional tests are executed through
the [Karma Testrunner](https://karma-runner.github.io/latest/index.html). The configuration for the Karma Testrunner
is located in `karma.functional.conf.cjs`. The streams to be tested are located in
dedicated JSON files in the `streams` subfolder. In addition, example configurations for executing the tests locally
and on Browserstack are provided.
- `content`: Locally hosted content files that are used by the testcases. As an example, this folder contains manifest
files with missing segments to create a gap in the media buffer.
- `results`: Contains the results of the testruns in HTML and JUnit format.
- `src`: Utility functions and classes used in the test implementations.
- `test`: Implementation of the testcases. The testfiles are divided into different folders depending on the
functionality they are verifying. For instance, tests related to subtitles and captions are located in the `text`
folder.
- `view`: Contains the landing page that is launched by the Karma Testrunner for the execution of the tests.

## Execution

### Via Karma

The execution of the functional testsuite is straight forward. Simply start Karma with the required configuration
options:

`karma start test/functional/config/karma.functional.conf.cjs --configfile=local --streamsfile=smoke`

* `test/functional/config/karma.functional.conf.cjs`: The Karma configuration file
* `--configfile=local`: Path to the test configuration file to define which browsers should be used for test execution
and
how the test reports should be saved. The path is relative to `test/functional/config/test-configurations`
* `--streamsfile=smoke`: Path to the streams configuration file defining the streams to be used.

## Example Configurations

dash.js ships with predefined configuration files. They are located in `test/functional/config/test-configurations`.

## Supported Testscases

| Testcase | Description |
|:-----------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `advanced/no-reload-after-seek` | Build up a backwards buffer and then seek into the backwards buffer range. Expect no redundant segment downloads for segments that are already in the buffer. |
| `advanced/seek-in-gaps` | Checks playback of MPDs that contain gaps in the timeline or create gaps in the media buffer. The player should recover from such situations. |
| `audio/initial-audio` | Define an initial audio language and expect the player to choose this language at playback start. |
| `audio/switch-audio` | Switch the audio language during playback and expect the player to download the right media segments. |
| `buffer/buffer-cleanup` | Play the stream for some time and expect the buffer level to stay withing the predefined tolerance. |
| `buffer/initial-buffer-target` | Set an initial buffer target and expect the player to build the buffer before starting playback. |
| `feature-support/emsg-triggered` | Checks if EMSG events are correctly parsed and dispatched to the application. |
| `live/latency-catchup` | Expect the player to apply the catchup logic to stick to a certain latency if enabled. |
| `live/live-delay` | Expect the live delay to correspond to the initial settings. |
| `playback/ended` | Expect the ended event to be thrown once playback is finished. |
| `playback/pause` | Expect the player to correctly pause playback. |
| `playback/play` | Expect the player to correctly trigger playback. |
| `playback/seek` | Expect the player to correctly seek to a target time. |
| `playback-advanced/attach-at-non-zero` | Check if the player uses the starttime provided via the `attachSource()` function |
| `playback-advanced/attach-with-posix` | Check if the player uses the starttime provided in `posix` format to `attachSource()` |
| `playback-advanced/cmcd` | Enable CMCD reporting and expect the media segment requests to have CMCD parameters. |
| `playback-advanced/mpd-anchor` | Use MPD anchors to define a starttime. |
| `playback-advanced/multiperiod-playback` | Verify that the player transitions to a new period when playing a multiperiod MPD. |
| `playback-advanced/preload` | Preload media segments to a virtual buffer before a video element is attached to the player. |
| `text/initial-text` | Set an initial language for the selection of a texttrack and expect the player to respect that setting. |
| `text/switch-text` | Switch the texttrack during playback and expect the player to download the right track. |
| `vendor/google-ad-manager-emsg` | Check the working integration of the Google Ad Manager |
| `video/switch-video` | Switch to a different video track e.g. switching between AdaptationSets with different codecs. |

5 changes: 5 additions & 0 deletions pages/testing/index.md
View file
Expand Up @@ -4,3 +4,8 @@ title: Testing
nav_order: 5
has_children: true
---

# Testing

dash.js ships with unit and functional tests that verify various features of the player to make sure that new additions
to the player do not introduce any regression.
34 changes: 34 additions & 0 deletions pages/testing/unit-test.md
View file
@@ -0,0 +1,34 @@
---
layout: default
title: Unit Tests
parent: Testing
---

# Unit Tests

dash.js ships with various unit tests for validating the correct behavior of the functions in different classes.

## Structure

The unit tests are located in `test/unit` and divided into different folders:

- `config`: Contains the configuration files to execute the unit tests. dash.js uses
the [Karma Testrunner](https://karma-runner.github.io/latest/index.html) to execute the unit tests directly in the
browsers.
- `data`: Contains additional files like manifests, license responses and subtitle files to be used in the unit tests.
- `helpers`: Contains helper classes that contain common functionality used by the testfiles.
- `mocks`: Contains mock implementations of specific classes. When instantiating a class for a test, mock dependencies
are injected to trigger a certain behavior.
- `results`: Contains the final results of the tests in JUnit format.
- `test`: Contains the concrete implementation of the testcases. The tests are divided into different folders based on
the location of the source file to be tested.

## Execution

To execute the unit tests simply run `npm run test` in the root folder of dash.js. Per default, the tests are then
executed in Chrome and Firefox (running in headless mode). The result for each test is printed in the terminal. The
final result similar to this:

![controlbar]({{site.baseurl}}/assets/images/unit-tests-result.png)


0 comments on commit ab70b8b

Please sign in to comment.