Design goal is to have a single test suite, that can detect if CLI is not working properly - before and after it's released. Defects it should catch are e.g. broken pkg builds for a specific platform, misaligned dependencies or issues with a specific installation targets.
CLI is being tested by a series of tests using Shellspec. See them in a test/smoke/spec folder.
Spec in this folder is used as a 1) "Smoke test" step in CircleCI to verify that built CLI can run 2) "Smoke Tests" GitHub Action to verify that our distribution channels are working.
Smoke tests should be fast. Ideally tests only things that could break when CLI is being built and packaged into binaries. Functionality should be tested our other tests.
Before you start adding specs, those files are bash scripts, it's recommended to have a ShellCheck installed and running in your IDE. See Shellspec reference for available commands.
It's recommended to have a branch named feat/smoke-test, as this branch will run the GitHub Action.
To run these tests locally, install shellspec, cd into test/smoke folder and run:
CI=1 SMOKE_TESTS_SNYK_TOKEN=$SNYK_API_TOKEN shellspec -f d- basics: version, help, config
- auth [TOKEN]
- test [--json][npm project, java-goof]
- policy, ignore
- monitor
- wizard - possibly impossible? (maybe a basic test that it even loads)
- Alpine binary
- Docker: current images can't output a clear stderr, because of an extraneous --json flag. Also released version is currently lagging behind the latest GitHub tag by a few hours
- yarn installation (see https://github.com/snyk/snyk/issues/1270)
- scoop package