Design goal is to have a single test suite, that can detect if CLI is not working properly - before and after it's released. Defects it should catch are e.g. broken pkg builds for a specific platform, misaligned dependencies or issues with a specific installation targets.
CLI is being tested by a series of tests using Shellspec. See them in a test/smoke/spec
folder.
Spec in this folder is used as a
- "Smoke test" step in CircleCI to verify that built CLI can run
- "Smoke Tests" GitHub Action to verify that our distribution channels are working.
Smoke tests should be fast. Ideally tests only things that could break when CLI is being built and packaged into binaries. Functionality should be tested our other tests.
Before you start adding specs, those files are bash scripts, it's recommended to have a ShellCheck installed and running in your IDE. See Shellspec reference for available commands.
It's recommended to have a branch named feat/smoke-test
, as this branch will run the GitHub Action.
To run these tests locally:
cd test/smoke
CI=1 SMOKE_TESTS_SNYK_TOKEN=$SNYK_API_TOKEN shellspec -f d
To run the Alpine test in Docker locally:
docker build -f ./test/smoke/alpine/Dockerfile -t snyk-cli-alpine ./test/ && docker run --rm -eCI=1 -eSMOKE_TESTS_SNYK_TOKEN=$SNYK_API_TOKEN snyk-cli-alpine
Note: Alpine image is not copying/mounting everything, so you might need to add anything new to the test/smoke/alpine/Dockerfile
- be able to run against PR
- basics: version, help, config
- auth [TOKEN]
- test [--json][npm project]
- test [--json][java-goof]
- policy, ignore
- monitor [--json][npm project]
- monitor [--json][java-goof]
- wizard - possibly impossible? (maybe a basic test that it even loads)
- Alpine binary
- Docker: current images can't output a clear stderr, because of an extraneous --json flag. Also released version is currently lagging behind the latest GitHub tag by a few hours
- yarn installation (see https://github.com/snyk/snyk/issues/1270)
- scoop package
- homebrew
- Needs to run in a Docker container because GitHub Actions don't support Alpine as a host OS. Using shellspec container, as it's based on alpine and ready to run the tests.
- Need to skip a test that normally tries to open browser for login, but that fails horribly on Alpine.