Skip to content

Latest commit

 

History

History
60 lines (39 loc) · 2.8 KB

README.md

File metadata and controls

60 lines (39 loc) · 2.8 KB
Raw

Snyk CLI Smoke Tests

Design goal is to have a single test suite, that can detect if CLI is not working properly - before and after it's released. Defects it should catch are e.g. broken pkg builds for a specific platform, misaligned dependencies or issues with a specific installation targets.

CLI is being tested by a series of tests using Shellspec. See them in a test/smoke/spec folder.

Spec in this folder is used as a 1) "Smoke test" step in CircleCI to verify that built CLI can run 2) "Smoke Tests" GitHub Action to verify that our distribution channels are working.

How to add a new smoke test

Smoke tests should be fast. Ideally tests only things that could break when CLI is being built and packaged into binaries. Functionality should be tested our other tests.

Before you start adding specs, those files are bash scripts, it's recommended to have a ShellCheck installed and running in your IDE. See Shellspec reference for available commands.

It's recommended to have a branch named feat/smoke-test, as this branch will run the GitHub Action.

To run these tests locally, install:

  • Shellspec
  • jq
  • timeout (if not available on your platform)

cd into test/smoke folder and run:

cd test/smoke
CI=1 SMOKE_TESTS_SNYK_TOKEN=$SNYK_API_TOKEN shellspec -f d

TODO

Wishlist

  • be able to run against PR

Missing scenarios

  • basics: version, help, config
  • auth [TOKEN]
  • test [--json][npm project]
  • test [--json][java-goof]
  • policy, ignore
  • monitor [--json][npm project]
  • monitor [--json][java-goof]
  • wizard - possibly impossible? (maybe a basic test that it even loads)

Missing environments

  • Alpine binary
  • Docker: current images can't output a clear stderr, because of an extraneous --json flag. Also released version is currently lagging behind the latest GitHub tag by a few hours
  • yarn installation (see https://github.com/snyk/snyk/issues/1270)
  • scoop package
  • homebrew

Current workarounds and limitations

Alpine

  • Needs to run in a Docker container because GitHub Actions don't support Alpine as a host OS. Using shellspec container, as it's based on alpine and ready to run the tests.
  • Need to skip a test that normally tries to open browser for login, but that fails horribly on Alpine.