Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polytracker CI should run all possible commands for regression purposes #6532

Open
kaoudis opened this issue Jan 3, 2023 · 0 comments
Open
Labels
CI/CD Continuous Integration / Continuous Delivery good first issue Good for newcomers maintenance Tasks that are not critical but would improve the overall maturity of PolyTracker usability Related to ease of use for end users

Comments

@kaoudis
Copy link
Collaborator

kaoudis commented Jan 3, 2023

We currently run unit tests as well as a couple of the example Dockerfiles in CI, but this existing coverage does not seem to cover all Polytracker commands. Some features such as grammar generation and now info (see #6530, https://github.com/trailofbits/polytracker/tree/fix_polytracker_info) have bitrotted over time.

While I'm not sure if functional testing comes from "agile" full stack development, that's where I'm familiar with it from!

If the existing test suite and example Dockerfiles running in CI are not sufficient to ensure all the Polytracker features continue to work (preventing regressions), some ideas to broaden test coverage:

  • use a "Does It All" Dockerfile which runs all the commands, proving each works on each CI run, or maybe:
  • add functional-esque tests to the Polytracker test suite (we currently primarily have unit tests, which to me are more about proving a single unit or function works correctly than proving WYSIWYG from the customer POV / feature level) to prove each command works on each CI run
@kaoudis kaoudis added good first issue Good for newcomers CI/CD Continuous Integration / Continuous Delivery usability Related to ease of use for end users maintenance Tasks that are not critical but would improve the overall maturity of PolyTracker labels Jan 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD Continuous Integration / Continuous Delivery good first issue Good for newcomers maintenance Tasks that are not critical but would improve the overall maturity of PolyTracker usability Related to ease of use for end users
Projects
None yet
Development

No branches or pull requests

1 participant