Declarative Testing of Policies

[knechtionscoding]

I'd like to be able to declaratively test that the policies.yml matches the desired rules and end state.

Similar to behavior driven tests I'd like to be able to define the .policy.yml, input conditions (comments, approvals, authors, etc), and an expected state: approved|pending|disapproved and then be able to run that suite against policy-bot.

This would allow for a lot more confidence in the policy intentions and a lot more confidence for auditors.

I'm happy to start writing a test runner for something like this if this is something others would find useful.

[bluekeyes]

I think this is an interesting idea and is similar to #455, which was a draft attempt at adding a way to simulate policies from branches or under different approval conditions.

I believe an implementation of this should be relatively straight-forward, since this is how Policy Bot's own tests work. Create a new implementation of pull.Context (or potentially use the test version), instantiate it using the input conditions, load the policy, and then run the approval logic against the simulated context instead of against GitHub.

The major open questions for me are about the user experience. Is this a subcommand of the server binary that you run locally or as part of a CI build? Is it an API endpoint, where you submit everything to a running server and it returns a result? Is it a new UI, like the simulate page? What did you have in mind?

[knechtionscoding]

@bluekeyes that's a great question.

From a user experience I think having a binary that you can run locally, like conftest or helm unittest would be helpful. Ideally that way you can both run it locally and in a CI system to ensure your policy is behaviorally correct as well as syntactically correct.

Specifically I'm imagining a yaml file that looks something like:

policy-file: <path to policy.yml>
scenarios:
- name: <name of scenario>
   result: approved|disapproved|pending
   authors:
     - <author>
   reviewers:
     - <reviewer>
   ...

Then perhaps a local binary whose job it is to translate that yaml/struct into a pull.Context and then submitted to either a remote URL (if testing against a live version of policy-bot) or the local server binary (if remote URL isn't provided).

That allows for a clean user experience of something like: policy-bot test-policy -f test-file.yaml

Benefits:

1. Versioning can be kept in sync. You are running against the same spec that you are running for approvals.
2. You can however run against new versions as well. Run against a local server binary allows you to ensure that an upgrade of policy-bot will keep the same behavior for each policy
3. Easy to write. Ensuring that minimal info is required to ensure a desired outcome. I don't want to have to define the entire pull.Context for each test, just the relevant pieces.
4. Can run both in CI, and as an endpoint.

I could see an argument for not allowing it to run against a remoteURL for security reasons. It would be another endpoint that would need to be opened up, or you would need to be able to authenticate as github in order to run the tests. It could also expose a variety of information about policy behaviors allowing someone to brute force an outcome (though security through obscurity isn't something that should be relied on).

[bluekeyes]

I like the idea of a local binary that can do everything without a server connection. This avoids possible security issues with the server, but also means your tests do not require a network connection and won't fail if the server or GitHub is unavailable. If you want to match the local behavior against a real Policy Bot server, the health check endpoint returns the version, so you can use that to make sure the local version matches.

One complication with a purely local system is that you'll have to specify the details of the PR and background information about teams and organizations in GitHub. For instance, you'll need to define which teams exist and which users are members of those teams.

This makes the scenario definitions longer but I think is probably better overall because it means your tests will not break as team and organization membership changes.

Maybe the YAML definition can include a global section that contains information, like team definitions, that is true for all of the scenarios?

[knechtionscoding]

I agree. While having to declare teams and orgs requires slightly more yaml, not a massive amount. And I think is vital to ensuring the tool could test the correct state.

Perhaps I don't have a user who is part of groups A and B but I want to ensure that their hypothetical approval under rule foo wouldn't be enough. Then the test would need that information anyway.

I don't think it needs to be global to a test run. Instead making it global to a single file (test or test suite definition) then we could allow for testing different scenarios.

[bluekeyes]

As a starting point towards something like this feature, #725 added a simulation API that allows testing a policy in different situations given a real repository and running instance of the server. This doesn't fulfill all of the same uses as a local tool, but does allow some new types of tests. It also provides a starting point for future expansion, since the types of things that can be simulated are kind of limited right now.