Custom Policy_Check always require approvals

[tweiss-mdm]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

With custom policy checks, approvals are required even if there is no failures.

Logs

{"level":"info","ts":"2024-04-22T14:36:31.966Z","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"conftest test plan.json variables.json --combine --all-namespaces -p /atlantis-data/opa-rules/policy\" in \"/atlantis-data/repos/common/terraform/formation/15/live_dev_infrastructure_compute/live/dev/infrastructure/compute\"","json":{"repo":"common/terraform/formation","pull":"15","duration":0.892278925}}  

{"level":"error","ts":"2024-04-22T14:36:31.966Z","caller":"events/project_command_runner.go:529","msg":"\n80 tests, 80 passed, 0 warnings, 0 failures, 0 exceptions\n","json":{"repo":"common/terraform/formation","pull":"15"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandRunner).doPolicyCheck\n\tgithub.com/runatlantis/atlantis/server/events/project_command_runner.go:529\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandRunner).PolicyCheck\n\tgithub.com/runatlantis/atlantis/server/events/project_command_runner.go:240\ngithub.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:74\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).PolicyCheck\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42\ngithub.com/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:29"}                                                                                                                                                                                                                                                                

{"level":"error","ts":"2024-04-22T14:36:31.966Z","caller":"events/instrumented_project_command_runner.go:84","msg":"Failure running policy_check operation: Some policy sets did not pass.","json":{"repo":"common/terraform/formation","pull":"15"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).PolicyCheck\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42\ngithub.com/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:29"}

Environment details

• Atlantis version: 0.27.2
• Deployment method: helm
• platform: Gitlab 16.8

Atlantis server-side config file:

---
  repos:
  - id: "/.*/"
    delete_source_branch_on_merge: true
    apply_requirements: [approved, mergeable]
    repo_locking: false
    policy_check: true
    custom_policy_check: true
    workflow: terragrunt
  metrics:
    prometheus:
      endpoint: /metrics
  workflows:
    terragrunt:
      plan:
        steps:
        - env:
            # Reduce Terraform suggestion output
            name: TF_IN_AUTOMATION
            value: 'true'
        - run: terragrunt hclfmt --terragrunt-check
        - run:
            command: terragrunt plan -input=false $(printf '%s' $COMMENT_ARGS | sed 's/,/ /g' | tr -d '\\') -no-color -out $PLANFILE
            output: hide
        - run: terragrunt show $PLANFILE
      policy_check:
        steps:
          - run:
              command: terragrunt show -json $PLANFILE | awk '/^\{/ {print}' > "plan.json"
              output: hide
          - run: 
              command: |
                cat <<-EOF > ./variables.json
                {
                   "project_id": "project",
                   "identifier": "ops",
                   "entity": "sre",
                   "environment": "dev",
                   "mode": "FULL"
                }
                EOF
              output: hide
          - run: conftest test plan.json variables.json --combine --all-namespaces -p /atlantis-data/opa-rules/policy
      apply:
        steps:
        - env:
            name: TF_IN_AUTOMATION
            value: 'true'
        - run: terragrunt apply -input=false $PLANFILE
---

[sourava01]

@tweiss-mdm
custom policy check does not ALWAYS require approvals. The fail condition for custom policy check is keyword fail inside the policy check output. Since your output contains the word 0 failures, it fails the policy check even when there are 0 failures.

On another note, have you tried using atlantis apply after the failed policy check comment? What is the output that you get in this case?

[tweiss-mdm]

@sourava01 thank you for the insight. atlantis apply does work (whether the policy check has 0 failures or severals).

[tweiss-mdm]

i managed to correct it by using a custom output when no failures.