[TS-161] rejecting SARIF, as there are more runs than allowed (16 > 15)

[phw]

Since recently we get the following error messages on SARIF upload in https://github.com/metabrainz/picard/ :

Error: Code Scanning could not process the submitted SARIF file:
rejecting SARIF, as there are more runs than allowed (16 > 15)

This happens without any obvious change on our side. I downloaded one of the generated SARIF files, and indeed the runs array contains 16 entries. But this also includes tools like spectral or markdownlint that we have disabled in the Codacy UI for this project.

It looks like this action will run all the tools available on Codacy, but then it fails because codeql does not allow so many runs in the sarif upload. That's a bit of an odd situation.

How can we get this action to generate a sarif files codeql then accepts?

Our action YML: https://github.com/metabrainz/picard/blob/master/.github/workflows/codacy-analysis.yml

[github-actions]

Internal ticket created : TS-161

[mirabilos]

⚠ This is unfortunately a showstopper! ⚠

There seems to be no way to disable tools that we don’t use anyway (like tsqllint, which seems to be for some Microsoft thing, whereas we use PostgreSQL); they still show up in the SARIF file. This prevents SARIF file upload, and all possible workarounds end up having flaws preventing their use.

[ben-manes]

I missed this issue and opened a duplicate in #121 a while back. I finally took a look and fixed it by removing the duplicate runs.

jq '.runs |= unique_by({tool, invocations})' results.sarif > codacy.sarif

original sarif

$ gron results.sarif.json | rg "tool.driver.name"
json.runs[0].tool.driver.name = "Checkov (reported by Codacy)";
json.runs[1].tool.driver.name = "Semgrep (reported by Codacy)";
json.runs[2].tool.driver.name = "Pmd (reported by Codacy)";
json.runs[3].tool.driver.name = "Semgrep (reported by Codacy)";
json.runs[4].tool.driver.name = "Cppcheck (reported by Codacy)";
json.runs[5].tool.driver.name = "Pmd (reported by Codacy)";
json.runs[6].tool.driver.name = "Markdownlint (reported by Codacy)";
json.runs[7].tool.driver.name = "Trivy (reported by Codacy)";
json.runs[8].tool.driver.name = "Semgrep (reported by Codacy)";
json.runs[9].tool.driver.name = "Shellcheck (reported by Codacy)";
json.runs[10].tool.driver.name = "Semgrep (reported by Codacy)";
json.runs[11].tool.driver.name = "Trivy (reported by Codacy)";
json.runs[12].tool.driver.name = "Jacksonlinter (reported by Codacy)";
json.runs[13].tool.driver.name = "Detekt (reported by Codacy)";
json.runs[14].tool.driver.name = "Flawfinder (reported by Codacy)";
json.runs[15].tool.driver.name = "Remark-lint (reported by Codacy)";
json.runs[16].tool.driver.name = "Spectral (reported by Codacy)";
json.runs[17].tool.driver.name = "Trivy (reported by Codacy)";
json.runs[18].tool.driver.name = "Trivy (reported by Codacy)";
json.runs[19].tool.driver.name = "Spectral (reported by Codacy)";
json.runs[20].tool.driver.name = "Checkov (reported by Codacy)";

fixed sarif

$ gron codacy.sarif | rg "tool.driver.name"
json.runs[0].tool.driver.name = "Checkov (reported by Codacy)";
json.runs[1].tool.driver.name = "Cppcheck (reported by Codacy)";
json.runs[2].tool.driver.name = "Detekt (reported by Codacy)";
json.runs[3].tool.driver.name = "Flawfinder (reported by Codacy)";
json.runs[4].tool.driver.name = "Jacksonlinter (reported by Codacy)";
json.runs[5].tool.driver.name = "Markdownlint (reported by Codacy)";
json.runs[6].tool.driver.name = "Pmd (reported by Codacy)";
json.runs[7].tool.driver.name = "Remark-lint (reported by Codacy)";
json.runs[8].tool.driver.name = "Semgrep (reported by Codacy)";
json.runs[9].tool.driver.name = "Shellcheck (reported by Codacy)";
json.runs[10].tool.driver.name = "Spectral (reported by Codacy)";
json.runs[11].tool.driver.name = "Trivy (reported by Codacy)";

[mirabilos]

@ben-manes oh, good idea…

… but lacking in the execution as this can hide results; my SARIF has (after sorting) parts like this:

[…]
    {
      "tool": {
        "driver": {
          "name": "Pmd (reported by Codacy)",
          "version": "6.55.0",
          "informationUri": "https://www.codacy.com",
          "rules": []
        }
      },
      "results": [],
      "invocations": [
        {
          "executionSuccessful": true,
          "workingDirectory": {
            "uri": "file:///codacy"
          }
        }
      ],
      "artifacts": []
    },
    {
      "tool": {
        "driver": {
          "name": "Pmd (reported by Codacy)",
          "version": "6.55.0",
          "informationUri": "https://www.codacy.com",
          "rules": []
        }
      },
      "results": [
        {
          "ruleIndex": -1,
          "ruleId": "PMD_category_ecmascript_codestyle_AssignmentInOperand",
          "message": {
            "text": "Avoid assignments in operands"
          },
          "level": "none",
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
[…]

[mirabilos]

jq '.runs |= unique_by({tool, invocations, results})' seems to behave, though

[mirabilos]

(ideally, there would be a SARIF merger, best contained within Codacy’s already-existing tools)