JSON & XML formaters #442

QuentinN42 · 2022-02-03T19:28:58Z

Add json and junitxml output format.

QuentinN42 · 2022-02-03T21:22:14Z

Trough this point there is only some refactors.
All tests still pass (except test_run_with_user_global_config_file (tests.test_cli.CommandLineTestCase) which fails also on master).

I start now adding some features.

QuentinN42 · 2022-02-03T21:35:24Z

python -m yamllint tests/yaml-1.2-spec-examples/example-2.28 -f json

[
    {
        "line": 29,
        "column": 1,
        "desc": "too many blank lines (3 > 0)",
        "rule": "empty-lines",
        "level": "error",
        "file": "tests/yaml-1.2-spec-examples/example-2.28"
    }
]

QuentinN42 · 2022-02-03T22:09:05Z

python -m yamllint tests/yaml-1.2-spec-examples/example-7.10 -f junitxml

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite name="pytest" errors="9" failures="1" skipped="0" tests="10" time="0" timestamp="2022-02-03T23:08:47.168125" hostname="E42">
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:2:1" name="document-start" time="0.0"><failure message="missing document start &quot;---&quot;"><\/failure><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:8:4" name="brackets" time="0.0"><error message="too many spaces inside brackets"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:8:4" name="colons" time="0.0"><error message="too many spaces before colon"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:8:5" name="None" time="0.0"><error message="syntax error: expected the node content, but found ':' (syntax)"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:8:6" name="indentation" time="0.0"><error message="cannot infer indentation: unexpected token"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:9:3" name="indentation" time="0.0"><error message="wrong indentation: expected 4 but found 2"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:10:3" name="indentation" time="0.0"><error message="wrong indentation: expected 4 but found 2"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:11:3" name="indentation" time="0.0"><error message="wrong indentation: expected 4 but found 2"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:12:3" name="indentation" time="0.0"><error message="wrong indentation: expected 4 but found 2"><\/error><\/testcase>
        <testcase classname="tests/yaml-1.2-spec-examples/example-7.10:12:29" name="brackets" time="0.0"><error message="too many spaces inside brackets"><\/error><\/testcase>
    </testsuite>
</testsuites>

adrienverge · 2022-02-05T08:03:00Z

Hello,

See #441 (comment) (and other issues) for why junitxml format won't be accepted.

About JSON, there are already 2 open pull requests: #245 and #68, their authors abandoned them. Maybe it's better to start from there?

As described in previous discussions, the output would be better like:

{
  "path": ...,
  "line": ...,
  "column": ...,
  "message": ...,
  "rule": ...,
  "level": "warning|error"
}

Also I see this pull request doesn't have tests and introduces small unrelated changes, please check out CONTRIBUTING.rst. Thanks!

QuentinN42 · 2022-02-05T09:22:56Z

As I can see the 'message' seems to be : [warning] missing starting space in comment (comments)
But it can be inferred from level, desc and rule...

I think it's better to keep all this fields separates if you want an output as JSON it's likely to be sent through an API or thx like that...

QuentinN42 · 2022-02-05T09:24:01Z

And yes tests will come this WE.

QuentinN42 · 2022-02-05T09:29:55Z

A lot of code modification comes form refactoring.

Trough this point there is only some refactors. All tests still pass (except test_run_with_user_global_config_file (tests.test_cli.CommandLineTestCase) which fails also on master).

I start now adding some features.

QuentinN42 · 2022-02-05T09:32:12Z

So the new code added are from this two commits :

JSON
junit

adrienverge#442 (comment)

QuentinN42 · 2022-02-05T09:44:37Z

If you want the "description" as formatted as #245, you can simply use jq :
python -m yamllint tests/yaml-1.2-spec-examples/example-2.28 -f json | jq '.[] | .message = ("["+.level+"] "+.desc+" ("+.rule+")")'

{
  "line": 29,
  "column": 1,
  "desc": "too many blank lines (3 > 0)",
  "rule": "empty-lines",
  "level": "error",
  "path": "tests/yaml-1.2-spec-examples/example-2.28",
  "message": "[error] too many blank lines (3 > 0) (empty-lines)"
}

Or I can add it to the current JSONFormater class, as you want.

QuentinN42 · 2022-02-05T09:49:02Z

The comment of @blui is valid (#245 (comment))...

I'll read the Codeclimate docs and then modify my output accordingly.

QuentinN42 · 2022-02-05T09:55:47Z

Here is the format: GL docs, codeclimate docs.

QuentinN42 · 2022-02-05T10:25:13Z

python -m yamllint tests/yaml-1.2-spec-examples/example-2.28 -f codeclimate

[
    {
        "type": "issue",
        "check_name": "empty-lines",
        "description": "too many blank lines (3 > 0)",
        "content": "too many blank lines (3 > 0) (empty-lines)",
        "categories": [
            "Style"
        ],
        "location": {
            "path": "tests/yaml-1.2-spec-examples/example-2.28",
            "positions": {
                "begin": {
                    "line": 29,
                    "column": 1
                }
            }
        },
        "remediation_points": 1000,
        "severity": "major"
    }
]

QuentinN42 · 2022-02-06T10:29:08Z

When we are ok with the desired format @adrienverge I'll start writting some tests.

adrienverge · 2022-02-08T08:05:40Z

The desired format is:

{
  "path": "dir/file.yaml",
  "line": 1337,
  "column": 42,
  "message": "duplication of key \"k\" in mapping",
  "rule": "key-duplicates",
  "level": "error"  # or "warning"
}

Can you please post only one message, when the PR is fully ready?

QuentinN42 · 2022-02-10T10:58:07Z

Is it normal that the test test_run_with_user_global_config_file failed on master ?

❯ python -m unittest discover
======================================================================
FAIL: test_run_with_user_global_config_file (tests.test_cli.CommandLineTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/n42/git/yamllint/tests/test_cli.py", line 315, in test_run_with_user_global_config_file
    self.assertEqual(ctx.returncode, 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 356 tests in 4.294s

FAILED (failures=1)

…e accordingly

QuentinN42 · 2022-02-11T13:00:41Z

@adrienverge I think it's ready, I've added 100 test to reach 98 coverage %.

QuentinN42 · 2022-02-11T13:44:20Z

tests/test_cli.py

+        # remove other env vars to make sure we are using the HOME config file.
+        os.environ.pop('YAMLLINT_CONFIG_FILE', None)
+        os.environ.pop('XDG_CONFIG_HOME', None)


I've also patched the test_run_with_user_global_config_file test by adding removing other env vars.

QuentinN42 · 2022-02-11T13:51:53Z

.github/workflows/ci.yaml

@@ -45,7 +45,7 @@ jobs:
          python-version: ${{ matrix.python-version }}
      - name: Append GitHub Actions system path
        run: echo "$HOME/.local/bin" >> $GITHUB_PATH
-      - run: pip install coveralls
+      - run: pip install coveralls ddt


I've added this module to make pytest.mark.parametrized like tests.
I don't konw if there is a built in unittest solution for that.

QuentinN42 · 2022-02-11T13:54:55Z

CONTRIBUTING.rst

+    pip install coveralls ddt
+    # all tests...
+    python -m coverage run --source=yamllint -m unittest discover
+    coverage report
+    # or just some tests (faster)
+    python -m unittest tests/rules/test_commas.py


Updated the docs to show coverage when running tests.

QuentinN42 · 2022-02-11T13:57:53Z

yamllint/linter.py

+        column,
+        desc='<no description>',
+        rule=None,
+        level=None


level can be passed as an optional parameter to simplify uses after.

QuentinN42 · 2022-02-11T14:14:07Z

yamllint/linter.py

+            "desc": self.desc,
+            "rule": self.rule,
+            "level": self.level,
+            "message": f"[{self.level}] {self.message}"


Message field match the parsable output style.

QuentinN42 · 2022-02-11T14:56:14Z

yamllint/format.py

+        if name == 'auto':
+            if run_on_gh():
+                name = 'github'
+            elif supports_color():
+                name = 'colored'
+            else:
+                name = 'standard'


This is the uncovered code (with also run_on_gh & supports_color).

I don't know how to test it because it's platform dependent...

KyleTryon · 2022-02-25T14:39:08Z

Very hopeful to see JUNIT formatting 🙇

QuentinN42 · 2022-02-28T22:52:54Z

@adrienverge the pr is ready, when do you think you'll be available to review it?

adrienverge · 2022-03-03T18:23:45Z

@QuentinN42 no the PR is not ready: it's very messy, very verbose and I cannot count the number of unrelated changes it adds. Contributing guidelines are still not followed, and my previous requests are not addressed. Sorry for saying it rough, but I am a volunteer and cannot spend so much time reviewing this kind of work. Sorry again.

chore: ignore venv

81951b3

QuentinN42 mentioned this pull request Feb 3, 2022

Create junitxml output #441

Closed

QuentinN42 added 5 commits February 3, 2022 20:33

refactor: moved formater inside another file

b858cbb

refactor: moved output at the end of the tests

3601817

fix: missed return statement

46a12a8

feat: max_level as separate function

c4463c7

refactor: use object inheritance for the formaters

6217241

feat: json formater

fbf8108

feat: junitxml formater

805b9a9

feat: changed json output file name to path

6a06e8d

adrienverge#442 (comment)

QuentinN42 added 2 commits February 5, 2022 11:05

typo

2839489

feat: add codeclimate output

d24519c

QuentinN42 added 2 commits February 5, 2022 11:26

feat: add message to the json format

4baaf12

feat: auto infer formats

e04f03a

feat: changed json format according to @adrienverge comment

4057451

QuentinN42 added 18 commits February 10, 2022 12:08

fix: test_run_with_user_global_config_file patched

dab7a80

test: escape_xml

b1ce2f2

test: severity_from_level

32de447

test: Base class test

eb48e1d

test: Class to test all fmt

1f6adf6

chore: rm coverage

b42ac81

tests: use ddt to parametrize tests

d666308

feat: add the possibility to pass the level as argument

4ae5487

refactor: match flake requirements

418952c

tests: tested formats with errors

48b5e2b

feat: fixed some isues

e018da1

ci: add ddt to the tests requirements and update the contributing fil…

0354c33

…e accordingly

tests: testing skipping warn with None level

3561a77

fix: filter None to exclude them

64e19e8

test: test with mixed errors / files

f0000a1

tests: use ddt to others classes

fba5b70

test: escape real text

d7f59fe

test: remaining tests done

ad46f33

QuentinN42 commented Feb 11, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JSON & XML formaters #442

JSON & XML formaters #442

QuentinN42 commented Feb 3, 2022 •

edited

QuentinN42 commented Feb 3, 2022

QuentinN42 commented Feb 3, 2022

QuentinN42 commented Feb 3, 2022

adrienverge commented Feb 5, 2022 •

edited

QuentinN42 commented Feb 5, 2022 •

edited

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022 •

edited

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 6, 2022

adrienverge commented Feb 8, 2022

QuentinN42 commented Feb 10, 2022

QuentinN42 commented Feb 11, 2022

QuentinN42 Feb 11, 2022

QuentinN42 Feb 11, 2022

QuentinN42 Feb 11, 2022

QuentinN42 Feb 11, 2022

QuentinN42 Feb 11, 2022

QuentinN42 Feb 11, 2022

KyleTryon commented Feb 25, 2022

QuentinN42 commented Feb 28, 2022

adrienverge commented Mar 3, 2022

JSON & XML formaters #442

Are you sure you want to change the base?

JSON & XML formaters #442

Conversation

QuentinN42 commented Feb 3, 2022 • edited

QuentinN42 commented Feb 3, 2022

QuentinN42 commented Feb 3, 2022

QuentinN42 commented Feb 3, 2022

adrienverge commented Feb 5, 2022 • edited

QuentinN42 commented Feb 5, 2022 • edited

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 5, 2022 • edited

QuentinN42 commented Feb 5, 2022

QuentinN42 commented Feb 6, 2022

adrienverge commented Feb 8, 2022

QuentinN42 commented Feb 10, 2022

QuentinN42 commented Feb 11, 2022

QuentinN42 Feb 11, 2022

Choose a reason for hiding this comment

QuentinN42 Feb 11, 2022

Choose a reason for hiding this comment

QuentinN42 Feb 11, 2022

Choose a reason for hiding this comment

QuentinN42 Feb 11, 2022

Choose a reason for hiding this comment

QuentinN42 Feb 11, 2022

Choose a reason for hiding this comment

QuentinN42 Feb 11, 2022

Choose a reason for hiding this comment

KyleTryon commented Feb 25, 2022

QuentinN42 commented Feb 28, 2022

adrienverge commented Mar 3, 2022

QuentinN42 commented Feb 3, 2022 •

edited

adrienverge commented Feb 5, 2022 •

edited

QuentinN42 commented Feb 5, 2022 •

edited

QuentinN42 commented Feb 5, 2022 •

edited