Is there a way to include rule level in error output?

[switchplane-tom]

I'm analysing our projects with PHPStan nightly. In order to know which level a project is passing at (i.e. zero errors), I need to analyse multiple times. If the rule level was included in the output, I'd be able to group together the number of errors at each level.

I've looked at the various error formatters and looked at the ErrorFormatter interface, but can't see a way to get the rule level for the error that was generated.

{
    "totals": {
        "errors": 0,
        "file_errors": 1
    },
    "files": {
        "/var/www/project/src/File.php": {
            "errors": 1,
            "messages": [
                {
                    "message": "Parameter #1 $array of static method ...",
                    "line": 21,
                    "ignorable": true,
                    "level": x
                }
            ]
        }
    }
}

Doing this would allow me to analyse at level 9, then count the number of errors at each specific level.

[ondrejmirtes]

All of this seems backwards to me.

I'm analysing our projects with PHPStan nightly. In order to know which level a project is passing

You should analyse the project after every commit and fail if there are any errors. You should set a specific level you want to pass (like level 8) and make sure there are never any errors on that level commited on your main branch.

I don't see how your workflow is useful. If one day your project passes on level 8, and another day doesn't pass on level 0, how is that useful and actionable?

[switchplane-tom]

You should analyse the project after every commit and fail if there are any errors. You should set a specific level you want to pass (like level 8) and make sure there are never any errors on that level commited on your main branch.

Yes, we do this already.

We have ~100 projects. The reason for looking into this is so we can gather metrics on project health. This would allow us to quickly see:

• What level each project is set to. i.e. 0 = bad, 5 = ok, 9 = good times
• What level each project is passing at.
• How many errors are at the next level (or some arbitrary level). This allows us to understand the gap in health.

By gathering this data, I can report to the engineering team on our overall project health metrics.

We also have internal packages that our projects make use of. These packages may introduce non-breaking changes that could flag up new errors - e.g. we introduce generics to a low level framework package class somewhere and implementation(s) in the project do not define the concrete implementation in the templates.

Additionally, some projects are really old - created pre PHP 7. So with those projects we set the level to X, even though the entire project does not pass at that level. This is to force / ensure devs are creating new code at that level. We then run Stan on the changed files only at the time of dev, but still would like to gather metrics to help us better understand where we're at overall.

[ondrejmirtes]

We then run Stan on the changed files only.

Please don't do this. It's wrong. See https://phpstan.org/blog/why-you-should-always-analyse-whole-project. Use the baseline instead.

[ondrejmirtes]

can't see a way to get the rule level for the error that was generated.

PHPStan's architecture does not allow this. When PHPStan itself needs to know on which level an error appears, it does the same thing - run on each level starting from 0 and compare the results. In fact, some errors may disappear when going to a higher level. The logic is implemented here: https://github.com/phpstan/phpstan-src/blob/1.11.x/src/Testing/LevelsTestCase.php

Here are the JSON files for expectation comparation: https://github.com/phpstan/phpstan-src/tree/1.11.x/tests/PHPStan/Levels/data - if you search for -missing.json you'll see the errors that can be missing. It's because for example on level 5 you can get an error like "parameter expects string, int given", but on level 8 the same call will give you "Cannot call method on Foo|null".

[switchplane-tom]

Please don't do this. It's wrong. See https://phpstan.org/blog/why-you-should-always-analyse-whole-project. Use the baseline instead.

This is excellent. I honestly have no idea how I didn't know the baseline feature existed.

Thanks for the explanation. 👍