Ability to print out the list of top disabled rules

[bmish]

The problem you want to solve.

In a large codebase, there can easily be hundreds or even thousands of places where inline disable directive comments (like // eslint-disable-line no-console) have been used.

There is not currently a convenient method to find out what rules developers are disabling like this other than manually searching the codebase or writing a custom regexp parsing script. In fact, I put together a custom script for exactly this purpose, but it's a bit buggy and not easily available across different projects.

Gaining an understanding / summary statistics of what rules are being most frequently disabled by contributors can be useful for a variety of reasons:

• determining what kinds of tech debt exist in a codebase
• determining what rules may be buggy and in need of improvements
• determining what issues developers need more education about
• etc

Your take on the correct solution to problem.

I'm proposing a new CLI option --list-disable-directives (or similar name) that would show the complete list of inline-disabled rules by count (descending order).

yarn eslint --list-disable-directives .

[normal eslint output goes here]

Rule                   | Count | Relative 
:----------------------|------:|--------:
no-console             |   125 |    40.1%
no-unused-vars         |   104 |    33.3%
radix                  |    43 |    18.8%
node/no-missing-import |    22 |     7.1%
import/order           |    15 |     4.8%
prettier/prettier      |     2 |     0.6%
no-undef               |     1 |     0.3%

This matches the output format of the TIMING environment variable which can be used to see summary statistics about rule performance.

Are you willing to submit a pull request to implement this change?

Yes

[nzakas]

This is an interesting idea. I’m not sure if this exact proposal is complete enough to dig in too deep, but I think we can at least discuss the overall direction and if the team is interested, we could progress to an RFC.

[btmills]

The reasons you listed for this being potentially useful are compelling. For codebases that don't configure globals, envs, or rule options inline, --no-inline-config can do this today: the new errors with that option would be the ones that were disabled inline.

[bmish]

Thanks for reviewing. Are there any areas to discuss/elaborate on before I start writing an RFC?

Note: I updated the output format to match that of the TIMING environment variable.

[nzakas]

@bmish i think you can start on the RFC. 👍

[github-actions]

Oops! It looks like we lost track of this issue. @eslint/eslint-tsc what do we want to do here? This issue will auto-close in 7 days without an update.

[bmish]

I still plan to work on this hopefully in the near future.

[nzakas]

Great, adding you as an assignee on this issue so the bot won't pick it up.

[mdjermanovic]

Rule	Count	Relative
no-console	125	40.1%
no-unused-vars	104	33.3%
radix	43	18.8%
node/no-missing-import	22	7.1%
import/order	15	4.8%
prettier/prettier	2	0.6%
no-undef	1	0.3%

Is the idea to count directives or suppressed problems? In the latter case, this could be closely related to eslint/rfcs#82 - once we merge #15459, I believe the data required for this table will be already available in LintResult#suppressedMessages, and the task would be just to summarize by rule and print out the table.

[bmish]

@mdjermanovic good call out. If I had to choose one, my guess is that actual suppressions are more important to analyze than just the disable directives. But I could see both being useful.

This raises the possibility that we could expose multiple metrics and so ideally we would come up with a scalable solution that would unify them all under a single framework.

The initial per-rule metrics I'm envisioning are:

Type	Description	Motivation
Violations	How many violations there are of each rule	Being able to see the number of violations for each rule in a large codebase is something that has been sorely lacking for me over the years. This data is needed to inform how much work enabling / addressing the violations of each rule will be or deciding whether it's practical to enable a rule at all, and I've had to resort to obscure workarounds to count these numbers from the current eslint output which can reach thousands of lines long for a large codebase with thousands of violations. This problem is also mentioned in this ticket for a similar linter ember-template-lint/ember-template-lint#1984.
Suppressions	How many violations are suppressed for each rule	Indicates how many violations are being ignored/silenced.
Directives	How many disable directives there are for each rule	Indicates how many times/places humans intervened to ignore/silence violations for a rule.
Performance	Time taken to run each rule	Currently offered by the TIMING environment variable. We could eliminate this one-off environment variable by moving this feature into this unified metrics framework.

Here's how it could work:

• eslint --summary . -- display all summary metrics
• eslint --summary=violations . -- display just violation summary metrics, for example
• eslint --summary=violations,suppressions . or eslint --summary=violations --summary=suppressions . -- display these two metrics

And then there's another question of whether we show a separate table for each metric, or whether we try to combine the displayed metrics into one table like this:

Rule	Violations	Violations Relative	Suppressions	Suppressions Relative	Directives	Directives Relative	Time (ms)	Time Relative
no-console	10	30%	2	40%	1	20%	30.312	5.3%
no-unused-vars	5	15%	1	20%	1	10%	24.421	4.4%

Or we could put values and relative percentages in the same column for better aesthetics and space-savings (my preference):

Rule	Violations	Suppressions	Directives	Time (ms)
no-console	10 (30%)	2 (40%)	1 (20%)	30.312 (5.3%)
no-unused-vars	5 (15%)	1 (20%)	1 (20%)	60.421 (10.4%)

The table would be sorted in descending order based on the first column. The first column would default to violations but the ordering / columns could be controlled by eslint --summary=violations,suppressions as mentioned above.

To summarize, a new CLI feature for displaying multiple metrics has the potential to expose powerful new data to users, as well as providing a better home for the existing rule performance metrics.

Let me know what people think.