feat: support for testing invalid rule schemas and runtime exceptions

[bmish]

Summary

Enable rule authors to write unit tests for invalid rule schemas and runtime exceptions in rules.

Related Issues

• Add ability to test rule schemas eslint#13434 - the issue triggering this RFC
• feat: ability to test rule errors and invalid schemas eslint#16823 - draft implementation

[ljharb]

yes please! it'd also be great to have test cases that fail to parse, intentionally

[jfmengels]

I like this 👍

Some prior art here: in elm-review rules can't crash (no such thing in Elm, and there are no exceptions here). The tool doesn't validate a schema for you (there is a type checker that does the most work). But it can happen that some values pass the type checker but are not valid inputs. For instance, a rule might take a string as an argument but being a given an empty string which would be a non-valid input.

This is how the rule would report that there is a configuration error: by returning a

rule : String -> Rule
rule functionName =
    if isValidFunctinoName functionName then
        Rule.configurationError "RuleName"
            { message = "`" ++ functionName ++ "` is not a valid function name"
            , details =
                [ "I was expecting the function name to be a valid Elm function name."
                , "When that is not the case, I am not able to function as expected."
				, "<more explanations on the constraints for instance>"
                ]
            }

	else
        Rule.newModuleRuleSchema "RuleName" ()
            -- ...the configuration of the rule, with the different visitors we want to have
            |> Rule.fromModuleRuleSchema

And we can write some tests for this as well:

someTest : Test
someTest =
    -- using a regular Elm test syntax
    test "should report a configuration error when function name is empty" <|
        \() ->
            rule ""
                |> Review.Test.expectConfigurationError
                    { message = "Configuration argument should not be empty"
                    , details =
                        [ "I was expecting the function name to be a valid Elm function name."
                        , "When that is not the case, I am not able to function as expected."
                        , "<more explanations on the constraints for instance>"
                        ]
                    }

whereas a regular error looks more like this

someTest : Test
someTest =
    test "should report an error when..." <|
        \() ->
            """module ModuleA exposing (a)
a = 1"""
                |> Review.Test.run rule
                |> Review.Test.expectErrors
                      [ Review.Test.error
                          { message = "Some message"
                          , details = [ "Some details" ]
                          , under = "a = 1"
                          }
                      ]

Error messages

The testing framework forces you to provide the expected error message (for configuration but for all the other errors as well). Not requiring so would create the possibility that the test is failing for an different or unexpected reason. So I would recommend forcing to give the expected error message (and have the test failure make that easy to copy/paste).

I don't know how I feel about a still helpful but less precise "CUSTOM_RULE_ERROR". If the error message will be shown to the user and is controllable by the rule author (you could catch the error and re-throw it with a nicer error message), then I would really recommend that authors have to provide it. Being able to see it in the test source code means you can see the error that will be presented to the user, and figure out how it could be better presented or made more understandable.

For SCHEMA_VALIDATION_ERROR, I imagine the rule author doesn't control the error message, but forcing the error message instead of using this generic id might catch some problems where you are getting an different/unexpected validation error. For instance if the schema changes and you forget to update one of the tests.

I imagine you'll find me a bit pedantic on this but it has worked really well for me and elm-review.

Separate category

In elm-review, we don't (can't) provide a source code for this kind of test API, which fits with your suggestion 👍

I concur with @mdjermanovic that it makes more sense to have a separate category. In a way, elm-review separates also forces these configuration error tests to be separate, and it worked well for me.

Also, to be clear, if the rule has a configuration error, then the testing framework tells you as such, forcing you to either fix the problem or to assert that it reports a configuration error like I've shown, and that's the only way.

I hope this gives some helpful tips. Anyway, I like the proposal, I think it goes in the right direction 👍

[nzakas]

I like this idea -- it's a use case we haven't really thought of before but I can see the value in having a way to test if a rule is throwing an error that breaks out of the normal linting flow.

[nzakas]

I'm happy with the general direction of this, and I like fatal as a separate category of test.

[bmish]

I've simplified the "implementation" section of the RFC based on my learnings from creating a working implementation of the proposed feature in this draft PR:

• feat: ability to test rule errors and invalid schemas eslint#16823

[mdjermanovic]

code is indeed irrelevant when testing whether the options will fail schema validations as these validations are run before using Linter.

However, when testing anything else, if code is not provided (i.e., it's undefined), Linter will throw:

TypeError: Cannot read properties of undefined (reading 'text')
      at Linter._verifyWithoutProcessors (lib\linter\linter.js:1307:37)
      at Linter.verify (lib\linter\linter.js:1496:57)
      at runRuleForItem (lib\rule-tester\rule-tester.js:748:35)
      at testFatalTemplate (lib\rule-tester\rule-tester.js:1116:28)
      at Context.<anonymous> (lib\rule-tester\rule-tester.js:1175:29)
      at processImmediate (node:internal/timers:466:21)

Perhaps the code property should be required when error.name in the test case is not set to "SchemaValidationError"?

[bmish]

In my draft PR (https://github.com/eslint/eslint/pull/16823/files#diff-1d5e36429b1a08e3baa8311c3c702f2edf5e3c5e542771dcd6a6bef4c43727f5R1173), I solved this issue by using a placeholder (Test Case #x) when there's no name nor code.

fatal.name || fatal.code || `(Test Case #${index + 1})`

I prefer this over requiring the user to come up with a name anytime they want to omit code, which feels like it would be a burden.

Is that okay?

[mdjermanovic]

This solves the test case name (first argument of it). I was thinking about the code we are passing to linter.verify.

[nzakas]

I think code can be an empty string.

[mdjermanovic]

I think code can be an empty string.

If code property isn't specified, we'll pass an empty string to linter.verify? Sounds good to me.

[nzakas]

Yup, exactly.

[mdjermanovic]

I think this is the last open question we have for this RFC.

@bmish what do you think about this? If you agree with passing an empty string to linter.verify() when the code property is not provided in the test case, can you update the RFC document with this detail?

[bmish]

Will address remaining issues in the next week.

[bmish]

I added a note about using an empty string for code.

[mdjermanovic]

I think we should document that these error messages are not guaranteed to be stable and can change even in non-major versions of ESLint, and suggest using error: { name: "SchemaValidationError" }.

[bmish]

I'm happy to document this. Note that the error message changing should be rare, only if:

• The user changes an error message in their own rule
• An upgraded version of ajv changes the validation text

[mdjermanovic]

• The user changes an error message in their own rule

I was thinking to add this note only for schema validation errors as those are the messages we produce.

• An upgraded version of ajv changes the validation text

That should be really rare, but still we can't guarantee it won't happen.

[mdjermanovic]

Seems we agree on this. Can you add a note to the RFC document so that we don't forget this? Perhaps in the Documentation section.

[bmish]

I added a note about documenting this in the "Drawbacks" section where I describe this issue.

[bmish]

@mdjermanovic since you commented recently, do you think SchemaValidationError is the ideal name here? It's a bit lengthy, but describes well that the rule was provided options that do not conform to the rule's schema. Alternatively, could use SchemaError or OptionsError. ValidationError would be ambiguous regarding what it's referring to. My goal is that we choose a good name that will make sense even if we add additional types of errors that can be tested in the future.

[mdjermanovic]

SchemaValidationError sounds perfect to me.

[nzakas]

@bmish I think we are close on this. Can you update with the last batch of suggestions?

[nzakas]

LGTM. Given the age and type of updates applied, moving to Final Commenting pending @mdjermanovic's final approval.

[mdjermanovic]

LGTM, thanks!