Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: RuleTester supports processor #31

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: RuleTester supports processor #31

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4af9338
New: RuleTester supports processor
mysticatea Jul 15, 2019
07fd2eb
Update RFC
btmills Nov 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
94 changes: 94 additions & 0 deletions designs/2019-rule-tester-processors/README.md
View file
Open in desktop
@@ -0,0 +1,94 @@
- Start Date: 2019-07-16
- RFC PR: (leave this empty, to be filled in later)
- Authors: Toru Nagashima ([@mysticatea](https://github.com/mysticatea)), Brandon Mills ([@btmills](https://github.com/btmills))

# RuleTester supports processor

## Summary

This RFC makes `RuleTester` class supporting `processor` option.

## Motivation

Currently, we cannot test rules with processors. This is inconvenient for plugin rules which depend on processors. For example, [vue/comment-directive](https://github.com/vuejs/eslint-plugin-vue/blob/6751ff47b4ecd722bc2e2436ce6b34a510f92b07/tests/lib/rules/comment-directive.js) rule could not use `RuleTester` to test. Rules that distinguish between physical and virtual filenames [cannot be tested without processors](https://github.com/eslint/eslint/issues/14800).

## Detailed Design

To add `processor` option and `processorOptions` (see #29) to test cases.

```js
const { RuleTester } = require("eslint")
const rule = require("../../../lib/rules/example-rule")
const exampleProcessor = require("../../../lib/processors/example-processor")

const tester = new RuleTester()

tester.run("example-rule", rule, {
valid: [
{
code: `
<script>
console.log("Hello")
</script>
`,
processor: exampleProcessor,
processorOptions: {},
},
],
invalid: [],
})
```

### § `processor` option

This is a definition object of a processor that the "[Processors in Plugins](https://eslint.org/docs/developer-guide/working-with-plugins#processors-in-plugins)" section describes.

If a processor is given, the tester passes its `preprocess`, `postprocess`, and optional `supportsAutofix` properties as part of `Linter#verify()`'s `filenameOrOptions` options object.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!processor.supportsAutofix value will be passed as filenameOrOptions.disableFixes?

This comment was marked as off-topic.

Sign in to view


The tester only applies fixes if the processor also has `supportsAutofix: true`.

### § `processorOptions` option

RFC #29 defines the `processorOptions`, though it has not yet been implemented.

If this option was given along with `processor` option, it will be given to the processor.

## Documentation

The [RuleTester](https://eslint.org/docs/developer-guide/nodejs-api#ruletester) section should describe the new `processor` and, if implemented, `processorOptions` properties.

## Drawbacks

This expands `RuleTester`'s purpose to include testing processor-aware rules.
Some could see this as out of scope for `RuleTester`.

It adds additional complexity to `RuleTester`, though it is minimal.

This design only supports a single processor.

## Backwards Compatibility Analysis

`RuleTester` currently accepts the `processor` key as a string in valid and invalid test case objects, but it does nothing with that string.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am struggling with this point. Fundamentally, it seems wrong to put this functionality into RuleTester, as it does seem beyond the scope of what testing a rule should do. Rules only run on JS code, so I’m not sure it makes sense to be able to pass non-JS code into a test.

I’m wondering if maybe we should provide a separate utility that applies processors to any text?

Maybe it’s worth taking a step back and thinking through what it is, exactly, we are trying to achieve here. Do we need a ProcessorTester?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, we picked this RFC back up as a solution to eslint/eslint#14800, and the meeting notes for that discussion are at https://github.com/eslint/tsc-meetings/blob/main/notes/2021/2021-07-15.md#add-physicalfilename-option-into-rule-tester.

Stepping back a bit first, are we still in agreement that we ought to provide a way to test rules that care about physicalFilename?

From an implementation perspective, RuleTester uses the Linter API, so the approach in this RFC takes advantage of existing API. If we were to allow setting physicalFilename from RuleTester directly, we'd need to add additional API surface to Linter that is only used by tests, which feels icky.

Making RuleTester aware of processors doesn't bother me too much because the focus is still on testing a rule, and the physicalFilename that the rule has access to is processor-related context. If we had an easy way to test rules with fully accurate context that didn't require running the processor (and perhaps even Linter), that might be preferable. Maybe that would look like replicating a minimal bit of parsing and traversal logic inside RuleTester, which would call rules directly? Then your idea of a utility to run a processor on arbitrary text could slot in nicely.

Can you elaborate more on what a ProcessorTester might do? It sounds like it might be useful for processors, though potentially without solving the physicalFilename in rules use case this RFC addresses.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t have good answers for your questions right now, I’m just concerned that we are going down a path that has led to pain the past, which is when we start adding more and more responsibility to an object to the point where it becomes unwieldy (see Linter) rather than taking a step back and thinking about how we might rethink the solution given what we know now.

So yes, the easiest solution is to add processor support into RuleTester — I just think that maybe there’s a more fundamental issue, like perhaps we need to rethink how Linter deals with physical file names to enable better testing.

(I’ll think more on this when I’m less tired.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably isn’t feasible, but throwing it out anyway: as you pointed out, RuleTester already does a lot internally, some of which parallels logic elsewhere in the engine. But its API surface is relatively small, which gives us a lot of freedom. What if we refactored RuleTester to use the ESLint API instead of Linter, then allow consumers to replace the default by passing in a pre-configured ESLint instance to the constructor? That second step might only work with flat config so that it doesn’t depend on modules being present in the file system. Then, if this works at all, if someone wanted to test a rule in a processor context, they could configure ESLint with a processor, then use the test case’s existing name option to trigger the processor.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure that changes the equation here. My concern is that I’m not sure RuleTester should know about processors at all; changing from Linter to ESLint just changes how processors might be implemented.

If the main concern is access to context.getPhysicalFilename(), it just seems like overkill to load a processor for such a narrow use case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure RuleTester should know about processors at all

Ah, now I understand your objection. Thank you. In theory, I'm inclined to agree. But we may have already crossed this bridge, so let's see if this counterargument is compelling:

Rules themselves already know about processors in the form of getFilename() vs getPhysicalFilename(). Therefore shouldn't RuleTester also know about processors?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the difference is that rules know about processor meta data, but not about processors themselves. In theory, the filename and physical filename could be populated by something other than a processor, and I’m more inclined to feel like that’s a better direction than running an actual processor to fill in those values.

It throws an error if `processor` is set to any non-string type.
If someone has set `processor` to a string value in any of their test cases, those tests would throw with this change.
Because the `processor` key does nothing prior to this change, having it is unlikely, and the fix is easy: deleting the `processor` key leaves the test logically unchanged.

## Alternatives

- To support testing rules that distinguish between physical and virtual filenames, we could instead add a `physicalFilename` option to test cases and modify `Linter` to use that option instead of computing filenames.

## Open Questions

- Does this design need to accept multiple nested processors?

## Help Needed

I can implement this myself.

## Frequently Asked Questions

None yet.
## Related Discussions

- https://github.com/eslint/rfcs/pull/25#issuecomment-499877621
- https://github.com/eslint/eslint/issues/14800