Remove createRuleTester

[hudochenkov]

Looks like createRuleTester is an abandoned piece of API. We stopped use it almost three years ago.

I've checked all plugins from https://stylelint.io/user-guide/plugins. None of them is using createRuleTester.

I would consider createRuleTester even harmful to keep.

1. It's not using stylelint for tests. Instead it creates PostCSS instance, and mimick few stylelint features. And than treat every rule as a PostCSS plugin. Which means it's not possible to test how rule really works, and it doesn't have access to context, that stylelint passes to a rule (if autofix enabled, etc).
2. It have performance impact on stylelint loading. We fought hard to reduce load time. Specifically we dynamically require syntax, which needed to parse current file. Instead of loading all syntaxes. createRuleTester loads every syntax. And because it's a public API it happens every stylelint load.

I propose to remove it. And then work on https://github.com/stylelint/jest-preset-stylelint, which stylelint itself could use for tests. And other plugins could use it as well.

[vankop]

I use only stylelint, so I don't know what impact will cause removing createRuleTester.
Maybe deprecate it and remove from public API will be enough for next release.

[vankop]

1. It's not using stylelint for tests. Instead it creates PostCSS instance, and mimick few stylelint features. And than treat every rule as a PostCSS plugin. Which means it's not possible to test how rule really works, and it doesn't have access to context, that stylelint passes to a rule (if autofix enabled, etc).

Also some developers trying to use it and getting problems using it #4120

[hudochenkov]

Also some developers trying to use it and getting problems using it #4120

Exactly, it's way behind stylelint development and cause more harm.

Maybe deprecate it and remove from public API will be enough for next release.

We're going to release major version, so it's safe to remove. I'm thinking we could make jest-preset-stylelint to match current stylelint's jest configuration in the coming month. It won't be a big problem for any user of createRuleTester (I believe their number is close to zero) to wait a little bit and not upgrade stylelint.

The other solution, is to remove it from the export and deprecate (what @vankop did in #4265). This way we improve performance, and users (if any) would have more time to migrate from this API. But on the other side this code will stick with us until the next major release. It would create more work, that just remove this function, and give proper replacement in the coming month.

I could take a lead on jest-preset-stylelint. This way I could see how it works for stylelint itself and for a stylelint plugin (stylelint-order).

I'm very interested what people, who worked on stylelint the most, would say. /cc @jeddy3 @ntwb @evilebottnawi @m-allanson

[ntwb]

I'd like to see createRuleTester deprecated in the next release, and remove it in the following major release if jest-presetstylelint` is ready to go 👌🏼

I quickly discovered lots of instances of createRuleTester in use, primarily my search results are copies of stylelint on GitHub:

https://github.com/search?q=createRuleTester+stylelint&type=Code

But it also did not take long before I discovered legit uses of createRuleTester in:

https://github.com/ben-eb/css-values

[vankop]

But it also did not take long before I discovered legit uses of createRuleTester in:

https://github.com/ben-eb/css-values

last commit 3 years ago.. stylelint@7 used

[hudochenkov]

Ben's project is 3 year without updates.

I found more alive projects:

https://github.com/alexlafroscia/stylelint-plugin-css-modules/blob/master/test/helpers/test-rule.js
https://github.com/niksy/stylelint-no-restricted-syntax/blob/master/test/util/index.js
https://github.com/liferay/liferay-frontend-source-formatter/blob/master/test/stylelint_rule_tester.js
https://github.com/jonathantneal/stylelint-logical-properties/blob/master/test.mjs

How shall we deprecate it? If we just show deprecation message, then we won't gain any performance until the next major release. So 99.9% of our users (users, who don't use createRuleTester) won't get it.

I would remove it completely. 10-20 people who might need it could wait few weeks. While tens of thousands users would benefit from performance improvements.

Because we follow semver no builds would be broken.

[ntwb]

This example is only ~5 weeks ago:

https://github.com/DefinitelyTyped/DefinitelyTyped/blob/579970f8b775ba086d8267df48b4e0ee0a8d21c1/types/stylelint/stylelint-tests.ts

---

Personally I think it is poor form to deprecate anything without notice, we can follow up with another major release immediately after releasing v11.0.0, theoretically we could release v12.0.0 one minute after releasing v11.0.0.

We can also make a note in the release notes that a v12 removing createRuleTester will be released shortly after v11 is released

The v12 release does not require any other features or bug fixes to be included.

How shall we deprecate it?

Deprecate it as per #4265

[vankop]

We can also make a note in the release notes that a v12 removing createRuleTester will be released shortly after v11 is released

The v12 release does not require any other features or bug fixes to be included.

If we follow #4265 we don't need to create release as fast as possible (we remove createRuleTester from public API so it does not matter much)

[hudochenkov]

While I was simplifying jest-setup.js, I was also thinking how we can extract it into a separate package.

I didn't came up with a good solution :(

Requirements:

1. Should have all logic from our jest-setup.js.
2. Should be possible to use in stylelint core repository.
3. Should be possible to use in any stylelint plugin repository.

Here's a tricky part: stylelint repository requires stylelint as a relative path. Plugin repository would use stylelint package.

The only solution I came up with is dependency injection.

In jest-preset-stylelint we would have a function which receives stylelint and returns testRule.

// jest-preset-stylelint
module.exports = function (stylelint) {
	return function testRule(rule, schema) {
		// here content of https://github.com/stylelint/stylelint/blob/master/jest-setup.js
		describe(`${schema.ruleName}`, () => {
			// ...
		}
	}
}

In our repository we could consume it like:

const testRule = require('jest-preset-stylelint');
const stylelint = require('./lib');

global.testRule = testRule(stylelint);

In plugin repository it would be:

const testRule = require('jest-preset-stylelint');
const stylelint = require('stylelint');

global.testRule = testRule(stylelint);

Pros:

• Plugins authour would receive powerful Jest configuration and wouldn't need to create their own.

Cons:

• For stylelint repository it would be harder to update test configuration. Because it would be in separate repository.
• One more package to worry about. Release it, maintain it, follow SemVer.

createTestRule is a good idea, but I don't know how to make it good and to have all functionality we have in jest-setup.

ESLint has similar thing: RuleTester. But there they kind of implement their own test framework with Node.js assert module.

Overall, I'm not sure if it worth it to create this shared Jest testRule. Maybe we shouldn't, and plugin authors could copy our setup if they use Jest, or adopt for their test runner.

[ntwb]

Pros:

Plugins authour would receive powerful Jest configuration and wouldn't need to create their own.

Two thumbs up on this

Cons:

For stylelint repository it would be harder to update test configuration. Because it would be in separate repository.
One more package to worry about. Release it, maintain it, follow SemVer.

Would it be worth considering switching to a monorepo using Lerna?

Hoisting dependencies might be a good alternative to dependency injection.

We could consolidate all the stylelint repos into a single repo

[hudochenkov]

I used Lerna for almost a year at work. I wouldn't use it unless it's really needed. While it helps in situations with inter dependencies, but overall it makes things more complex. For users it's hard to find where the package, where is changelog (I always have hard time to find changelog from React team on things other than React itself). For developers lerna bootstrap takes more time, when in fact you only need one project. Issues are in one big pile.

Plus we would have cyclic dependency: stylelint <-> jest-preset-stylelint. It's bad.

[hudochenkov]

We could expose Jest configuration as our Node.js package (require('stylelint/jest-setup')). But I'm not sure it's a good idea. It will require releasing new major stylelint if test configuration changes.

[filipekiss]

Hey guys, regarding this changes in the test environment for the stylelint ecosystem, how does this affects stylelint-test-rule-tape? I've been using it in stylelint-color-format and I wanted to know if I should move to a different approach, since stylelint-test-rule-tape uses the createRuleTester API

[jeddy3]

Pros:

• Plugins authour would receive powerful Jest configuration and wouldn't need to create their own.

Two thumbs up on this

Ditto. I've copied the stylelint jest setup into too many plugin repos already.

I'd love to see a jest-preset-stylelint in the stylelint org as the blessed test runner (like we have vscode-stylelint as the blessed editor extension).

The dependency injection route looks good to me.

Cons:

• For stylelint repository it would be harder to update test configuration. Because it would be in separate repository.
• One more package to worry about. Release it, maintain it, follow SemVer.

It doesn't seem that often we need to update the test configuration. I think the overhead is worth it.

Hey guys, regarding this changes in the test environment for the stylelint ecosystem, how does this affects stylelint-test-rule-tape?

I believe stylelint-test-rule-tape will need to be deprecated in favour of either jest-preset-stylelint (if we go there) or (if you still want to use tape) stylelint-tape, which is a 3rd party tape runner similar in functionality, I believe, to the jest-setup script.

[asapach]

If stylelint-test-rule-tape is now deprecated could you please update the docs and recommend an alternative solution?

[jeddy3]

Shall we create and publish jest-preset-stylelint with dependency injection and have the community provide alternative runners, like stylelint-tape? It seems like the best option having weighed up the pros and cons.

It would also be in line with having a one "blessed" editor integration (stylelint/vscode-stylelint).