Add automated regression testing against old TS versions

[bradzacher]

#1698 introduced a regression for TS<3.8.x (#1746).
This wasn't caught by the type checker, nor was it caught by the tests because both were using TS3.8.

It was a simple mistake, but one that shouldn't have happened.
We should add an integration test in order to make sure that this doesn't happen again.

We cannot run builds against older versions of TS, because our code relies upon property types etc added by the latest versions, but we can easily run tests against the old versions.
I don't think we need to test every version we support, but at least testing against the previous one or two minors and the minimum supported version should be good enough.

Regressions that would have been caught:

• Regression in 2.24.0: TypeError: Cannot read property 'name' of undefined #1746 - feat(typescript-estree): support 3.8 export * as ns #1698 used an enum value added in TS3.8.
• Receiving Parsing error: not implemented on v2.24.0 #1759 - fix: unecessary program updates by removing timeout methods #1693 relied upon functionality added in TS3.3.
• [no-unsafe-return] TypeError: checker.getTypeArguments is not a function #1807 - feat(eslint-plugin): add rule no-unsafe-return #1644 used an API added in TS3.7.
• chore: upgrade local ts version to 4.0.0-beta #2305 produced an incorrect AST for <TS4.0.
• Bug: [no-unused-vars] 'Component' is defined but never used #5571 - feat: support TypeScript 4.8 #5551 broke exported decorators for <TS4.8.

[JoshuaKGoldberg]

Copying the useful bits from my dup issue:

The simplest approach might be extending CI's unit_tests_on_other_node_versions to also switch on different TypeScript package versions (example syntax).

It'd be great as a next step (followup issue?) to also make use of CodeCov's report merging to combine unit test coverage from all of them. #2516 is an example PR where code under test changes behavior on the TypeScript version.

[bradzacher]

The thing that makes this difficult is ensuring that the rules that have variant functionality on TS versions will have to pass on the other TS versions (this is the same problem that makes #2464 difficult)

To make this work we'll have to extend and customise the core ESLint RuleTester and design a config system for it.

---

There's a lot of codecov features we should totally leverage more of - I've just never bothered to read the docs 😅

[JoshuaKGoldberg]

To start, can that be as simple as checking the TypeScript version at Jest runtime when constructing the .test.ts file tests?

tests/rules/some-rule.test.ts:

ruleTester.run('some-rule', rule, {
  // ...
});

if (util.isAboveTsVersion('3.9')) {
  ruleTester.run('some-rule typescript@>=3.9', rule, {
    // ...
  });
} else {
  ruleTester.run('some-rule typescript@<3.9', rule, {
    // ...
  });
}

util/versions.ts:

// (may need to add overrides or some logic if any TS releases had unusual version numbers...)
export const isAboveTsVersion = (version: string) =>
  semver.satisfies(
    ts.version,
    `>= ${version}.0 || >= ${version}.1-rc || >= ${version}.0-beta`,
    {
      includePrerelease: true,
    },
  );

Are there other testing features needed?

[bradzacher]

It should be pretty trivial for us to extend the rule tester and add a property to the config object. It'd set a good precedence for doing a similar thing with ESLint version

We can chuck it in here

Something like:

type RunTests<TMessageIds extends string, TOptions extends Readonly<unknown[]>> = MakeSomeChangesToTheType<TSESLint.RunTests<TMessageIds, TOptions>>;

class RuleTester extends RuleTester {
  run<TMessageIds extends string, TOptions extends Readonly<unknown[]>>(
    name: string,
    rule: TSESLint.RuleModule<TMessageIds, TOptions>,
    testsReadonly: RunTests<TMessageIds, TOptions>,
  ): void {
    const tests = { ...testsReadonly };

    // standardize the valid tests as objects
    tests.valid = tests.valid.map(test => {
      if (typeof test === 'string') {
        return {
          code: test,
        };
      }
      return test;
    });

    const filter = test => isValidTsVersion(test.tsVersion);

    tests.valid = tests.valid.filter(filter);
    tests.invalid = tests.valid.filter(filter);
  }
}

[voxpelli]

Following up on original suggestion, I just pushed a suggestion to type-fest that re-utilizes the matrix for testing on many ts-versions: sindresorhus/type-fest@3a4bb56

[bradzacher]

I think in the next major we need to consider filling DefinitelyTyped's example and bump our minimum TS version as well?

We support a super old one and it's getting to be a but out of date at this point!
We go back as TS 3.1 which is 4 years old!

DT supports 4.0 and above for a few more days (at which point 4.1 will be EOL)
https://github.com/DefinitelyTyped/DefinitelyTyped/#support-window

@JoshuaKGoldberg

[JoshuaKGoldberg]

Proposal: let's support the same time range as DT? I don't see any particular reason to go older.