Compilation errors aren't silenced in a .tst.ts file when running typescript, but the test still passes with tstyche

[erheron]

Hi, and thanks for this awesome library!
We noticed that .toRaiseError sometimes doesn't quite work as we expect it to with respect to compilation errors.
We're not sure if there is a bug, or some better documentation is needed 🤔

Reproduction

/// file sample.tst.ts

import { expect, test } from 'tstyche';

function id(arg: number): number {
	return arg;
}

test('Cannot call "id" function with an argument of type string', () => {
	// Full error underneath: "Argument of type `string` is not assignable to parameter of type `number`."
	expect(id('123')).type.toRaiseError('Argument'); 
	expect(id('123')).type.toRaiseError(); 
});

And then when we run yarn tsc, the result is

src/__typetests__/sample.tst.ts:9:12 - error TS2345: Argument of type 'string' is not assignable to parameter of type 'number'.

9  expect(id('123')).type.toRaiseError('Argument');
             ~~~~~

src/__typetests__/sample.tst.ts:10:12 - error TS2345: Argument of type 'string' is not assignable to parameter of type 'number'.

10  expect(id('123')).type.toRaiseError();

However, running yarn tstyche gives an expected output:

❯ yarn tstyche
yarn run v1.22.19
$ /Users/..../.bin/tstyche
uses TypeScript 5.2.2 with ./tsconfig.json

pass ./src/__typetests__/sample.tst.ts
  + Cannot call id function with an argument of type string

Targets:    1 passed, 1 total
Test files: 1 passed, 1 total
Tests:      1 passed, 1 total
Assertions: 2 passed, 2 total
Duration:   11.1s

Ran all test files.
✨  Done in 12.66s.

Packages

typescript@5.2.2
tstyche@1.0.0-beta.1

[mrazauskas]

Do I get it right, you would expect to see no error in tscs output? Hm.. It does not know anything about tstyche, so it reports the error.

Why do you run tsc on a type test? (;

[mrazauskas]

Thanks for giving a try for TSTyche! It is a fresh library, can be documentation is missing some clarity. Let’s figure this out.

[erheron]

Do you get it right, you would expect to see no error in tscs output? Hm.. It does not know anything about tstyche, so it reports the error.

You're right. After giving it a second thought, that was a bit of an unfair expectation 😬

Why do you run tsc on a type test? (;

It's valuable for us to get autocomplete and other help from Typescript when writing a test (think of a normal unit test -- would you like it to write it without TSC?). Please note that if this would be another type test (not the one with .toRaiseError), running typescript on it probably wouldn't make any difference.

I see your suggestion to not run TSC on type test files 🙌🏻
I'd like to think of it as a last resort, due to a worse DevEx.
If nothing more comes to mind, though, we'll give it a go. Maybe I'm being too pessimistic about that "bad DevEx"

[mrazauskas]

Wait.. Autocompletions should be there. TypeScript provides these through language service. That is just a technical detail, tsc is the TypeScript compiler.

Are you trying to say that the errors should not be highlighted in IDE, if they are matched by .toRaiseError()? This will be fixed in some 2-3 months. I have a working prototype.

[erheron]

Wait.. Autocompletions should be there. TypeScript provides these through language service. That is just a technical detail, tsc is the TypeScript compiler.

Yeah sorry, I mixed things up a little, but here's what I meant:

1. I thought it would be best if I can both compile this file with tsc and run it with tstyche. That does not seem to be possible (cause' as you said, TS does not know nothing about tstyche).
2. Yes, it would be nice if errors are not highlighted in the IDE and language service still works. If there's a working prototype for that, I'm more than happy to wait 🚀

[mrazauskas]

Thanks!

It still feels puzzling for me why you want to pass test files through tsc? Are you trying to type check them? There is no need, because tstyche does that for you. It will pick up all syntactic and semantic errors just like tsc. Try it out (also on different versions of compiler, e.g. tstyche --target 4.8,latest).

And there is more:

import { expect, test } from "tstyche";

test.skip("is skipped?", () => {
  const a: string = 123; // obviously a type error, but it will be skipped

  // ...
});

If a test is skipped, tstyche does not report type errors in that particular test. Less noise while debugging. This is why it is a type test runner.

[erheron]

It still feels puzzling for me why you want to pass test files through tsc? Are you trying to type check them?

Yep exactly, type-checking code on-the-fly is something that boosts the development, from my experience.

But anyway, it sounds like we just can't achieve that here, and the alternative way is to use IDE Language Server (we'll be waiting for the improvement about .toRaiseError! 🚀 ) + exclude .tst files from compilation and include them in tstyche run. We'll give it a go 🙌🏻

One more thought (unrelated to the original topic haha) -- is the --watch mode planned on the roadmap?

[mrazauskas]

Yes, the watch mode is planned for version 2. I have a working draft, but it needs more thinking.

[mrazauskas]

To make it easier to track the progress, I open these issues:

• Add watch mode #16
• Silence errors matched by .toRaiseError() #17

Add a couple of sentences will be added to the documentation website as well.