Add coverage report

[voxpelli]

I'm not sure if this is possible at all, but it would be very neat to get a coverage report for the type tests, to avoid a situation like with HasMultipleCallSignatures in sindresorhus/type-fest#839, where an untested type is what causes some other much more complex types to fail.

[mrazauskas]

Thanks for idea. It definitely worth to think about.

A good start could be to agree on what coverage is in this case. There are libraries checking if there are no implicit anys in code and naming this 100% coverage. Hm.. That sounds more like lint, not type coverage.

I think, TSTyche could make sure that all paths of a conditional type or all members of a union type are tested. These are most obvious, but there should be other cases.

This will not be easy to find a way to implement a feature like this one, but it sounds very interesting to try.

[voxpelli]

Yeah, eg type-coverage checks for the use of any

I would be mostly interested in knowing whether all exports of a file at all has been directly imported/used from a test file.

Another thing to consider is whether this is something to be found at "runtime" or as a separate static analysis job (eg: does all X from Y appear in the Z of ABC)

[voxpelli]

If one looks for all symbols exported from every non-test file and compares that to every symbol used within a test(), then that could possibly be a first step.

[mrazauskas]

Ah.. right! Checking if all exports were imported is very good first step and this is doable. Harder to know if they are actually tested, but also doable and that’s the second step.

I think this must be behind --coverage flag and must be checked during test run. In a way user wants to know, if current tests are covering all features or not.

This is more flexible than it might sound, because any file selected by the testFileMatch patterns is a test file. It is not required to use test() or any other API. For instance, a file with usage examples can be a type test. Running tstyche --coverage type checks the file and makes sure that all APIs have usage examples.

Also some option like sourceFileMatch should be implemented to allow selecting source files for coverage check. Having it you should be able to check if all exports in files selected by sourceFileMatch are imported/used in files selected by testFileMatch. Et voilà!

Thanks for the input. I will work on this.