feat: improve expectTypeOf error messages (#4206)

Co-authored-by: Misha Kaletsky <mmkal@users.noreply.github.com>
Co-authored-by: Vladimir Sheremet <sleuths.slews0s@icloud.com>

README.md

@@ -42,6 +42,7 @@ Next generation testing framework powered by Vite.
 - Workers multi-threading via [Tinypool](https://github.com/tinylibs/tinypool) (a lightweight fork of [Piscina](https://github.com/piscinajs/piscina))
 - Benchmarking support with [Tinybench](https://github.com/tinylibs/tinybench)
 - [Workspace](https://vitest.dev/guide/workspace) support
+- [expect-type](https://github.com/mmkal/expect-type) for type-level testing
 - ESM first, top level await
 - Out-of-box TypeScript / JSX support
 - Filtering, timeouts, concurrent for suite and tests

docs/guide/testing-types.md

@@ -31,17 +31,62 @@ You can see a list of possible matchers in [API section](/api/expect-typeof).
 
 ## Reading Errors
 
-If you are using `expectTypeOf` API, you might notice hard to read errors or unexpected:
+If you are using `expectTypeOf` API, refer to the [expect-type documentation on its error messages](https://github.com/mmkal/expect-type#error-messages).
+
+When types don't match, `.toEqualTypeOf` and `.toMatchTypeOf` use a special helper type to produce error messages that are as actionable as possible. But there's a bit of an nuance to understanding them. Since the assertions are written "fluently", the failure should be on the "expected" type, not the "actual" type (`expect<Actual>().toEqualTypeOf<Expected>()`). This means that type errors can be a little confusing - so this library produces a `MismatchInfo` type to try to make explicit what the expectation is. For example:
+
+```ts
+expectTypeOf({ a: 1 }).toEqualTypeOf<{ a: string }>()
+```
+
+Is an assertion that will fail, since `{a: 1}` has type `{a: number}` and not `{a: string}`.  The error message in this case will read something like this:
+
+```
+test/test.ts:999:999 - error TS2344: Type '{ a: string; }' does not satisfy the constraint '{ a: \\"Expected: string, Actual: number\\"; }'.
+  Types of property 'a' are incompatible.
+    Type 'string' is not assignable to type '\\"Expected: string, Actual: number\\"'.
+
+999 expectTypeOf({a: 1}).toEqualTypeOf<{a: string}>()
+```
+
+Note that the type constraint reported is a human-readable messaging specifying both the "expected" and "actual" types. Rather than taking the sentence `Types of property 'a' are incompatible // Type 'string' is not assignable to type "Expected: string, Actual: number"` literally - just look at the property name (`'a'`) and the message: `Expected: string, Actual: number`. This will tell you what's wrong, in most cases. Extremely complex types will of course be more effort to debug, and may require some experimentation. Please [raise an issue](https://github.com/mmkal/expect-type) if the error messages are actually misleading.
+
+The `toBe...` methods (like `toBeString`, `toBeNumber`, `toBeVoid` etc.) fail by resolving to a non-callable type when the `Actual` type under test doesn't match up. For example, the failure for an assertion like `expectTypeOf(1).toBeString()` will look something like this:
+
+```
+test/test.ts:999:999 - error TS2349: This expression is not callable.
+  Type 'ExpectString<number>' has no call signatures.
+
+999 expectTypeOf(1).toBeString()
+                    ~~~~~~~~~~
+```
+
+The `This expression is not callable` part isn't all that helpful - the meaningful error is the next line, `Type 'ExpectString<number> has no call signatures`. This essentially means you passed a number but asserted it should be a string.
+
+If TypeScript added support for ["throw" types](https://github.com/microsoft/TypeScript/pull/40468) these error messagess could be improved significantly. Until then they will take a certain amount of squinting.
+
+#### Concrete "expected" objects vs typeargs
+
+Error messages for an assertion like this:
 
 ```ts
-expectTypeOf(1).toEqualTypeOf<string>()
-//             ^^^^^^^^^^^^^^^^^^^^^^
-// index-c3943160.d.ts(90, 20): Arguments for the rest parameter 'MISMATCH' were not provided.
+expectTypeOf({ a: 1 }).toEqualTypeOf({ a: '' })
 ```
 
-This is due to how [`expect-type`](https://github.com/mmkal/expect-type) handles type errors.
+Will be less helpful than for an assertion like this:
 
-Unfortunately, TypeScript doesn't provide type metadata without patching, so we cannot provide useful error messages at this point, but there are <a href="https://github.com/microsoft/TypeScript/pull/40468" target="_blank">works in TypeScript project</a> to fix this. If you want better messages, please, ask TypeScript team to have a look at mentioned PR.
+```ts
+expectTypeOf({ a: 1 }).toEqualTypeOf<{ a: string }>()
+```
+
+This is because the TypeScript compiler needs to infer the typearg for the `.toEqualTypeOf({a: ''})` style, and this library can only mark it as a failure by comparing it against a generic `Mismatch` type. So, where possible, use a typearg rather than a concrete type for `.toEqualTypeOf` and `toMatchTypeOf`. If it's much more convenient to compare two concrete types, you can use `typeof`:
+
+```ts
+const one = valueFromFunctionOne({ some: { complex: inputs } })
+const two = valueFromFunctionTwo({ some: { other: inputs } })
+
+expectTypeOf(one).toEqualTypeof<typeof two>()
+```
 
 If you find it hard working with `expectTypeOf` API and figuring out errors, you can always use more simple `assertType` API:
 

packages/vitest/package.json

@@ -171,7 +171,7 @@
     "birpc": "0.2.14",
     "chai-subset": "^1.6.0",
     "cli-truncate": "^4.0.0",
-    "expect-type": "^0.16.0",
+    "expect-type": "^0.17.3",
     "fast-glob": "^3.3.2",
     "find-up": "^6.3.0",
     "flatted": "^3.2.9",

packages/vitest/src/typecheck/typechecker.ts

@@ -187,7 +187,9 @@ export class Typechecker {
       const suiteErrors = errors.map((info) => {
         const limit = Error.stackTraceLimit
         Error.stackTraceLimit = 0
-        const error = new TypeCheckError(info.errMsg, [
+        // Some expect-type errors have the most useful information on the second line e.g. `This expression is not callable.\n  Type 'ExpectString<number>' has no call signatures.`
+        const errMsg = info.errMsg.replace(/\r?\n\s*(Type .* has no call signatures)/g, ' $1')
+        const error = new TypeCheckError(errMsg, [
           {
             file: filepath,
             line: info.line,
@@ -201,7 +203,7 @@ export class Typechecker {
           error: {
             name: error.name,
             nameStr: String(error.name),
-            message: info.errMsg,
+            message: errMsg,
             stacks: error.stacks,
             stack: '',
             stackStr: '',

test/typescript/test/__snapshots__/runner.test.ts.snap

@@ -1,18 +1,18 @@
 // Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
 
 exports[`should fail > typecheck files 1`] = `
-"TypeCheckError: Expected 1 arguments, but got 0.
-TypeCheckError: Expected 1 arguments, but got 0.
-TypeCheckError: Expected 1 arguments, but got 0.
-TypeCheckError: Expected 1 arguments, but got 0.
-TypeCheckError: Expected 1 arguments, but got 0.
+"TypeCheckError: Type 'string' does not satisfy the constraint '"Expected string, Actual number"'.
+TypeCheckError: This expression is not callable. Type 'ExpectArray<number>' has no call signatures.
+TypeCheckError: This expression is not callable. Type 'ExpectUndefined<number>' has no call signatures.
+TypeCheckError: This expression is not callable. Type 'ExpectVoid<number>' has no call signatures.
+TypeCheckError: Type 'string' does not satisfy the constraint '"Expected string, Actual number"'.
 TypeCheckError: Unused '@ts-expect-error' directive.
-TypeCheckError: Expected 1 arguments, but got 0."
+TypeCheckError: This expression is not callable. Type 'ExpectVoid<number>' has no call signatures."
 `;
 
 exports[`should fail > typecheck files 2`] = `
 " FAIL  fail.test-d.ts > nested suite
-TypeCheckError: Expected 1 arguments, but got 0.
+TypeCheckError: This expression is not callable. Type 'ExpectVoid<number>' has no call signatures.
  ❯ fail.test-d.ts:15:19
      13|   })
      14| 
@@ -34,18 +34,18 @@ TypeCheckError: Unused '@ts-expect-error' directive.
 
 exports[`should fail > typecheck files 4`] = `
 " FAIL  fail.test-d.ts > failing test
-TypeCheckError: Expected 1 arguments, but got 0.
- ❯ fail.test-d.ts:4:19
+TypeCheckError: Type 'string' does not satisfy the constraint '"Expected string, Actual number"'.
+ ❯ fail.test-d.ts:4:33
       2| 
       3| test('failing test', () => {
       4|   expectTypeOf(1).toEqualTypeOf<string>()
-       |                   ^
+       |                                 ^
       5| })"
 `;
 
 exports[`should fail > typecheck files 5`] = `
 " FAIL  fail.test-d.ts > nested suite > nested 2 > failing test 2
-TypeCheckError: Expected 1 arguments, but got 0.
+TypeCheckError: This expression is not callable. Type 'ExpectVoid<number>' has no call signatures.
  ❯ fail.test-d.ts:10:23
       8|   describe('nested 2', () => {
       9|     test('failing test 2', () => {
@@ -56,7 +56,7 @@ TypeCheckError: Expected 1 arguments, but got 0.
 
 exports[`should fail > typecheck files 6`] = `
 " FAIL  fail.test-d.ts > nested suite > nested 2 > failing test 2
-TypeCheckError: Expected 1 arguments, but got 0.
+TypeCheckError: This expression is not callable. Type 'ExpectUndefined<number>' has no call signatures.
  ❯ fail.test-d.ts:11:23
       9|     test('failing test 2', () => {
      10|       expectTypeOf(1).toBeVoid()
@@ -67,7 +67,7 @@ TypeCheckError: Expected 1 arguments, but got 0.
 
 exports[`should fail > typecheck files 7`] = `
 " FAIL  js-fail.test-d.js > js test fails
-TypeCheckError: Expected 1 arguments, but got 0.
+TypeCheckError: This expression is not callable. Type 'ExpectArray<number>' has no call signatures.
  ❯ js-fail.test-d.js:6:19
       4| 
       5| test('js test fails', () => {
@@ -78,12 +78,12 @@ TypeCheckError: Expected 1 arguments, but got 0.
 
 exports[`should fail > typecheck files 8`] = `
 " FAIL  only.test-d.ts > failing test
-TypeCheckError: Expected 1 arguments, but got 0.
- ❯ only.test-d.ts:4:19
+TypeCheckError: Type 'string' does not satisfy the constraint '"Expected string, Actual number"'.
+ ❯ only.test-d.ts:4:33
       2| 
       3| test.only('failing test', () => {
       4|   expectTypeOf(1).toEqualTypeOf<string>()
-       |                   ^
+       |                                 ^
       5| })"
 `;
 
@@ -99,8 +99,8 @@ Error: error TS18003: No inputs were found in config file '<root>/tsconfig.vites
 `;
 
 exports[`should fail > typecheks with custom tsconfig 1`] = `
-"TypeCheckError: Expected 1 arguments, but got 0.
-TypeCheckError: Expected 1 arguments, but got 0.
-TypeCheckError: Expected 1 arguments, but got 0.
-TypeCheckError: Expected 1 arguments, but got 0."
+"TypeCheckError: This expression is not callable. Type 'ExpectUndefined<number>' has no call signatures.
+TypeCheckError: This expression is not callable. Type 'ExpectVoid<number>' has no call signatures.
+TypeCheckError: Type 'string' does not satisfy the constraint '"Expected string, Actual number"'.
+TypeCheckError: This expression is not callable. Type 'ExpectVoid<number>' has no call signatures."
 `;