Upgrade TypeDoc

[jasongrout]

References

After the upgrade to TypeScript 4.1 in #9476, I think we need to upgrade TypeDoc. This PR does that.

Code changes

Upgrade TypeDoc to 0.20.0-beta.27.

Previously we had pinned to exactly 0.17.0-3 since that was the version that had the library mode.

This includes #9476, so that should be merged first.

User-facing changes

Backwards-incompatible changes

[jupyterlab-dev-mode]

Thanks for making a pull request to JupyterLab!

To try out this branch on binder, follow this link:

[telamonian]

Hmm, so far I've been able to locally reproduce the same error we're seeing in the CI for jlpm docs, so that's good, as far as it goes. More worringly, when I just ran a normal jlpm build it took forever and exhausted my system's memory:

webpack 5.3.1 compiled successfully in 1051 ms
✨  Done in 643.55s.

Probably 600 of those seconds were spent just in tsc, which is downright unusual.

@jasongrout Have you noticed any increase in build times or resources used during build after upgrading typescript to 4.1.3?

[telamonian]

@jasongrout The v0.20.0-beta.27 typedoc build works! You probably won't love what I had to do to make it work, though:

• hacks (hopefully temporary)
  • no @jupyterlab/application-extension docs
  • all test-related .ts sources have to be temporarily deleted immediately before the typedoc build and then restored immediately afterwards

update: that second (extra ugly) hack is thankfully no longer needed, due to @Gerrit0's very helpful suggestion about the exclude field in tsconfigdoc.json.

For now I think the above is liveable. As for the docs themselves, they look good. So far I've noticed a few regressions (eg there's some kind of issue with cross-package inheritance, so some parent classes aren't listed/linked correctly), and a few improvements (eg the TOC is slightly nicer), so I think it's fine.

Longer term, I'll be submitting related issues upstream to typedoc, so all of this can eventually get cleaned up

[telamonian]

when I just ran a normal jlpm build it took forever and exhausted my system's memory:

webpack 5.3.1 compiled successfully in 1051 ms
✨  Done in 643.55s.

I just closed my other programs and tried another clean build. This time it was only 148 s, which yeah, pretty normal

What I saw before during the slow build was 4 separate node procs that were using > 100% CPU and also sucked up all available system memory (which is prob where the real slowdown came from). This time there was only one high cpu node. So I think what I saw before might have actually been the node procs associated with vscode panicking for some reason. Perhaps some new behavior of tsc forced them into a confused state

anyway, I can't repro it, so it's a non-issue

[Gerrit0]

all test-related .ts sources have to be temporarily deleted immediately before the typedoc build and then restored immediately afterwards

This worried me, so I spent some time looking into it. TypeDoc's behavior with errors should be identical to either npx tsc -p tsconfigdoc.json or npx tsc -b tsconfigdoc.json, depending on your tsconfig setup. In this case, npx tsc -p tsconfigdoc.json produces the same errors.

If I add an exclude section to the tsconfig to tell it to ignore those files - both tsc and typedoc will run as expected.

diff --git a/tsconfigdoc.json b/tsconfigdoc.json
index bf46a7779..7f959a811 100644
--- a/tsconfigdoc.json
+++ b/tsconfigdoc.json
@@ -1,6 +1,7 @@
 {
   "$schema": "http://json.schemastore.org/tsconfig",
   "extends": "./tsconfigbase",
+  "exclude": ["**/test/**", "**/testutils/**"],
   "compilerOptions": {
     "paths": {
       "@jupyterlab/*": [

beta 27 doesn't remove src from the end of module names (I think I should probably make it do this though since it is common). However, you can change the module name displayed for each package with a @module tag if in a rush:

diff --git a/packages/application/src/index.ts b/packages/application/src/index.ts
index 3ed540439..f3571ff9f 100644
--- a/packages/application/src/index.ts
+++ b/packages/application/src/index.ts
@@ -1,3 +1,5 @@
+/** @packageDocumentation
+@module application */
 // Copyright (c) Jupyter Development Team.
 // Distributed under the terms of the Modified BSD License.

It looks like I need to introduce more logic for determining references, application -> ILabShell has the same TS symbol as the interface, which results in the variable type linking to the variable instead of the interface.

Hopefully this helps a bit! Probably won't have time to look at any other issues until the weekend...

[jasongrout]

Wow, @Gerrit0, thank you very much for not only your great work on typedoc, but also for coming over to look at the issues we've been having!

The other day, when initially working on upgrading from 0.17.0-3 to 0.20 beta, I was trying to ignore these test files by adding to our exclude section of typedoc.js, but it didn't seem to do anything. I didn't think about changing the tsconfig that typedoc was using.

[telamonian]

@Gerrit0 This is awesome, thank you for coming to help! 👍 to everything @jasongrout just said

I was trying to ignore these test files by adding to our exclude section of typedoc.js, but it didn't seem to do anything. I didn't think about changing the tsconfig that typedoc was using.

Yeah I had the same experience. The exclude field in our typedoc.js definitely does work with typedoc v0.17, but I couldn't make heads or tails of it while working on this PR. Between that and the way that v0.20 picks up file paths from the references field in our tsconfig.json files, I was starting to get very confused about the interactions of entryPoints, exclude, and references in v0.20

[telamonian]

@Gerrit0's Your explanation about npx tsc -p tsconfigdoc.json is helpful. I wonder if the difference between this command and our build's normal invocation of tsc:

cd packages/metapackage && tsc -b

might also explain why I was seeing several dozen strange type errors when including the application-extension package in the typedoc build:

Error: packages/application-extension/src/index.tsx:138:13 - error TS2339: Property 'registerPluginErrors' does not exist on type 'never'.
  The intersection 'JupyterFrontEnd<IShell, "desktop" | "mobile"> & JupyterLab' was reduced to 'never' because property '_contextMenuEvent' exists in multiple constituents and is private in some.

138     if (app.registerPluginErrors.length !== 0) {
                ~~~~~~~~~~~~~~~~~~~~

The issue here seems to boil down to an instanceof type guard somehow getting interpreted very differently in our typedoc build vs our standard build:

jupyterlab/packages/application-extension/src/index.tsx

Lines 92 to 112 in 979dd37

	/**
	* The main extension.
	*/
	const main: JupyterFrontEndPlugin<ITreePathUpdater> = {
	id: '@jupyterlab/application-extension:main',
	requires: [IRouter, IWindowResolver, ITranslator],
	optional: [ICommandPalette, IConnectionLost],
	provides: ITreePathUpdater,
	activate: (
	app: JupyterFrontEnd,
	router: IRouter,
	resolver: IWindowResolver,
	translator: ITranslator,
	palette: ICommandPalette | null,
	connectionLost: IConnectionLost | null
	) => {
	const trans = translator.load('jupyterlab');
	if (!(app instanceof JupyterLab)) {
	throw new Error(`${main.id} must be activated in JupyterLab.`);
	}

As far as I can tell the tsconfig.json settings are the same for both builds, so it's a bit unclear what's going on here. I'll open an issue for this on the typedoc repo

[jasongrout]

I think we can also delete the exclude settings in typedoc.js.

[telamonian]

I agree. But I still want to know more about exactly what exclude in tsconfigdoc.json (which presumably is just acting as the standard tsconfig option here) is supposed to do vs exclude in typedoc-specific config

[jasongrout]

@telamonian - I think this looks good now. Do you think it is ready to go? We'll be missing the application-extension docs until we solve that never typing issue mentioned above, but I think that's a relatively minor issue all things considered, the library api documentation being the really important thing here.

[blink1073]

Thanks!

[jasongrout]

Thanks!

[telamonian]

give me a couple more hours with this. The more I look at the generated docs, the more issues I see. For example, in some places it's pulling defs from lib/*.d.ts instead of the real src/*.ts, which then seems to break some links and refs

whelp, I guess maybe that fix can be a new PR. But yeah, I think the current state is good enough for 3.0.0, but not for, say, 3.0.1

[jasongrout]

Sorry for rushing this, and thanks for follow-up PRs to fix those sorts of issues.

[telamonian]

actually, for some reason it looks like all of the -extension packages are fine, but all others fetch their symbols from /lib, which seems to break a bunch of links. weird

[Gerrit0]

The exclude option in 0.20 is used to filter entry points - but only if you give typedoc a directory as an entry point.
In previous versions, it was used to filter what files were given to typescript as "root names" of a program. This had problems with more complicated projects, resulting in weird compiler errors at times.

From the documentation side of things - tsc -b / project references is a really painful feature to support. tsc implements this by getting each project according to an order derived from the project reference graph, and building just those packages which are outdated. This is great for build speed, but means that the root project doesn't actually have references to the source files - they instead consume the built declaration files. There is probably a way around this... but I haven't found it yet. (Maybe using the language service? Does go to definition work properly with project references?)

The errors with application-extension... no idea there. As a wild guess, typedoc probably does something wrong with project references so the program isn't created correctly.

[telamonian]

Does go to definition work properly with project references?

@Gerrit0 Yes, "go to def" works very reliably (at least in vscode and intellij, which both use the standard tsserver language service).

On the other hand, there are at least some language service features that are a bit spottier. For example, in order to get "find all references" to actually search in all of the packages in our monorepo, I've found that I have to have at least one file open from each package.

But even with our weird project structure "go to def" has always worked well

[jasongrout]

I opened #9495 to deal with the broken links in the docs.