Suggestion: CompilerHost should be able to "plug into" a LanguageServiceHost

[TheLarkInn]

Background

The traditional method of transpiling a typescript application is through ts.CompilerHost. This is awesome, and there are slowly becoming more and more CompilerHost based "flavors" of tsc that developers are wanting to use. I spoke about this with @alexeagle and @mhegazy briefly on this as well:

Webpack Loaders

A webpack loader in its most simple form does the following:

function takeASingleFileThatIsServedToThisFunctionViaWebpack(rawFileStringContent) {
  let newlyUpdatedRawFileString = rawFileStringContent + 'console.log(WOW THIS MAKES SENSE);'


  return newlyUpdatedRawFileString;
}

As you can see all a loader does is take a raw source, and return it. On top of this webpack handles file resolution through some crafty regex:

loaders: [
  {
    test: /\.ts/,  // this regex string tells webpack which files to 'send' to the loader
    loader: ‘ts-loader’ // this specifies the loader to use (resolves to node modules)
  }
]

Webpack Typescript Loaders Cannot & Should Not Use ts.CompilerHost

So knowing what we know about how a file is tranformed and resolved through webpack, ts.CompilerHost is not an option for a ts-webpack loader. Why? Loaders are a powerful tool in webpack because they can be chained together with other loaders. For example: This Angular2 TS Loader I wrote. CompilerHost (as you know probably) runs a build/transpiliation against a whole project, and not one file per time. Therefore if you tried to plop a CompilerHost in a webpack loader, it would lose the context between each file.

Webpack TS Loaders & ts.LanguageServiceHost.

Because of this, the current typescript webpack loaders that are being used (ts-loader, awesome-typescript-loader) are both using ts.LanguageServiceHost. Why? Because LSG's support transpilation at a long-term lifetime one-file-at-a-time transpilation, persisting the transpile context back to the ts.Program. (This is tranditionally useful for IDE'S yes, but its a match made in heaven for a webpack loader). I would call this an 'edge-case' request, however, with 5+ million downloads a month, webpack is increasingly more popular than any bundler at the moment.

The Problem:

Trying to use ts.CompilerHost implementations in a ts.LanguageServiceHost is a huge pain, and doesn't port well into an existing LSH. Its true that they share some functions, however there lacks a way to use a CompilerHost in a LanguageService incrementally.

The Solution.

Create some way to attach the functionality of a custom CompilerHost implementation, and then perform it on a per-file basis, updating the LanguageServiceHost to help persist the transpilation/program contex. This will allow not only @s-panferov and @TypeStrong able to create plugin's systems that allow for super powerful TS builds in webpack. Or instead a plugin system where you add/override certain functionalities to an existing LSH.

Please refer to this conversation thread of me explaining the issue to @DanielRosenwasser with lots of examples:

Start: https://gitter.im/Microsoft/TypeScript?at=5757ab04662b042b7e596b70
End: https://gitter.im/Microsoft/TypeScript?at=5757b77445cf128e5f1d7429

References

#6508 (comment)
angular/angular#8759
TypeStrong/ts-loader#223
https://github.com/TypeStrong/ts-loader/issues/
s-panferov/awesome-typescript-loader#153
TypeStrong/ts-loader#224

[DanielRosenwasser]

Thanks for this awesome writeup and summarizing the points of the conversation! 💯

[TheLarkInn]

No problem I'm glad I could clarify!!!

[mhegazy]

CC @chuckjaz as we had a chat with about this issue.

So why do you use LanguageServiceHost to do the loading. i would expect you just use CompilerHost all the time, given that there are no language service operations needed here.

[TheLarkInn]

CC @s-panferov @jbrantly, please chime up if I'm not explaining this right.

@mhegazy @chuckjaz so I said above that webpack is responsible for serving individual files one at a time to a loader. In that case, you can't use CompilerHost on one TS file at a time (because you lose program context between transpilation).

LSH provides the ability to build up your Program of files, and then additionally update them only compiling that one file in the context of the Program. Is there something I am missing?

[aluanhaddad]

This may be relevant to plugin-typescript as well.
cc @frankwallis

[mhegazy]

The LanguageService API (LS), and the matching host, LanguageServiceHost (LSH), are built for IDE/Editor scenarios. The LS is mainly designed to be able to answer questions like get completions at position, quick info, formatting edits, etc.. one thing that the LS has to do well is update the state after an edit.

The Compiler API, (i.e. createProgram), and the matching host implementation, CompilerHost, are designed for generating outputs and errors given a set of sources.

Obviously there is an overlap. the LS is a higher level abstraction than the Compiler API. but, assuming i understand your needs correctly, you do not really need it.

If i understand correctly, what you need is given some source file, generate js/map/declarations, and compute errors.
And once you built that state, you want to keep it in memory and reuse it if one or more files change, and quickly regenerate errors and outputs.
if this is not an accurate representation of your scenario please let me know.

generating outputs and errors are done through program.emitFiles and program.get*Diagnostics.

Caching state, and reusing it is done through two things:

1. caching SourceFiles (AST's), i.e. if a file did not change, do not parse it again.
2. reusing program structure: i.e. do not resolve modules on disk if the file has not changed and we resolved them before.

For reusing the SourceFiles, the CompilerHost API has getSourceFile method specifically for this. it allows you to chose either to parse it, i.e. call createSourceFile or get it from a cache, say a previous Program object that has parsed this file, e.g. Program.getSourceFile(filename). This is what the --watch implementation does. see https://github.com/Microsoft/TypeScript/blob/master/src/compiler/tsc.ts#L460-L473.

The second is achieved by passing the old program to the createProgram call, see https://github.com/Microsoft/TypeScript/blob/master/src/compiler/program.ts#L1049.

The LS can do this for you as well, as part of other things, but as noted before, you do not need the LS to do this.

I think the request here is for an API, that given a Program instance, and a set of edits, generate a new program with reusing as much as possible. if this is accurate, then we can definitely do that.

[TheLarkInn]

First, I want to thank you and @DanielRosenwasser for the support while you guys are in the midst of a crazy busy and awesome TS2.0 timeline. I apoligize if this response is messy but its hard for me to track everything in both loaders to ensure that the right things are done here.

Obviously there is an overlap. the LS is a higher level abstraction than the Compiler API. but, assuming i understand your needs correctly, you do not really need it.

If i understand correctly, what you need is given some source file, generate js/map/declarations, and compute errors.
And once you built that state, you want to keep it in memory and reuse it if one or more files change, and quickly regenerate errors and outputs.
if this is not an accurate representation of your scenario please let me know.
I think the request here is for an API, that given a Program instance, and a set of edits, generate a new program with reusing as much as possible. if this is accurate, then we can definitely do that.

Not quite, what's needed is: (again @s-panferov and @jbrantly please correct me on this)
That given a Program instance, and a single edit, generate the following:

• the single transpiled file
• languageService.getEmitOutput(filepath)
  • gives access to outputFiles
• the dependencyGraph for the transpiled file (I believe this is languageService.dependencyGraph[filePath])
  • this allows loaders to add dependencies into webpacks dependency system

https://github.com/TypeStrong/ts-loader/blob/master/index.ts#L587-L633

[s-panferov]

awesome-typescript-loader right now has two basic scenarios of work (for watches):

1. If declaration enabled.
   1.1 Update program (pass all changed files)
   1.2 Emit changed file: transpile source, get source map, get declaration emit. Uses program.emit
   1.3. Extract dependency graph, pass it to webpack.
   1.4 Check that full program is correct.
2. If declaration disabled and isolatedModules enabled.
   2.1 Transpile changed files: transpiled source, get source map. Uses transpileModule, because it's much faster.
   2.2. Extract dependency graph, pass it to webpack.
   2.3 Update program and check that full program is correct.
3. awesome-typescript-loader also can do type-checking it a separate process, when all module resolution data and source files are send via IPC to the process which runs type checking and gets program diagnostics.

[s-panferov]

To simplify requirements:

1. We need to be able to create a Program reusing as much as possible.
2. We need the way to get file's dependencies (especially elided ones).
3. The main scenario of work in webpack is isolatedModules, so we can take benefits from it. But sometimes we need to generate declarations.
4. We need to be able to output all the errors after a watch cycle. Here we already know our new files state.

@TheLarkInn wants some kind of plugins, that are able to modify source code using semantic information (if I'm correct). So maybe we need a fast way to update Program's semantic information from the new file independently of diagnostics output (to get the new semantic state available for the plugin during a transpile phase)

[TheLarkInn]

@s-panferov Thank you so much for the much more accurate translation! Your support in this has been invaluable.

[mhegazy]

The dependency graph is not something that is provide today by the LS or the Compiler API. so we can use this issue to track adding it.

[mhegazy]

one thing to note, as isolatedModules and transpileModule has been mentioned here.

There are two operations, 1. transpilation, and 2. typechecking

transpilation can be done most of the time on a single file. you can use transpileModule to do that. but there are no semantic errors generated here, and no grantees about the correctness of the program.

typechecking is a project wide operation. can not be done on a single file. but it does generate errors.

obviously, transpiling a single file is significantly faster than typechecking the whole project. but the two are not the same.

so if you do not care about errors, e.g. you report them from a different channel e.g. IDE, full build step, then use transpileModule. if you do care about the errors, transpileModule is not the correct API.

[erikbarke]

@mhegazy: your explanation of LanguageService vs CompilerHost, caching etc should be in the Compiler Api docs :)

[TheLarkInn]

@DanielRosenwasser wanted to bump this issue for relevance. This is still a definite need for not only webpack loaders, but also angular and potentially broccoli-typescript plugin.

[erikbarke]

@TheLarkInn, chiming in here - this is relevant to karma-typescript too since the Karma test runner serves its files to its preprocessors asynchronously one at a time. My current implementation collects the files sent to the preprocessor using lodash.debounce, and then performs a deferred compilation of the whole project after a small delay.

The upside of this approach is that I can use the CompilerHost and keep state in a cached Program between compilations when running Karma in watch mode, making updates incremental.

The downside is that it's such a clunky workaround imo: it adds an unwanted delay to the compilation, there's always the risk of race conditions introducing subtle bugs etc. So this would be a more than welcome addition to the Typescript API.

[mhegazy]

With #20234, i think the underlying issues in this thread should all be addressed. #20234 introduces two new concepts, a Builder and a Watcher.
A Builder can take an old program and a new one (after an edit), and answer questions like getDiagnostics and emit efficiently. it does that by caching the state from the old program and only recomputes things that needs to be computed, based on the dependency graph of the project.
A Watcher, can watch either a tsconfig.json or a set of input files. it caches most of the module resolution/file IO operations needed, and triggers the creation of a new program on edit.
combined together you can get the same output tsc --watch and compiler-on-save in VS achieves today.
We are working on docs at the moment, but here is the PR for the docs: microsoft/TypeScript-wiki#169. you can also find an example of the new API being used in TypeStrong/ts-loader#685