Reduce typescript package size #27891
Comments
|
Some backstory in #23339. |
DanielRosenwasser
added
Suggestion
|
Interesting reading, thanks.
It was "fixed" by #25901, released in 3.1.1, which was 40MB. 🙁 It won't be hard at all to shrink the package size. For example, And 99% of And And 99% of And 80% of That's nearly 30MB of duplication just in those few files (and this doesn't even include declaration files). I can't begin to guess at the kinds of design decisions that produce this (or what kind of compatibilities the TS team needs to support), but I trust there is a solution the maintainers would be happy with. |
It's done this way so that every file can be used by itself without having to deal with the nastiness of modules in JavaScript. Every file is a functional library/program in itself. I think that is a great thing, at the cost of some disk space. |
|
Reading the linked issue #23339, it appears that it desire is in fact to (eventually) use modules.
ES module systems in general can be hit-and-miss, but reminder that we're talking specifically about an npm package. npm, npm packages, node_modules, package.json, etc. are relate to Node.js (or clones) which supports CommonJS. Right? |
|
I have two ideas, but I am not sure which one is better.
|
|
ping @DanielRosenwasser |
|
I am skeptical that tree-shaking is useful for shipping our own package because presumably everything we ship is used in some capacity, or is part of our public API - at which point, our consumers would actually be the ones winning from tree-shaking. Splitting source on its own can help, but practically speaking the larger components like services and TSServer will need the entire core compiler. I think that converting to modules is the most practical and obvious way to avoid duplicating most of the contents of |
|
A simpler solution: inspired from Busybox. Combine N near-duplicate files into 1 polymorphic file that can do N things based on a parameter passed in. It would introduce a performance overhead of parsing tiny % of unnecessary JS code, but can make the tool integration story way simpler. Maybe worth it? One trivial way to know which feature is expected would be to directly copy Busybox approach: symlink all the duplicate files and differentiate at runtime based on the |
|
From speaking with @RyanCavanaugh, it sounded like @orta was interested in working on this. |
|
+1 for splitting up typescript into multiple packages. One major benefit would be that these individual packages (other than the "typescript" package) could use semantic versioning on at least their APIs then other libraries could just depend on the packages they need. Right now it's kind of a pain to maintain a library that has a peer dependency on the typescript package (without being super strict about the supported version). |
|
Yeah, I'm chatting with folks internally this week, but my goal is roughly:
Then have subset packages which are smaller and focused on a specific task:
I doubt I can offer any useful semver on them, as they link to the main TS version. That'd need the API to actually be classed as "stable" which doesn't look like that's happening soon. Figuring out how/if we can reduce the main |
|
Removing tools from the package doesn't reduce overall size. Compilation, dev tools etc reuse a lot of the same code that is now copied to multiple commands without changes. The issue is how to share the very duplicated part between the tools, reduce the duplication, or pack the tools into one bundle. |
Oh, we're generally for it (and have been for years, provided we still provide a services bundle for our (browser) consumers who use it) - we just need an automated way to remap the current namespace-based code layout into modules, this way we can keep a PR doing the migration up to date and not stop development on other things. I have a branch from two years ago that migrated all of With respect to said automation, I think we could probably write a kind of codemod for it using the APIs we have today, but nobody's put in the effort yet. |
|
@orta VS Code is very interested in this work. Right now we consume TypeScript in two ways:
Each of those files is around 8MB on disk. Additionally, are interested in shipping built-in support for tsc ( Let me know if you would like any additional info about how VS Code consumes TS As a side note, |
|
I brought this up during the most recent design meeting - #34899 Where the end result was basically, we're meeting about trying to get modules happening again As mentioned above - all of these files are basically the same but with a bit of flavor difference because they represent different sets of the compiler + services - for example I think you can probably use |
|
I am filing followup issues now that the modules PR has been merged. One such issue of interest here is #51440; the TL;DR is that if we raise our minimum supported Node version to Node 12, we could safely ship our executables as ESM, which would save us roughly 7 MB more on top of the 43% reduction above. |
|
The reduction from modules is very significant. (Thanks!!!!!!) If your math is correct, that reduces the package size from 65MB to 36MB. Which is still larger than it was when #23339 was filed, asking for it to be smaller. But alas, such is progress. This was the largest possible improvement to the size. More could be done, but it's not gonna cut in half again. |
|
Eventually, we may be able to ship as ESM and achieve the smallest possible package. Or, go further and publish individual packages for parts of our repo. That goal's a long way off, but there is work left o be done here. |
|
Confirmed, TS nightly is much smaller now, thanks!
|
|
Following the migration to modules in
|
|
I mentioned minification in the module conversion PR; we are restricted on that front because so many people still patch our package. If we minify, patching becomes difficult to impossible. I'd love to be able to do so, but we have to figure out what to do about that first. (We'd also probably not go "full" minify; we need to keep names for backtraces.) |
|
Minify only saves space if you don't include source maps. And excluding source maps seems like deal-breaker. |
|
We already exclude source maps in the package, but our output is left "pretty" so that stack traces are meaningful when provided by downstream users. If we were enabling minification, we would likely only have it remove whitespace and optimize syntax, leaving names in the output. |
|
Re: ES Modules, I think we have to take performance as a serious goal. We get a big speed boost from esbuild's whole-program-aware bundling and giving that up for a better sticker number isn't a good trade-off for most users. People who want to vendor TS and get the smallest possible final output should pick up our mid-build artifacts and tree shake them. |
|
Yeah, this is something I want to performance test; my impression is that ESM imports should be as fast as the whole-program bundling. I think that the differences were really down to variance + load time. |
It's worth noting that vendoring has some big tradeoffs which might leave a user worse off. If someone still installs TypeScript (due to another dependency, for custom build tasks, or for having their editor use a workspace version), that person gets even more duplication of TypeScript, possibly with mismatched versions. |
|
This is closed, but since people do still follow this issue, #55273 is on the docket for an early 5.3 merge; this PR effectively replaces
As for our executables (and potentially an ESM API); that'll be handled by #51440 when I get to dealing with the long set of changes that are required to make that happen. |
|
Hi! First of all, thanks @jakebailey and the rest of the typescript team for constantly working on this matter to reduce the typescript install size 💙 With the awareness of all these efforts, I made an experimental project
I hope this project will be helpful rather than something conflicting with the future roadmap of install size optimizations from the core package. |
|
There is still more size work that can be done, specifically #51440. However, I will note that the problem of package sizes is really not as bad as people think these days; every modern package manager uses hardlinks to a global cache, meaning that every install of TypeScript on a system will share the same backing files on disk. The "apparent" size may seem duplicative, but it's really all shared. That and the install size seen on packagephobia is the unpacked size; the actual bits transferred from the registry are much, much smaller. Even gzip brings the tarball to about 6MB. tslite is smaller on that front at about 3MB, but overall most people only download each version of TypeScript once. That combined with the hardlinking really means that we're talking about a few MB per system, paid once. One spends more network and disk space loading up Twitter or even GitHub via images and scripts that change often than the TS package. I'm still going to try and make it smaller because I find it fun to do so, but it's a little moot IMO. |
|
This matters when opening a repo on an online IDE where there is no cache. My home connection is ~2MB/s, so even in tarball TS still adds few seconds when I open a Stackblitz repro for Vite. |
Neither npm nor yarn use a global cache. (Unless Yarn is PnP mode, which brings a number of issues.)
There are over 2,800 versions of TypeScript. The chance that two different projects happen to install the same exact version is very low. Even for a single npm install which dedups as much as possible, right now I'm looking at a project with 5 TypeScript versions. (Why? jsii, postcss-loader, prettier-plugin-organize-imports, puppeteer-core, cosmiconfig-typescript-loader, plus the version for the project itself.) |
That's certainly true. It's a shame that these systems do not cache their artifacts.
Yarn 3 supports hard linking (https://yarnpkg.com/configuration/yarnrc#nmMode). If you're still using Yarn v1, you're not going to get any new features at all. I was wrong about npm; it has a global cache but it copies the files.
There should really only be one TS version in a project; if this is happening, then some package is over-restricting what version of TS it needs. All modern package managers allow you to override versions within a workspace, and I would think it'd be safe to do that if space is a concern and your package manager can't hardlink. It's also misleading to say that there are 2,800 versions of TypeScript; there are only a handful of stable releases. The rest are nightly builds. |
|
People shouldn’t have to override Typescript versions. The project I’m working on now has 70 dependencies and if they all required post-install customization npm would be pretty unusable. |
I'm referring specifically to doing this in "overrides": {
"typescript@*": "$typescript"
},Or in "resolutions": {
"typescript@*": "$typescript"
},Or in "pnpm": {
"overrides": {
"typescript@*": "$typescript"
},
}I am not referring to any sort of post-install patching, but just asking the package manager to resolve to a single version. |
|
The point is that an override only seems reasonable because other dependencies don’t require any extra setup. NPM repos are supposed to be low-effort installs and typescript should be no exception. |
Yes, as you say, IDEs, package maintainers, and package managers should be aggressively deduplicating redundancies. .... .... .... .... And TypeScript should be doing the same. (Right now it's something crazy like ~75% duplicate code.) |
|
Yes, again, #27891 (comment) removed one more copy, and #51440 will remove even more (down to the absolute minimum of 2 copies one can have when shipping both CJS and ESM). I'm not sure what else I can say, I was just originally attempting to explain that a large bulk of situations do not benefit from the effort to lower the package size. |
|
I think it'd be useful for people to be a bit more specific about what they care about so we can tailor our efforts. For time-over-wire, deduplication isn't a great savings, since each additional copy is a tiny increment (compressed checker.ts (2 MB) is 416k, compressed 4x checker.ts is 418k) For space-on-disk, uh, I'm going to need some more details. It's not 1998 anymore. 40 MB is 0.04% of a terabyte. If the problem is that there are 35 copies of TS due to how a package manager behaves, going from 35 to even 10 is going to be a much bigger than anything we could plausibly do. That's a package manager problem, not a TypeScript problem, it's unrealistic to expect projects to put their effort into slimming down instead of having package managers duplicate less. For bundling into other projects like web IDEs, treeshaking is going to be a big part of any successful strategy here. Identifying places where we can be more shakeable is a good thing. |
I see where you're coming from, but npm is still the most commonly used package manager. The problem is npm, definitely, but who knows when this will change? Anything that reduces TS size definitely has impact on the ecosystem. Same is true for the "time-over-wire" thing, shipping 1 MB less is probably not doing anything for one person, but if you multiply 1000 kb by 43 million downloads weekly, the picture looks totally different again. As far as I can tell, the details you need are written down here: https://github.com/pi0/tslite#how |
|
There are plenty of cases where cache isn't available and the size of the package matters – for over the network size, number of files that need to be written, and amount of JS that needs to be parsed at execution time.
|
35 to 10 is a 71% reduction. The latest version (5.2.2) has tsc, tsserver, tsserverlibrary, and typescript which total 32MB but have only 9MB of unique content. Removing the duplicate code drops the package from 41MB to 18MB, a 56% reduction. So.....actually, there is a lot that TS can do.
The problem is largely a synthetic one introduced by TS's bundling. The source code (excluding tests) is only 32MB. |
Tangential, but if you want to go that route @RyanCavanaugh , there's a four-year PR open for PnP to dedup installs, maybe it could be get some eyeballs :) |
and others
Search Terms
size, bloat, install
Suggestion
The typescript package is large, and it only getting larger.
Version 3.1.3 is a whopping 40MB.
Use Cases
TypeScript is used in many contexts.
A TypeScript formatter (e.g. prettier) does not need an entire compiler. It only needs a parser. And 45MB scripted parser is orders of magnitude larger than one would normally expect. (For reference, the installed npm package for Esprima -- the most compatible and compliant ES parser in the ecosystem -- is a mere 0.3MB.)
Examples
Solution 1: Split up packages
Optionally, there could be separate packages for typescript-config and typescript-i18n.
Solution 2: Don't duplicate code
There is a lot of code duplication between
Don't duplicate the code.
Checklist
My suggestion meets these guidelines:
The text was updated successfully, but these errors were encountered: