Pain points while migrating over a large project

[lewisl9029]

Hi, I've been working on migrating a large webpack project over to snowpack for the past week, and feel like I'm almost at the end of the tunnel, but wanted to document some of the issues I've encountered along the way. These are in rough chronological order:

1. OOM errors with large projects with lots of dependencies

I've posted about this here since there was an existing thread for it: #857 (reply in thread)

2. Poor handling of source folders containing multiple projects

When a source folder contains multiple apps with different entry points (in our case there's a backend and frontend app sharing the same folder), snowpack would try to scan the entire source folder and "install" all dependencies referenced in all source files. This is not ideal because usually we're trying to use snowpack to build a single app at a time, but all of the other apps' dependencies end up having to be built as well (especially problematic if a backend app is involved as in our case).

Also, dependencies for things like tests and storybook also end up get included by default as well, also significantly contributing to build times.

I tried to use the exclude option in snowpack.config.js to remove the backend project's source files and test files from the build, but it didn't seem to work. Dependencies in backend files that matched the exclude glob still ended up getting built. This may be a bug of some kind, happy to set up a repro if it's not already a known issue.

In the end I ended up just temporarily removing all the backend project files to unblock myself.

Longer term I feel it would be helpful to consider actually analyzing the dependency graph starting at the index.html entry point to figure out what dependencies to install, rather than naively scanning every single file in some folder for dependencies. This would greatly improve the migration experience coming from something like webpack that is already entrypoint based and doesn't touch any files that are not referenced in the dependency graph starting from the entry points.

3. Extremely long iteration cycles when running with --reload

Migrating an existing project currently involves a lot of whack-a-mole when it comes to configuring and reconfiguring installOptions, which requires rerunning snowpack dev with --reload. This seems to involve rebuilding every single dependency in the project, and currently takes about 3 minutes for the project I'm working on (I believe this is at least somewhat exacerbated by issue 2) above).

This feels extremely inefficient, because a vast majority of my dependencies never change between builds (when I'm only tweaking installOptions to add to the namedExports list), yet they all end up getting rebuilt regardless. Would be great if we had a way to install specific dependencies that we know to have been added/changed, or better yet, snowpack should try to figure out what needs to be rebuilt and what doesn't, at least in the case of simple config changes like adding to namedExports.

4. namedExports whack-a-mole

Related to 3) above, currently errors like Uncaught SyntaxError: The requested module ‘/web_modules/XXXXXX.js’ does not provide an export named ‘YYYYYY’ only show up after the page loads, and they show up 1 at a time. The solution is to add them to namedExports, but for a large project that takes 3 minutes to change namedExports every time, the process becomes extremely frustrating. I was actually about to just throw in the towel here because I couldn't see myself being able to go through this for dozens of dependencies, but thankfully I found out there were only 4 dependencies that needed this for my project. (Never mind, looks like it was hidden behind another error, back to whack-a-mole)

Addressing 3) can definitely alleviate this somewhat, but is there anything we can do to better detect which modules might cause these errors and do the namedExports transform on behalf of the user?

5. Importing file contents as string

I couldn't find an existing supported way to import raw file contents as strings using snowpack (we're using https://webpack.js.org/loaders/raw-loader/ for this in webpack). Our use cases for this involves importing raw css files as strings and using emotion's Global component to inject them into the page. This is a lot more flexible than importing css files directly as you can apply and unapply the global styles dynamically by mounting and unmounting the component in response to state changes.

I imagine we could probably solve this by writing a custom plugin, but for my use case it was easier to just copy the file contents and inline them into the js for now. I'm probably not the only one with this use case though, so an official solution would likely be helpful for many. Found at least 1 other use case here: #721

6. Dealing with file extension ambiguity

We've become accustomed to excluding file extensions using webpack, and for the most part things still work as expected for snowpack, but I did find one interesting edge case:

When we have files using the following folder structure:

/hooks/useSomething.ts
/hooks.ts

import * as hooks from './hooks' works fine in webpack, but the same import statement in snowpack attempts to load /hooks/index.js and ignores the /hooks.ts file. There are likely some heuristics in the loader that need to be adjusted to account for this edge case.

7. Graceful handling of type imports that don't exist at runtime

This feels like a more challenging problem than the others, and the answer very likely could be for us to bite the bullet and fix our code across the board, but I wanted to mention it and see what input folks here might have regardless.

In our Typescript codebase, we have a bunch of imports that look like this:

// BlahType is a typescript type that doesn't exist at runtime
import { BlahType } from './Blah'

In webpack, this seems to work just fine. In snowpack however, it will complain about the fact that BlahType doesn't exist in ./Blah.js since it's a type declaration that doesn't exist at runtime. Obvious solution on our side is to just change the import to this:

import type { BlahType } from './Blah'

Which will cause the type import to be removed by babel altogether. This kind of pattern is everywhere in our codebase though, and it has been a huge pain so far to actually migrate them, so I would really appreciate it if snowpack could do something to make our lives easier here.

Typescript itself seems somewhat hostile towards tools that want to transform each module in isolation (or really any tool that doesn't use tsc for doing the transformation for that matter), it'd be nice for instance if the importsNotUsedAsValues=error option can categorically rule out this class of issues (or better yet be able to auto-fix them across our codebase), but from my experience so far it's definitely not exhaustive (and doesn't even try to be microsoft/TypeScript#39432 (comment)).

I'm not completely finished with the migration yet, so I'm sure I'll have more things to report later. Will add those in follow up comments as I come across them.

[drwpow]

This is SO amazing! Thank you! 🙏🏻 And so well organized.

I’ve given this a quick skim, but will be rereading this more carefully and may ask some questions for each section. Really, really appreciate the time you’ve taken to outline your experience. This is valuable, and we really want to make the experience as smooth as possible.

[melissamcewen]

Thanks for this! I've definitely encountered some of these issues myself, especially #3 and #4. Hopefully we can improve both Snowpack and the docs to help with some of these.

[netaisllc]

We are facing doing the same, so this is quite timely. Thanks!
Is there any way you can characterize large as it is so easy to equivocate on that aspect of a s/w project. :)

[lewisl9029]

I'd love to be able to quantify this better, but it's a bit difficult to get a good LOC count on it since it's part of a larger monorepo and the frontend code is mixed with backend.

If I had to give a ballpark estimate I'd say it's on the order of 500k - 5m lines of source code, and as mentioned in #857 (reply in thread) the node_modules directory is slightly over 2GB on disk. The full production bundle with source + deps totals to about 10MB uncompressed.

Hopefully this gives a better sense of scale for comparisons.

[rodoabad]

I would typically say a "large" project is around 500K file entries.

Here's an example from a Webpack progress plugin that's been running for more than an hour now.

[lewisl9029]

Finally managed to get the project itself working, but had to stub out a few problematic dependencies in the process. Wanted to get some thoughts from people here on what it might take to support these:

1. react-pdf

For react-pdf, the problem seems to be the fact that it requires a web worker to be loaded, and it seems to have special entry points for different popular bundlers for compatibility with how they expect to work with web workers:

https://github.com/wojtekmaj/react-pdf/blob/master/src/entry.webpack.js
https://github.com/wojtekmaj/react-pdf/blob/master/src/entry.parcel.js

The parcel one looks like it should work with snowpack as long as we can get the react-pdf/dist/esm/pdf.worker.entry.js file served at /pdf.worker.entry.js somehow, but I haven't been able to figure out how we can leverage the alias system to do that yet. An entry like this doesn't seem to have any effect:

  alias: {
    public: './public',
    './pdf.worker.entry.js': './public/test',
  },

Which makes sense since new Worker('./pdf.worker.entry.js') doesn't seems like something snowpack is designed to rewrite today.

Thinking about longer term, does Snowpack have a story for how web workers should be handled, especially those included in dependencies?

2. contentful

Or more specifically contentful-sdk-core. When trying to install it, I get the following error from snowpack:

[snowpack] installing dependencies...
[snowpack] Failed to load node_modules/contentful-sdk-core/dist/index.es-modules.js
  'platform' is not exported by node-resolve:empty.js, imported by node_modules\contentful-sdk-core\dist\index.es-modules.js
[snowpack] Install failed.

This is because contentful-sdk-core attempts to import the os module: https://github.com/contentful/contentful-sdk-core/blob/02d64ef28e8d396dfa019cef22f0c88b4cd6a80a/lib/get-user-agent.js#L1, which doesn't make sense in a browser context. The ideal solution would likely be to send a PR to them changing it to a dynamic import, but I would expect it to be polyfilled in by snowpack, perhaps with shim functions that throw when called, given that I'm running with the polyfill-node flag. It would have no effect on the runtime behavior of the library since I believe that code path is never exercised in the browser.

Curious if folks are familiar with how webpack might handle cases like this?

[FredKSchott]

for os, you can use --polyfill-node to automatically polyfill this. You actually should have gotten a hint about that directly in the console, but it looks like the actual error is for "platform" instead of "os" for some reason, which might explain that.

For Workers, we expect them to work directly in the browser. We've considered adding automatic bundling in the past, but both Chrome & Edge now support unbundled ESM in web workers, so that new Worker('./foo.js') would just work and Webpack 5 will bundle it for production.

For Workers in dependencies, I'm actually not sure how we handle them. esinstall (our internal package installer) uses Rollup under the hood, so I'd assume whatever react-pdf does for Rollup support would work for us as well.

[lewisl9029]

for os, you can use --polyfill-node to automatically polyfill this. You actually should have gotten a hint about that directly in the console, but it looks like the actual error is for "platform" instead of "os" for some reason, which might explain that.

Unfortunately I was already using --polyfill-node. For some reason it's not working for this library. I was able to repro with a fresh create-snowpack-app project:

1. npx create-snowpack-app test --template @snowpack/app-template-react
2. add import "contentful" to top of index.jsx
3. npm start -- --polyfill-node

[FredKSchott]

Oh strange, the package says has this in its package.json:

  "browser": {
    "os": false
  },

Which I think is why "os" is treated as "empty.js".

If they made this change:

- import { platform, release } from 'os';
+ import os from 'os';

And then referenced things off of the os object, it wouldn't break if os: false. I'd recommend reaching out to that repo with a PR. This problem affects all Rollup projects, and not just Snowpack