Unable to compile pyodide using webpack and typescript

[szymswiat]

🐛 Bug

When I'm trying to use pyodide in typescript project I'm getting bunch of errors during compilation:

• error-stack-parser does not export StackFrame from ErrorStackParser in version 2.0.7 which is currently installed with pyodide. It has been fixed in 2.1.0 so I'm assuming that updating version will fix this issue.

ERROR in ../../node_modules/pyodide/error_handling.gen.ts 193:49-59
TS2694: Namespace 'ErrorStackParser' has no exported member 'StackFrame'.
    191 | Tests.convertCppException = convertCppException;
    192 |
    193 | function isPyodideFrame(frame: ErrorStackParser.StackFrame): boolean {
        |                                                 ^^^^^^^^^^
    194 |   const fileName = frame.fileName || "";
    195 |   if (fileName.includes("pyodide.asm")) {
    196 |     return true;

• if it comes to three below errors I'm not sure what is the reason

ERROR in ../../node_modules/pyodide/api.ts 13:21
Module parse failed: Unexpected token (13:21)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
|  * from JavaScript.
|  */
> export let pyodide_py: PyProxy; // actually defined in loadPyodide (see pyodide.js)
| 
| /**

ERROR in ../../node_modules/pyodide/load-package.ts 6:11
Module parse failed: Unexpected token (6:11)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| 
| /** @private */
> let baseURL: string;
| /**
|  * Initialize the packages index. This is called as early as possible in

ERROR in ../../node_modules/pyodide/pyodide.ts 29:7
Module parse failed: Unexpected token (29:7)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| } from "./pyproxy.gen";
| 
> export type Py2JsResult = any;
| 
| let runPythonInternal_dict: PyProxy; // Initialized in finalizeBootstrap

To Reproduce

Please use following repository.
Errors related to ErrorStackParser are not showing using this project (I installed newer version) however is still should be fixed.

Expected behavior

Pyodide package is compiling.

Environment

• Pyodide Version: 0.20.0
• Browser version: N/A
• Any other relevant information:

Additional context

[hoodmane]

error-stack-parser does not export StackFrame from ErrorStackParser in version 2.0.7 which is currently installed with pyodide. It has been fixed in 2.1.0 so I'm assuming that updating version will fix this issue.

Could you open a PR for this? Currently we seem to have 2.0.6 in the lockfile:
https://github.com/pyodide/pyodide/blob/main/src/js/package-lock.json#L2367
but I guess the ^2.0.6 requirement will install 2.0.7 with npm i and 2.0.6 with npm ci which may explain why we don't see the error in the main branch.

if it comes to three below errors I'm not sure what is the reason

Could you try with Pyodide v0.21.0a2 and see if you get different errors? There has been some work by @jasongrout and other folks on the js package / webpack bundling support so there may be some improvement or different behavior now.

[hoodmane]

Seems like the new version still doesn't work but there's a different error.

[szymswiat]

Yep, there are three different errors coming from src/js/compat.ts related to non-static imports:

WARNING in ../../node_modules/pyodide/pyodide.mjs 1:8171-8182
Critical dependency: the request of a dependency is an expression
 @ ../annotator/src/pyprocessing/index.ts 7:0-38 16:20-31
 @ ../annotator/src/components/AnnotatorApp/index.tsx 13:0-49 21:4-15
 @ ../annotator/index.ts 3:0-58 4:15-27
 @ ./src/routes/router.tsx 36:64-91
 @ ./src/routes/index.js 3:0-39 3:0-39
 @ ./src/app/LumradApp.tsx 11:0-31 17:26-32
 @ ./src/app/index.js 3:0-42 3:0-42
 @ ./src/index.tsx 6:0-30 40:90-99

WARNING in ../../node_modules/pyodide/pyodide.mjs 1:8325-8336
Critical dependency: the request of a dependency is an expression
 @ ../annotator/src/pyprocessing/index.ts 7:0-38 16:20-31
 @ ../annotator/src/components/AnnotatorApp/index.tsx 13:0-49 21:4-15
 @ ../annotator/index.ts 3:0-58 4:15-27
 @ ./src/routes/router.tsx 36:64-91
 @ ./src/routes/index.js 3:0-39 3:0-39
 @ ./src/app/LumradApp.tsx 11:0-31 17:26-32
 @ ./src/app/index.js 3:0-42 3:0-42
 @ ./src/index.tsx 6:0-30 40:90-99

WARNING in ../../node_modules/pyodide/pyodide.mjs 1:8593-8625
Critical dependency: the request of a dependency is an expression
 @ ../annotator/src/pyprocessing/index.ts 7:0-38 16:20-31
 @ ../annotator/src/components/AnnotatorApp/index.tsx 13:0-49 21:4-15
 @ ../annotator/index.ts 3:0-58 4:15-27
 @ ./src/routes/router.tsx 36:64-91
 @ ./src/routes/index.js 3:0-39 3:0-39
 @ ./src/app/LumradApp.tsx 11:0-31 17:26-32
 @ ./src/app/index.js 3:0-42 3:0-42
 @ ./src/index.tsx 6:0-30 40:90-99

[hoodmane]

I've periodically looked into this but I have no idea how to fix it...

[hoodmane]

It's been an ongoing issue for us that we don't have any developers with much JavaScript/webpack expertise.

[himself65]

I expected a same error and I would help on this

[himself65]

The problem is import syntax in webpack

https://webpack.js.org/api/module-methods/#dynamic-expressions-in-import

The import() must contain at least some information about where the module is located. 

[alexmojaki]

Is this still a problem in version 0.21.*? I believe it was fixed.

[alexmojaki]

Same for these related issues:

• Unable to import pyodide in react native project: Invalid call at line 15: import(url) #2549
• Node package improvements #2403
• npm package should not include .ts files #2393

[hoodmane]

Thanks @alexmojaki. I don't think this issue was fixed. #2393 is fixed. Not sure about the others.

[alexmojaki]

A few days after the last comment above by @himself65, version 0.21.0 was released. I was able to use the new npm module in pyodide-worker-runner, which uses webpack and typescript, having previously failed with 0.20.x. And pyodide-worker-runner was then used as an npm module in futurecoder, which uses create-react-app, i.e. webpack, so it worked in two 'layers' of webpack. Same with 0.21.1.

The one caveat is that initially I got an error like this:

ERROR in ../node_modules/pyodide/pyodide.mjs 1:11502-11515
Module not found: Error: Can't resolve 'url' in '/home/alex/work/pyodide-worker-runner/node_modules/pyodide'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "url": require.resolve("url/") }'
        - install 'url'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "url": false }
resolve 'url' in '/home/alex/work/pyodide-worker-runner/node_modules/pyodide'
  Parsed request is a module
  using description file: /home/alex/work/pyodide-worker-runner/node_modules/pyodide/package.json (relative path: .)
    resolve as module
      looking for modules in /home/alex/work/pyodide-worker-runner/node_modules/pyodide/node_modules
        single file module
          using description file: /home/alex/work/pyodide-worker-runner/node_modules/pyodide/package.json (relative path: ./node_modules/url)
            no extension
              /home/alex/work/pyodide-worker-runner/node_modules/pyodide/node_modules/url doesn't exist
            .ts
...

Based on that, I added this to my webpack.config.js when I upgraded to 0.21.0:

module.exports = {
  resolve: {
    fallback: {url: false},

[hoodmane]

Huh I don't know which commit would have fixed this but that's good news. Maybe we can reenable the webpack test