fix(deps): update undici to 4.x #8842

janpio · 2021-08-20T15:56:21Z

Fixes #6564
Fixes #6899
Fixes #6925

Co-authored-by: Joël Galeran <Jolg42@users.noreply.github.com>

janpio · 2021-08-20T16:23:36Z

Available for manual testing as 2.30.0-integration-undici-4.2

packages/engine-core/package.json

packages/cli/helpers/build.js

packages/client/helpers/build.js

Jolg42 · 2021-08-23T07:47:59Z

Note: we founds that undici was using Blob https://github.com/nodejs/undici/search?q=require%28%27buffer%27%29&type=code which was only added with Node v15.7.0 https://nodejs.org/docs/latest-v16.x/api/buffer.html#buffer_class_blob (Node PR nodejs/node#36811)

We need to make sure this doesn't impact our supported versions of Node (12, 14)

packages/engine-core/src/binary/Connection.ts

pnpm-lock.yaml

packages/cli/helpers/build.js

packages/migrate/src/__tests__/DbCommand.test.ts

packages/migrate/src/__tests__/MigrateDeploy.test.ts

pnpm-lock.yaml

packages/client/helpers/build.ts

helpers/compile/plugins/replaceWithPlugin.ts

Jolg42

Thank you for pushing this through the door, it might very well be useful for other wasm projects 💚

Jolg42 · 2022-01-12T08:07:30Z

This PR is current blocked because the following action job times out CI / client (library, ubuntu-latest, 16) (pull_request) Cancelled after 25m

Some investigation might be needed there first, what I found earlier was that something gets really slow during tests.
Example:
https://github.com/prisma/prisma/runs/4608802724?check_suite_focus=true#step:9:1493

 PASS src/__tests__/integration/happy/binary/test.ts (34.847 s)
  ✓ binary (34030 ms)

vs on main branch

https://github.com/prisma/prisma/runs/4605713109?check_suite_focus=true#step:9:1047
PASS src/__tests__/integration/happy/binary/test.ts (5.121 s)
  ✓ binary (4215 ms)

@millsp saw the heap size getting big too, so that might be linked.

juanzgc · 2022-03-13T15:34:04Z

Is there any news on the state of this PR and whether it is still being blocked 😇 ?

I believe this PR would fix the following Issue: #6899
Which is currently preventing me from using Prisma in Turborepo as a package

vincent-thomas · 2022-03-17T15:02:23Z

I have an issue with _http_common and I think this pr is going to fix that problem! Error here:

ERROR in ./apps/thunderbolt/prisma/client/runtime/index.js 27999:21-44
Module not found: Error: Can't resolve '_http_common' in '[path]\Codebase\apps\[app_name]\prisma\client\runtime'

…ci-4

Jolg42 · 2022-03-23T14:06:09Z

helpers/compile/plugins/replaceWithPlugin.ts

+/**
+ * Example
+ */
+// const inlineUndiciWasm = replaceWithPlugin([
+//   [
+//     /(await WebAssembly\.compile\().*?'(.*?)'\)\)\)/g,
+//     async (regex, contents) => {
+//       for (const match of contents.matchAll(regex)) {
+//         if (match[2].includes('simd') === false) {
+//           // we only bundle lhttp wasm files that are not simd compiled
+//           const engineCoreDir = resolve.sync('@prisma/engine-core')
+//           const undiciPackage = resolve.sync('undici/package.json', { basedir: engineCoreDir })
+//           const lhttpWasmPath = path.join(path.dirname(undiciPackage), 'lib', match[2])
+//           const wasmContents = (await fs.promises.readFile(lhttpWasmPath)).toString('base64')
+//           const inlineWasm = `${match[1]}(Buffer.from("${wasmContents}", "base64")))`
+
+//           contents = contents.replace(match[0], inlineWasm)
+//         }
+//       }
+
+//       return contents
+//     },
+//   ],
+// ])


What is this example for @millsp ?

It used to show how to inline wasm when the wasm file is not directly imported. However, in the mean time undici changed how they load the wasm and now puts the wasm contents in a .js file instead of doing a readFileSync so this is not needed anymore and works well when bundled.

Should we remove this then in a follow up PR?

We can probably trim it a bit, but I'd like to keep an example for how to use replaceWithPlugin.

Jolg42 · 2022-03-23T14:08:16Z

packages/engine-core/src/binary/Connection.ts

+// because undici lazily loads llhttp wasm which bloats the memory
+// TODO: hopefully replace with `import` but that causes segfaults
+const undici = () => require('undici')


So that's our current workaround for the segfaults for the Binary? any idea why is this happening?

I initially wanted to do () => import('undici') and I could not understand why this could cause segfaults in jest and only on CI. So I swapped for () => require('undici').

Worth opening a PR again to see if it still happens, and if so maybe point it out to Undici/Node ppl?

I tried this today. I made undici not to load lazily like it does by re-wrapping it in () => . Somehow, their lazy loading of the wasm (ie. on module load) was also causing memory leaks in jest. It would make sense to debug where it segfaults when doing import instead of require so that we can report to them.

michalkvasnicak and others added 12 commits June 29, 2021 17:09

fix(deps): update unidici

fea217c

build: copy undici wasm files to dist

e79239e

build: copy llhttp

4cc8333

fix: mark _http_common external

b92f97d

chore: remove fs-extra

206c0ab

Merge remote-tracking branch 'origin/main' into pr/michalkvasnicak/7931

3644418

Delete undici.ts

c6b0286

Delete build.js

6ddf052

add runtime/llhttp to package.json

487d6a8

copy wasm to generator-build as well

22b805c

remove unused import

3a2df2c

fix(deps): update undici to 4.4.6 (#8831)

6079f14

Co-authored-by: Joël Galeran <Jolg42@users.noreply.github.com>

janpio requested review from Jolg42, millsp and williamluke4 as code owners August 20, 2021 15:56

janpio mentioned this pull request Aug 20, 2021

fix(deps): update undici #8823

Closed

janpio changed the title ~~fix(deps): update unidici~~ fix(deps): update unidici to 4.x Aug 20, 2021

janpio changed the title ~~fix(deps): update unidici to 4.x~~ fix(deps): update undici to 4.x Aug 20, 2021

Merge branch 'main' into integration/undici-4

50b79d6

millsp reviewed Aug 20, 2021

View reviewed changes

packages/engine-core/package.json Outdated Show resolved Hide resolved

millsp reviewed Aug 20, 2021

View reviewed changes

packages/cli/helpers/build.js Outdated Show resolved Hide resolved

millsp reviewed Aug 20, 2021

View reviewed changes

packages/client/helpers/build.js Outdated Show resolved Hide resolved

millsp mentioned this pull request Aug 20, 2021

fix(deps): update undici to 4.0.0 #7931

Closed

Jolg42 mentioned this pull request Aug 23, 2021

Prisma Client (Binary): upgrade to undici@4 #7539

Closed

Jolg42 linked an issue Aug 23, 2021 that may be closed by this pull request

Prisma Client (Binary): upgrade to undici@4 #7539

Closed

janpio commented Aug 23, 2021

View reviewed changes

packages/engine-core/src/binary/Connection.ts Outdated Show resolved Hide resolved

janpio commented Aug 23, 2021

View reviewed changes

pnpm-lock.yaml Show resolved Hide resolved

janpio commented Aug 23, 2021

View reviewed changes

packages/cli/helpers/build.js Outdated Show resolved Hide resolved

janpio commented Dec 22, 2021

View reviewed changes

packages/migrate/src/__tests__/DbCommand.test.ts Outdated Show resolved Hide resolved

janpio commented Dec 22, 2021

View reviewed changes

packages/migrate/src/__tests__/MigrateDeploy.test.ts Outdated Show resolved Hide resolved

janpio commented Dec 22, 2021

View reviewed changes

pnpm-lock.yaml Show resolved Hide resolved

millsp added 2 commits December 22, 2021 15:55

chore(engine-core): bump undici

4dd5fb0

chore: cleanup

a811b01

aqrln approved these changes Dec 22, 2021

View reviewed changes

packages/client/helpers/build.ts Outdated Show resolved Hide resolved

packages/client/helpers/build.ts Outdated Show resolved Hide resolved

packages/client/helpers/build.ts Outdated Show resolved Hide resolved

Jolg42 reviewed Dec 22, 2021

View reviewed changes

helpers/compile/plugins/replaceWithPlugin.ts Outdated Show resolved Hide resolved

Jolg42 approved these changes Dec 22, 2021

View reviewed changes

smasala approved these changes Jan 4, 2022

View reviewed changes

Merge branch 'main' into integration/undici-4

ac83530

smasala approved these changes Jan 9, 2022

View reviewed changes

Jolg42 mentioned this pull request Jan 10, 2022

fix(deps): update dependency undici to v4 #11086

Closed

1 task

juanzgc mentioned this pull request Mar 18, 2022

I get "Can't resolve '_http_common'" when I use NestJS with monorepo (webpack) #6899

Closed

Merge branch 'main' of github.com:prisma/prisma into integration/undi…

fb7d30d

…ci-4

millsp removed this from the 3.8.0 milestone Mar 22, 2022

millsp force-pushed the integration/undici-4 branch from 4a1e395 to fb7d30d Compare March 23, 2022 09:20

millsp added 5 commits March 23, 2022 11:11

fix(client): unlazify undici

c2a8ed8

fix(client): defer undici loading

347dd06

fix(client): load unidici with require

fcc6f06

chore(client): cleanup

6b860b3

docs(client): add comment

6c82dc0

millsp added this to the 3.12.0 milestone Mar 23, 2022

millsp merged commit 6fc821f into main Mar 23, 2022

millsp deleted the integration/undici-4 branch March 23, 2022 13:51

Jolg42 reviewed Mar 23, 2022

View reviewed changes

lucasbogerd mentioned this pull request Apr 5, 2022

Prisma Client doesn't work outside of node_modules when using custom output dir #8326

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(deps): update undici to 4.x #8842

fix(deps): update undici to 4.x #8842

janpio commented Aug 20, 2021 •

edited by millsp

janpio commented Aug 20, 2021

Jolg42 commented Aug 23, 2021

Jolg42 left a comment

Jolg42 commented Jan 12, 2022

juanzgc commented Mar 13, 2022

vincent-thomas commented Mar 17, 2022

Jolg42 Mar 23, 2022

millsp Mar 23, 2022

janpio Mar 23, 2022

millsp Mar 23, 2022

Jolg42 Mar 23, 2022

millsp Mar 23, 2022

janpio Mar 23, 2022

millsp Mar 23, 2022

fix(deps): update undici to 4.x #8842

fix(deps): update undici to 4.x #8842

Conversation

janpio commented Aug 20, 2021 • edited by millsp

janpio commented Aug 20, 2021

Jolg42 commented Aug 23, 2021

Jolg42 left a comment

Choose a reason for hiding this comment

Jolg42 commented Jan 12, 2022

juanzgc commented Mar 13, 2022

vincent-thomas commented Mar 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janpio commented Aug 20, 2021 •

edited by millsp