Add support for `zeromq` #391

eulersson · 2024-02-10T08:45:31Z

Some libraries like zeromq.js need runtime libraries with bindings which they ship along. Those binaries exist under node_modules/zeromq/prebuilds.

Simply by creating a new Next.js app, installing https://github.com/zeromq/zeromq.js and trying to import it and running npm run dev and use one of its classes would throw a No native build was found for platform=linux arch=x64 runtime=node abi=115 uv=1 libc=glibc node=20.11.0 webpack=true loaded from: /app/.next/server. I found out that if I print this dir variable used (among other things) to search for binaries in node-gyp-builder it print .next/server/... instead of node_modules/... and since the binaries have not been copied it fails.

Note

I provided a reproducible example and a container, but it's easily reproducible on mac and linux: https://github.com/eulersson/zeromq.js-next.js-errors

I tried using the outputFileTracingIncludes option as explained in https://nextjs.org/docs/pages/api-reference/next-config-js/output#caveats but without success.

Here's the files I would like to have in .next/server.

❯ tree node_modules/zeromq/prebuilds
node_modules/zeromq/prebuilds
├── darwin-arm64
│   └── node.napi.glibc.node
├── darwin-x64
│   └── node.napi.glibc.node
├── linux-x64
│   ├── node.napi.glibc.node
│   └── node.napi.musl.node
├── win32-ia32
│   └── node.napi.glibc.node
└── win32-x64
    └── node.napi.glibc.node

6 directories, 6 files

Since I don't know in what area work should be done I posted the same problem as an issue to the various tools involved:

Thanks for the help and apologies if I have been generating noise by posting so many issues.

The text was updated successfully, but these errors were encountered:

styfle · 2024-02-12T14:29:38Z

You can try npx @vercel/nft@latest print path/to/my/file.js to see which files are included.

If its missing prebuilds, then it is likely an issue with @vercel/nft.

You can create a PR with a test and special case, similar to PR #37.

Or if you want to fix the general case (likely a dynamic require somewhere) that would be great too (albeit much harder).

eulersson · 2024-02-12T15:51:26Z

@styfle Many thanks for the guidance. I will investigate that route this afternoon! :)

eulersson · 2024-02-12T19:16:28Z

@styfle Indeed it certainly does not bring in the binaries:

❯ npx @vercel/nft@latest print ./node_modules/zeromq/lib/index.js
FILELIST:
node_modules/@aminya/node-gyp-build/index.js
node_modules/@aminya/node-gyp-build/package.json
node_modules/zeromq/lib/draft.js
node_modules/zeromq/lib/index.js
node_modules/zeromq/lib/native.js
node_modules/zeromq/lib/util.js
node_modules/zeromq/package.json

eulersson · 2024-02-12T19:19:27Z

@styfle I will provide a PR tomorrow.

styfle · 2024-02-12T19:49:35Z

Also see this related PR that mentions node-gyp-build

fix: node-gyp-build binary locating in different versions #286

That might lead to the more general purpose fix instead of a special case.

eulersson · 2024-02-13T12:18:48Z

I will do the general purpose one. I think I understood the tracing logic a bit.

…evel where de require() is run at. Fixes zeromq.js. Closes vercel#391.

eulersson · 2024-02-13T16:41:22Z

@styfle Can you have a look? 👀 🙇

@aminya

- Fixes #391 Three things I want to point out: 1. In zeromq the way node-gyp-build is called is by passing `path.join(__dirname, "..")` instead of `__dirname` to the `require("node-gyp-build")(...)` call. 2. In zeromq they use `@aminya/node-gyp-build` instead of `node-gyp-build`, particular-li version `4.5.0-aminya.5`. 3. The zeromq version that's recommended for new project is the `6` which is beta according to @aminya https://github.com/zeromq/zeromq.js. After those changes it resolves OK. ``` ❯ node ./node_modules/@vercel/nft/out/cli.js print ./node_modules/zeromq/lib/index.js FILELIST: node_modules/@aminya/node-gyp-build/index.js node_modules/@aminya/node-gyp-build/package.json node_modules/zeromq/lib/draft.js node_modules/zeromq/lib/index.js node_modules/zeromq/lib/native.js node_modules/zeromq/lib/util.js node_modules/zeromq/package.json node_modules/zeromq/prebuilds/darwin-x64/node.napi.glibc.node <---- It worked! ``` --------- Co-authored-by: Steven <steven@ceriously.com>

github-actions · 2024-02-13T22:42:46Z

🎉 This issue has been resolved in version 0.26.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

eulersson · 2024-02-14T16:54:24Z

@styfle how can I test this? I am trying to use the overrides npm feature without success:

https://github.com/eulersson/zeromq.js-next.js-errors/blob/b0e24cbf471d5f454cbe3f6b89f7083d0027474a/package.json#L28-L30

I also set next and eslint-config-next to ^14.1.1-canary.52 which seems to have the @vercel/nft to 0.26.4 but in reality, if after npm installing I check node_modules/next/package.json I see 0.26.3 and not 0.26.4.

I'm very confused because at this commit https://github.com/vercel/next.js/blob/07c652a1200445ebfed8ef13426e36a150a04a67/packages/next/package.json#L194 (Next.js 14.1.1-canary.52) I can see the nft version set at 0.26.4.

Can you help me set up a project with a next app that has my changes?

eulersson · 2024-02-14T17:07:56Z

@styfle I can see the tagged next git repo 14.1.1-canary.52 does have 0.26.3 but the canary branch carrying the same package.json version carries 0.26.4. I was thinking of using something like this:

  "dependencies": {
    "next": "https://github.com/vercel/next.git#0c21654845f78717a8198a69d968671477adc7f8",

0c21654845f78717a8198a69d968671477adc7f8 is the commit of the canary branch that has the changes I want.

Would that work?

The next.js repo has many packages: next, eslint-config-next, etc... I don't know if putting a git url in packages.json is the best solution for testing my changes....

styfle · 2024-02-14T20:44:16Z

@eulersson A new canary was just released with the fix. You can run npm i next@14.1.1-canary.53 to try it out now.

https://github.com/vercel/next.js/releases/tag/v14.1.1-canary.53

eulersson · 2024-02-15T08:12:05Z

Still not working @aminya @styfle. Do you have any idea what might be the issue?

zeromq.js-next.js-errors git:main* 
❯ npm run build
> my-app@0.1.0 build
> next build

   ▲ Next.js 14.1.1-canary.53

   Creating an optimized production build ...

warn - No utility classes were detected in your source files. If this is unexpected, double-check the `content` option in your Tailwind CSS configuration.
warn - https://tailwindcss.com/docs/content-configuration
 ✓ Compiled successfully
 ✓ Linting and checking validity of types
   Collecting page data  .Error: No native build was found for platform=darwin arch=x64 runtime=node abi=115 uv=1 libc=glibc node=20.11.0 webpack=true
    loaded from: /Users/ramon/Playground/zeromq.js-next.js-errors/.next/server

    at g.path (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:1:5427)
    at g (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:1:3762)
    at 9140 (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:6:36323)
    at t (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/webpack-runtime.js:1:127)
    at 6874 (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:6:29810)
    at t (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/webpack-runtime.js:1:127)
    at 1100 (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:1:2991)
    at Function.t (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/webpack-runtime.js:1:127)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async collectGenerateParams (/Users/ramon/Playground/zeromq.js-next.js-errors/node_modules/next/dist/build/utils.js:920:21)

> Build error occurred
Error: Failed to collect page data for /
    at /Users/ramon/Playground/zeromq.js-next.js-errors/node_modules/next/dist/build/utils.js:1268:15
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  type: 'Error'
}
   Collecting page data  .%                                                                                                                    zeromq.js-next.js-errors git:main*  15s

However the nft version is correct and brings in the binary.

❯ npx @vercel/nft print ./node_modules/zeromq/lib/index.js
FILELIST:
node_modules/@aminya/node-gyp-build/index.js
node_modules/@aminya/node-gyp-build/package.json
node_modules/zeromq/lib/draft.js
node_modules/zeromq/lib/index.js
node_modules/zeromq/lib/native.js
node_modules/zeromq/lib/util.js
node_modules/zeromq/package.json
node_modules/zeromq/prebuilds/darwin-x64/node.napi.glibc.node

It breaks here at the throw error:

https://github.com/aminya/node-gyp-build/blob/4cd1eb0f1ef2ed7d4c36b82e3d547e33cd83c9a0/index.js#L25-L60

The variable dir in line 26 is evaluated to '/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server when debugging, this is what the folder looks like:

zeromq.js-next.js-errors/.next/server git:main*
❯ tree
.
├── app
│   ├── _not-found_client-reference-manifest.js
│   ├── not-found.js
│   ├── not-found_client-reference-manifest.js
│   ├── page.js
│   └── page_client-reference-manifest.js
├── app-paths-manifest.json
├── interception-route-rewrite-manifest.js
├── middleware-build-manifest.js
├── middleware-manifest.json
├── middleware-react-loadable-manifest.js
├── next-font-manifest.js
├── next-font-manifest.json
├── pages-manifest.json
├── server-reference-manifest.js
├── server-reference-manifest.json
├── vendor-chunks
│   ├── @aminya.js
│   ├── @swc.js
│   ├── next.js
│   └── zeromq.js
└── webpack-runtime.js

3 directories, 20 files

I will attach here the contents of .next/server/vendor-chunks/zeromq.js. zeromq.js.txt

To reproduce simply import zeromq@6.0.0-beta.19 into your next canary app greater than 14.1.1-canary.53 and place import { Push } from "zeromq"; and const socket = new Push() anywhere .

styfle · 2024-02-15T16:31:23Z

This will fix it:

fix: add zeromq to server-external-packages.json next.js#62105

eulersson · 2024-02-16T05:24:12Z

@styfle I confirm it works now! Thanks for your help 🙏

eulersson mentioned this issue Feb 9, 2024

Investigate how to make zeromq.js work in a Next.js application eulersson/taconez#107

Closed

This was referenced Feb 12, 2024

Next.js does not bring over prebuilds folder into .next/server: "No native build was found for ..." prebuild/node-gyp-build#65

Closed

zeromq.js failing on Next.js "No native build was found for ..." zeromq/zeromq.js#600

Closed

eulersson added a commit to eulersson/nft that referenced this issue Feb 13, 2024

Fix node-gyp-build not including prebuilds that are not at the same l…

c094d28

…evel where de require() is run at. Fixes zeromq.js. Closes vercel#391.

eulersson mentioned this issue Feb 13, 2024

fix(analyze): node-gyp-build not including zeromq prebuilds #392

Merged

eulersson mentioned this issue Feb 13, 2024

zeromq works standalone but not when used within Next.js: "No native build was found for ..." vercel/next.js#61844

Closed

styfle closed this as completed in #392 Feb 13, 2024

github-actions bot added the released label Feb 13, 2024

austinwoon mentioned this issue Apr 18, 2024

Add support for @datadog/pprof #410

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for `zeromq` #391

Add support for `zeromq` #391

eulersson commented Feb 10, 2024

styfle commented Feb 12, 2024

eulersson commented Feb 12, 2024

eulersson commented Feb 12, 2024

eulersson commented Feb 12, 2024

styfle commented Feb 12, 2024 •

edited

eulersson commented Feb 13, 2024

eulersson commented Feb 13, 2024 •

edited

github-actions bot commented Feb 13, 2024

eulersson commented Feb 14, 2024

eulersson commented Feb 14, 2024

styfle commented Feb 14, 2024 •

edited

eulersson commented Feb 15, 2024 •

edited

styfle commented Feb 15, 2024

eulersson commented Feb 16, 2024

Add support for zeromq #391

Add support for zeromq #391

Comments

eulersson commented Feb 10, 2024

styfle commented Feb 12, 2024

eulersson commented Feb 12, 2024

eulersson commented Feb 12, 2024

eulersson commented Feb 12, 2024

styfle commented Feb 12, 2024 • edited

eulersson commented Feb 13, 2024

eulersson commented Feb 13, 2024 • edited

github-actions bot commented Feb 13, 2024

eulersson commented Feb 14, 2024

eulersson commented Feb 14, 2024

styfle commented Feb 14, 2024 • edited

eulersson commented Feb 15, 2024 • edited

styfle commented Feb 15, 2024

eulersson commented Feb 16, 2024

Add support for `zeromq` #391

Add support for `zeromq` #391

styfle commented Feb 12, 2024 •

edited

eulersson commented Feb 13, 2024 •

edited

styfle commented Feb 14, 2024 •

edited

eulersson commented Feb 15, 2024 •

edited