AWS CRT binary not present in any of the following locations issue on Next.js app while using aws-iot-device-sdk-v2

[hari-prasad-a]

Describe the bug

I facing an issue after installing the aws-iot-device-sdk-v2 package in a Next.js app using Module federation. After importing iot and mqtt, I attempt to run the Next.js app using npm run dev but encounter the below errors. I am facing this issue only when I am using @module-federation/nextjs-mf, and if I remove the module federation configuration the issue is not occuring

Error: AWS CRT binary not present in any of the following locations:
        /bin/darwin-arm64-cruntime/aws-crt-nodejs.node
    at eval (webpack-internal:///./node_modules/aws-crt/dist/native/binding.js:110:11)
    
- error node_modules/aws-crt/dist/native/binding.js (109:0) @ eval
- error Error: AWS CRT binary not present in any of the following locations:
        /bin/darwin-arm64-cruntime/aws-crt-nodejs.node
    at eval (webpack-internal:///./node_modules/aws-crt/dist/native/binding.js:110:11)

Reproduction Steps

1. Install aws-iot-device-sdk-v2 on the Nextjs app which is using Module federation.
2. When I import iot and mqtt from 'aws-iot-device-sdk-v2' and iot and mqtt APIs are used on the Nextjs app component.
3. run npm run dev on terminal
4. The above-mentioned mention error throwing on the terminal

Reproduction link

Container App: https://github.com/hari-prasad-a/mqtt-container-app
Child App: https://github.com/hari-prasad-a/mqtt-child-app

Expected Behavior

After importing iot and mqtt from 'aws-iot-device-sdk-v2' the nextjs app running as we expect normally

Current Behavior

Currently, it's throwing the below error we can import { iot, mqtt } from 'aws-iot-device-sdk-v2' on the Nextjs app component and run on it.

Error: AWS CRT binary not present in any of the following locations:
        /bin/darwin-arm64-cruntime/aws-crt-nodejs.node
    at eval (webpack-internal:///./node_modules/aws-crt/dist/native/binding.js:110:11)
    
- error node_modules/aws-crt/dist/native/binding.js (109:0) @ eval
- error Error: AWS CRT binary not present in any of the following locations:
        /bin/darwin-arm64-cruntime/aws-crt-nodejs.node
    at eval (webpack-internal:///./node_modules/aws-crt/dist/native/binding.js:110:11)

Used Package Manager

npm

System Info

| Environment  | version |
| macOS        | 14.4.1  |
| processer    | m2 chip |
| node         | 20.12.1 |
| nextjs       | 14.1.4  |

Validations

• Read the docs.
• Read the common issues list.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Module federation issue and not a framework-specific issue.
• The provided reproduction is a minimal reproducible example of the bug.

[ScriptedAlchemy]

node Binaries cannot be shared over federation. You will have to mark them as externals and install them into applications for use at runtime.
Also since this is a next.js app, its even more limited in whats possible

[prakashmallow]

@ScriptedAlchemy The Next.js app, integrating 'aws-iot-device-sdk-v2' without the module federation package, runs seamlessly.

[ScriptedAlchemy]

yes because you are not trying to pull a binary file over the internet, its on disk like normal.

.node extension is loaded via fs.readFile() in the node module, not require(), so if you run this code on another machine without it npm installed on that machine, its going to fs.readFile and not find the binary. fs.readFile is not bundled by bundlers, only import and require is. This is beyond the scope of bundlers as a whole, not just federation.

It works in nextjs because its installed there on the app so it can find it since next server makes all node_modules externals and makes a big zip file of the node module dir.

When sharing code, i disable externalization of the node modules so that its bundled, but i cant put node binarys in .js files. Id have to patch the filesystem of node as a whole plus you would need to exose all node modules to the internet incase theres a binary somewhere in there that would be needed.

Dont share whatever uses the package binary, add it as "externals" and install it seperately into all apps

[prakashmallow]

@ScriptedAlchemy We need the aws-iot-device-sdk-v2 package only in the root app. How can we add it as an "external" dependency and install it in the root app?

[ScriptedAlchemy]

config.externals.push('aws-iot-device-sdk-v2') and dont put it in shared of module federation.

[prakashmallow]

@ScriptedAlchemy

const nextConfig = {
  reactStrictMode: true,
  webpack(config, options) {
    const { isServer } = options;
    config.plugins.push(
      new NextFederationPlugin({
        name: 'home',
        remotes: {
          services: `services@${process.env.NEXT_PUBLIC_SERVICES_APP}/_next/static/${isServer ? 'ssr' : 'chunks'}/remoteEntry.js`
        },
        filename: 'static/chunks/remoteEntry.js',
        exposes: {},
        extraOptions: {
          exposePages: true,
          automaticAsyncBoundary: true
        },
        shared: {
          antd: {
            requiredVersion: false,
            singleton: true
          },
        },
      }),
    );
    config.externals.push('aws-iot-device-sdk-v2')

    return config;
  },
};

export default nextConfig;

The line config.externals.push('aws-iot-device-sdk-v2') has been added to the next.config.js file. The following error is being thrown. Could you please check and address this?

[prakashmallow]

@ScriptedAlchemy

const nextConfig = {
  reactStrictMode: true,
  webpack(config, options) {
    const { isServer } = options;
    config.plugins.push(
      new NextFederationPlugin({
        name: 'home',
        remotes: {
          services: `services@${process.env.NEXT_PUBLIC_SERVICES_APP}/_next/static/${isServer ? 'ssr' : 'chunks'}/remoteEntry.js`
        },
        filename: 'static/chunks/remoteEntry.js',
        exposes: {},
        extraOptions: {
          exposePages: true,
          automaticAsyncBoundary: true
        },
        shared: {
          antd: {
            requiredVersion: false,
            singleton: true
          },
        },
      }),
    );
    config.externals.push('aws-iot-device-sdk-v2')

    return config;
  },
};

export default nextConfig;

The line config.externals.push('aws-iot-device-sdk-v2') has been added to the next.config.js file. The following error is being thrown. Could you please check and address this?

@ScriptedAlchemy Could you please give any example for webpack config.externals on Nextjs app?

[ScriptedAlchemy]

Only add externals to server build

[prakashmallow]

@ScriptedAlchemy The issue persists. I need to use the aws-iot-device-sdk-v2 package for local development. When trying to build, the same issue occurs.

[brenden-js]

@ScriptedAlchemy The issue persists. I need to use the aws-iot-device-sdk-v2 package for local development. When trying to build, the same issue occurs.

Were you ever able to get it working? Running into the same issue.