mock-aws-s3, aws-sdk, nock dependencies vs devDependencies issue #661
Comments
|
I've encountered the same issue while trying to compile a back-end app with webpack for limited distribution. @johnmanko's analysis definitely looks correct to me. |
|
I encountered this issue when depending on |
|
I ended up having to remove |
|
I encountered this issue using nest js. I solved it by removing // I remove this block
"compilerOptions": {
"webpack": true
} |
|
+1 |
|
+1, encountered depending on node-sqlite3 |
|
I have encountered the same problem and opened an issue at bcrypt yesterday so they improve their README, but my bad I didn't know that was a recent |
|
Is there a workaround for this issue? I face this problem |
|
Bad workaround, add this to your webpack.config.js: const config = {
// Some other config
externals: [
{
'nock': 'commonjs2 nock',
'mock-aws-s3': 'commonjs2 mock-aws-s3'
}
],
}This will tell webpack to let the |
Downside of this is that you need to handle all the runtime errors (like nested runtime dependencies) can take a long long time |
|
I stumbled upon this as well when trying to bundle the DuckDB Node.js package. This additionally attempts to require Using esbuild is seemingly also not an option, as this produces incorrect builds as well, due to |
|
some error here... |
|
Hmmm. So i got same error while using bcrypt in a nuxt app. And i just kept on thinking what nuxt had to do with aws. It was frustrating. I just kept on searching online and stumbled at this |
|
I got this error using Vite and sqlite in an Electron.js app |
|
+1 with sqlite3 |
|
+1 with Vite and sqlite (Sveltekit app) |
|
+1 electron + sqlite |
|
I solved it by adding externals: ['nock', 'mock-aws-s3', 'aws-sdk'], in webpack config |
I guess that won‘t help you if the code is actually run, because then the dependencies are missing IMO |
|
Having the same issue depending on bcrypt inside of an Astro project. |
+1 with sqlite3 |
|
+1 depending on |
|
+1 sqlite3 |
|
+1 sqlite3. send help |
|
Can't those |
|
@springmeyer I can propose a PR to fix this issue. I am proposing a |
|
I got the same issue while trying to install swaggiffy and it was caused by sqlite3 |
|
+1 swaggify |
|
I too encountered this when using sqlite3 in NodeJS. My simple workaround is to just add the unnecessary dependencies to my project, it builds fine now. |
|
Was there a fix for this? |
No doubt this is going to work, but I don't want to bundle a bunch of useless things in my app (and you shouldn't either). I found a very ugly but functional work around for Vite. For info, I am using the latest version of vite-plugin-electron. What I did is that I created an alias for all the libraries that were causing errors to an empty file. // vite.config.ts
export default defineConfig({
plugins: [
// ...
electron([
{
entry: resolve('./src/main/electron.ts'),
vite: {
build: {
// Whatever you have here
},
resolve: {
alias: {
// Your other aliases if you have some
"sqlite3": resolve(__dirname, 'src/main/empty.ts'),
"pg": resolve(__dirname, 'src/main/empty.ts'),
"pg-query-stream": resolve(__dirname, 'src/main/empty.ts'),
"tedious": resolve(__dirname, 'src/main/empty.ts'),
"mysql": resolve(__dirname, 'src/main/empty.ts'),
"mysql2": resolve(__dirname, 'src/main/empty.ts'),
"oracledb": resolve(__dirname, 'src/main/empty.ts'),
},
}
},
},
]),
]
})And in /**
* This file is used to override problematic dependencies when using node-pre-gyp.
*
* These dependencies are imported but are never used and cause the application to crash at runtime.
*
* See this issue: https://github.com/mapbox/node-pre-gyp/issues/661
*/I hope we'll get a better fix than that real soon, cause this is without a doubt one of the worst hack I've had to do in a long time. |
Worked for me temporally! Using electron-forge and sqlite3 here. |
|
I made a workaround for this issue that completely removes node-pre-gyp from the packaged electron application. I am no expert in making native packages for nodejs. I use them, as many people do, and I recently tried to package one of them, node-sqlite3, to be a part of electron app. I use webpack and electron-forge for packaging. This tools produces a self-contained package, in the form of zip archive, debian, or rpm package. The package is OS and CPU architecture-specific (e.g. linux-x64) and it contains, as a part of electron binary, a specific version of nodejs. In the usual setup, webpack collects the bundle which contains the node-sqlite3 package. That bundle is collected from the local files. At that point, the node-sqlite3 package already contains platform-specific binary files. As a next step, that webpack bundle is packaged along with the electron runtime to the resulting application package. When running the resulting electron application on the end user machine, it is not expected to download any additional native binaries, compile them, cross-compile binaries to target other platforms, or upload compiled native binaries to the cloud storage. Problem is, that functionality is a part of node-pre-gyp and it's dependencies, and all that, being the runtime dependency of node-sqlite3, is packed by the webpack to the resulting application bundle. That packaging process is not without issues: in the current version of node-pre-gyp it may require to have a webpack configuration workaround of The bigger picture is, as I see it, that node-pre-gyp should not be a part of electron application bundle at all. It have it's place in the build pipeline, but carrying it further to the end-users should be avoided. I may not see some cases when it do required, but as far as I could tell, native modules, as soon as they are installed, could work fine without node-pre-gyp. At runtime, node-sqlite3 uses node-pre-gyp to resolve native module filename. There is a The electron application is already packaged for a specific environment, so I believe that template string could be resolved to a static path at the time of application packaging. That will effectively remove the need to include node-pre-gyp in the electron application package. The following code could be added to a webpack configuraton file to do exactly that. const path = require('path');
const fs = require('fs');
const nodePreGyp = require('@mapbox/node-pre-gyp');
const webpack = require('webpack');
// The following block assumes that the current file is in the project root, so we could calculate paths as relative to __dirname
const sqliteBindingPath = nodePreGyp.find(path.join(__dirname, 'node_modules', 'sqlite3', 'package.json'));
const sqlitePatchFileName = 'sqlite3-binding-with-binary-find-patch.js';
const sqlitePatchFilePath = path.resolve(path.join(__dirname, 'node_modules', 'sqlite3', 'lib', sqlitePatchFileName));
// Here we might write an absolute path. Webpack will take care to produce a portable bundle, in which this
// absolute path will be changes to a path, relative to the bundle itself.
fs.writeFileSync(sqlitePatchFilePath, `module.exports = require(${JSON.stringify(sqliteBindingPath)});`);
// Add this to your webpack config
module.exports = {
plugins: [
new webpack.NormalModuleReplacementPlugin(
/sqlite3\/lib\/sqlite3-binding\.js$/,
`./${sqlitePatchFileName}`
)
]
};The resulting bundle will not contain node-pre-gyp and it's dependencies. That will solve the current issue with packing ( I hope that this workaround might be of an inspiration to someone to create a more universal solution. It could be a lightweight runtime as a part of node-pre-gyp, or, if that is not possible, as a form of some alternative lightweight project. Maybe it could be solved on a side of packages such as node-sqlite3, and not on the node-pre-gyp side. Maybe package managers such as npm could came up with some abstractions to handle native modules that could help in this regard. |
|
Same problem with the bcrypt package, any chance of this bug being fixed? |
|
Solved for me! |
|
Same problem using NextJS.13 with Mongo and Prisma. And I actually found that it's happening due to Bcrypt. |
|
Found a fix for my case. Now it works, compiles sqlite3 with no additional issues. |
|
Not sure if this is an accepted approach but i did this to get it work, my stack is currently using and now I can at least run my application in dev mode which I couldn't before, we will see how it goes later when I'm gonna ship my app. ggwp |
|
I worked around this by putting the packages that depend on this in the "dependencies" section instead of "devDependencies" in Also, if you use "bcrypt", you can try Bun, which has Bcrypt (and Argon2 too) in the standard library without installing any packages (https://bun.sh/guides/util/hash-a-password). |
|
Experienced the same problem when attempting to build with Remix, happens with both argon2 and bcrypt packages. For me, the fix was to install |
|
Also worth trying the As a plus feature, these are faster than the |
|
This issue has caused me an incredible amount of lost time trying to work around it. There are some suggested solutions to get sqlite3 (which depends on this package) and vite work, but they didn't work for me. To continue the development, I am falling back into implementing a custom file storage without SQL until this is fixed and I can revert to sqlite3. I'm not mentioning this not because I think I am entitled to having this software working, but to provide visibility on the impact it's having. |
|
@fcanela despite Bun is in a very experimental stage, it's worth keeping an eye on it, as it has SQLite in the standard library, out of the box, without any precompiled dependency. Its standard library provides solutions to a big part of the problems described in this thread. |
|
A snippet for anyone if you rather just migrate from bcrypt to scrypt: import { randomBytes, scrypt, timingSafeEqual } from 'node:crypto';
/**
* https://dev.to/advename/comment/24a9e
*/
const keyLength = 32;
/**
* Has a password or a secret with a password hashing algorithm (scrypt)
* @param {string} password
* @returns {string} The salt+hash
*/
export const hash = async (password: string) => {
return new Promise((resolve, reject) => {
// generate random 16 bytes long salt - recommended by NodeJS Docs
const salt = randomBytes(16).toString('hex');
scrypt(password, salt, keyLength, (error, derivedKey) => {
if (error) reject(error);
// derivedKey is of type Buffer
resolve(`${salt}.${derivedKey.toString('hex')}`);
});
});
};
/**
* Compare a plain text password with a salt+hash password
* @param {string} password The plain text password
* @param {string} hash The hash+salt to check against
* @returns {boolean}
*/
export const compare = async (password: string, hash: string) => {
return new Promise((resolve, reject) => {
const [salt, hashKey] = hash.split('.');
// we need to pass buffer values to timingSafeEqual
const hashKeyBuff = Buffer.from(hashKey, 'hex');
scrypt(password, salt, keyLength, (error, derivedKey) => {
if (error) reject(error);
// compare the new supplied password with the hashed password using timeSafeEqual
resolve(timingSafeEqual(hashKeyBuff, derivedKey));
});
});
}; |
|
I use bcryptjs instead of bcrypt and it worked |
Exacly! I installed bycryptjs instead of bycrypt and the problem disappeared |
|
This bug looks suffer a lot of people and seems still no PR to resolve it. I think it should be easy to solve it by changing dependencies declaration. Should I try to raise the PR? |
and others
Usage of
mock-aws-s3,aws-sdk, andnockare used in the call stack originating fromlib/node-pre-gyp.js(which is the packagemain), but they are only listed in"devDependencies".This causes a problem with webpack:
Either those modules should be moved to
"dependencies", or the main call chain should never hit them.The text was updated successfully, but these errors were encountered: