Error running with webpack: ENOENT: no such file or directory, open 'iam_service.json'

[hgardneriv]

Environment details

• OS: Linux & MacOS
• Node.js version: v14.15.0
• npm version: 6.14.8
• @google-cloud/pubsub version: 2.10.0

Steps to reproduce

1. When running a pubsub consumer or producer with webpack, the following error occurs:

"err":{"message":"ENOENT: no such file or directory, open 'iam_service.json'"," stack":"Error: ENOENT: no such file or directory, open 'iam_service.json'\n at Object.openSync (fs.js:476:3)\n at Object.readFileSync (fs.js:377:35)\n at fetch (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:403345:34) at Root.load (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:403379:13) at Root.loadSync (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:403420:17) at Object.loadSync (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:105027:29) at GrpcClient.loadFromProto (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245393:48) at GrpcClient.loadProto (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245434:21) at new IamClient (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245684:38) at new SubscriberClient (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:86374:26)"

NOTE: This issue only happens when running with webpack, it works expected when webpack is not in use.

2. This problem can be worked around by creating directory ../../protos and place the following files in it:

$ ls ../../protos
iam_service.json
protos.json

Unfortunately ../../protos falls one level up of our install directory, so it's not a valid work around.

3. Using the {fallback: true} option in the PubSub constructor, as suggested in this post: Webpack and other bundler support google-cloud-node#2933, results in another error:

`"fetch is not a function","stack":"TypeError: fetch is not a function
at Service.newServiceStub. [as testIamPermissions] (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:244522:17)
at /Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245730:29
at /Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:246661:16
at OngoingCall.call (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:244030:27)
at NormalApiCaller.call (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:246452:19)
at /Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:244163:30\n at processTicksAndRejections (internal/process/task_queues.js:93:5)

Is there a work around for this issue?

[feywind]

@hgardneriv Thanks for the report! I think for now our take on webpack / rollup / et al is that we don't officially support it, but there is some talk about moving toward making it work at some point. We're putting the (badly named) web label on issues that have to do with that.

[amiracle]

@feywind is there any way to escalate this issue? We are running into this same issue would like to know if there is a path forward from GCP / Google. Thanks again!

[feywind]

@amiracle The quickest way I know of to start some sort of escalation is through a support case, but I can say that there are actually some higher level discussions going on about the webpack support right now, so that might not be necessary. I can't promise anything, but it's more and more on the radar lately.

[feywind]

@alexander-fenster is taking a look at this now, he's our gax maintainer, so hopefully he'll have something for you.

[alexander-fenster]

Hi @hgardneriv,

Judging by your stack, I'm assuming you run webpack with target: "node", is this assumption correct?

In this case, don't use the fallback option (which is for browsers and other environments with no gRPC support).

The problem is caused by this snippet in google-gax:

    const nodejsProtoPath = path.join(
      __dirname,
      '..',
      '..',
      'protos',
      'iam_service.json'
    );

The default behavior of webpack changes how __dirname works (you can check it by doing console.log(__dirname) in your script and comparing the result with and without webpack), so this logic does not work properly when webpacked. The workaround is to add the following into your webpack config:

module.exports = {
  mode: 'production',
  target: 'node',
  node: {
    __dirname: true,  // this makes all the difference
  },
};

Please let us know if this workaround is acceptable for you.

[alexander-fenster]

To add to what I just said - since this code actually uses fs.readFileSync to read the JSON file, the webpack bundle produced won't be self contained: it will still need to be able to read node_modules/@google-cloud/pubsub/build/protos/protos.json and node_modules/google-gax/protos/iam_service.json from the filesystem so you will need to put at least these two files in your distribution along with the bundle.

[alexander-fenster]

Adding one more thing :) This is my test project that worked for me: https://gist.github.com/alexander-fenster/33c0142f0d62c2d78d0fa3e76559e26b

[hgardneriv]

Hello @alexander-fenster thank you very much for the info. Yes, we are running webpack with 'target: "node"'.

Unfortunately enabling __dirname will not work in this case, we use the following:

      node: {
        __dirname: false,
        __filename: false,
      },

Something that would be nice is the ability to pass in the config directly vs. reading from the filesystem. With this option we could easily work around the issue. Given our webpack config, do you see any other viable work arounds?

[alexander-fenster]

@hgardneriv OK, I'll try to see if I can make an option for proto files root and will get back to you.

[hgardneriv]

Also note that we have the same issue with protos.json which I think pulled is from @google-cloud/pubsub.

[alexander-fenster]

Yes, those are very similar places in the code and the fix would affect all of them. The second place is in https://github.com/googleapis/nodejs-pubsub/blob/master/src/v1/publisher_client.ts#L156-L162 and the third is in https://github.com/googleapis/nodejs-pubsub/blob/master/src/v1/subscriber_client.ts#L157-L163.

[alexander-fenster]

@hgardneriv Just an update - I actually found a better way to make it work; I'll basically replace loading the proto JSON from the filesystem with a require call. This change will take some time to be released (since several packages are affected) but I'll be able to release the tagged next version of @google-cloud/pubsub with this change so you could evaluate it. I'll keep this issue updated when it's done (hopefully within a day or so). Thank you!

[alexander-fenster]

@hgardneriv Before I start pushing the fix through several libraries, here's the test version for you to check:

$ rm -rf package-lock.json node_modules
$ npm install @google-cloud/pubsub@load-proto-json

The following command should give the pre-released versions for these two packages:

$ npm ls @google-cloud/pubsub google-gax
pubsub-repro@ /Users/fenster/pubsub-repro
└─┬ @google-cloud/pubsub@2.11.1-pre
  └── google-gax@2.12.0-pre.1

With these versions, I'm able to run the test code both with and without webpack. Could you please check that this fix works for you?

Thank you!

[hgardneriv]

@alexander-fenster - I'm happy to report that the build is working as expected, nice work!!!

[alexander-fenster]

Thank you! I'll go ahead and push this changes to the generator and google-gax. The wheels will start turning today and the releases will likely come out early next week.

[dmarczydlo]

I have the same problem with google-translate. The dist version of the app (builded by webpack) always return ENOENT: no such file or directory, open '/protos/protos.json

@alexander-fenster any idea how to solve also this case?

[dmarczydlo]

@alexander-fenster it looks like something similar to @hgardneriv case with @google-cloud/pubsub, but right now it's for translation.

[alexander-fenster]

@dmarczydlo Translate library is generated by the same code generator so the fix will make it work too. Please wait a few days and this will go away when I update the generator and the dependency chain to use require instead of the filesystem modules.

[dmarczydlo]

Awesome @alexander-fenster please let me know when the new approach will be available. I'm looking forward.

[dmarczydlo]

@alexander-fenster any update?

[alexander-fenster]

The code is ready, the releases are pending, please expect it soon. I'll update this issue.

[alexander-fenster]

@dmarczydlo PR is sent (#1283). If you would like we can pre-release it for you, or you can wait till Monday when we can make an official release. Thank you for waiting, and thank you for asking this question! I believe the code is much less fragile now that we use the regular require there.

[dmarczydlo]

@alexander-fenster will be great to test pre-relese if you can. As I check your RP is still open.

[alexander-fenster]

@dmarczydlo Sure, npm install @google-cloud/pubsub@next will do it (the exact version is 2.11.1-pre.1).

[alexander-fenster]

@dmarczydlo This is not the right place to talk about the translate library, but just wanted to let you know that it just got released (v6.2.1). In case if you see any issue with it, please create an issue in its own repo. Thank you!

[feywind]

This is pending this release PR:
#1281