[Bug] Issue using aws-otel-nodejs in Lambda

[inception-cto]

Context

I am trying to configure ADOT and use it with the current implementation of Lambda through AWS CDK. AWS CDK offers a ready-to-use NodeJS lambda constructs @aws-cdk/aws-lambda-nodejs.

Unfortunately, it seems that there is an error/conflict between the otel-handler and the way @aws-cdk/aws-lambda-nodejs builds the lambda function. I am not sure if it is related to esbuild or not.

Update: It seems that the issue is similar to #97 .

Configuration

I am using the following code to create a Lambda function.

this.adotLayer = lambda.LayerVersion.fromLayerVersionArn(
      this,
      "adot-layer",
      "arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-ver-0-19-0:1"
    );

const lambdaFn = new NodejsFunction(this, `sample-fn`, {
      runtime: lambda.Runtime.NODEJS_14_X,
      description: props.description,
      handler: "handler",
      entry: "function/sample.ts",
      tracing: lambda.Tracing.ACTIVE,
      environment: {
        AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-handler", 
      },
      layers: [this.adotLayer],
    });

Error

The Lambda function is deployed properly. However, when running the function, I receive the following error:

{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '/var/task/package.json'\nRequire stack:\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/index.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/index.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/index.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/aws-lambda.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/index.js\n- /opt/wrapper.js\n- internal/preload",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module '/var/task/package.json'",
        "Require stack:",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/index.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/index.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/index.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/aws-lambda.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/index.js",
        "- /opt/wrapper.js",
        "- internal/preload",
        "    at _loadUserApp (/var/runtime/UserFunction.js:100:13)",
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
        "    at Object.<anonymous> (/var/runtime/index.js:43:30)",
        "    at Module._compile (internal/modules/cjs/loader.js:1068:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)",
        "    at Module.load (internal/modules/cjs/loader.js:933:32)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:774:14)",
        "    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)",
        "    at internal/main/run_main_module.js:17:47"
    ]
}

⚠️ Update

I added a dummy package.json to the project, and now I receive the same error in #97 .

"TypeError: Cannot redefine property: handler",
        "    at Function.defineProperty (<anonymous>)",
        "    at defineProperty (/opt/nodejs/node_modules/shimmer/index.js:14:10)",
        "    at AwsLambdaInstrumentation.wrap [as _wrap] (/opt/nodejs/node_modules/shimmer/index.js:56:3)",
        "    at InstrumentationNodeModuleFile.patch (/opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/aws-lambda.js:67:26)",
        "    at AwsLambdaInstrumentation._onRequire (/opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js:75:33)",
        "    at /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js:102:29",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:154:32)",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:80:39)",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:80:39)",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:80:39)"

[srprash]

Seems like it could be related to aws/aws-xray-sdk-node#433 which is an esbuild issue with CDK aws-lambda-nodejs bundling.

[anuraaga]

Thanks @melek-inception - I have filed an issue upstream on whether it's possible to monkey patch libraries created with esbuild or CDK (not sure which is causing it to be non-redefinable).

open-telemetry/opentelemetry-js#2260

[willarmiros]

w.r.t. the original ImportModuleError, this is discussed in open-telemetry/opentelemetry-js#2193, and we've reached an agreement with the maintainers of the OTel JS SDK for an approach that won't require an empty package.json. It is not yet implemented, but should be a quick fix once out of the backlog.

For the newer TypeError issue, sounds like that might be fixable with a patch to the shimmer library used for monkey-patching. We'll continue discussion in the opentelemetry-js repo most likely.

[longility]

For anyone facing the issue related to TypeError: Cannot redefine property: handler, @melek-inception found a workaround for the esbuild gotcha.

For the lambda handler, instead of

export const handler = () => {
  // removed for brevity
};

Do this

const handler = () => {
  // removed for brevity
};
module.exports = { handler }

[inception-cto]

@longility , you beat me to update the GitHub issue 💯

The error seems to be linked to the esbuild immutability of the exported objects. Enforcing a mutable export of the handler fixes the problem.

Hope this help!

[inception-cto]

@willarmiros , I will close the ticket unless you think of a different workaround.