Node - TypeError: Cannot read properties of undefined (reading 'forEach') #1950

Starefossen · 2024-02-21T22:00:39Z

What happened?

Steps to Reproduce

Follow the guide for automatic instrumentation: https://opentelemetry.io/docs/languages/js/automatic/

The source code for this application is here: https://github.com/nais/unleash/tree/unleash-v5 and is based on express ^4.18.2.

Environment variables:

    - name: NODE_OPTIONS                                                                                                                      
      value: --require @opentelemetry/auto-instrumentations-node/register                                                                               
    - name: OTEL_EXPORTER_OTLP_PROTOCOL                                     
      value: grpc                                                           
    - name: OTEL_NODE_RESOURCE_DETECTORS                                    
      value: env,host,container                                                                                                                         
    - name: OTEL_SERVICE_NAME                                                                                                                           
      value: my-demo                                                                                                                                  
    - name: OTEL_EXPORTER_OTLP_ENDPOINT                                                                                                                 
      value: http://opentelemetry-collector.my-system:4317     
    - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME                               
      valueFrom:                  
        fieldRef:                                                           
          apiVersion: v1                                                    
          fieldPath: metadata.name
    - name: OTEL_RESOURCE_ATTRIBUTES_NODE_NAME                              
      valueFrom:                                                                                                                                        
        fieldRef:                                                                                                                                       
          apiVersion: v1                                                                                                                                
          fieldPath: spec.nodeName                                          
    - name: OTEL_PROPAGATORS 
      value: tracecontext,baggage,b3                                        
    - name: OTEL_RESOURCE_ATTRIBUTES                                        
      value: k8s.container.name=unleash,k8s.deployment.name=my-demo,k8s.namespace.name=my-unleash,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_N
AME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=my-demo-f95cddd45,service.version=v5.9.6-20240221-210653-d96e3be           
    image: europe-north1-docker.pkg.dev/nais-io/nais/images/unleash-v4:v5.9.6-20240221-210653-d96e3be

Expected Result

It should start properly.

Actual Result

Application

Additional Details

OpenTelemetry Setup Code

// automatic instrumentation

package.json

// https://github.com/nais/unleash/blob/unleash-v5/package.json
{
  "name": "nais-unleash",
  "version": "4.0.0",
  "description": "",
  "main": "src/index.js",
  "scripts": {
    "all": "yarn lint && yarn test && yarn build",
    "start": "node ./dist/index.js",
    "build-and-start": "tsc && node ./dist/index.js",
    "build": "tsc",
    "test": "jest --coverage --no-cache --silent src/*.test.ts",
    "test:debug": "jest --runInBand --coverage --no-cache --detectOpenHandles src/*.test.ts",
    "lint": "eslint ./src --ext .ts,.tsx",
    "lint:fix": "eslint ./src --ext .js,.jsx,.ts,.tsx --fix"
  },
  "author": "NAIS <aura@nav.no>",
  "license": "MIT",
  "engines": {
    "node": "^20.0.0"
  },
  "dependencies": {
    "@opentelemetry/api": "^1.7.0",
    "@opentelemetry/auto-instrumentations-node": "^0.41.1",
    "google-auth-library": "^9.6.3",
    "jsonwebtoken": "^9.0.2",
    "log4js": "^6.9.1",
    "unleash-server": "5.9.6"
  },
  "devDependencies": {
    "@types/cors": "2.8.17",
    "@types/express": "4.17.21",
    "@types/express-session": "1.17.10",
    "@types/faker": "6.6.9",
    "@types/jest": "^29.5.12",
    "@types/js-yaml": "4.0.9",
    "@types/jsonwebtoken": "^9.0.5",
    "@types/make-fetch-happen": "10.0.4",
    "@types/node": "20.11.19",
    "@types/semver": "7.5.7",
    "@types/stoppable": "1.1.3",
    "@types/supertest": "6.0.2",
    "@types/type-is": "1.6.6",
    "@types/uuid": "9.0.8",
    "@typescript-eslint/eslint-plugin": "7.0.2",
    "@typescript-eslint/parser": "7.0.2",
    "eslint": "^8.56.0",
    "eslint-config-airbnb-base": "^15.0.0",
    "eslint-config-airbnb-typescript": "^17.1.0",
    "eslint-config-prettier": "^9.1.0",
    "eslint-plugin-import": "^2.29.1",
    "eslint-plugin-no-only-tests": "^3.1.0",
    "eslint-plugin-prettier": "^5.1.3",
    "jest": "^29.7.0",
    "nock": "^13.5.3",
    "prettier": "^3.2.5",
    "supertest": "^6.3.4",
    "ts-jest": "^29.1.2",
    "ts-node": "^10.9.2",
    "typescript": "5.3.3"
  }
}

Relevant log output

TypeError: Cannot read properties of undefined (reading 'forEach')
    at iterateStack (/app/node_modules/@wesleytodd/openapi/lib/generate-doc.js:74:24)
    at /app/node_modules/@wesleytodd/openapi/lib/generate-doc.js:17:5
    at Array.forEach (<anonymous>)
    at generateDocument (/app/node_modules/@wesleytodd/openapi/lib/generate-doc.js:16:26)
    at OpenApiMiddleware (/app/node_modules/@wesleytodd/openapi/index.js:42:29)
    at /app/node_modules/@opentelemetry/instrumentation-express/build/src/instrumentation.js:214:37
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/app/node_modules/express/lib/router/index.js:328:13)
    at /app/node_modules/express/lib/router/index.js:286:9
    at Function.process_params (/app/node_modules/express/lib/router/index.js:346:12)
    at next (/app/node_modules/express/lib/router/index.js:280:10)
    at arguments.<computed> (/app/node_modules/@opentelemetry/instrumentation-express/build/src/instrumentation.js:210:41)
    at SendStream.error (/app/node_modules/serve-static/index.js:121:7)
    at SendStream.emit (node:events:518:28)
    at SendStream.error (/app/node_modules/send/index.js:270:17)
    at SendStream.onStatError (/app/node_modules/send/index.js:417:12)

The text was updated successfully, but these errors were encountered:

dyladan · 2024-02-28T17:23:40Z

If you remove the otel instrumentation, does the issue persist? I am not confident that this is an issue caused by otel.

Starefossen · 2024-02-28T20:55:36Z

The app runs completely fine without otel instrumentation, removing the --require removes the errors.

pichlermarc · 2024-03-06T15:57:41Z

The app runs completely fine without otel instrumentation, removing the --require removes the errors.

Thanks for the extra info 👍

I've marked it as P1 as it's causing an error in end-user apps. Like many open-source projects we have a limited bandwidth so a more minimal reproducer would be immensely useful and would allow us to tackle that issue more quickly.

Starefossen · 2024-03-06T18:05:45Z

No worries, I am still debugging this myself as well, will update the issue if I find something. Also tracking this in the Upstream project I am instrumenting https://github.com/orgs/Unleash/discussions/6455

david-luna · 2024-04-22T12:08:17Z

Some extra context on this after checking out the project ad try it myself

use the following steps to reproduce after cloning the repo

# install dependencies based on the lock file & build
yarn && yarn build

# start db
docker compose up -d postgres

# start the app with the required env (for app & OTEL)
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
  OTEL_EXPORTER_OTLP_PROTOCOL="grpc" \
  OTEL_NODE_RESOURCE_DETECTORS="env,host,container" \
  OTEL_SERVICE_NAME="my-demo" \
  OTEL_PROPAGATORS="tracecontext,baggage,b3" \
  OTEL_RESOURCE_ATTRIBUTES="k8s.container.name=unleash,k8s.deployment.name=my-demo,k8s.namespace.name=my-unleash,k8s.replicaset.name=my-demo-f95cddd45,service.version=v5.9.6-20240221-210653-d96e3be" \
  DATABASE_USERNAME=unleash \
  DATABASE_PASSWORD=unleash \
  DATABASE_NAME=unleash \
  DATABASE_HOST=localhost \
  DATABASE_SSL=false \
  INIT_ADMIN_API_TOKENS=*:*.unleash4all \
  GOOGLE_IAP_AUDIENCE=/projects/123/global/backendServices/123 \
  node ./dist/index.js

nodejs process starts without error
the error described happens when a request is made to http:localhost:4242

Express instrumentation may be affecting/modifying some route internals. The code throwing the error is from @wesleytodd/openapi package and is using express internally and inspection route layers to generate the swagger specs.

david-luna · 2024-04-22T14:27:13Z

@wesleytodd/openapi is inspecting the layer handle function for an internal property called stack. That property gets lost when doing the patch. ~~I guess exposing it as an own property of the patched function will solve the issue.~~

Code checking for that prop is here:
https://github.com/wesleytodd/express-openapi/blob/a55f50f71fed8f9f60e68a9c2c870a1e504043f0/lib/generate-doc.js#L73

Edit: router handler has some properties that get hidden when patching it. By having them exposed as properties of the patched function these libs that make use of them could work properly.

I'm going to create a PR for this one with a proposal @Starefossen

cc: @pichlermarc

Starefossen added bug Something isn't working triage labels Feb 21, 2024

pichlermarc transferred this issue from open-telemetry/opentelemetry-js Feb 22, 2024

pichlermarc added the pkg:instrumentation-express label Feb 22, 2024

dyladan added information-requested needs:reproducer This bug/feature is in need of a minimal reproducer labels Feb 28, 2024

pichlermarc added priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies and removed triage information-requested labels Mar 6, 2024

david-luna self-assigned this Apr 22, 2024

david-luna removed the needs:reproducer This bug/feature is in need of a minimal reproducer label Apr 22, 2024

david-luna linked a pull request Apr 22, 2024 that will close this issue

fix(instr-express): keep hidden properties in layer handlers #2137

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Node - TypeError: Cannot read properties of undefined (reading 'forEach') #1950

Node - TypeError: Cannot read properties of undefined (reading 'forEach') #1950

Starefossen commented Feb 21, 2024

dyladan commented Feb 28, 2024

Starefossen commented Feb 28, 2024 •

edited

pichlermarc commented Mar 6, 2024 •

edited

Starefossen commented Mar 6, 2024 •

edited

david-luna commented Apr 22, 2024

david-luna commented Apr 22, 2024 •

edited

Node - TypeError: Cannot read properties of undefined (reading 'forEach') #1950

Node - TypeError: Cannot read properties of undefined (reading 'forEach') #1950

Comments

Starefossen commented Feb 21, 2024

What happened?

Steps to Reproduce

Expected Result

Actual Result

Additional Details

OpenTelemetry Setup Code

package.json

Relevant log output

dyladan commented Feb 28, 2024

Starefossen commented Feb 28, 2024 • edited

pichlermarc commented Mar 6, 2024 • edited

Starefossen commented Mar 6, 2024 • edited

david-luna commented Apr 22, 2024

david-luna commented Apr 22, 2024 • edited

Starefossen commented Feb 28, 2024 •

edited

pichlermarc commented Mar 6, 2024 •

edited

Starefossen commented Mar 6, 2024 •

edited

david-luna commented Apr 22, 2024 •

edited