Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistency in lambda instrumentation across languages. #831

Open
biswajit-nanda opened this issue Aug 5, 2023 · 0 comments
Open

Inconsistency in lambda instrumentation across languages. #831

biswajit-nanda opened this issue Aug 5, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@biswajit-nanda
Copy link

biswajit-nanda commented Aug 5, 2023 •
edited

I have three lambda functions exactly with the same business logic written in java, node.js and python instrumented with the latest version of OpenTelemetry lambda auto-instrumentation packages. Each of these lambda functions make two outbound HTTP calls after invocation.

All the three lambda functions are being invoked exactly same way via API Gateway triggers. While they are being instrumented successfully, the spans generated from them are in completely different ways triggering inconsistencies.

Steps to reproduce
a. All the three Lambda functions are attached.
b. Just auto-instrument them with latest version of opentelemetry auto instrumentation libraries for lambda and configure them to send the traces to an Otel Collector.
c. Configure logging exporter in the Otel Collector for trace.
e. Configure API Gateway trigger for each of the lambda functions.
f. Make a single invocation to each lambda function via the API gateway trigger.
g. Check the OpenTelemetry collector logs.

What I expect to See:
a. All three lambdas are expected to generate one SERVER type span for the handler triggered by the HTTP gateway and then generate at least two CLIENT type spans for each of the two HTTP outbound calls made.
b. All the spans generated should be part of the same TraceID.

What I really see:
A. Java Lambda:
a. Three SERVER spans are generated from the API gateway trigger, each showing a completely different TraceID. One from instrumentation scope io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha and two from instrumentation scope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha.
b. Two CLIENT spans for two outbound HTTP calls generated from io.opentelemetry.apache-httpclient-4.0 1.26.0-alpha with the same TraceID as the SERVER Span generated from io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha.

Below is the Otel Collector log snippet for a single invocation:

Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 3, "spans": 5}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z debug otlphttpexporter@v0.77.0/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/metrics"}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z info ResourceSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.apache-httpclient-4.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 4fbdb5019fb40322
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : GET
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Client
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.384374039 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.443981299 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.method: Str(GET)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.version: Str(1.1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.url: Str(http://internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:30/ecs/ec2/java/hello)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.name: Str(http)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.status_code: Int(200)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.peer.port: Int(30)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.response_content_length: Int(76)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 806aede6fce14c90
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : GET
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Client
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.48442222 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.503998599 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.method: Str(GET)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.version: Str(1.1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.url: Str(http://ec2-52-43-189-49.us-west-2.compute.amazonaws.com/json/products.json)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.name: Str(http)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.status_code: Int(200)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.response_content_length: Int(2485)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.365332 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.504745744 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ResourceSpans #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 91228e8b067f5e2d4e92607bda5cd4ba
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 095c02b9b8f0090d
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.365288 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.64412236 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Links:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: SpanLink #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Trace ID: 64cd7295464c71963515de8403d5b235
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> ID: 79fb02975ab75964
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> TraceState:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> DroppedAttributesCount: 0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Attributes::
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> source: Str(x-ray-env)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ResourceSpans #2
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 571cc1e464cf9a5a8edaecfc3c25a6b8
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 0ba08cd04d96c2bf
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.365175 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.706600342 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Links:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: SpanLink #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Trace ID: 64cd7295464c71963515de8403d5b235
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> ID: 79fb02975ab75964
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> TraceState:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> DroppedAttributesCount: 0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Attributes::
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> source: Str(x-ray-env)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:20.006Z debug otlphttpexporter@v0.77.0/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}

B. Node.js Lambda:
a. No SERVER type spans are being generated at all.
b. Two INTERNAL spans from instrumentation scope @opentelemetry/instrumentation-net 0.32.0 and two CLIENT spans from instrumentation scope @opentelemetry/instrumentation-http 0.41.1 are being generated. The TraceId of one INTERNAL span matches with one of the CLIENT spans and the TraceId of another INTERNAL span matches with the TraceID of the other CLIENT span.

Below is the Otel Collector log snippet for a single invocation:

Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.841Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 4}
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.842Z info ResourceSpans #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.name: Str(bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.language: Str(nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.version: Str(1.15.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.provider: Str(aws)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.platform: Str(aws_lambda)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.region: Str(us-west-2)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.name: Str(bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.version: Str($LATEST)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.namespace: Str(aws-lambda)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.pid: Int(8)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.executable.name: Str(/var/lang/bin/node)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.executable.path: Str(/var/lang/bin/node)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.command_args: Slice(["/var/lang/bin/node","--expose-gc","--max-http-header-size","81920","--max-semi-space-size=6","--max-old-space-size=116","/var/runtime/index.mjs"])
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.runtime.version: Str(18.16.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.runtime.name: Str(nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.runtime.description: Str(Node.js)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.command: Str(/var/runtime/index.mjs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.owner: Str(sbx_user1051)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.port: Int(443)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope @opentelemetry/instrumentation-net 0.32.0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 33bd0b8f9241c6c77b8fe9b43954a1a1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID : 365a6c4cc24d2415
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : 23382ec398a02d05
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : tcp.connect
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Internal
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.536999936 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:42:24.876628992 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Int(80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.10.214)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.ip: Str(169.254.76.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.port: Int(53462)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 516fe8d1ccd786cf6fc7b1e44180c3c5
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID : c3f4123cfa5757fa
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : eea68d81fee6bea3
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : tcp.connect
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Internal
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.536 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:42:24.877544192 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Str(40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.40.22)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.ip: Str(169.254.76.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.port: Int(37510)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope @opentelemetry/instrumentation-http 0.41.1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 33bd0b8f9241c6c77b8fe9b43954a1a1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : 365a6c4cc24d2415
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : GET
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Client
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.536999936 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:42:24.879968768 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.url: Str(http://ec2-52-43-189-49.us-west-2.compute.amazonaws.com/json/books.json)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.method: Str(GET)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.target: Str(/json/books.json)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.host: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com:80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.10.214)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Int(80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.response_content_length_uncompressed: Int(848)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_code: Int(200)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_text: Str(OK)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.flavor: Str(1.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 516fe8d1ccd786cf6fc7b1e44180c3c5
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : c3f4123cfa5757fa
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : GET
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Client
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.535000064 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:43:06.36994816 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.url: Str(http://internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:40/ecs/ec2/nodejs/hello)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.method: Str(GET)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.target: Str(/ecs/ec2/nodejs/hello)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.host: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.40.22)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Int(40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_code: Int(200)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_text: Str(OK)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.flavor: Str(1.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.842Z debug otlphttpexporter@v0.77.0/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}

B. Python Lambda:
a. One SERVER type span is generated corresponding to the API Gateway trigger from instrumentation scope opentelemetry.instrumentation.aws_lambda 0.40b0.
b. No CLIENT type spans are generated at all. So the two outbound HTTP calls are not being tracked.

Below is the Otel Collector log snippet for a single invocation:

Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.829Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.830Z info ResourceSpans #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource SchemaURL:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource attributes:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.language: Str(python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.version: Str(1.19.0)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.region: Str(us-west-2)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.provider: Str(aws)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.name: Str(bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.version: Str($LATEST)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.instance: Str(2023/08/04/[$LATEST]6e7687d2ff6a4c7e8bc577dacf4625b4)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.namespace: Str(aws-lambda)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.name: Str(bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.auto.version: Str(0.40b0)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.port: Int(443)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope opentelemetry.instrumentation.aws_lambda 0.40b0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : e5508d4afeb8280e1370ad838c9dd044
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID :
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : 09a7ddff9c33a5b1
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : lambda_function.lambda_handler
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Server
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:39:06.313833645 +0000 UTC
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:39:06.448371414 +0000 UTC
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.execution: Str(a1581204-a256-4c00-937b-a4d6428d4555)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.trigger: Str(http)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.method: Str(GET)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.user_agent: Str(Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.scheme: Str(https)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.name: Str(ssn9gz3t4l.execute-api.us-west-2.amazonaws.com)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.route: Str(/bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.target: Str(/bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_code: Int(200)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.829Z debug otlphttpexporter@v0.77.0/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}

What version of collector/language SDK version did you use?
Collector: Latest Version
Auto instrumentation library: Latest version
Runtime: Java 11, Node.js 18.x, Python 3.11

sourcecode.zip
otelcol_log_snippet.zip
@biswajit-nanda biswajit-nanda added the bug Something isn't working label Aug 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@biswajit-nanda