http.response.status_code vs 'response code' in X-Ray

[ig16022]

Describe the question

Hello, we are attempting to push OpenTelemetry traces from a .NET 8.0 application running in ECS Fargate with AWS Distro for OpenTelemetry running as a sidecar for all services. I am trying to understand the relationship between X-Ray trace information and the metadata that's coming in from our services. As an example, we have "http.response.status_code": 200 in the metadata of a segment, but the X-Ray 'Response Code' in the user interface is always empty. Please see attached screenshot.

We have configured the application to use AWSXRayPropagator, and XRay Trace Id:
.AddXRayTraceId()
Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());

Here is an example of a complete trace displayed by X-Ray:

{
"Id": "1-ee967e4d-1856e32d1b0302ea3f6c599f",
"Duration": 0.3,
"LimitExceeded": false,
"Segments": [
{
"Id": "28f47fa7ea0ecebf",
"Document": {
"id": "28f47fa7ea0ecebf",
"name": "Calendar",
"start_time": 1705694954.8016534,
"trace_id": "1-ee967e4d-1856e32d1b0302ea3f6c599f",
"end_time": 1705694954.8029037,
"parent_id": "d5809d2a3518e5b5",
"fault": false,
"error": false,
"throttle": false,
"aws": {
"account_id": "MY_ACCOUNT_ID",
"ecs": {
"task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/515ef0272285473bbd7d7ab5ed56490e",
"availability_zone": "us-west-1a",
"cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"task_family": "ecs-fargate-my_app-api-calendar-task-definition",
"launch_type": "fargate"
},
"xray": {
"auto_instrumentation": false,
"sdk_version": "1.7.0",
"sdk": "opentelemetry for dotnet"
},
"cloudwatch_logs": [
{
"log_group": "/ecs/ecs-fargate-my_app-api-calendar-task-definition",
"arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition"
}
]
},
"metadata": {
"default": {
"otel.resource.aws.log.stream.arns": [
"arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition:log-stream:ecs/mpa-my_app-calendar-v1_0-container/515ef0272285473bbd7d7ab5ed56490e"
],
"otel.resource.service.instance.id": "ip-10-225-0-154",
"otel.resource.aws.log.group.names": [
"/ecs/ecs-fargate-my_app-api-calendar-task-definition"
],
"otel.resource.aws.ecs.cluster.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"otel.resource.aws.ecs.launchtype": "fargate",
"url.scheme": "http",
"otel.resource.aws.ecs.task.revision": "28",
"otel.resource.service.name": "Calendar",
"otel.resource.aws.log.group.arns": [
"arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition"
],
"server.address": "internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com",
"http.request.method": "GET",
"otel.resource.cloud.platform": "aws_ecs",
"otel.resource.cloud.region": "us-west-1",
"http.response.status_code": 200,
"otel.resource.aws.ecs.task.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/515ef0272285473bbd7d7ab5ed56490e",
"otel.resource.telemetry.sdk.name": "opentelemetry",
"otel.resource.service.version": "1.0.0.0",
"otel.resource.cloud.availability_zone": "us-west-1a",
"network.protocol.version": "1.1",
"otel.resource.aws.ecs.task.family": "ecs-fargate-my_app-api-calendar-task-definition",
"otel.resource.cloud.account.id": "MY_ACCOUNT_ID",
"otel.resource.telemetry.sdk.version": "1.7.0",
"url.path": "/calendar/index.html",
"otel.resource.telemetry.sdk.language": "dotnet",
"otel.resource.aws.log.stream.names": [
"ecs/mpa-my_app-calendar-v1_0-container/515ef0272285473bbd7d7ab5ed56490e"
],
"otel.resource.cloud.provider": "aws"
}
},
"service": {
"version": "1.0.0.0"
},
"origin": "AWS::ECS::Fargate"
}
},
{
"Id": "9266d561577b88d1",
"Document": {
"id": "9266d561577b88d1",
"name": "Dashboard",
"start_time": 1705694954.6465578,
"trace_id": "1-ee967e4d-1856e32d1b0302ea3f6c599f",
"end_time": 1705694954.9463103,
"fault": false,
"error": false,
"throttle": false,
"aws": {
"account_id": "MY_ACCOUNT_ID",
"ecs": {
"task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be",
"availability_zone": "us-west-1a",
"cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"task_family": "ecs-fargate-my_app-api-dashboard-task-definition",
"launch_type": "fargate"
},
"xray": {
"auto_instrumentation": false,
"sdk_version": "1.7.0",
"sdk": "opentelemetry for dotnet"
},
"cloudwatch_logs": [
{
"log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition",
"arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition"
}
]
},
"metadata": {
"default": {
"otel.resource.aws.log.stream.arns": [
"arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition:log-stream:ecs/mpa-my_app-dashboard-v1_0-container/a51a6e88ad5341a4b2374774571096be"
],
"otel.resource.host.name": "ip-10-225-0-238.us-west-1.compute.internal",
"otel.resource.service.instance.id": "ip-10-225-0-238",
"otel.resource.aws.log.group.names": [
"/ecs/ecs-fargate-my_app-api-dashboard-task-definition"
],
"otel.resource.aws.ecs.cluster.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"otel.resource.aws.ecs.launchtype": "fargate",
"url.scheme": "http",
"otel.resource.aws.ecs.task.revision": "36",
"otel.resource.service.name": "Dashboard",
"otel.resource.aws.log.group.arns": [
"arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition"
],
"server.address": "l04v5pn392.execute-api.us-west-1.amazonaws.com",
"url.query": "?userId=22",
"http.request.method": "GET",
"otel.resource.cloud.platform": "aws_ecs",
"otel.resource.os.type": "linux",
"otel.resource.cloud.region": "us-west-1",
"http.response.status_code": 200,
"otel.resource.aws.ecs.task.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be",
"otel.resource.telemetry.sdk.name": "opentelemetry",
"user_agent.original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"otel.resource.service.version": "1.0.0.0",
"otel.resource.cloud.availability_zone": "us-west-1a",
"network.protocol.version": "1.1",
"otel.resource.aws.ecs.task.family": "ecs-fargate-my_app-api-dashboard-task-definition",
"otel.resource.cloud.account.id": "MY_ACCOUNT_ID",
"otel.resource.telemetry.sdk.version": "1.7.0",
"url.path": "/dashboard/api/v1.0/dashboard",
"http.route": "api/v{version:apiVersion}/dashboard",
"otel.resource.telemetry.sdk.language": "dotnet",
"otel.resource.aws.log.stream.names": [
"ecs/mpa-my_app-dashboard-v1_0-container/a51a6e88ad5341a4b2374774571096be"
],
"otel.resource.cloud.provider": "aws"
}
},
"service": {
"version": "1.0.0.0"
},
"origin": "AWS::ECS::Fargate",
"subsegments": [
{
"id": "fe055153f19488ea",
"name": "Get Online Application Data",
"start_time": 1705694954.8839161,
"end_time": 1705694954.8855886,
"fault": false,
"error": false,
"throttle": false,
"aws": {
"account_id": "MY_ACCOUNT_ID",
"ecs": {
"task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be",
"availability_zone": "us-west-1a",
"cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"task_family": "ecs-fargate-my_app-api-dashboard-task-definition",
"launch_type": "fargate"
},
"xray": {
"auto_instrumentation": false,
"sdk_version": "1.7.0",
"sdk": "opentelemetry for dotnet"
},
"cloudwatch_logs": [
{
"log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition",
"arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition"
}
]
}
},
{
"id": "3c7c5884e1f95e50",
"name": "GET",
"start_time": 1705694954.7782323,
"end_time": 1705694954.798333,
"fault": false,
"error": false,
"throttle": false,
"aws": {
"account_id": "MY_ACCOUNT_ID",
"ecs": {
"task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be",
"availability_zone": "us-west-1a",
"cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"task_family": "ecs-fargate-my_app-api-dashboard-task-definition",
"launch_type": "fargate"
},
"xray": {
"auto_instrumentation": false,
"sdk_version": "1.7.0",
"sdk": "opentelemetry for dotnet"
},
"cloudwatch_logs": [
{
"log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition",
"arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition"
}
]
},
"metadata": {
"default": {
"server.address": "internal-my_app-dev-alb1-123499307.us-west-1.elb.amazonaws.com",
"http.request.method": "GET",
"network.protocol.version": "1.1",
"http.response.status_code": 301,
"url.full": "http://internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com/calendar/?userId=22"
}
},
"namespace": "remote"
},
{
"id": "cfd82c6836bc3bc1",
"name": "Get Navigatables Data",
"start_time": 1705694954.8856056,
"end_time": 1705694954.8867292,
"fault": false,
"error": false,
"throttle": false,
"aws": {
"account_id": "MY_ACCOUNT_ID",
"ecs": {
"task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be",
"availability_zone": "us-west-1a",
"cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"task_family": "ecs-fargate-my_app-api-dashboard-task-definition",
"launch_type": "fargate"
},
"xray": {
"auto_instrumentation": false,
"sdk_version": "1.7.0",
"sdk": "opentelemetry for dotnet"
},
"cloudwatch_logs": [
{
"log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition",
"arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition"
}
]
}
},
{
"id": "d5809d2a3518e5b5",
"name": "GET",
"start_time": 1705694954.8006032,
"end_time": 1705694954.8034613,
"fault": false,
"error": false,
"throttle": false,
"aws": {
"account_id": "MY_ACCOUNT_ID",
"ecs": {
"task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/a51a6e88ad5341a4b2374774571096be",
"availability_zone": "us-west-1a",
"cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"task_family": "ecs-fargate-my_app-api-dashboard-task-definition",
"launch_type": "fargate"
},
"xray": {
"auto_instrumentation": false,
"sdk_version": "1.7.0",
"sdk": "opentelemetry for dotnet"
},
"cloudwatch_logs": [
{
"log_group": "/ecs/ecs-fargate-my_app-api-dashboard-task-definition",
"arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-dashboard-task-definition"
}
]
},
"metadata": {
"default": {
"server.address": "internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com",
"http.request.method": "GET",
"network.protocol.version": "1.1",
"http.response.status_code": 200,
"url.full": "http://internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com/calendar/index.html"
}
},
"namespace": "remote"
}
]
}
},
{
"Id": "d8281f3d5cf86a7f",
"Document": {
"id": "d8281f3d5cf86a7f",
"name": "Calendar",
"start_time": 1705694954.7947981,
"trace_id": "1-ee967e4d-1856e32d1b0302ea3f6c599f",
"end_time": 1705694954.796209,
"parent_id": "3c7c5884e1f95e50",
"fault": false,
"error": false,
"throttle": false,
"aws": {
"account_id": "MY_ACCOUNT_ID",
"ecs": {
"task_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/2cadcc3b1ab1465298d69a5c9675ce34",
"availability_zone": "us-west-1c",
"cluster_arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"task_family": "ecs-fargate-my_app-api-calendar-task-definition",
"launch_type": "fargate"
},
"xray": {
"auto_instrumentation": false,
"sdk_version": "1.7.0",
"sdk": "opentelemetry for dotnet"
},
"cloudwatch_logs": [
{
"log_group": "/ecs/ecs-fargate-my_app-api-calendar-task-definition",
"arn": "arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition"
}
]
},
"metadata": {
"default": {
"otel.resource.aws.log.stream.arns": [
"arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition:log-stream:ecs/mpa-my_app-calendar-v1_0-container/2cadcc3b1ab1465298d69a5c9675ce34"
],
"otel.resource.service.instance.id": "ip-10-225-2-60",
"otel.resource.aws.log.group.names": [
"/ecs/ecs-fargate-my_app-api-calendar-task-definition"
],
"otel.resource.aws.ecs.cluster.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:cluster/MY_APP-Development-Cluster",
"otel.resource.aws.ecs.launchtype": "fargate",
"url.scheme": "http",
"otel.resource.aws.ecs.task.revision": "28",
"otel.resource.service.name": "Calendar",
"otel.resource.aws.log.group.arns": [
"arn:aws:logs:us-west-1:MY_ACCOUNT_ID:log-group:/ecs/ecs-fargate-my_app-api-calendar-task-definition"
],
"server.address": "internal-my_app-dev-alb1-287599307.us-west-1.elb.amazonaws.com",
"url.query": "?userId=22",
"http.request.method": "GET",
"otel.resource.cloud.platform": "aws_ecs",
"otel.resource.cloud.region": "us-west-1",
"http.response.status_code": 301,
"otel.resource.aws.ecs.task.arn": "arn:aws:ecs:us-west-1:MY_ACCOUNT_ID:task/MY_APP-Development-Cluster/2cadcc3b1ab1465298d69a5c9675ce34",
"otel.resource.telemetry.sdk.name": "opentelemetry",
"otel.resource.service.version": "1.0.0.0",
"otel.resource.cloud.availability_zone": "us-west-1c",
"network.protocol.version": "1.1",
"otel.resource.aws.ecs.task.family": "ecs-fargate-my_app-api-calendar-task-definition",
"otel.resource.cloud.account.id": "MY_ACCOUNT_ID",
"otel.resource.telemetry.sdk.version": "1.7.0",
"url.path": "/calendar/",
"otel.resource.telemetry.sdk.language": "dotnet",
"otel.resource.aws.log.stream.names": [
"ecs/mpa-my_app-calendar-v1_0-container/2cadcc3b1ab1465298d69a5c9675ce34"
],
"otel.resource.cloud.provider": "aws"
}
},
"service": {
"version": "1.0.0.0"
},
"origin": "AWS::ECS::Fargate"
}
}
]
}

What did you expect to see?
Expecting to see Response Code, Request URl, Request method, etc. populated, but these are always emtpy in X-Ray user interface. These values can be seen in the metadata only.

Environment
Describe any aspect of your environment.
If this is related to a deployment of the ADOT Collector please
provide your Collector config file.

I have tried several variations of the predefined collector config files, but the results are exactly the same.

"--config=/etc/ecs/ecs-default-config.yaml"
--config=/etc/ecs/ecs-xray.yaml
"--config=/etc/ecs/container-insights/otel-task-metrics-config.yaml"

[rafar55]

@ig16022 we are having exactly the same issue as you. We're you able to find a solution to this?

[zeva-us]

I am facing the same issue. It looks like the format expected by Xray is like this (directly in the segment document) :

"http": { "request": { "url": "http://certificates.godaddy.com/repository/gdig2.crt", "method": "GET" }, "response": { "status": 200, "content_length": 0 } },

Meanwhile the collector is producing the http information in the metadata:
"http.request.method": "POST",
"http.route": "GetCertificatesSummary",
"http.response.status_code": 200,

Let me know if this is a problem in our configuration or this is something that should be fixed in the collector itself.

[julianh-y]

I'm having the same issue.

According to Semantic Conventions for HTTP Spans the key-values are correct, so maybe there is an issue with the exporter to X-Ray?. X-Ray seems to want this information as nested objects...

[zeva-us]

I think the issue is in the collector, since it translates the Otel attributes ("http.request.method": "POST" for example) to metadata attributes and not in a form the Xray Portal understands it.

[jorgeparavicini]

I am having the same problem. Our .net OTel produces the attributes in the form {"http.request.method": "POST"} but according to the aws-otel-collector docs it expects the attribute to be http.method instead. So the issue seems to be in the collector

[zeva-us]

I am upvoting this issue again.

[kcrandall]

I'm having the same issue with this. ASP.netCore otel outputs with http.request.method and no meta data is showing up in xray (method, url, etc)

[ig16022]

aws-otel-collector team - Are you able to provide any feedback?

[jorgeparavicini]

Any news on this? I believe that the actual implementation of the collector is located here maybe we have to check there

[Elter71]

Here is commit that should fix this issue: open-telemetry/opentelemetry-collector-contrib@fe35716

Release: https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.96.0

[zeva-us]

Thank you Piotr! thats good new, but we should update aws-otel-collector to use the new fixed opentelemetry collector. is that correct?

[Elter71]

Thank you Piotr! thats good new, but we should update aws-otel-collector to use the new fixed opentelemetry collector. is that correct?

Unfortunately, we have to wait until aws-otel-collector releases a new version with that fix.

[EduardoMRB]

Also having the same problem. Hopefully that gets released soon 🙏

[Elter71]

Hi, all! The OTel Collector has finally been updated to the new version v0.98.0 in the latest release v0.39.0. Everything should be working now.