Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aws X-Ray sampling rules are being ignored #2698

Open
StiviiK opened this issue Mar 28, 2024 · 1 comment
Open

Aws X-Ray sampling rules are being ignored #2698

StiviiK opened this issue Mar 28, 2024 · 1 comment

Comments

@StiviiK
Copy link

StiviiK commented Mar 28, 2024

Describe the bug
Hi, i have deployed the AWS-OTEL-COLLECTOR as a sidecar for my regular Java container. The Java container uses the automatic-instrumentation via the javaagent.
I wan't to disable sampling for my health check endpoints, so i configured that within the AWS Portal under Cloud Watch -> Settings -> Traces -> X-Ray Sampling rules. (At first i tried to limit to a specific path /actuator/* but as this did nothing i tried limiting it completly)
image
But traces are still comming in as normal on X-Ray. Also as you can see on the Trend it seems like these rules do nothing at all.

According to the AWS Documentation the otel-collector supports the console sampling configuration.
"You can use the AWS X-Ray console to configure sampling rules for your services. The X-Ray SDK and AWS services that support active tracing with sampling configuration use sampling rules to determine which requests to record."

Steps to reproduce
I am running ECS Tasks on a EC2 instance (ec2 is not that relevant i guess). Here is a snipped from the most important configurations:

Task configuration:

container_definitions = jsonencode([
    {
      "healthCheck" : {
        "command" : ["/healthcheck"],
        "interval" : 5,
        "timeout" : 6,
        "retries" : 5,
        "startPeriod" : 1
      },
      "command" : [
        # https://aws-otel.github.io/docs/adot-collector-using-ecs#understanding-your-configuration
        "--config=/etc/ecs/ecs-cloudwatch-xray.yaml"
      ],
      "image" : "amazon/aws-otel-collector",
      "name" : "aws-otel-collector"
    },
    {
      "image" : "${var.aws_ecr_host}:${var.image_tag}",
      "links" : [
        "aws-otel-collector" # Can probably remove this when using awsvpc for network_mode
      ],
      [...]
    }
]

Application Dockerfile with auto-instrumentation:

# AWS Distro for OpenTelemetry
ADD --chown=nonroot:nonroot ${AWS_OTEL_SOURCE} /opt/aws-opentelemetry-agent.jar
ENV OTEL_RESOURCE_ATTRIBUTES "service.namespace=WorkflowPlatform,service.name=WFPBackend"
ENV OTEL_EXPORTER_OTLP_ENDPOINT "http://aws-otel-collector:4317"
ENV OTEL_TRACES_EXPORTER "otlp"
ENV OTEL_METRICS_EXPORTER "otlp"
ENV OTEL_TRACES_SAMPLER "xray"
# ENV OTEL_PROPAGATORS "tracecontext,baggage,xray"

# Run the application
ENTRYPOINT ["java", "-javaagent:/opt/aws-opentelemetry-agent.jar", "org.springframework.boot.loader.launch.JarLauncher"]

Environment
Collector configuration file: https://github.com/aws-observability/aws-otel-collector/blob/main/config/ecs/ecs-cloudwatch-xray.yaml

@StiviiK
Copy link
Author

StiviiK commented Apr 2, 2024

Update: I have enabled and configured the awsproxy extension but it still seems to get ignored.
This is the documentation I followed: https://aws-otel.github.io/docs/getting-started/remote-sampling.
My config is basically the same (as above) but adapted with the extension configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant