Error: NoCredentialProviders when using AWS X-Ray Exporter

[smarza]

Describe the bug
I am encountering an issue with the AWS OpenTelemetry Collector when trying to export traces to AWS X-Ray. The collector fails with a NoCredentialProviders: no valid providers in chain error. Below are the details of my setup and the error logs.

Steps to reproduce

Docker Compose Configuration

version: '3.9'

services:
  aws-otel-collector:
    image: public.ecr.aws/aws-observability/aws-otel-collector:latest
    command: ["--config=/etc/otel-agent-config.yaml"]
    environment:
      - AWS_PROFILE=erm-development
      - AWS_REGION=eu-west-1
    volumes:
      - ./config.yaml:/etc/otel-agent-config.yaml
      - ${USERPROFILE}/.aws:/home/aoc/.aws
    ports:
      - "4317:4317"
      - "13133:13133"

OpenTelemetry Collector Configuration (config.yaml)

extensions:
  health_check:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch/traces:
    timeout: 1s
    send_batch_size: 50
  batch/metrics:
    timeout: 60s

exporters:
  logging:
    loglevel: debug
  awsxray:
    region: 'eu-west-1'
  awsemf:
    region: 'eu-west-1'
    log_group_name: 'log_group_name'
    log_stream_name: 'log_stream_name'
    namespace: 'namespace'

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch/traces]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch/metrics]
      exporters: [awsemf]

  extensions: [health_check]
  telemetry:
    logs:
      level: debug

Error Logs

Here are the relevant logs from the AWS OpenTelemetry Collector:

2024-05-16T11:02:15.002Z        error   exporterhelper/common.go:292    Exporting failed. Rejecting data.       {"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors", "rejected_items": 1}

AWS ~/.aws/config:

[profile erm-development]
sso_start_url = https://xxxxxxx/start
sso_region = eu-west-1
sso_account_id = xxxxxxxxxxxx
sso_role_name = xxxdev-Dev

AWS ~/.aws/credentials:

[iam-user]
aws_access_key_id=XXXXXXXXXXXXXXXX
aws_secret_access_key=xxxxxx
aws_session_token=xxxxx
region=eu-west-1

[erm-dev]
region=eu-west-1
role_arn=arn:aws:iam::xxxxxxxxxxxx:role/DeveloperRole
source_profile=iam-user
toolkit_artifact_guid=xxxxxxxx

Make sure you have logged with aws sso login using the correct profile.

Start the Collector with Docker Compose:

podman-compose down
podman-compose up -d

Send a Trace:

You can use otel-cli or a similar tool to send a trace to the collector. Below is an example using otel-cli:

otel-cli span --service-name test-service --endpoint http://localhost:4317 --name "test-span"

Check Logs

podman-compose logs -f aws-otel-collector

What did you expect to see?
I expected the AWS OpenTelemetry Collector to successfully authenticate using the provided AWS profile (erm-development) and to send trace data to AWS X-Ray without encountering any credential errors.

What did you see instead?

Instead of the expected behavior, I encountered the following error in the logs:

2024-05-16T11:02:15.002Z        error   exporterhelper/common.go:292    Exporting failed. Rejecting data.       {"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors", "rejected_items": 1}

Environment
AWS OpenTelemetry Collector version: latest
Operating System: Windows 11

Client: Podman Engine
Version: 5.0.1
API Version: 5.0.1
Go Version: go1.22.1
Git Commit: 946d055df324e4ed6c1e806b561af4740db4fea9
Built: Mon Apr 1 16:15:15 2024
OS/Arch: windows/amd64

Server: Podman Engine
Version: 5.1.0-dev-4817811cb
API Version: 5.1.0-dev-4817811cb
Go Version: go1.21.9
Built: Wed Apr 17 02:00:00 2024
OS/Arch: linux/amd64

[smarza]

I managed to get it working by adding some environment variables in the collector container, specifying the locations for AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE.

The Docker Compose file now looks like this:

version: '3.9'

services:
  aws-otel-collector:
    image: public.ecr.aws/aws-observability/aws-otel-collector:latest
    command: ["--config=/etc/otel-agent-config.yaml"]
    environment:
      - AWS_PROFILE=erm-dev
      - AWS_REGION=eu-west-1
      - AWS_SDK_LOAD_CONFIG=1
      - AWS_CONFIG_FILE=/home/aoc/.aws/config
      - AWS_SHARED_CREDENTIALS_FILE=/home/aoc/.aws/credentials
    volumes:
      - ./config.yaml:/etc/otel-agent-config.yaml
      - ${USERPROFILE}/.aws/config:/home/aoc/.aws/config:ro
      - ${USERPROFILE}/.aws/credentials:/home/aoc/.aws/credentials:ro
    ports:
      - "4317:4317"     # Port for OTLP gRPC
      - "13133:13133"   # Port for health check

I noticed that the getting started documentation does not mention the need to specify these variables.

Does anyone know what is the reason for requiring this extra configuration?