XRay not working #832
Comments
|
@SamirHafez @schanjr do you have any insights that you could share here? |
|
More information! After playing with the example (https://github.com/open-telemetry/opentelemetry-ruby/tree/main/examples/http) and doing some instrumentation (logging, haha). Seems like environment var should be OTEL_PROPAGATORS: 'xray', well in the simpler example this seems to be happy with faraday instrumented client. (I can fork and push this example if that'd help.) However with the rack (auto) instrumentation for installed i'm getting the stack trace: This is happening for all health checks from the Kubernetes readiness/liveness checks. So in the case of a client that isn't instrumented, is there a different method that I should be using? |
|
Even more info! Modified the server example to be a little simpler and include XRay: When curling: Seems like the same error. If I add a "return context unless header" between like 40 and 41 on https://github.com/open-telemetry/opentelemetry-ruby/blob/main/propagator/xray/lib/opentelemetry/propagator/xray/text_map_propagator.rb#L41 I wonder if I'm allowed to PR this? |
|
Yes! We would appreciate any contributions. Please open PR! 🥳 |
|
|
|
Unfortunately it's still angry. Collector logs(just a subset, lots of the same thing over and over): Wondering if the contrib package isn't picking up the correct ID? Hmm. So that PR definitely fixes an issue, but the fundamental problem still exists. (Traces aren't making it to XRay) |
|
AWS XRay has specific requirements around trace ID generation - see https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html for details. The default trace ID generation in the SDK is incompatible with AWS XRay requirements, but a valid XRay trace ID can be encoded as a valid OpenTelemetry trace ID, so you just need to replace the generator. I apologize for the complete lack of documentation (at least, that I could find), but you can plugin an alternative ID generator in your |
|
Threw together a PR that seems to work. I'm a ruby noob, so, any guidance is appreciated. Have questions in the PR (#840) hopefully someone point me in the correct direction here. |
|
@scar-lovevery late to the conversation, I was able to successfully create correct trace ids, but was not able to get it pushed to a real AWS environment in the Jruby world. Might be slightly different from your use case, I had issues with the collector. My last update on pushing the traces to the collector was like this: OTLP receiver --> Xray Exporter (broken in JRUBY. protobuf gem is not working, protocolbuffers/protobuf#7923) Jaeger receiver --> Xray Exporter (broken at the ADOT collector. It does not transform trace id format to Xray format. aws-observability/aws-otel-collector#562) I got around with creating the correct traces doing some manual instrumentation at the Rack level, mainly utilizing the xray propagator extract and inject methods manually. # a middleware for rack
# Trace::GlobalPropagator is using OpenTelemetry::Propagator::XRay::TextMapPropagator.new
def call(env)
context = nil
trace_id_provided = false
# if trace id is given at the header level, extract it and use it
unless env[Trace::Constants::RACK_CONTEXT_KEY].nil?
context = Trace::GlobalPropagator.extract(env)
trace_id_provided = true
end
status = 200
headers = {}
response_body = ''
# For attribute naming, see
# https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/data-semantic-conventions.md#http-server
# Span kind MUST be `:server` for a HTTP server span
span = @tracer.start_span(
env['PATH_INFO'],
attributes: {
'component' => 'http',
'http.method' => env['REQUEST_METHOD'],
'http.route' => env['PATH_INFO'],
'http.url' => env['REQUEST_URI']
},
kind: :server,
with_parent: context
)
OpenTelemetry::Trace.with_span(span) do |child_span|
# if trace id was not given, generate one using the inject method, and use it
unless trace_id_provided
id = {}
Trace::GlobalPropagator.inject(id)
env[Trace::Constants::RACK_CONTEXT_KEY] = id[Trace::Constants::OUTGOING_TRACE_ID_HEADER_KEY]
end
# Run application stack.
status, headers, response_body = @app.call(env)
child_span.set_attribute('http.status_code', status)
end
[status, headers, response_body]
ensure
span&.finish
endrequire 'opentelemetry-propagator-xray'
module Trace
class GlobalPropagator
@propagator ||= OpenTelemetry::Propagator::XRay::TextMapPropagator.new
class << self
def extract(carrier,
context: OpenTelemetry::Context.current,
getter: OpenTelemetry::Context::Propagation.rack_env_getter)
@propagator.extract(carrier, context: context, getter: getter)
end
def inject(carrier,
context: OpenTelemetry::Context.current,
setter: OpenTelemetry::Context::Propagation.text_map_setter)
@propagator.inject(carrier, context: context, setter: setter)
end
end
end
end |
|
@schanjr Using: Fixes this issue for me. I'm able to get traces from my instrumented app through the aws-otel-collector into XRay. I wonder if I'm having a similar issue to you and haven't noticed? It's possible. Since the root of my issue is fixed, I'm going to close this. |
Not sure if it's related to:
open-telemetry/opentelemetry-collector#3405
or
aws-observability/aws-otel-collector#537
Seems like both of these indicate that the trace id wasnt correctly generated, which i'm also guessing is the issue i'm facing.
I'm getting a similar behavior to the above issues.
When sending data to the aws-otel-collector I'm getting the "Permanent error: SerializationException" much like they are. I have tried many different configurations.
I know that the aws-otel-collector is configured correctly if I run a java example, I get traces forwarded to XRay:
I'm attempting to use the XRay propagator (0.18.0)
Tried(environment var) OTEL_PROPAGATORS:
value: xray,tracecontext,baggage
&
value: tracecontext,baggage,xray
& (this one is very angry)
value: xray
Is that all that needs to be configured to use that?
The ruby app is configured correctly to export using otlp. With the collector I'm able to print the trace data if I just use the logging exporter, just doesnt work with XRay.
Wondering if the XRay propagator works? Or more likely I'm using it incorrectly? I'd love some help or ideas to try.
Seems like it's not sending 'X-Amzn-Trace-Id' in the logs of the collector, Is there an example somewhere how to configure the ruby sdk with XRay correctly?
The text was updated successfully, but these errors were encountered: