Otel Tracing API vs Tokio-Tracing?

[TommyCpp]

Problem

We are currently support two sets of APIs. The Opentelemetry API(Otel for short) and Tokio tracing API(through tracing-opentelemetry crate). Given the popularity of the tracing eco-system we certainly want to support an integration between tracing and opentelemetry. However, we need to decide whether to support Otel API on the long run.

Difference

This section will cover the difference between tracing API and opentelemetry API to my best knowledge. Feel free to chime in if you think I made a mistake and/or missed anything

Span

Both models has the concept of Span. They both have the concept of the SpanId, parent span. However, the metadata between those two models varies.

Tracing span's metdata usually include Metadata like callsite, verbosity level and a list of fields that contains key value pairs.

The Otel spans contains SpanContext, which covers the SpanId and esbalished the link between spans and it's partent span. Its attributes are similar to tracing span's fields. Additionaly it contains links to other spans and other metadata.

So far we have been handlding those difference by converting the fields from tracing API to Otel span using magic words. For example if user puts otel.kind as the key of the field. We will convert the value to span kind in Otel span. For the field that doesn't map to metadata of the Otel spans, we will emit them as Attribute of the span.

Event

Another important concept within tracing API is Event, which

represent single points in time where something occurred during the execution of a program

But it exists within the context of span and have a list of fields. This concept is almost identical with the Event in Otel spans.

[djc]

Problem

We are currently support two sets of APIs. The Opentelemetry API(Otel for short) and Tokio tracing API(through tracing-opentelemetry crate). Given the popularity of the tracing eco-system we certainly want to support an integration between tracing and opentelemetry. However, we need to decide whether to support Otel API on the long run.

Where is the problem? Is there any particular reason to bring this up now? My inclination would be to wait until someone shows up who wants to implement the Otel API, and then they can implement it and we'll review it?

[cijothomas]

The problem as I see it is the fact that there is (and will be) 2 APIs for instrumenting my app/library - the Otel API, and the Tokio/Tracing one. As an end user, I have to pick between two competing APIs. As of today, there is no written doc comparing both the options, so I requested for a writeup comparing these two. Once the comparison doc is available, it'd be easy for end users to pick which one they want. Also, it may be possible for the OTel Rust maintainers and the owners of Tokio/Tracing to explore if it is feasible to merge both into single one long term.

Additional background: This situation is very similar to what OpenTelemetry .NET encountered. OTel .NET made tracing API, but there was an existing tracing API in .NET runtime already. Eventually, OTel .NET maintainers collaborated with .NET runtime team to modify/enhance the tracing API from .NET runtime to be more closely aligned with OTel spec. Once that was done, OTel .NET recommends using the API from the .NET runtime itself, and the OTel API became just a simple shim on top of .NET runtime API.
The above approach was later followed for Metrics too - Metrics API was fully implemented in .NET runtime. OTel .NET recommends people to simply use .NET runtime for their Metric API needs (Meter, Counter, Histogram all comes from .NET runtime without installing any OTel packages!).

[cijothomas]

https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/getting-started-console#opentelemetry-net-and-relation-with-net-activity-api Link to OTel .NET doc explaining the situation.

[davidbarsky]

Also, it may be possible for the OTel Rust maintainers and the owners of Tokio/Tracing to explore if it is feasible to merge both into single one long term.

Speaking as a person who helps maintain the tracing crate, it would be nice to see what parts we can merge and/or make easier on the OTel side of things. Note that some of the differences between the tracing and OTel APIs boil down to different constraints, as I've mentioned in my other comment.

This situation is very similar to what OpenTelemetry .NET encountered. OTel .NET made tracing API, but there was an existing tracing API in .NET runtime already. Eventually, OTel .NET maintainers collaborated with .NET runtime team to modify/enhance the tracing API from .NET runtime to be more closely aligned with OTel spec.

What sort of approach/enhancements for tracing do you have in mind here, if have any? (I assume that you do, but totally fine if not!)

I ask because one of things that gives me pause about a potential, long-term merger with OpenTelemetry is that—at least to my understanding—the OpenTelemetry standard appears to mandate API design standards. This results in an experience that is coherent across OpenTelemetry SDKs, but isn't necessarily coherent within a language. Some of those mandated (?) design decisions come into conflict with tracing's other goals. That being said, there appears to be some flexibility, especially when it came to the .NET activity API. Is my understanding correct?

[TommyCpp]

@cijothomas can probably speak more about this but per my understanding, we have some wiggle room in terms of changing certain behavior of the Opentelemetry API to better suit languages' needs

[davidbarsky]

Just some additional details:

Both models has the concept of Span. They both have the concept of the SpanId, parent span.

tracing::span::Id are best thought of opaque identifier that is unique within the process and the lifetime of the span until it exits. Under the hood, the span ID is a usize that (most often) indexes into an arena. See here for some additional details: https://docs.rs/tracing/latest/tracing/trait.Subscriber.html#tymethod.new_span.

Put differently, tracing::span::Id has no global uniqueness property like a UUID or the W3C trace-context: it is optimized for fast, in-process lookups.

Tracing span's metdata usually include Metadata like callsite, verbosity level and a list of fields that contains key value pairs.

The Otel spans contains SpanContext, which covers the SpanId and esbalished the link between spans and it's partent span. Its attributes are similar to tracing span's fields. Additionaly it contains links to other spans and other metadata.

A large portion of these differences boil down to tracing wanting to be:

• almost entirely 'static to maximize performance for both active and inactive spans/events.
• support no_std + alloc use-cases, of which we have a non-trivial amount.

Most differences in the tracing versus OTel APIs boil down to tracing adhering those constraints.

But it exists within the context of span and have a list of fields. This concept is almost identical with the Event in Otel spans

tracing::Event doesn't need to strictly exist within the context of a span—events can be standalone without any parents. Spans are simply additional context and flavor.

So far we have been handlding those difference by converting the fields from tracing API to Otel span using magic words. For example if user puts otel.kind as the key of the field. We will convert the value to span kind in Otel span.

A thing that wanted to do for a while was to come up with a solution better than "magic" strings. The rough idea I was thinking of was something along the lines of a type registry such that a crate such as tracing-opentelemetry can register what types it's providing and allow the user to record said values through an extension trait (as tracing-opentelemetry currently does). I haven't thought exactly how this should happen, but the intent is certainly there—I'm not a fan of magic strings either.

[cijothomas]

Pasting something from the today's SIG calls:

.NET Reasons for favoring the existing tracing API instead of newly making OTel Tracing API:

1. Familiarity with existing -- Applicable to Rust as well
2. Strong back-compat (as .NET Tracing API is part of .NET Runtime (aka BCL, base class library) itself) -- Not applicable to Rust
3. Part of .NET runtime (no need of installing any extra packages) -- Not fully applicable. Tokio itself is very common, most projects have it??
   Note: tracing does not require Tokio runtime. It is just part of Tokio org, but does not require Tokio runtime.

[shaun-cox]

I like the tracing ecosystem and think it's a natural fit for instrumentation of my Rust code. IOW, I'd rather stick with tracing alone, than rip and replace or add additional API calls to OTel.

I love the OTel model and vision, especially around distributed tracing. I would love to see tracing handle the idea of context propagation within a process in a manner that composes with context propagation across processes. For example, I use channels and actors within a process and would like to be able to create a span inside each actor for the processing of each of its messages, as a child of a span (context) that was delivered with each message and following from the "current" span which is that of the running actor. And when the final actor sends a message across the network, I'd like to be able to extract and send the Otel-compatible span context in my network message and construct a new tracing span on the receiving side with that context as parent, kept intact.

I also think it would be cool to see tracing subscriber filters expose or compose with OTel's notion of "sampling" to enable collection of traces (i.e. root spans) that are marked at runtime during the "tail" of processing as anomalous or interesting.

[shaun-cox]

Small update from me on API choice. I'm currently exploring using opentelemetry-rust for instrumentation rather than tracing-opentelemetry after doing a quick experiment to evaluate both.

Overall, I like the macro interface offered by tracing for a lot less boilerplate. But, surprisingly, the code path length of tracing is quite a bit longer than that of opentelemetry-rust, just for building spans.

vs.

[davidbarsky]

Small update from me on API choice. I'm currently exploring using opentelemetry-rust for instrumentation rather than tracing-opentelemetry after doing a quick experiment to evaluate both.

Overall, I like the macro interface offered by tracing for a lot less boilerplate. But, surprisingly, the code path length of tracing is quite a bit longer than that of opentelemetry-rust, just for building spans.

vs.

What do you mean by "the code path length of tracing"? Is it the longer durations?

(If I recall correctly, the reason you're seeing this timing difference is that spans from tracing might get buffered—in other words, this isn't an inherent architectural difference.)

[shaun-cox]

What do you mean by "the code path length of tracing"? Is it the longer durations?

Yes. I have the same workload of sending ten requests through a two-stage pipeline of actors. The only differences are in the instrumentation used to generate the spans representing the duration of each request in each stage of the pipeline, which are children of a root span for the whole run. The wall-clock time of the whole run is longer when tracing::trace_span! is used to create the spans vs. opentelemetry::SpanBuilder.start_with_context.

I'd like to understand more of what you mean by "might get buffered" in tracing. Could you point me to an area in the tracing repo where this is happening? I'd assumed the time-stamping of span open/close events was all synchronous w.r.t. to the call-sites for create/drop. Thanks for any insight to help me understand this better.

[davidbarsky]

I was mistaken about buffering. I'll need to take a closer look at both.

[jtescher]

@shaun-cox you can take a look at the benchmarks in https://github.com/tokio-rs/tracing-opentelemetry/blob/v0.1.x/benches/trace.rs to see where they may be overhead as a jumping off point.

[shaun-cox]

I finally got around to looking at these benchmarks. My conclusion after looking at the data is that the expense seems to reside in the OpenTelemetryLayer implementation, mainly due to accumulating information in the SpanBuilder then extracting it back out when the tracing span is closed. It's the gap between the "baseline" and "full" labels below:

Here's a flame graph of the full run:

The use of hash maps (EvictedHashMap, IndexMap) are pretty expensive to accumulate into and then just drop after export... unless I'm missing where their O(1) lookup properties are leveraged in this scenario.