feat(RFC): Add Internal Trace Model.

[hdost]

Provides a path forward for integrating sources and sinks for Tracing.

Relates #1444, #17307, #17308

[jszwedko]

Thanks for this @hdost . We'll try to review it next week in more detail.

[bruceg]

Suggested change

	Vector fields shown using VRL.
	Vector fields are shown using VRL path notation.

[bruceg]

Is this (and below) valid VRL path notation? I'm thinking it should be more like this:

Suggested change

	| TraceID | `.["scope_spans"][i]["spans"][j]["trace_id"]` | |
	| TraceID | `.scope_spans[i].spans[j].trace_id` | |

[bruceg]

nit: Remove this extraneous heading.

[bruceg]

Can we not receive traces through the datadog_agent interface? Even if not, I would think at some point we will want to be able to pass them through as well.

[bruceg]

Are you planning to fill this in?

[bruceg]

I don't see why this must be out of scope, if the desired data model requires a change.

[jszwedko]

I'm also a bit confused on this one. It seems like the RFC here is proposing a strongly-typed representation, which I think makes sense, but means we'd also have an actual struct representing the trace event.

[bruceg]

Some general description of what is in the data model would be useful here. Also, is this data model so generic that only a Map can describe it, or are more dedicated data types relevant?

[bruceg]

The remaining mappings of what?

[bruceg]

Suggested change

	The keys for `message` come the field names for `Value::Object`, and MUST follow a "Snake Case" format.
	As an example, instead of `traceId` and `droppedEventCount`, use `trace_id` and `dropped_event_count`.
	The keys for `message` come the field names for `Value::Object`, and MUST follow a "snake case" format.
	As an example, instead of `traceId` and `droppedEventCount`, use `trace_id` and `dropped_event_count`.

[bruceg]

This warrants some expansion as well.

[tsloughter]

Wouldn't the advantage be to do datadog agent -> vector -> trace collector be it one that supports OTLP or Zipkin or whatever?

[tsloughter]

Isn't the proposal this? It just does conversions of the proto types to Vector's internal Rust types but keeps the structure of Resource -> InstrumentationScope -> Spans the same?

[jszwedko]

I think we need both directions, no? The datadog_agent source needs to decode it and the datadog_traces sink needs to encode it.

[jszwedko]

I'm also a bit confused on this one. It seems like the RFC here is proposing a strongly-typed representation, which I think makes sense, but means we'd also have an actual struct representing the trace event.

[bruceg]

We discussed this internally and agree with the general direction of this proposal. It makes sense to adopt the the OpenTelemetry data types as the internal model, since the OTel project has defined many mappings to and from their data model for other trace types. So it stands as a useful intermediate representation for many kinds of traces.

However, if we are going to adopt that data model, we would prefer it be implemented in a more structured form. Specifically, we would want all of the data structures, were possible, to become fixed Rust structs instead of a generic map type. This would echo our metric types, which can represent a variety of different kinds of metrics with a fixed structure. This will simplify all kinds of code managing conversion to and from other data types, such as protobufs or other trace models, since they can rely on the presence of certain fields at a source code level.

As such, we would like to see the above reflected in the RFC. It should contain source code specifics of what the structures would look like and, if appropriate, what methods would be made available for manipulating them. Since this isn't a completely different direction, we don't think it requires a complete overhaul of this document but it does require changing the core data types, which means that consideration is very much in scope.

Some examples from existing RFCs to show what that might look like:

• https://github.com/vectordotdev/vector/blob/master/rfcs/2022-10-12-14742-flexible-metric-tags.md
• https://github.com/vectordotdev/vector/blob/master/rfcs/2022-07-28-13691-registered-internal-events.md
• https://github.com/vectordotdev/vector/blob/master/rfcs/2022-02-23-9481-config-schema.md

Many other RFCs contain Rust code samples, as we intend for these documents to be very opinionated as to the target solution, with many of the implications worked out.

[hdost]

Thank you for the thorough review, I'll take some time to reformulate with all that in mind.