Support nested Attribute values (#376)

[shengxil]

Related issue #376

Nested Attribute is useful for representing multi-level values. For example, HTTP headers:

'http': {
    'headers': [
        'x-forwarded-for': 'foo.bar'
        'keep-alive': 'timeout=5'
    ]
}

Currently such cases are usually represented by a period delimited key chain (e.g. 'http.headers.x-forwarded-for': 'foo.bar'). This does not work if any key contains a period itself. It is also difficult for SDKs to iterate the Attributes under a certain level.

This change allows nested Attributes by adding an Attribute value type - child Attribute. An attribute can also contain a homogeneous array of children attributes.

Why not map?

1. Map is somehow a high level data structure. It has different definitions in different languages. In some languages putting a map value replaces the original value, while in other languages it becomes a sibling of the original value.
2. Programming-wise, it is hard to keep the homogeneity in map values.

• Actually, the OT spec never used map in any span data type

[Oberon00]

HTTP headers can be represented using an attribute http.request.headers.HEADER-NAME per header.

[Oberon00]

A single child attribute? That does not sound useful. So instead of Attribute(3), I could have Attribute(Attribute(Attribute(3)))? That's probably not what you meant to say.

[Oberon00]

Ah, I think I understand: The attribute does include the key! Then I would allow the child attribute only in the array because Attribute(key=foo, value=Attribute(key=bar, value=42)) sounds useless.

[shengxil]

Appreciate the feedback! Yeah that makes sense. Even if some use cases need a single child attribute, it can still be placed in the array. I'll updating the PR.

[tigrannajaryan]

Note that due to upcoming Log Data model we will likely support maps and arrays for attribute values. Logs need this (most importantly for the body). It is very likely that this enhanced attributes capability will be applied to other signals as well, there is no reason to limit this to logs only.

[Oberon00]

The array restrictions (no nested arrays, no different types) were made to ease implementation and APIs in statically typed languages. This PR would mostly negate this, because by using an array of attributes you could have different values inside these attributes, including arrays.

[shengxil]

I would argue that the current homogeneous convention is made because some languages can't accept array with different member types. If we consider the whole Attribute as a firm type, and consider its value type and value as its property, then within an array all sibling Attribute are still the same type.

Programming-wise IMHO, Attribute encapsulates the data so from the array's point of view, the value type is invisible.

[Oberon00]

This does not work if any key contains a period itself.

Note that attribute keys are still subject to (undecided) restrictions, see #504.

Also, I suggest the following: When parsing such an attribute (e.g. http.header.*) ignore any dots in the * part. Maybe even use a syntax like http.header:* (note the colon instead of the period) instead to make it clear that dots are ignored from now on.

[shengxil]

HTTP headers can be represented using an attribute http.request.headers.HEADER-NAME per header.

It is feasible at the trace generating side (client), but not a good experience at the consumer side (tracer provider), especially for those deep nested data. For example, AWS traces has data like

AWS: [
    Lambda1: {
        Id: Id1,
        Name: Foo
    },
    EC2: {
        Id: Id2,
        Metadata: [
            Key1: Value1,
            Key2: Value2
        ]
    }
]

It becomes very difficult to parse the data if we use a list of flatten String keys. We have to do a lot of String joining/delimiting. Nested attribute or map will provide a much better flexibility on dealing with such scenario

The nested attribute support shall also apply to the span Resource (#579) and log (open-telemetry/oteps#97). A span may contain multiple resources from different origin.

[shengxil]

This does not work if any key contains a period itself.

Note that attribute keys are still subject to (undecided) restrictions, see #504.

Also, I suggest the following: When parsing such an attribute (e.g. http.header.*) ignore any dots in the * part. Maybe even use a syntax like http.header:* (note the colon instead of the period) instead to make it clear that dots are ignored from now on.

IMHO the restriction makes it more controversial. If we don't allow period or colon in the key, we can't use it as a delimiter. If we allow it, a single key itself may contain a delimiter, which breaks the parser. Today we have a semantic HTTP keys that works well. But for feature scalability concern, I don't think banning period/colon in attribute key is a good idea.

On the other hand, nested attribute or map can mitigate this issue

[shengxil]

Note that due to upcoming Log Data model we will likely support maps and arrays for attribute values. Logs need this (most importantly for the body). It is very likely that this enhanced attributes capability will be applied to other signals as well, there is no reason to limit this to logs only.

Thanks for bringing this up. I'm trying to represent the multi-level value here. I'm wondering if the log map support nested map. If yes, does every language support it?

[tigrannajaryan]

Thanks for bringing this up. I'm trying to represent the multi-level value here. I'm wondering if the log map support nested map. If yes, does every language support it?

Data model requires that nested maps are supported for log body. Language implementations of don't exist yet.

[anuraaga]

Another interesting part of the AWS trace model is metadata

https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html#api-segmentdocuments-metadata

Metadata allows users to store arbitrary JSON-like structures, which can provide a nice treelike UI, as opposed to the common two-column one for simple K-V pairs, and more structure for automated workflows.

One thought is that it's relatively easy to flatten a JSON-like structure, e.g, these nested attributes, into key-value pairs by using something like the dot delimiters, during export for backends that don't support rich attributes and they would still support the normal K-V pair functionality. But the reverse, unflattening K-V pairs with delimiter, is ambiguous unless restricting the key character values, so backends with rich attributes would just have to drop support. I guess making the instrumentation layer more general and simplifying in the exporters could support more use cases? Given the popularity of JSON even as a data storage format (e.g., MongoDB and similar stores), it seems like many backends could want to expose the richness in metadata.

The array restrictions (no nested arrays, no different types) were made to ease implementation and APIs in statically typed languages.

Supporting dynamic structure could definitely make language implementations more complex. But considering how prevalent JSON is, maybe it's not so bad since I guess pretty much all languages have a good story for handling it nowadays.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[SergeyKanzhelev]

This PR being small in a text didn't receive a single approve. This may only mean no contributors needs this support.

Since this is the case, closing this PR. Perhaps it can be discussed after GA.