Telemetry filters on span events

[ivan-zaitsev]

Is your feature request related to a problem? Please describe.

Currently only filtering on attributes is supported

ApplicationInsights-Java/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/processors/SpanExporterWithAttributeProcessor.java

Lines 38 to 47 in 547ec61

	IncludeExclude include = attributeProcessor.getInclude();
	if (include != null && !include.isMatch(span.getAttributes(), span.getName())) {
	// If not included we can skip further processing
	return span;
	}
	IncludeExclude exclude = attributeProcessor.getExclude();
	if (exclude != null && exclude.isMatch(span.getAttributes(), span.getName())) {
	// If excluded we can skip further processing
	return span;
	}

Describe the solution you would like
It would be good to have filtering also on span events. For example, if span event contains some fields and it is specified as include or exclude option the span processor actions such as adding new attribute should be executed.

Example (azure blob storage deleteIfExists):

Code:

BlobContainerClient blobContainerClient = blobServiceClient.getBlobContainerClient("container");
blobContainerClient.deleteIfExists();

Produced spans:

SpanData{
    spanContext=ImmutableSpanContext{
        traceId=0, spanId=0, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, 
        parentSpanContext=ImmutableSpanContext{traceId=0, spanId=0, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, 
        resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.20.0, attributes={telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.28.0"}}, 
        instrumentationScopeInfo=InstrumentationScopeInfo{name=azure-storage-blob, version=12.23.0, schemaUrl=https://opentelemetry.io/schemas/1.17.0, attributes={}}, 
        name=AzureBlobStorageCont.delete, kind=INTERNAL, startEpochNanos=0, endEpochNanos=0, 
        attributes=AttributesMap{data={
            az.namespace=Microsoft.Storage, thread.id=59, applicationinsights.internal.operation_name=DELETE /api/v1/operation, thread.name=http-nio-8080-exec-1, applicationinsights.internal.item_count=1
        }, 
        capacity=128, totalAddedValues=5}, totalAttributeCount=5,
        events=[ImmutableExceptionEventData{epochNanos=0, exception=
            com.azure.storage.blob.models.BlobStorageException: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.</Message></Error>", 
        additionalAttributes={}, spanLimits=SpanLimitsValue{maxNumberOfAttributes=128, maxNumberOfEvents=128, maxNumberOfLinks=128, maxNumberOfAttributesPerEvent=128, maxNumberOfAttributesPerLink=128, maxAttributeValueLength=2147483647}}], 
        totalRecordedEvents=1, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=ERROR, description=}, hasEnded=true
    }

SpanData{
    spanContext=ImmutableSpanContext{
        traceId=0, spanId=0, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, 
        parentSpanContext=ImmutableSpanContext{traceId=0, spanId=0, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, 
        resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.20.0, attributes={telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.28.0"}}, 
        instrumentationScopeInfo=InstrumentationScopeInfo{name=azure-storage-blob, version=12.23.0, schemaUrl=https://opentelemetry.io/schemas/1.17.0, attributes={}}, 
        name=HTTP DELETE, kind=CLIENT, startEpochNanos=0, endEpochNanos=0, 
        attributes=AttributesMap{data={
            az.namespace=Microsoft.Storage, http.user_agent=az-sp-sb/5.4.0 azsdk-java-azure-storage-blob/12.23.0, http.status_code=404, thread.id=61, 
            http.url=https://storage.blob.core.windows.net/container?restype=container, applicationinsights.internal.operation_name=DELETE /api/v1/operation, 
            thread.name=http-nio-8080-exec-3, http.method=DELETE, applicationinsights.internal.item_count=1, az.client_request_id=0, az.service_request_id=0
        }, 
        capacity=128, totalAddedValues=11}, totalAttributeCount=11, 
        events=[],
        totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=ERROR, description=}, hasEnded=true}
...

• First span attributes contain only operation name. and events contain the error. In this case it is not possible to create filter if operation is not successful, because useful information is located in events parameter.
• Second span contains http status parameter, and it is possible to filter that.

My use case is to add additional attribute if any of these spans are found. As azure marks these spans as "success = false", and I want to add additional attribute to mark this as "controlled error".

---

Also, for all attributes operation "and" is applied (all include filters should be true), it would be good to configure it somehow to have ability to specify some "or" conditions. Currently it is possible to do that only by creating duplicate sections for processors, what leads to complexity. This would give ability to filter multiple spans using only one processor. See example with azure blob storage deleteIfExists which produces multiple spans.

[heyams]

@ivan-zaitsev span #2 above, current telemetry processor supports it.

{
    "type": "attribute",
    "include": {
        "matchType": "strict",
        "attributes": [
            {
                "key": "http.status_code",
                "value": 404
            }
        ]
    },
    "actions": [
        {
            "action": "insert",
            "key": "controlledError",
            "value": true
        }
    ],
    "id": "mark as not an error for status codes"
}

Are you asking for supporting filtering span.events via telemetry processors?

[ivan-zaitsev]

Are you asking for supporting filtering span.events via telemetry processors?

Yes. Second span is there just as an example. The first one has actual events.

[ivan-zaitsev]

From different thread @heyams said that it won`t be supported.

It's sad that this won't be supported, because currently it is not possible to filter on events or exceptions (they added as span events). If I would submit merge request, would you consider adding this feature?

I think it is pretty easy to add, because EventData already contains getAttributes() method, and everything is already in place to handle attributes, only new processor similar to SpanExporterWithAttributeProcessor should be added to process span.getEvents().stream().forEach(event -> event.getAttributes()) instead of span.getAttributes().

How i think it can be added:

1. New ProcessorType value = EVENT_ATTRIBUTE should be added. Here:

   ApplicationInsights-Java/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/configuration/Configuration.java

   Line 127 in e3eccab

   ATTRIBUTE("an attribute"),

2. New EventSpanExporterWithAttributeProcessor class should be added. Similar to this: SpanExporterWithAttributeProcessor to process attributes for each event from span.getEvents().

Then it would be possible to filter on event attributes like this (only new type is added, everything else is the same):

{
  "sampling" : {
    "percentage" : 100
  },
  "preview" : {
    "processors": [
      {
        "id" : "processors/attribute/strict",
        "type": "event-attribute",                  --- new type
        "include": {
          "matchType": "regexp",
          "attributes": [
            {
              "type": "string",
              "key": "exception.type",
              "value": "com.azure.storage.blob.models.BlobStorageException"
            },
            {
              "type": "string",
              "key": "exception.message",
              "value": "Status code 404"
            }
          ]
        },
        "actions": [
          {
            "action": "insert",
            "key": "strictKey",
            "value": "strictValue"
          }
        ]
      }
    ]
  }
}

It is possible to filter on "exception.type" and "exception.message" event attributes because they are added automatically by ImmutableExceptionEventData.

[heyams]

@ivan-zaitsev I'm going to work on the sample app using OTEL extensions. if it doesn't work, i will discuss it with my team and then get back to you.

[ivan-zaitsev]

Hello @heyams

Tried to create OTEL extension, but it didn't work. I can see that my custom addTracerProviderCustomizer and addSpanExporterCustomizer are executed before SecondEntryPoint, and tracer exporter processor AgentSpanExporter.export is executed after my custom SimpleSpanProcessor. But by some reason tracer exporter processor does not export those changes.

Code example:

resources/META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider

otel.OpentelemetryExporterCustomizer

package otel;

import java.util.Collection;

import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributesBuilder;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizer;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import io.opentelemetry.sdk.common.CompletableResultCode;
import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder;
import io.opentelemetry.sdk.trace.data.DelegatingSpanData;
import io.opentelemetry.sdk.trace.data.SpanData;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;

public class OpentelemetryExporterCustomizer implements AutoConfigurationCustomizerProvider {

    @Override
    public int order() {
        return -1;
    }

    @Override
    public void customize(AutoConfigurationCustomizer autoConfiguration) {
        autoConfiguration.addTracerProviderCustomizer((tracer, otelConfig) -> wrapTracerProvider(tracer));
        autoConfiguration.addSpanExporterCustomizer((exporter, otelConfig) -> wrapSpanExporter(exporter));
    }

    private SdkTracerProviderBuilder wrapTracerProvider(SdkTracerProviderBuilder tracerProviderBuilder) {
        SpanExporter spanExporter = wrapSpanExporter(SpanExporter.composite());
        tracerProviderBuilder.addSpanProcessor(SimpleSpanProcessor.create(spanExporter));
        return tracerProviderBuilder;
    }

    private SpanExporter wrapSpanExporter(SpanExporter spanExporter) {
        return new ExtendedSpanExporter(spanExporter);
    }

    private static class ExtendedSpanExporter implements SpanExporter {

        private final SpanExporter delegate;

        private ExtendedSpanExporter(SpanExporter delegate) {
            this.delegate = delegate;
        }

        @Override
        public CompletableResultCode export(Collection<SpanData> spans) {
            return delegate.export(spans.stream().map(this::process).toList());
        }

        @Override
        public CompletableResultCode flush() {
            return delegate.flush();
        }

        @Override
        public CompletableResultCode shutdown() {
            return delegate.shutdown();
        }

        private SpanData process(SpanData span) {
            AttributesBuilder builder = Attributes.builder();
            builder.putAll(span.getAttributes());
            builder.put("test-key", "test-value");

            SpanData result = new ExtendedSpanData(span, builder.build());
            System.out.println(result);
            return result;
        }

    }

    public static class ExtendedSpanData extends DelegatingSpanData {

        private final Attributes attributes;

        public ExtendedSpanData(SpanData delegate, Attributes attributes) {
            super(delegate);
            this.attributes = attributes;
        }

        @Override
        public Attributes getAttributes() {
            return attributes;
        }

    }

}

Run app

java -jar app.jar -Dotel.javaagent.extensions=otel-extension.jar

[ivan-zaitsev]

Figured it out. Usual otel extension creation guidelines with span exporters are not applicable here. ApplicationInsights-Java uses span processor as last sink for data exporting. Exporters from addSpanExporterCustomizer are combined together and being used in addTracerProviderCustomizer as separate processor. Processors don't change span data by chaining and delegating like exporters do, so one processor is not affected by other one, because of that any modification of span data in exporters from addSpanExporterCustomizer will not simply work for other processors. As last export of data is performed by different processor the exporters from addSpanExporterCustomizer will not be applied. Therefore, to make it work the modification in span processors should be made.

But this way it is not possible to do any conditions on spans, because:

1. onStart is executed right away when span is started, so some data is missing as it can be added later after that. onEnd has only ReadableSpan which is not possible to change.
2. SimpleSpanProcessor is executed before ApplicationInsights-Java processors and exporters, therefore spans are missing some data from there.

Code example:

package otel;

import io.opentelemetry.context.Context;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizer;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import io.opentelemetry.sdk.trace.ReadWriteSpan;
import io.opentelemetry.sdk.trace.ReadableSpan;
import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder;
import io.opentelemetry.sdk.trace.SpanProcessor;

public class OpentelemetryExporterCustomizer implements AutoConfigurationCustomizerProvider {

    @Override
    public int order() {
        return -1;
    }

    @Override
    public void customize(AutoConfigurationCustomizer autoConfiguration) {
        autoConfiguration.addTracerProviderCustomizer((tracer, otelConfig) -> wrapTracerProvider(tracer));
    }

    private SdkTracerProviderBuilder wrapTracerProvider(SdkTracerProviderBuilder tracerProviderBuilder) {
        tracerProviderBuilder.addSpanProcessor(new SimpleSpanProcessor());
        return tracerProviderBuilder;
    }

    private static class SimpleSpanProcessor implements SpanProcessor {

        @Override
        public boolean isStartRequired() {
            return true;
        }

        @Override
        public void onStart(Context parentContext, ReadWriteSpan span) {
            span.setAttribute("test-key", "test-value");
        }

        @Override
        public boolean isEndRequired() {
            return false;
        }

        @Override
        public void onEnd(ReadableSpan span) {
        }

    }

}

Either I missed something, or it seems pretty limited that it is not possible to add span processors/exporters which would modify span for the ApplicationInsights-Java processing/exporting flow.

I think if it a custom exporter as a delegate for existing one would be added here then it should be possible to check/modify spans right away before exporting.

[heyams]

Happy new year! Will get back to you soon. Sorry was out of office during the holidays. Thank you for trying :)

[heyams]

@ivan-zaitsev I tried something similar here.

My extension code gets applied, but it requires some code change in the Java agent in order to make it full-functional.
Please stay tune and I will try to work on the java agent to make this sample works for your use case.

[ivan-zaitsev]

Hello @heyams

Thank you for your help.
Interesting, everything seems similar to my first example. But in my case the exporters were executed but it was not applied when I checked logs in applications insights. I think that all SpanExporters which are added by addSpanExporterCustomizer have no effect on the actual http exporter which is configured as span processor - here.

[heyams]

@ivan-zaitsev yes, that's the part I need to fix in our java agent. But addTracerProviderCustomizer works.

[ivan-zaitsev]

Hello @heyams, any update on this? Seems that there no other way at the moment to add any custom span processing on top of this java agent.

[heyams]

@ivan-zaitsev yes, #3493 is intended to support OpenTelemetry Extension. I will test it in my sample and get back to you soon.

[ivan-zaitsev]

@heyams Thanks! Sorry for pinging you too often, I just thought it was abandoned :)

[heyams]

@ivan-zaitsev no, we're working on it.. It's a bit tricky though. I will make sure i test it and then confirm it's working in my sample.. then once this is released next week, you can use it. you have been super helpful assisting me getting to the bottom of this. I appreciate your patience.

[heyams]

@ivan-zaitsev good news! it seems to be working. Azure-Samples/ApplicationInsights-Java-Samples#272 will generate this payload:

{
  "ver": 1,
  "name": "RemoteDependency",
  "time": "2024-01-16T22:55:00.503Z",
  "iKey": "<REDACTED>",
  "tags": {
    "ai.internal.sdkVersion": "java:3.5-SNAPSHOT",
    "ai.operation.id": "d1d540898b5bc6697637561c9dea25db",
    "ai.cloud.roleInstance": "<REDACTED>"
  },
  "data": {
    "baseType": "RemoteDependencyData",
    "baseData": {
      "ver": 2,
      "id": "018c905b75ea7b3f",
      "name": "mySpan",
      "type": "InProc",
      "duration": "00:00:10.297740",
      "success": true,
      "properties": {
        "controlledError": "true", **// this is added via the Telemetry Processor**
        "myCustomAttributeKey2": "myCustomAttributeValue2", **// added via the extension via the Sample above**
        "myCustomAttributeKey": "myCustomAttributeValue"
      }
    }
  }
}

You can reproduce it if you follow my readme .

It will be part of 3.5 GA release probably next week if no delay. I'll give you an update.

[heyams]

@ivan-zaitsev just a heads up: 3.5 GA release is delayed to the week of Feb 21. We are waiting for a couple of fixes in upstream to be released in OpenTelemetry 2.1. Thank you for your patience.

[heyams]

3.5.0 is live today.

[ivan-zaitsev]

3.5.0 is live today.

Thanks, it means we can close this issue?

[heyams]

3.5.0 is live today.

Thanks, it means we can close this issue?

i can close it for you. i will update the sample too. thank you!

[heyams]

@ivan-zaitsev I've updated the sample app using the latest java agent. Azure-Samples/ApplicationInsights-Java-Samples#272. It is working as expected. Please follow the readme for detail on build command and kusto query instructions.

[ivan-zaitsev]

@heyams Thank you.