This approach has one considerable downside over the annotation. The annotation changes pod spec, therefore it triggers re-deployment which is needed by the OTEL operator to inject the auto-instrumentation (the operator uses pod mutating webhook)

therefore it triggers re-deployment which is needed by the OTEL operator to inject the auto-instrumentation (the operator uses pod mutating webhook)

Yeah, you're right.

However, we hope that all the configurations associated with OTel can be handed over to Operator, and if the annotation are used, we also need to maintain them ourselves during the deployment.

This is our usage scenario; I'll submit a PR if appropriate.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  selector:
    app: appname
  propagators:
    - tracecontext
  java:
    image: image

@crossoverJie we discussed this issue at the SIG (and realized we should have probably done this prior to your PR) and had a few follow up questions.

I saw the link you shared for how skywalking handles this, and at first glanced it seemed very similar, however, there's a key difference. Skywalking seems to be a java-only auto-instrumentation solution whereas opentelemetry can inject instrumentation for multiple languages. That results in what we agreed to be a confusing experience for users where a cluster admin needs to set a piece of configuration AND a tenant in the cluster needs to set configuration as well.

1. What are the user stories for this request and how do they differ from today? With the change you've made, it seems to be a very similar end state where both cluster admin and tenant need to do actions which feels against the goal of the issue which is to put the onus of instrumentation solely in the hands of the cluster admin.

2. What are the reasons for having multiple instrumentation resources for the same language in the same cluster? I'm not sure I understand the exact issue today that the current solution solves.

I was imagining it would be more beneficial for both personas if the cluster admin could specify a set of rules that specify a selector for applications to target for injection and specify which languages (and optionally container names) to inject. Let me know your thoughts here, and if you'd like to meet more to discuss the above I'd be happy to chat.

@jaronoff97

Thank you for following up on this issue.

That results in what we agreed to be a confusing experience for users where a cluster admin needs to set a piece of configuration AND a tenant in the cluster needs to set configuration as well.

Can you explain the relationship between cluster admin and tenants here?
In my scenario, I'm the only one maintaining Instrumentation.

2. What are the reasons for having multiple instrumentation resources for the same language in the same cluster? I'm not sure I understand the exact issue today that the current solution solves.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation-sit-grpc-consumer
  namespace: sit
spec:
  env:
    - name: OTEL_SERVICE_NAME
      value: grpc-consumer
  selector:
    matchLabels:
      app: grpc-consumer
  java:
    image: autoinstrumentation-java:v1
    extensions:
      - image: extensions:v1
        dir: /extensions
    env:
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: service.name=grpc-consumer

---
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation-sit-grpc-provider
  namespace: sit
spec:
  env:
    - name: OTEL_SERVICE_NAME
      value: grpc-provider
  selector:
    matchLabels:
      app: grpc-provider
  java:
    image: autoinstrumentation-java:v1
    extensions:
      - image: extensions:v2
        dir: /extensions

    env:
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: service.name=grpc-provider

This is our usage scenario, and there are some application-specific configurations that need to be defined in different Instrumentations.

There are some other examples:

• New versions of javaagent are only tested on specific applications first.
• Some specific applications only need to report metrics but do not want to report traces.
• For resource isolation, the OTEL_EXPORTER_OTLP_ENDPOINT reported by some applications may be different.

Therefore we like that each application can maintain its own Instrumentation file independently.