Instrumentations support select

[crossoverJie]

Description:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  selector:
    app: appname
  propagators:
    - tracecontext
  java:
    image: image

Support select.

Link to tracking Issue(s): #2744

• Resolves: Instrumentation support selector #2744

Testing:

Documentation:

[swiatekm-sumo]

I'd also like to see a E2E test for this behaviour.

[crossoverJie]

I'd also like to see a E2E test for this behaviour.

I'll add it later.

[swiatekm-sumo]

LGTM, minus some unnecessary changes from E2E tests.

[swiatekm-sumo]

I noticed that we're not following the standard semantics of v1.LabelSelector:

A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.

Whereas the way we have this implemented, null means "everything" and there's no way to select nothing. Could we switch to the standard semantics? I don't think there's a way to get kubebuilder to set this default for us, so we'd need to do it in the defaulting webhook, but we'd get to skip the conditional instructions in the Pod mutator. What do you think?

[crossoverJie]

Could we switch to the standard semantics?

Good suggestion. I agree with you.

An empty label selector matches all objects. A null label selector matches no objects.

If we want to implement this semantics, the current test case will need to add empty selectors to pass the e2e test.

The implement code:

This will change the default behavior; is this necessary?

[swiatekm-sumo]

Could we switch to the standard semantics?

Good suggestion. I agree with you.

An empty label selector matches all objects. A null label selector matches no objects.

If we want to implement this semantics, the current test case will need to add empty selectors to pass the e2e test.

The implement code:

This will change the default behavior; is this necessary?

You can add the following annotation to the field: // +kubebuilder:default:={}, and it will work as intended. I also made the following change to your logic:

	for _, ins := range otelInsts.Items {
		labelSelector, err := v1.LabelSelectorAsSelector(ins.Spec.Selector)
		if err != nil {
			return nil, err
		}
		set := labels.Set(pod.GetLabels())
		if labelSelector.Matches(set) {
			availableInstrument = append(availableInstrument, ins)
		}
	}

and all the e2e tests passed.

The only potential problem here, is that this default value is only reflected in the CRD itself. So if a user were to upgrade the operator without upgrading their Instrumentation CRD, the field would be nil by default, and their Instrumentations would select nothing. So maybe it's better to keep your current logic and make this change when we promote Instrumentation to beta? @pavolloffay @jaronoff97 @TylerHelmuth WDYT?

[crossoverJie]

Are there any new updates now?

[jaronoff97]

@swiatekm-sumo i think it may be okay for the instrumentations to select none by default, given that's technically the current behavior right?

[swiatekm-sumo]

@swiatekm-sumo i think it may be okay for the instrumentations to select none by default, given that's technically the current behavior right?

I don't think that's the case? Looking at the code being modified in this PR, it just picks the instrumentation defined in the namespace if the annotation isn't specific. Setting the default to nil would stop this from working.

Incidentally, we should document that this selector only applies to this situation. As is, if the selector doesn't fit the Pod, and the Pod asks for a specific Instrumentation via the annotation, it will still get it.

[crossoverJie]

So maybe it's better to keep your current logic and make this change when we promote Instrumentation to beta?

I prefer this approach; maybe we can finish the current issue first.

[swiatekm-sumo]

I'm ok keeping this as is, but we should document how we diverge from standard LabelSelector behavior.

[pavolloffay]

Are all pods from all namespaces matched?

What is the precedence order when annotation instrumentation.opentelemetry.io/ is also present on the pod/namespace?

[swiatekm-sumo]

I believe this feature only comes into play when the annotation is not set and we try to pick an Instrumentation from the namespace the Pod is in. The annotation has precedence. But we should document this behaviour somewhere.

[crossoverJie]

I believe this feature only comes into play when the annotation is not set and we try to pick an Instrumentation from the namespace the Pod is in

opentelemetry-operator/pkg/instrumentation/podmutator.go

Lines 368 to 377 in c65629b

	func (pm *instPodMutator) getInstrumentationInstance(ctx context.Context, ns corev1.Namespace, pod corev1.Pod, instAnnotation string) (*v1alpha1.Instrumentation, error) {
	instValue := annotationValue(ns.ObjectMeta, pod.ObjectMeta, instAnnotation)
	if len(instValue) == 0 || strings.EqualFold(instValue, "false") {
	return nil, nil
	}
	if strings.EqualFold(instValue, "true") {
	return pm.selectInstrumentationInstanceFromNamespace(ctx, ns)
	}

Did I understand it wrong? I think the premise for this feature to take effect is that the instrumentation.opentelemetry.io/inject-xx is equal to true.

[swiatekm-sumo]

What I meant to say was that this mechanism only applies if the annotation doesn't specify which instrumentation should be used. Apologies for the confusion.

[crossoverJie]

Thanks; I understand that the documentation has been supplemented.

[crossoverJie]

@swiatekm-sumo The E2E test failed, but I can pass it locally, any suggestions?

[swiatekm-sumo]

@swiatekm-sumo The E2E test failed, but I can pass it locally, any suggestions?

Can you try recreating the test cluster locally to see if tests still pass? It's possible you're using an older docker image in tests.

[crossoverJie]

Can you try recreating the test cluster locally to see if tests still pass? It's possible you're using an older docker image in tests.

Thanks; I have changed the e2e test.

[swiatekm-sumo]

@crossoverJie I think you're running into trouble merging changes in generated manifests. I suggest running make precommit and including the output in the merge commit.

[crossoverJie]

@crossoverJie I think you're running into trouble merging changes in generated manifests. I suggest running make precommit and including the output in the merge commit.

Thanks, done.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: crossoverJie / name: crossoverJie (4025131, bac91bf, 4f69d5b, 60540f0, 377b51f, 04badc2, 1a53108, 50ab7a2, 686f1c6, 8b7609a, b11cb13, 8d25853, 80bc887, 0a7d640, 787e7fe, 73eb99e, f531eb9, efd91de, fff0c2a, 7b754f4, a4500da)

[swiatekm-sumo]

@crossoverJie can you rebase on main? That should get rid of the EasyCLA complaint.

[crossoverJie]

@crossoverJie can you rebase on main? That should get rid of the EasyCLA complaint.

Thanks for your reminder, done.

[swiatekm-sumo]

I don't get why EasyCLA complains about my email address. Feel free to just drop the Co-authored-by: Mikołaj Świątek <mail+sumo@mikolajswiatek.com> messages from the commits it complains about. I don't want this to block this PR.

[pavolloffay]

I don't understand the use-case to have the Selector in the CR while the inject annotation is still needed.

What is the use-case this feature solves? Why do we need to maintain two approaches to solve the same use-case?

[crossoverJie]

This test case is to test that when there are multiple Instrumentation (they all use Selector), the Selector feature can still work as expected.

[swiatekm-sumo]

/easycla

Edit: FYI, the problem was communitybridge/easycla#4329.