OpenCensus Agent not honoring a deny sampling decision for traces

[jdgeisler]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

We are using the opencensus agent for tracing so that we can support both B3 and W3C trace headers in the same trace.

Sometimes, for testing and debugging purposes, we need to override the trace headers in requests to force or deny sampling.

However, when using the opencensus agent, we notice that a deny sampling decision X-B3-Sampled: 0 is not always honored. Similarly, for W3C, using traceparent: 00-0af7651916cd43dd8448eb211c80312d-00f067aa0ba902b7-00 is also not always honored. Inversely, forcing sampling to be 100% is always honored.

I also tested this behavior using the zipkin tracer instead and it worked correctly. Using X-B3-Sampled: 0 successfully forced a deny sampling decision 100% of the time. So it seems the inconsistent sampling on a deny decision only happens when using opencensus.

Version

$ istioctl version
client version: 1.19.7
control plane version: 1.19.7
data plane version: 1.19.7 (36 proxies)

$ kubectl version
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.11-eks-b9c9ed7

Additional Information

It can be reproduced by the following:

1. Set your meshConfig to use the openCensusAgent for tracing, specifying both B3 and W3C

defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  tracing:
    openCensusAgent:
      address: opentelemetry-collector.istio-system:55678
      context:
      - B3
      - W3C_TRACE_CONTEXT
    sampling: 25

2. Generate traces to your service that use B3 or W3C trace contexts and force sampling to be 0

curl -k -H "X-B3-Sampled: 0" -H "traceparent: 00-0af7651916cd43dd8448eb211c80312d-00f067aa0ba902b7-00" https://fortio/debug

3. See that on some requests the trace is still sampled (X-B3-Sampled: 1), even though sampling should be denied via the X-B3-Sampled and traceparent headers

GET /fortio/debug/ HTTP/2.0

headers:

Host: fortio
Accept: */*
Traceparent: 00-0af7651916cd43dd8448eb211c80312d-6b9b9d03aaaa6bf4-01
User-Agent: curl/8.4.0
X-B3-Sampled: 1
X-B3-Spanid: 6b9b9d03aaaa6bf4
X-B3-Traceid: 0af7651916cd43dd8448eb211c80312d
X-Envoy-Attempt-Count: 1
X-Envoy-Internal: true
X-Forwarded-Proto: https
X-Request-Id: 52b34acb-cc19-92f2-b9c9-4e714c453277

4. Instead set X-B3-Sampled: 1 and traceparent: 00-0af7651916cd43dd8448eb211c80312d-00f067aa0ba902b7-01 and see that traces are sampled 100% of the time

[zirain]

cc @lei-tang

[lei-tang]

FWIW, OpenCensus is deprecated. The OpenCensus related features will be removed from Istio. Related discussion in #50808.

[jdgeisler]

@lei-tang Is there an alternative solution in Istio that permits this then?

interoperability between services that use different headers, e.g. one service that propagates B3 headers and one that propagates W3C Trace Context headers can participate in the same trace?

[zirain]

@lei-tang Is there an alternative solution in Istio that permits this then?

interoperability between services that use different headers, e.g. one service that propagates B3 headers and one that propagates W3C Trace Context headers can participate in the same trace?

you should use zipkin or maybe move to OpenTelemetry?

[jdgeisler]

you should use zipkin or maybe move to OpenTelemetry?

Well I mean we have some services in our mesh that use zipkin instrumentation with B3 and others use opentelemetry instrumentation with W3C.

Is there another way in istio, besides opencensus, that allows those services to participate in the same trace if they are propagating different headers? The reason we use the opencensus agent is so we don’t have disjoint traces in that scenario.

[lei-tang]

@lei-tang Is there an alternative solution in Istio that permits this then?

interoperability between services that use different headers, e.g. one service that propagates B3 headers and one that propagates W3C Trace Context headers can participate in the same trace?

AFAIK, besides opencensus, Istio currently does not have an alternative that permits this.

[jdgeisler]

AFAIK, besides opencensus, Istio currently does not have an alternative that permits this.

What would you recommend in our case then? Could we configure and set two trace providers in istio (both zipkin and opentelemetry), but only one can be used at a time right?

I'm assuming this issue in opencensus won't actually get fixed, considering its being fully removed soon.

[zirain]

with telemetry API, you van use different trace provider for different workloads/services, is that help you?

[jdgeisler]

@zirain Yep I understand that, thanks. It doesn't accomplish our use case of interoperability between services that use different headers, but it seems it is the best option we'll have in istio.