Use OpenTelemetry tracer, bridge OpenTracing and OpenCensus

[kvrhdn]

What this PR does:
Adds a new feature flag use_otel_tracer which replaces the OpenTracing tracer with the OpenTelemetry tracer. The OTel tracer will install two bridges (OpenTracing and OpenCensus) so all existing spans will be collected and sent.

Why?

• This will allows us to gather OpenCensus spans, these are for instance emitted by the GCS SDK.
• OpenTracing has merged into OpenTelemetry

The feature flag defaults to false, i.e. keep using the OpenTracing tracer as before. There are a couple of issue we still need to iron out and not all features match (i.e. remote sampling is missing from the Jaeger exporter).

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[kvrhdn]

Issues:

• context is not propagated between instances OpenTracing bridge: BridgeTracer.Inject errors with ErrInvalidCarrier open-telemetry/opentelemetry-go#2137
  • ✅, will be fixed: OpenTracing Bridge: allow more generic carriers open-telemetry/opentelemetry-go#2141
• we are getting misconfiguration warnings from the OpenTracing bridge:

  SDK should have deferred the context setup, see the documentation of go.opentelemetry.io/otel/bridge/opentracing/migration
  

  • I asked how we can avoid this: OpenTracing bridge: BridgeTracing warns about deferred context setup, but how can this be avoided? open-telemetry/opentelemetry-go#2142
  • 🤷🏻, traces seem fine, so probably okay to ignore
• we are getting warnings from the OpenCensus bridge, this is originating from the Prometheus exporter built into OpenCensus

  ignoring custom sampler for span "ExportMetrics" created by OpenCensus because OpenTelemetry does not support creating a span with a custom sampler
  

  • this creates a log line for every /metrics scrape of the distributor
  • why this happens:
    • the Otel receiver uses the OpenCensus stats package
    • In OpenCensus metric exporter: Reader creates a span using a Sampler
    • this Sampler is hardcoded (can't be disabled)
  • 🤷🏻, probably safe to ignore?
  • the excessive logging is problematic though, it's logging a line for every /metrics scrape of a distributor 😬

[kvrhdn]

Example trace with spans from the GCS SDK

[kvrhdn]

Process tags have changed slightly:

OpenTracing

OpenTelemetry