Jaeger v2 based on OpenTelemetry collector

[yurishkuro]

Proposal: https://docs.google.com/document/d/1s4_6VgAS7qAVp6iEm5KYvpiGw3h2Ja5T5HpKo29iv00/view

This replaces the previous attempt: #3500.

Work tracked in the project https://github.com/orgs/jaegertracing/projects/3

Roadmap

Proof of concept

• A single binary with memory storage and all-in-one functionality
  • Add jaeger-v2 single binary based on OTEL collector #4766
• Validate that the current storage configuration works for primary / archive storage.
• Default configuration solution (do not require config file for all-in-one)
  • feat(jaeger-v2): Create default config for all-in-one #4842
• Demonstrate Kafka exporter / receiver (ideally from OTEL), compatible with existing Jaeger Kafka writer
  • [Jaeger v2] Add Kafka exporter and receiver #4868
• Define v2 storage API that operates on OTLP batches, see if internal & remote APIs can be identical
  • Define internal version of OTLP proto types using Gogo optimizations #5052
  • [Feature]: Define v2 storage API that operates on OTLP batches #5079
• Benchmarks [Jaeger v2] Run benchmarks #4869

Feature parity

• Extend configuration of jaeger_storage_exporter to support all existing backends -- project in Jaeger's LFX Mentorship - 2024 Term 01, March - May #5084
• Metrics & Traces -- project in Jaeger's LFX Mentorship - 2024 Term 01, March - May #5084
• What to do with admin port / services
• Adaptive sampling processor -- project in Jaeger's LFX Mentorship - 2024 Term 01, March - May #5084

Prepare for Beta

• Add to integration tests
• Add to release pipeline
• Documentation

Prepare for GA

• Helm chart [Feature]: Support Jaeger-v2 helm-charts#534
• Operator [Feature]: Support Jaeger-v2 jaeger-operator#2411
• Clarify https://github.com/jaegertracing/jaeger#version-compatibility-guarantees

Enhancements

• SPM built-in
• Integrate with OTEL collector builder ocb
• Print active configuration (unresolved: Print final configuration OTEL #1791), Auto Generation of Component Configuration from metadata.yaml  open-telemetry/opentelemetry-collector-contrib#27003

[pavolloffay]

The V2 is as well affected by open-telemetry/opentelemetry-collector#8768, It can cause startup failure.

[yurishkuro]

The new jaeger binary is available as Docker image https://hub.docker.com/repository/docker/jaegertracing/jaeger/tags

[yurishkuro]

I wrote down some thoughts on what Jaeger Storage API v2 might look like. Interested in feedback.

[hdost]

I wrote down some thoughts on what Jaeger Storage API v2 might look like. Interested in feedback.

So one thing that I think might be worth thinking about as well is search complexity. One issue I know we had was making search efficient on large span storage.
Time range to limit is useful, but painful for users. Being able to search on attributes is a must. It might be good add criteria to compare about query patterns as well.

[88vishnu]

Appx by when Jaeger-v2 will be available with tail-based-sampling??

[yurishkuro]

I am hoping to make good progress on v2 in the Spring if we get 3 interns, and then in the summer to get it to GA.

[yasharne]

Is this open to contributing? if yes, where can I find the open tasks/issues?

[yurishkuro]

@yasharne it is definitely open, but I'd say most open issues here are medium complexity, not good-first-issue. The open issues are listed in this ticket (and more in Google docs).

[tmeckel]

When I try to use the OLTPLogExporter with Jeager and GRPC,

details collased

```python get_logger_provider().add_log_record_processor( BatchLogRecordProcessor( exporter=OTLPLogExporter( endpoint="http://jaeger:4317", insecure=True, ), ) ) ```

I recieve the following error:

2024-02-27 13:41:34,233 ERROR [opentelemetry.exporter.otlp.proto.grpc.exporter] [exporter.py:306] - Failed to export logs to jaeger:4317, error code: StatusCode.UNIMPLEMENTED

An indication that https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/logs/v1/logs_service.proto is not implemented by Jaeger. Is this correct? Or did I configured Jaeger incorrectly, ref. to following Docker Compose snippet (ports 4317 and 4318 are used only internally using the network ipam)?

---
# This is a Docker Compose override file to add jeager tracing
version: "3.9"

services:
  jaeger:
    image: jaegertracing/all-in-one:${JAEGER_VERSION:-latest}
    ports:
      - "16686:16686"
    environment:
      - LOG_LEVEL=debug
      - COLLECTOR_OTLP_ENABLED=true
    networks:
      - ipam

  network-managemnent:
    depends_on:
      - jaeger

open-telemetry/opentelemetry-collector#6363

[yurishkuro]

@tmeckel Jaeger is not a logging backend, it only accepts traces.