Jaeger exporter creates too big UDP packets

[jflipts]

Description

Jaeger exporter logs errors like these: 'data does not fit within one UDP packet; size 65006, max 65000, spans 440'. This leads to spans missing from our traces.

Environment

• OS: alpine base image
• Architecture: x86
• Go Version: 1.17
• opentelemetry-go versions:
  go.opentelemetry.io/otel v1.3.0
  go.opentelemetry.io/otel/exporters/jaeger v1.3.0
  go.opentelemetry.io/otel/trace v1.3.0

Steps To Reproduce

1. Produce traces with a thousand spans.
2. Error 'data does not fit within one UDP packet; size 65006, max 65000, spans 440' is logged occasionally.

Expected behavior

Jaeger exporter does not create UDP packets that are too big.

[Aneurysm9]

The exporter already has some logic that attempts to ensure a transmission will not exceed the UDP max packet size. It looks like there may be some serialized data not accounted for in this calculation.

[jflipts]

I've done some more digging and in the original jaeger-client-go where this jaeger exporter is based on, they compensate with an extra 70 bytes. https://github.com/jaegertracing/jaeger-client-go/blob/master/transport_udp.go

// Empirically obtained constant for how many bytes in the message are used for envelope.
// The total datagram size is:
// sizeof(Span) * numSpans + processByteSize + emitBatchOverhead <= maxPacketSize
//
// Note that due to the use of Compact Thrift protocol, overhead grows with the number of spans
// in the batch, because the length of the list is encoded as varint32, as well as SeqId.
//
// There is a unit test TestEmitBatchOverhead that validates this number, it fails at <68.
const emitBatchOverhead = 70

I see no similar compensation happening in the exporter of this library. Based on the quote, I believe we are seeing this error due to our high number of small spans which causes the length encoding being longer than normal.

[Jownathan]

We started using the new Jaeger agent UDP exporter from the latest 'main' commit which contains the fix from this pull request.

Even with this fix, the problem still occurs:

data does not fit within one UDP packet; size 64932, max 64930, spans 453

It seems that the emitBatchOverhead variable is not only used for limiting the amount of spans that get added (which is what was required to fix this issue) but it is also used for the maximum size of an UDP packet (which is incorrect).
As seen in the error message max 64930 should be max 65000, which would fix the issue.