Traces send by http protocol fail in new Otel collector receiver

[jufab]

What version of OpenTelemetry are you using?

0.23.0

What version of Node are you using?

v14.15.2

Please provide the code you used to setup the OpenTelemetry SDK

All code is here https://github.com/open-telemetry/opentelemetry-js/tree/main/examples/tracer-web

What did you do?

I used this project example and new Otel collector (0.29.0).

What did you expect to see?

Traces OK in collector

What did you see instead?

All traces send to the Otel Collector failed (404).

Additional context

No additional.

What I think about this issue

Traces are send with beacon with a body string, so the content-type is text/plain.
New collector (0.29.0) is less permissive and expect an application/json

[dgarciarubio]

This could be because the open telemetry collector has been updated in version 0.29.0 to require the Content-Type: application/x-protobuf request header as mentioned in open-telemetry/opentelemetry-collector#3540.

[jufab]

I think application/x-protobuf is for GRPC protocol. application/json is for HTTP protocol

or "Content-Type: application/json" request header when sending JSON encoded Protobuf payload

[niko-achilles]

yes i tried also yesterday the 0.29.0 release of the image of otel/opentelemetry-collector:0.29.0 .

When using for tracing web, zone.js makes a POST Request type of xhr and the response is 404 not found.
The Preflight requests OPTIONS are responding with a status of 200, OK and allowed methods POST.

also after reading the specs, i read that The client MUST set "Content-Type: application/x-protobuf" request header when sending binary-encoded Protobuf or "Content-Type: application/json" request header when sending JSON encoded Protobuf payload.

So, in code:

const provider = new WebTracerProvider({ resource });

  provider.addSpanProcessor(
    new SimpleSpanProcessor(
      new CollectorTraceExporter({
        url: "http://localhost:55681/v1/traces",
        headers: { "Content-Type": "application/json" },
      })
    )
  );

The result is again a 404.

Spec location HTTP Request: https://github.com/open-telemetry/oteps/blob/main/text/0099-otlp-http.md#otlphttp-protocol-details

For Nodejs application with NodeTraceProvider and otel/opentelemetry-collector:0.29.0 the communication is ok (no issues), because byx default if i use the library of Collector Exporter and not apply headers explicitly the library defaults to "Content-Type": "application/json". To test i changed the headers to "Content-Type": "text/plain;charset=UTF-8" and of course the traces are not send. So assuming according to the specs the collector needs a "Content-Type" header of application/json or application/x-protobuf for OTLP HTTP.

My assumption is that zone.js that comes with the registration of WebTraceProvider needs a content-type of text/plain;charset=UTF-8 for request header.

[jufab]

To help
tracer-web generate this http request

curl 'http://localhost:55681/v1/traces' \
-H 'Content-Type: text/plain;charset=UTF-8' \
--data-raw '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"unknown_service"}},{"key":"telemetry.sdk.language","value":{"stringValue":"webjs"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"0.23.0"}}],"droppedAttributesCount":0},"instrumentationLibrarySpans":[{"spans":[{"traceId":"5661215315a87ad7dd8448b4101a59a9","spanId":"29f50492db6b0ced","name":"files-series-info-0","kind":1,"startTimeUnixNano":1625600800211400200,"endTimeUnixNano":1625600800700400000,"attributes":[],"droppedAttributesCount":0,"events":[{"timeUnixNano":1625600800700400000,"name":"fetching-span1-completed","attributes":[],"droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0}],"instrumentationLibrary":{"name":"example-tracer-web"}}]}]}'

and not working (404).

But when you change content-type by `application/json``

curl 'http://localhost:55681/v1/traces' \
-H 'Content-Type: application/json' \
--data-raw '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"unknown_service"}},{"key":"telemetry.sdk.language","value":{"stringValue":"webjs"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"0.23.0"}}],"droppedAttributesCount":0},"instrumentationLibrarySpans":[{"spans":[{"traceId":"5661215315a87ad7dd8448b4101a59a9","spanId":"29f50492db6b0ced","name":"files-series-info-0","kind":1,"startTimeUnixNano":1625600800211400200,"endTimeUnixNano":1625600800700400000,"attributes":[],"droppedAttributesCount":0,"events":[{"timeUnixNano":1625600800700400000,"name":"fetching-span1-completed","attributes":[],"droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0}],"instrumentationLibrary":{"name":"example-tracer-web"}}]}]}'

it works.

[niko-achilles]

yes it wokrs with curl. In code where you use for web try this:

provider.addSpanProcessor(
    new SimpleSpanProcessor(
      new CollectorTraceExporter({
        url: "http://localhost:55681/v1/traces",
        headers: {},
      })
    )
  );

Note do not specify in headers object nothing headers: {} .
In this way it works also !

Mysteriously when i use headers: {"Content-Type": "application/json"} in code i see in the network console that the zone.js xhr request applies Content-Type: "application/json" application/json

And also if you delete the headers: {} object in code then zone.js applies as Content Type plain/text ...

[jufab]

if you specify headers config, you force to use xhr and not beacon.

In the code

https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-exporter-collector/src/platform/browser/CollectorExporterBrowserBase.ts#L44-L45

You use xhr if an headers config is present.
In send method

https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-exporter-collector/src/platform/browser/CollectorExporterBrowserBase.ts#L94-L98

you send with xhr method or you send with navigator.sendBeacon otherwise.

In
https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-exporter-collector/src/platform/browser/util.ts#L45
you find xhr method with application/json header definition

In
https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-exporter-collector/src/platform/browser/util.ts#L25
you find sendbeacon method with a body string.

So, without headers config, you use beacon (if your navigator support it :) )

[niko-achilles]

thanks @jufab for the insights. It is a surprise for the developer point o view that the Collector Trace Exporter behaves in that way with and without headers config. But good know since now with your insights i can experiment with otel/collector version 0.29.0.

[preston]

Does this appear to be the same issue? https://stackoverflow.com/questions/68552810/opentelemetry-collector-404ing-at-http-localhost55681-v1-traces

If so, I suspect that pretty much everyone attempting to use this lib is failing. :-/ I know I've already spent more than a few hours with various example projects and still haven't successfully submitted any traces. Looks like @niko-achilles PR will solve this?

[mogest]

I've found a terrible workaround which is working for my particular use case. The code falls back to using XHR if there's no sendBeacon available, so I've added this to my codebase for the time being until it's fixed up:

navigator.sendBeacon = undefined;

[niko-achilles]

@mogest
if you use opentelemetry-collector of version 0.29.0 then,
an elegant workaround to fallback to xhr without mutating the navigator.sendBeacon property is to define the config of Collector Trace exporter as follows:

new CollectorTraceExporter({
        headers: {},
      })

or if you want to be explicit, update npm packages to version 0.24.0 :

new CollectorTraceExporter({
        headers: {
              "Content-Type": "application/json"
             },
      })

note opentelemetry-collector of version 0.29.0 is released and that workaround may not be needed.