How do I add request body as span attributes using the instrumentation-http?

[Ankcorn]

Helloo 👋

I am trying to trace some http requests my application is sending and was wondering what is the prefered way to add the request body to the trace, I can't seem to find the http request body in any of the options functions.

I can the the ClientReqest object from the requestHook function but cannot get the body from that, am I doing this wrong?

new HttpInstrumentation({
      startOutgoingSpanHook(request) {
        console.log("outgoing span", request);
        return {};
      },
      startIncomingSpanHook(request) {
        console.log("incoming req", request),
          request.addListener("data", (data) => {
            console.log("incoming!!!!", data.toString());
          });
        return {};
      },
      responseHook(span, response) {
        if (response instanceof IncomingMessage) {
          response.addListener("data", (data) => {
            const contentType = response.headers["content-type"];
            const rawResponseBody = data.toString();
            let body;
            if (contentType?.includes("application/json")) {
              body = JSON.parse(rawResponseBody);
            }

            // TODO support all content types
            if (body) {
              span.setAttributes(flattenObject(body, "http.res.body"));
            }
          });
        }
      },
      headersToSpanAttributes: {
        client: {
          requestHeaders: ["authorization"],
        },
      },
    })
    ```

[dyladan]

Hmm I've never really thought about getting the outgoing body from a ClientRequest object. It may not be possible or easy. The request object itself doesn't save the outgoing request body and the write method writes it to the kernel buffer. I'm not sure if any events are emitted (there is at least no documentation that any are). The ClientRequest is a child of OutgoingMessage which is a write-only stream according to the docs of OutgoingMessage.pipe().

My quick search seems to indicate it is not possible. If it is possible, requestHook is likely the right place to be hooking. It may depend on undocumented internals in order to access it if it is possible at all. We do not intercept the body at all so I don't think there is any place we would be interfering with normal operation there.

[Flarna]

storing the body on default in the instrumentation might be problematic as body can be really big.
To capture response body write/end need to be instrumented. For request body data/end/readable events and read API would be likely needed to instrument
This should be done with care as HTTP req/res body are streams and streams are used at a lot other places.

If body is compressed this would capture the compressed body.

I guess something like this would be by far easier done on application layer for specific routs only as there are likely infos regarding content, compression, expected size,... already available. Maybe the body is also available there for other reasons.

[gmreads]

@Ankcorn I was looking to something similar. Tried quite a lot by adding listeners to data, end events in responseHook but it did not work.

Eventually managed to do it by patching the send method of response object from express in the otel/instrumentation-express package.

[eliasfeijo]

I've managed to get the request body by using the requestHook and overwriting the write and end request methods:

new HttpInstrumentation({
  requestHook: (span, request: ClientRequest) => {
    const requestBodyChunks = [];
    const oldWrite = request.write.bind(request);
    request.write = (data: any) => {
      requestBodyChunks.push(decodeURIComponent(data.toString()));
      return oldWrite(data);
    };
    const oldEnd = request.end.bind(request);
    request.end = (data: any) => {
      if (data) {
        requestBodyChunks.push(decodeURIComponent(data.toString()));
      }
      console.log('request body:', requestBodyChunks.join());
      return oldEnd(data);
    };
  },
})

[jishnumuralikrishna]

@eliasfeijo I tried this. But it isn't working. Am I missing something?

[eliasfeijo]

@jishnumuralikrishna Could you provide more information? It could be related to your Node.js environment, system specs, or even a problem specific to your project (such as broken dependencies), so I can't say for sure.

[Ankcorn]

Apologies for the extremely slow response on this.

I have just implemented this - it works!

[Ankcorn]

The solution here no longer works - I can't attach the data to the span because the span is no longer active, I will take a look to see if we can find another way

It's been a good run. Thanks @eliasfeijo