Spans aren't being marked as errors in Cloud Trace

[andypwarren]

Hi,

I'm instrumenting a gRPC server with OpenTelemetry and Google Cloud Trace. I can see spans in my Trace dashboard but they aren't being coloured red if an rpc returns an internal error. I'm using the otelgrpc.UnaryServerInterceptor() (code here) which calls span.SetStatus with the otel error code and the grpc message if any of these statuses are returned

• grpc_codes.Unknown
• grpc_codes.DeadlineExceeded
• grpc_codes.Unimplemented
• grpc_codes.Internal
• grpc_codes.Unavailable
• grpc_codes.DataLoss

I've also tried calling span.SetStatus outside the interceptors and Cloud Trace doesn't colour them red either so I don't think the problem is with the interceptor code.

I've created a simple demo app to reproduce this using the example grpc-go Greeter service with the addition of tracing using otel and cloudtrace.

When forcing a request to fail this is what I see in cloud trace

The interceptor has added the attribute rpc.grpc.status_code: 13 but the span status isn't showing up.

Ideally this would produce a red dot in the trace graph and the span would be coloured red.

Many thanks,

Andy

[dashpole]

This seems suspicious...

opentelemetry-operations-go/exporter/trace/trace_proto.go

Lines 162 to 163 in 6b51e22

	case codes.Error:
	sp.Status = &statuspb.Status{Code: int32(codepb.Code_UNKNOWN), Message: s.Status().Description}

[dashpole]

Seems potentially related to #143

[dashpole]

@aabmass do you remember why we set codes.Error to codepb.Code_UNKNOWN?

[aabmass]

Unknown represents an unknown error, along the lines of HTTP 500 status code. Since OTel only has two possible statuses (OK and ERROR), gRPCs UNKNOWN (error) seems reasonable.

Do you know what status codes actually show red in Cloud Trace?

[dashpole]

It does seem like we are doing the right thing based on https://pkg.go.dev/google.golang.org/genproto/googleapis/rpc/code#Code

// Unknown error. For example, this error may be returned when
// a Status value received from another address space belongs to
// an error space that is not known in this address space. Also
// errors raised by APIs that do not return enough error information
// may be converted to this error.
//
// HTTP Mapping: 500 Internal Server Error
Code_UNKNOWN Code = 2

Do you know what status codes actually show red in Cloud Trace?

I'll see if I can find the answer to that question.

[dashpole]

I tested all status codes, and none appear to make the span look like an error

[dashpole]

I'll reach out to the trace UI team.

[BradleyChatha]

For further context, the way we're getting around this currently is by setting the attribute /http/status_code to 500 regardless of whether the context is for a HTTP server or not.

It seems to be the only way to make the trace UI render it as an error.

[andypwarren]

Hi @dashpole, is there any update on this?

[dashpole]

The cloud trace folks are aware of the issue, and suggested the same workaround pointed out above: #730 (comment). I'm not sure about timelines, but i'll post here when there are updates.

[shraddhaag]

+1. We are facing this problem as well. Thanks for pointing to the workaround!