api: add new pendingStreamCreated on clientStreamTracer

[YifeiZhuang]

b/259730220
go/grpc-java-new-trace-lines

Problem

Direct Path users (Big Table, Spanner) occasionally see spikes of RPC latencies and it is hard to debug the cause due to the lack of information. Users look for more information in tracing. Specifically, it requires a few timestamps in the RPC lifecycle .

Background

Before name resolution returns service configs, the ClientCalls are pendingCalls, the steam creation calls are saved.
After name resolution is done, calls are drained: this triggers stream creation, and at the same time the stream tracer is created from all the stream tracer factories. The stream created may not be real stream, instead it forks to two situations:

• If a picker is there, a real stream is immediately created from the transport.
• If the picker is not available yet, a pending stream is created. And creating a real stream is delayed until all transports are ready, i.e. connection is ready, handshake and protocol negotiation completed.

Client transport calls back streamCreate() on stream tracer (via StatsTraceContext).

Solution

1. Add a new method on client stream tracer. The new method is called when pendingStream is created. When this method is invoked, it indicates picker gets delayed for whichever reason.

public abstract class ClientStreamTracer extends StreamTracer {
public void pendingStreamCreated() {

}
…
}

2. Add span annotation in current streamCreated(). It indicates that connection is ready, i.e. protocol negotiation and handshake is done.

cc. @mohanli-ml

[ejona86]

The existing annotation didn't include a period. We should do the same.

Is "real stream" meaningful to users? Should we instead say something like "Transport selected" or "Load balancer provided a transport"? Do we know what string C used here?

[YifeiZhuang]

C use "Delayed name resolution complete." and "Delayed LB pick complete."
It is a good idea to use the same descriptions, at least for the this new one.
grpc/grpc#31913

[ejona86]

I don't this annotation gains us anything, as this will be the same as the start of the span.

[YifeiZhuang]

For The start of the span you mean when the stream tracer is created by the census interceptor, right?
That is around the same time as pending stream is created. The difference is that every stream has start of span moment, but only every stream has createPendingStream momemt.

[YifeiZhuang]

discussed offline, this needs to be added on parent span. Plus creatingPendingStream is not sufficiently mean name resolution delay. Maybe we will add follow up PR about name resolution delay timestamp.