gRPCLB fails RPCs if gRPCLB server unavailable

[ejona86]

This is split out of #7643 (comment):

The initial RPCs seem to fail because the picker is immediately failed when lb RPC fails and there are no backends. Changing the logic to try fallback backends first should be possible, but it also seems it will be a hard to define/implement all the edge cases.

@apolcyn, does it seem wrong that initial RPCs fail if gRPCLB server is unavailable, when fallback is available but hasn't been tried yet? It appears later RPCs would succeeded as the picker would be replaced to buffer further RPCs and eventually the fallback would come online. It seems like we may need to change this error behavior.

[apolcyn]

@apolcyn, does it seem wrong that initial RPCs fail if gRPCLB server is unavailable, when fallback is available but hasn't been tried yet? It appears later RPCs would succeeded as the picker would be replaced to buffer further RPCs and eventually the fallback would come online. It seems like we may need to change this error behavior.

Yes I agree that this current behavior seems not ideal. The behavior when the LB call fails should IMO be handled the same as when the connection to the LB service fails -- that is, I think we should enter fallback mode, and only fail RPCs if there is then a problem with the picker over the fallback addresses.

[temawi]

@apolcyn Do you think this one is still a relevant issue?