Universal debug logging enable (environment variable) for GCP client libraries

[feywind]

Summary

The tl;dr for this is: I'm proposing a GCP-wide environment variable that lets users opt in to receiving debug logging for various parts of the stack/libraries/etc. This is an AIP because the environment variable and its format are a specification we'd like to standardize on; the implementation (if/how) are left up to library authors.

I'm working on an AIP template for this (at the instigation of @bcoe), but following the instructions to make an issue here first.

Overview

As a client library author, it's a frequent frustration to help customers debug problems when we can't consistently get debug logging output, or the logging enables work in different ways. My canonical example is the library I work most with, nodejs-pubsub:

• Client surface (nodejs-pubsub) - no specific mechanism for debug logging enables
• gax-node - no debug logging available
• grpc-js - has its own specific mechanism that cuts across grpc impls

All three of these layers are relevant to debugging hard problems for customers. Additionally, we have customers who use different languages in different parts of their services, and each of those languages has its own mechanisms (or nothing). Back and forth with front-line support and customers wastes everyone's time, and it would be preferable if we could just tell them a single set of instructions to get more info.

The whole proposal as a "should", not a "must"; it's optional, but since there haven't been good platform-wide guidelines on this before, it should help make the experience more coherent.

Proposal

This follows the general guidelines of the Node logging variables, but it could be expanded (or swapped) if needed, to support other languages or features.

The proposed environment variable is GCP_DEBUG. This is short and sweet, and should be pretty unique, but if we want to match up with other envvars (GOOGLE_CLOUD_DEBUG or something) that'd be fine.

The value of the variable is a comma-delimited list of systems and subsystems, where either can be a wildcard (*). So for example:

• GCP_DEBUG=* - turn on all debugging logs
• GCP_DEBUG=pubsub:*,bigquery:queries - turn on all pubsub logs, and queries related BQ logs
• GCP_DEBUG=gax:retries - turn on only gax retry logging

That is where this AIP stops - it's up to each library and each language to decide what the system/subsystem tags mean, and how to implement the logging. A sample implementation for Node has been created that adds minimal code to a library and optionally hooks into the debug npm module for colour and timestamps.

Shims / compat

Some libraries do have their own logging (specifically mentioning grpc) that we could fairly easily shim into this system, to retain a single facade for customers and support. e.g.:

• GCP_DEBUG=grpc:*

The environment variable parser could take this and convert it into the appropriate GRPC_OTHER environment variables. It's not a 100% match, since grpc can support multiple log levels, but again, this is just trying to raise the bar on ease of enabling debug logging to solve customer issues.

[Cachiman]

Summary

The tl;dr for this is: I'm proposing a GCP-wide environment variable that lets users opt in to receiving debug logging for various parts of the stack/libraries/etc. This is an AIP because the environment variable and its format are a specification we'd like to standardize on; the implementation (if/how) are left up to library authors.

I'm working on an AIP template for this (at the instigation of @bcoe), but following the instructions to make an issue here first.

Overview

As a client library author, it's a frequent frustration to help customers debug problems when we can't consistently get debug logging output, or the logging enables work in different ways. My canonical example is the library I work most with, nodejs-pubsub:

• Client surface (nodejs-pubsub) - no specific mechanism for debug logging enables
• gax-node - no debug logging available
• grpc-js - has its own specific mechanism that cuts across grpc impls

All three of these layers are relevant to debugging hard problems for customers. Additionally, we have customers who use different languages in different parts of their services, and each of those languages has its own mechanisms (or nothing). Back and forth with front-line support and customers wastes everyone's time, and it would be preferable if we could just tell them a single set of instructions to get more info.

The whole proposal as a "should", not a "must"; it's optional, but since there haven't been good platform-wide guidelines on this before, it should help make the experience more coherent.

Proposal

This follows the general guidelines of the Node logging variables, but it could be expanded (or swapped) if needed, to support other languages or features.

The proposed environment variable is GCP_DEBUG. This is short and sweet, and should be pretty unique, but if we want to match up with other envvars (GOOGLE_CLOUD_DEBUG or something) that'd be fine.

The value of the variable is a comma-delimited list of systems and subsystems, where either can be a wildcard (*). So for example:

• GCP_DEBUG=* - turn on all debugging logs
• GCP_DEBUG=pubsub:*,bigquery:queries - turn on all pubsub logs, and queries related BQ logs
• GCP_DEBUG=gax:retries - turn on only gax retry logging

That is where this AIP stops - it's up to each library and each language to decide what the system/subsystem tags mean, and how to implement the logging. A sample implementation for Node has been created that adds minimal code to a library and optionally hooks into the debug npm module for colour and timestamps.

Shims / compat

Some libraries do have their own logging (specifically mentioning grpc) that we could fairly easily shim into this system, to retain a single facade for customers and support. e.g.:

• GCP_DEBUG=grpc:*

The environment variable parser could take this and convert it into the appropriate GRPC_OTHER environment variables. It's not a 100% match, since grpc can support multiple log levels, but again, this is just trying to raise the bar on ease of enabling debug logging to solve customer issues.

[leahecole]

Mentioned in an internal doc, but I strongly feel we need to use GOOGLE_CLOUD_DEBUG instead of GCP_DEBUG to be in line with current branding guidelines. Otherwise, I love this.

[feywind]

This spec is currently deprecated in favour of some new work, but a lot of the concepts will be reused. So I'll leave it here for now and come back later to update.