Add support in google_logging_organization_sink / google_logging_folder_sink for intercepting log sinks

[danekantner]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Description

GCP recently added the ability to set up aggregated log sinks that are marked as intercepting log sinks in that they will then intercept all logs in the children folder hierarchy, namely for example those that would otherwise route to the _Default logging bucket in the original project. The logging destination for the interception to be enabled must be another project; intercepted logs then are managed according to the log sinks within that destination project's log sinks (e.g., the _Default log sink within the destination project instead).

Supporting this feature enables cleaner redirection of logs without having to manually manage both the inclusions to redirect and the exclusions elsewhere. This feature allows us to enable chargeback against log volume (log storage SKU) in a shared GKE cluster to an actual engineering service's destination project. This also enhances security by allowing easier methods to isolate viewership of logs.

New or Affected Resource(s)

• google_logging_organization_sink

Potential Terraform Configuration

resource "google_logging_organization_sink" "my-sink" {
  name   = "my-sink"
  description = "some explanation on what this is"
  org_id = "123456789"

  destination = "logging.googleapis.com/projects/[PROJECT_ID]"
  include_children = true
  intercept = true

  # Log all WARN or higher severity messages relating to instances
  filter = "resource.type=k8s_container AND resource.labels.cluster_name=prod-env"
} 

Presently the aggregated sink resource supports 2 of 3 available options by toggling the include_children on or off. The 3rd option, to allow interception, also requires include_children to be enabled. This option is only compatible with project i.d. as the destination. Fundamentally in TF there may be a wiser move than adding another optional toggle such as a configuration block, this example was used to follow the existing formats.

References

https://cloud.google.com/logging/docs/export/aggregated_sinks#overview

An intercepting aggregated sink prevents logs from being routed through sinks in child resources, except for _Required sinks. This sink can be useful in preventing duplicate copies of logs from being stored in multiple places.

For example, consider [Data Access audit logs](https://cloud.google.com/logging/docs/audit#data-access), which can be large in volume and expensive to store multiple copies of. If you've enabled Data Access audit logs, you might create a folder-level intercepting sink that routes all Data Access audit logs to a central project for analysis. This intercepting sink also prevents sinks in child resources from routing copies of the logs elsewhere.

Intercepting sinks prevent logs being passed through the Log Router of child resources, unless the logs also match the _Required sink. Because the logs are intercepted, the logs don't count towards log-based metrics or log-based alerts in the child resources. You can view intercepting sinks in the Log Router page of child resources.

The UX for this shows this as

b/337873401