Include instance labels in COMPUTE_CONSTANTS for Google Compute Engine

[eastlondoner]

This is a feature suggestion:

GCE allows administrators to apply resource labels.

One of the main reasons for using labels is to create arbitrary groupings of resources.

It would be really great if the resource labels for a compute instance were automatically included in the Stackdriver logs for that instance so users could easily search for all logs from instances with a particular label.

[eastlondoner]

hello?? Anyone there?

[qingling128]

Hi @eastlondoner, the resource labels for a compute instance were automatically included in the Stackdriver logs for that instance should already happen today. Which labels are missing?

[eastlondoner]

@qingling128 It doesn't happen today.
I have an instance with an environment label. When I look at logs from that instance I only see:

{
 insertId:  "1o7kg3tg2iumqcb"   
 jsonPayload: {
  MESSAGE:  "I1120 09:46:31.278769    3083 server.go:796] GET /metrics: (20.703208ms) 200 [[python-requests/2.19.1] 10.8.2.5:59990]"    
  PRIORITY:  "6"    
  SYSLOG_FACILITY:  "3"    
  SYSLOG_IDENTIFIER:  "kubelet"    
  _BOOT_ID:  "b4215ff6aa9648f3b9dd10b1328e6b53"    
  _CAP_EFFECTIVE:  "3fffffffff"    
  _CMDLINE:  ...
  _COMM:  "kubelet"    
  _EXE:  "/home/kubernetes/bin/kubelet"    
  _GID:  "0"    
  _HOSTNAME:  ...
  _MACHINE_ID:  ...
  _PID:  "3083"    
  _SYSTEMD_CGROUP:  "/system.slice/kubelet.service"    
  _SYSTEMD_INVOCATION_ID:  "8aee3f7f32e54b86a4ff49eb85b2292f"    
  _SYSTEMD_SLICE:  "system.slice"    
  _SYSTEMD_UNIT:  "kubelet.service"    
  _TRANSPORT:  "stdout"    
  _UID:  "0"    
 }
 labels: {
  compute.googleapis.com/resource_name:  "fluentd-gcp-v3.1.0-bsw9d"    
 }
 logName:  "projects/neo4j-cloud/logs/kubelet"   
 receiveTimestamp:  "2018-11-20T09:46:35.393328228Z"   
 resource: {
  labels: {
   instance_id:  ...
   project_id:  ...
   zone:  ...
  }    
  type:  "gce_instance"    
 }
 timestamp:  "2018-11-20T09:46:31Z"   
}

[eastlondoner]

@qingling128 perhaps you could show me the place in the code of this plugin where the labels are attached?

[qingling128]

I see. The client side agent only associate logs with the monitored resource defined here. It comes with minimum set of labels to identify the resource. Metadata is ingested at the backend. To support that, we need to file a feature request with the backend to add additional metadata for gce_instance monitored resource.

Alternatively, there are a few ways to set the labels that are documented here.

[eastlondoner]

the alternatives are basically impossible with GKE. As a GKE user we have not got access to the fluentd configuration on the agent.

[igorpeshansky]

Labels (more precisely, pod and node annotations) are already ingested on GKE with the beta monitoring experience. @qingling128's comment was specifically about GCE.

[qingling128]

Yeah, it sounded like the request was about GCE. If it's for GKE, those labels should have been handled already.

[eastlondoner]

@igorpeshansky pod and node annotations are not the same thing as GCE labels

[jkohen]

@qingling128 is there anything left to do here?