BigQuery metrics not collected

[mseiwald]

Hello,

I'm running prometheus stackdriver exporter to collect Dataflow and BigQuery metrics. The strange thing is that Dataflow metrics (and others) work fine, however BigQuery metrics simply do not show up. No BigQuery metrics show up on the prometheus search and curl localhost:9255/metrics | grep bigquery to the stackdriver exporter pod also doesn't yield any results.

stackdriver exporter args:

        - --google.project-id=XXXXXXX
        - --log.level=debug
        - --monitoring.metrics-interval=5m
        - --monitoring.metrics-offset=0s
        - --monitoring.metrics-type-prefixes=dataflow.googleapis.com/job,bigquery.googleapis.com
        - --stackdriver.backoff-jitter=1s
        - --stackdriver.http-timeout=10s
        - --stackdriver.max-backoff=5s
        - --stackdriver.max-retries=0
        - --stackdriver.retry-statuses=503
        - --web.listen-address=:9255
        - --web.telemetry-path=/metrics

stackdriver exporter logs (debug):

ts=2022-09-19T07:47:20.393Z caller=stackdriver_exporter.go:188 level=info msg="Starting stackdriver_exporter" version="(version=0.12.0, branch=HEAD, revision=cd3d54a2724b1a59dfa91231d4f067182d37f85f)"
ts=2022-09-19T07:47:20.394Z caller=stackdriver_exporter.go:189 level=info msg="Build context" build_context="(go=go1.17.6, user=root@60f827bcf91f, date=20220209-12:21:37)"
ts=2022-09-19T07:47:20.394Z caller=stackdriver_exporter.go:190 level=info msg="Using Google Cloud Project ID" projectID=XXXXXXX
ts=2022-09-19T07:47:20.396Z caller=stackdriver_exporter.go:212 level=info msg="Listening on" address=:9255
ts=2022-09-19T07:47:34.145Z caller=monitoring_collector.go:335 level=debug msg="listing Google Stackdriver Monitoring metric descriptors starting with" prefix=bigquery.googleapis.com
ts=2022-09-19T07:47:34.145Z caller=monitoring_collector.go:335 level=debug msg="listing Google Stackdriver Monitoring metric descriptors starting with" prefix=dataflow.googleapis.com/job
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/scanned_bytes_billed
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/scanned_bytes_billed\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/statement_scanned_bytes
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/execution_times
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/execution_times\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/inserted_rows_count
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/inserted_rows_count\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated_for_project
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated_for_project\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/assigned
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/assigned\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/job/num_in_flight
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/job/num_in_flight\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/table_count
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/table_count\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated_for_project_and_job_type
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated_for_project_and_job_type\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/max_assigned
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/max_assigned\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/total_available
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/total_available\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/scanned_bytes
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/scanned_bytes\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/capacity_committed
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/capacity_committed\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/statement_scanned_bytes_billed
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/statement_scanned_bytes_billed\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/stored_bytes
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/stored_bytes\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/count
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/count\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/uploaded_bytes_billed
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/uploaded_bytes_billed\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated_for_reservation
ts=2022-09-19T07:47:34.336Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated_for_reservation\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/uploaded_row_count
ts=2022-09-19T07:47:34.336Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/uploaded_row_count\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/total_allocated_for_reservation
ts=2022-09-19T07:47:34.337Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/total_allocated_for_reservation\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/uploaded_bytes
ts=2022-09-19T07:47:34.338Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/uploaded_bytes\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/statement_scanned_bytes\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/inserted_bytes_count
ts=2022-09-19T07:47:34.338Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/inserted_bytes_count\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/streaming_pull_connection_status
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/streaming_pull_connection_status\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/backlog_bytes
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/backlog_bytes\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/gpu_utilization
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/gpu_utilization\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/read_count
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/read_count\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/system_lag
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/system_lag\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_streaming_data_processed
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_streaming_data_processed\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/user_counter
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/user_counter\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/billable_shuffle_data_processed
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/billable_shuffle_data_processed\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/bundle_user_processing_latencies
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/bundle_user_processing_latencies\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/element_count
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/element_count\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_memory_usage_time
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_memory_usage_time\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_pd_usage_time
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_pd_usage_time\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_vcpu_time
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/elements_produced_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/estimated_byte_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/per_stage_system_lag
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/processing_parallelism_keys
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/thread_time
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/timers_processed_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/data_watermark_age
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/current_shuffle_slots
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/gpu_memory_utilization
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/streaming_engine/persistent_state/read_bytes_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/streaming_engine/persistent_state/write_latencies
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/bigquery/write_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/estimated_bytes_produced_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/is_failed
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/status
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/backlog_elements
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/estimated_backlog_processing_time
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/write_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/streaming_engine/persistent_state/write_bytes_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/memory_capacity
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/per_stage_data_watermark_age
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/write_latencies
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_shuffle_data_processed
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/disk_space_capacity
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/timers_pending_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/current_num_vcpus
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/duplicates_filtered_out_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/read_latencies
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/elapsed_time
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/timers_processed_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/elements_produced_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/data_watermark_age\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/processing_parallelism_keys\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_vcpu_time\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/streaming_engine/persistent_state/read_bytes_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/current_shuffle_slots\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/is_failed\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/status\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/gpu_memory_utilization\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/backlog_elements\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/current_num_vcpus\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/per_stage_system_lag\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/streaming_engine/persistent_state/write_latencies\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/bigquery/write_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/estimated_bytes_produced_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/estimated_byte_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/per_stage_data_watermark_age\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/duplicates_filtered_out_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/estimated_backlog_processing_time\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/write_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/read_latencies\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/streaming_engine/persistent_state/write_bytes_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/elapsed_time\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/thread_time\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/memory_capacity\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_shuffle_data_processed\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/write_latencies\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/timers_pending_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/disk_space_capacity\""
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"

Any ideas?

[mseiwald]

When setting --monitoring.metrics-interval to e.g. 24h the metrics show up in the /metrics endpoint but Prometheus does not accept them because they are too old:

ts=2022-09-22T07:10:12.622Z caller=scrape.go:1600 level=warn component="scrape manager" scrape_pool=serviceMonitor/logging-monitoring/prometheus-prometheus-stackdriver-exporter/0 target=http://10.140.5.111:9255/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=113

[EladDolev]

you can try and use the following, then adjust with offset when you're drawing the graph

serviceMonitor:
  honorTimestamps: false