Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add list of metrics by level to Collector's internal telemetry docs #4278

Closed
tiffany76 opened this issue Apr 10, 2024 · 9 comments · Fixed by #4322
Closed

Add list of metrics by level to Collector's internal telemetry docs #4278

tiffany76 opened this issue Apr 10, 2024 · 9 comments · Fixed by #4322

Comments

@tiffany76
Copy link
Contributor

tiffany76 commented Apr 10, 2024

Desired feature or idea: The documentation for configuring internal metrics for the Collector should include a list of the metrics available at each level (basic, normal, detailed, etc.). Work is being done on this list in open-telemetry/opentelemetry-collector#9767. Once the list is finalized, it should be added to the website documentation.

Additional context:

@theletterf
Copy link
Member

Pretty interesting change. Thanks for raising this, @tiffany76 !

@open-telemetry/collector-maintainers how would you like this information to be presented? What's the default state and under which circumstances would each level be used?

@tiffany76
Copy link
Contributor Author

Now that the Collector PR 9767 has been merged, I'm going to work on this issue in parallel with the other internal telemetry docs. I'll put up a fresh PR this week.

@theletterf Thanks for posing those questions. @open-telemetry/collector-approvers, if you have any thoughts before I get started, please share!

@tiffany76
Copy link
Contributor Author

Hi folks, I need some help with this one.

I've got my Collector instance up and running, and I'm feeding it telemetry with telemetrygen. I have a list of internal metrics from my local metrics endpoint (pasted below), but I'm not sure if it represents a complete list of all internal metrics.

I've also checked the Semantic Conventions docs, but I don't see internal metrics called out.

Where could I find or how could I generate a complete list of internal metrics?

Relatedly, does the list of internal metrics change depending on which components are being used? And if so, do we want to list all internal metrics or only those that have a verbosity level other than the now-default normal?

# HELP otelcol_exporter_send_failed_metric_points Number of metric points in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_metric_points counter
otelcol_exporter_send_failed_metric_points{exporter="debug",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 0
# HELP otelcol_exporter_send_failed_spans Number of spans in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_spans counter
otelcol_exporter_send_failed_spans{exporter="debug",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 0
# HELP otelcol_exporter_sent_metric_points Number of metric points successfully sent to destination.
# TYPE otelcol_exporter_sent_metric_points counter
otelcol_exporter_sent_metric_points{exporter="debug",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 244
# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
# TYPE otelcol_exporter_sent_spans counter
otelcol_exporter_sent_spans{exporter="debug",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 248320
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 3.79
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 2.24362496e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 8.0079232e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 9.83230608e+08
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 1.49384488e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 138.772669691
# HELP otelcol_processor_batch_batch_send_size Number of units in the batch
# TYPE otelcol_processor_batch_batch_send_size histogram
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="25"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="50"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="75"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="100"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="250"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="500"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="750"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="1000"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="2000"} 0
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="3000"} 1
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="4000"} 1
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5000"} 1
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="6000"} 1
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="7000"} 1
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="8000"} 1
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="9000"} 31
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10000"} 31
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="20000"} 31
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="30000"} 31
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="50000"} 31
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="100000"} 31
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="+Inf"} 31
otelcol_processor_batch_batch_send_size_sum{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 248320
otelcol_processor_batch_batch_send_size_count{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 31
# HELP otelcol_processor_batch_batch_size_trigger_send Number of times the batch was sent due to a size trigger
# TYPE otelcol_processor_batch_batch_size_trigger_send counter
otelcol_processor_batch_batch_size_trigger_send{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 30
# HELP otelcol_processor_batch_metadata_cardinality Number of distinct metadata value combinations being processed
# TYPE otelcol_processor_batch_metadata_cardinality gauge
otelcol_processor_batch_metadata_cardinality{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 1
# HELP otelcol_processor_batch_timeout_trigger_send Number of times the batch was sent due to a timeout trigger
# TYPE otelcol_processor_batch_timeout_trigger_send counter
otelcol_processor_batch_timeout_trigger_send{processor="batch",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 1
# HELP otelcol_receiver_accepted_metric_points Number of metric points successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_metric_points counter
otelcol_receiver_accepted_metric_points{receiver="prometheus",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",transport="http"} 244
# HELP otelcol_receiver_accepted_spans Number of spans successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_spans counter
otelcol_receiver_accepted_spans{receiver="otlp",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",transport="grpc"} 248320
# HELP otelcol_receiver_refused_metric_points Number of metric points that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_metric_points counter
otelcol_receiver_refused_metric_points{receiver="prometheus",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",transport="http"} 0
# HELP otelcol_receiver_refused_spans Number of spans that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_spans counter
otelcol_receiver_refused_spans{receiver="otlp",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",transport="grpc"} 0
# HELP otelcol_rpc_server_duration Measures the duration of inbound RPC.
# TYPE otelcol_rpc_server_duration histogram
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="0"} 0
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5"} 456
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10"} 456
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="25"} 456
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="50"} 456
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="75"} 456
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="100"} 456
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="250"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="500"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="750"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="1000"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="2500"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5000"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="7500"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10000"} 485
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="+Inf"} 485
otelcol_rpc_server_duration_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 4664.201886999997
otelcol_rpc_server_duration_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 485
# HELP otelcol_rpc_server_request_size Measures size of RPC request messages (uncompressed).
# TYPE otelcol_rpc_server_request_size histogram
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="0"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="25"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="50"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="75"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="100"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="250"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="500"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="750"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="1000"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="2500"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5000"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="7500"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10000"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="+Inf"} 485
otelcol_rpc_server_request_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 3.332249e+07
otelcol_rpc_server_request_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 485
# HELP otelcol_rpc_server_requests_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_requests_per_rpc histogram
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="0"} 0
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="25"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="50"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="75"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="100"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="250"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="500"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="750"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="1000"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="2500"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5000"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="7500"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10000"} 485
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="+Inf"} 485
otelcol_rpc_server_requests_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 485
otelcol_rpc_server_requests_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 485
# HELP otelcol_rpc_server_response_size Measures size of RPC response messages (uncompressed).
# TYPE otelcol_rpc_server_response_size histogram
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="0"} 0
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="25"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="50"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="75"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="100"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="250"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="500"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="750"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="1000"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="2500"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5000"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="7500"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10000"} 485
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="+Inf"} 485
otelcol_rpc_server_response_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 970
otelcol_rpc_server_response_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 485
# HELP otelcol_rpc_server_responses_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_responses_per_rpc histogram
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="0"} 0
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="25"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="50"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="75"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="100"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="250"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="500"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="750"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="1000"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="2500"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="5000"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="7500"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="10000"} 485
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev",le="+Inf"} 485
otelcol_rpc_server_responses_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 485
otelcol_rpc_server_responses_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 485
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_instance_id="af99d4d2-38d9-4377-87ea-230c81490265",service_name="otelcontribcol",service_version="0.98.0-dev"} 1```

@mx-psi
Copy link
Member

mx-psi commented Apr 24, 2024

@dmitryax @codeboten do we have an easy way to grep for this?

@tiffany76
Copy link
Contributor Author

Thanks to @theletterf's guidance, I have the process metrics file.

Based on that file, I was able to grep for and find the following metrics:

I know I'm missing the rpc_server_*, rpc_client_*, http_server_*, and http_client_* metrics. I've seen the files changed in open-telemetry/opentelemetry-collector/pull/9767, but they don't seem to spell out all the metric names. I'll look again later, but if you can point me in the right direction, I'd appreciate the help!

Next week, I'll pull everything I've found into a list and then compare it against the metrics my own Collector is emitting and against vendor distribution lists (see Splunk's and DataDog's). Then I'll put up a PR.

If you think I over-included or missed anything, please let me know. Thanks!

@tiffany76
Copy link
Contributor Author

After considering this content, I think it fits in well with the documentation covered in the second internal observability PR (#4322), so rather than put up a separate PR for this issue, I'm solving it as part of the work I'm already doing. I've put up a table, but it's still a WIP.

@codeboten
Copy link
Contributor

Just a note on this issue, there's work underway in the collector repo to generate documentation for all internal metrics produced automatically. You can see the result here: https://github.com/open-telemetry/opentelemetry-collector/blob/9cbcacca8d1a307317096ad7ed39d5f3acb10575/processor/batchprocessor/documentation.md

It would be great at some point to pull in this data in the main docs, or link to it

@tiffany76
Copy link
Contributor Author

Just a note on this issue, there's work underway in the collector repo to generate documentation for all internal metrics produced automatically. You can see the result here: https://github.com/open-telemetry/opentelemetry-collector/blob/9cbcacca8d1a307317096ad7ed39d5f3acb10575/processor/batchprocessor/documentation.md

It would be great at some point to pull in this data in the main docs, or link to it

That's great news, @codeboten, and it will directly address @chalin's concern about maintaining the list I generated.

I will keep an eye on your PR's progress, and then we'll see if we can rework the List of metrics section on opentelemetry.io.

@tiffany76
Copy link
Contributor Author

Raised an issue to explore the possibility of automating the list of metrics: #4523

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

4 participants