Problem with probe based metrics || my misunderstanding?

[ypakhomenko]

Hi everyone,

I would like to start using cloudprober in GCP environment and since I am new to it, I started with the simplest setup:

Downloaded cloudprober-v0.12.6-linux-x86_64 from https://github.com/cloudprober/cloudprober/releases on Debian GNU/Linux 11 VM and created the simplest configuration:

cat simple.cfg
surfacer {
type: STACKDRIVER
}

probe {
name: "testing_simple"
type: HTTP
targets {
host_names: "www.google.com"
}
interval_msec: 5000
timeout_msec: 2000
}

Then started the cloudprober as root:
./cloudprober simple.cfg
cloudprober 1683011945325933759 1683011955 labels=ptype=sysvars,probe=sysvars external_ip="" hostname="cloud-prober1" instance="cloud-prober1" instance_id="2495614540300042738" instance_template="undefined" internal_ip="10.104.3.106" machine_type="e2-medium" nic_0_ip="10.104.3.106" project="de0360-e2e-monitoring" project_id="280119929030" region="europe-west3" start_timestamp="1683011945" version="v0.12.6" zone="europe-west3-b"
......

I would expect to see in the output some metrics from the defined probe, but I only see the sysvars metrics.
Also the curl returns only sysvars metrics:

curl -s http://localhost:9313/metrics| grep -v '^#'
external_ip{ptype="sysvars",probe="sysvars",val=""} 1 1683012325328
hostname{ptype="sysvars",probe="sysvars",val="cloud-prober1"} 1 1683012325328
instance{ptype="sysvars",probe="sysvars",val="cloud-prober1"} 1 1683012325328
instance_id{ptype="sysvars",probe="sysvars",val="2495614540300042738"} 1 1683012325328
instance_template{ptype="sysvars",probe="sysvars",val="undefined"} 1 1683012325328
internal_ip{ptype="sysvars",probe="sysvars",val="10.104.3.106"} 1 1683012325328
machine_type{ptype="sysvars",probe="sysvars",val="e2-medium"} 1 1683012325328
nic_0_ip{ptype="sysvars",probe="sysvars",val="10.104.3.106"} 1 1683012325328
project{ptype="sysvars",probe="sysvars",val="de0360-e2e-monitoring"} 1 1683012325328
project_id{ptype="sysvars",probe="sysvars",val="280119929030"} 1 1683012325328
region{ptype="sysvars",probe="sysvars",val="europe-west3"} 1 1683012325328
start_timestamp{ptype="sysvars",probe="sysvars",val="1683011945"} 1 1683012325328
version{ptype="sysvars",probe="sysvars",val="v0.12.6"} 1 1683012325328
zone{ptype="sysvars",probe="sysvars",val="europe-west3-b"} 1 1683012325328
cpu_usage_msec{ptype="sysvars",probe="sysvars"} 164.224 1683012325328
uptime_msec{ptype="sysvars",probe="sysvars"} 380081.695 1683012325328
gc_time_msec{ptype="sysvars",probe="sysvars"} 0.740 1683012325328
mallocs{ptype="sysvars",probe="sysvars"} 89043 1683012325328
frees{ptype="sysvars",probe="sysvars"} 73502 1683012325328
goroutines{ptype="sysvars",probe="sysvars"} 43 1683012325328
mem_stats_sys_bytes{ptype="sysvars",probe="sysvars"} 20530184 1683012325328

On the GCP side, in the same project, I would expect the metrics like:
custom.googleapis.com/cloudprober/http/testing_simple/failure
metric 'custom.googleapis.com/cloudprober/http/testing_simple/total
But they are not there.

The service account on the VM has the roles 'roles/logging.logWriter' and 'roles/monitoring.metricWriter' (even tried 'roles/monitoring.admin').

No error messages are shown.

Most probably I misunderstand something, so any hints / suggestions are highly appreciated.

Best regards,
Yury

[manugarg]

Yeah, so it's not really using the config file in this case. You need to provide the config file using the flag --config_file:

./cloudprober --config_file=simple.cfg

Ref: https://cloudprober.org/docs/overview/getting-started/#configuration

However Cloudprober should result in an error if non-flag argument is given to avoid confusion like this. I'll file a follow up issue.

[ypakhomenko]

Oh, sorry -- now I made sure the config is used:

./cloudprober --config_file simple.cfg
(no output to stderr or stdout)

from other terminal:
curl http://localhost:9313/config
surfacer {
type: STACKDRIVER
}

probe {
name: "testing_simple"
type: HTTP
targets {
host_names: "www.google.com"
}
interval_msec: 5000
timeout_msec: 2000
}

But then it does not provide any metrics at all:

curl http://localhost:9313/metrics
Found.

If I check status, I can only see "No data yet":

curl http://localhost:9313/status

.....
.No data yet

Actually, it was my status from yesterday when my brain still worked a bit and I did not forget to use "--config_file " key.

The funny thing is that if I comment out the surfacer like:

#surfacer {

type: STACKDRIVER

#}

probe {
name: "testing_simple"
type: HTTP
targets {
host_names: "www.google.com"
}
interval_msec: 5000
timeout_msec: 2000
}

Then it starts working as expected, just no metrics are exposed to GCP stack driver:
./cloudprober --config_file simple.cfg
cloudprober 1683016425303413783 1683016431 labels=ptype=http,probe=testing_simple,dst=www.google.com total=2 success=2 latency=185400.523 timeouts=0 resp-code=map:code,200:2
cloudprober 1683016425303413784 1683016435 labels=ptype=sysvars,probe=sysvars external_ip="" hostname="cloud-prober1" instance="cloud-prober1" instance_id="2495614540300042738" instance_template="undefined" internal_ip="10.104.3.106" machine_type="e2-medium" nic_0_ip="10.104.3.106" project="de0360-e2e-monitoring" project_id="280119929030" region="europe-west3" start_timestamp="1683016425" version="v0.12.6" zone="europe-west3-b"
cloudprober 1683016425303413785 1683016435 labels=ptype=sysvars,probe=sysvars cpu_usage_msec=112.155
cloudprober 1683016425303413786 1683016435 labels=ptype=sysvars,probe=sysvars uptime_msec=10090.911 gc_time_msec=3.142 mallocs=74550 frees=50452
cloudprober 1683016425303413787 1683016435 labels=ptype=sysvars,probe=sysvars goroutines=60 mem_stats_sys_bytes=20530184

---

curl -s http://localhost:9313/metrics| grep http
total{ptype="http",probe="testing_simple",dst="www.google.com"} 18 1683016511613
success{ptype="http",probe="testing_simple",dst="www.google.com"} 18 1683016511613
latency{ptype="http",probe="testing_simple",dst="www.google.com"} 8105823.103 1683016511613
timeouts{ptype="http",probe="testing_simple",dst="www.google.com"} 0 1683016511613
resp_code{ptype="http",probe="testing_simple",dst="www.google.com",code="200"} 18 1683016511613

What am I doing wrong with the surfacer definition?
Are there any ways to get debug output from cloudprober to see what is wrong?

(When yesterday I forgot to give logging.logWriter to the VM's service account, then the cloudprober reported rpc errors, currently no errors are reported)

[manugarg]

https://cloudprober.org/docs/surfacers/overview/ sheds some light on this.

You can configure multiple surfacers at the same time. If you don’t specify any surfacer, prometheus and file surfacers are enabled automatically.

If you specify any surfacer in your config, only that surfacer will be enabled. So if you want both, StackDriver and Prometheus (/metrics thing), you'll need to add the following to your config stanza:

surfacer {
  type: PROMETHEUS
}

surfacer {
  type: STACKDRIVER
}

and if you want to see metrics on stdout as well, add file surfacer too:

surfacer {
  type: FILE
}

[ypakhomenko]

Thank you so much for clarification!

My last problem was that I could not find the metrics on the GCP operations suite side.
I just copy/pasted your examples and changed the names of the metrics.
For some reason, I cannot see the metrics in "global" source, so I changed the source to gce_instance and then it immediately worked.

Now I can start having fun with QML :-)

Thanx a lot!

[ypakhomenko]

So as it often happens, the problem was sitting in front of the monitor (the 8th layer problem) :-)