Elasticsearch Exporter not reporting master node as down

[priyanshurohilla]

Hi,
We have got two issues in our cluster in which the master node went down but elasticexporter was not reporting the master node as down.

Currently i am scraping elasticsearch metrics from one of the master node using elasticsearch exporter. In our case that master node from where we were scraping metrics, it went down. We were getting connection refused error on elasticsearch exporter logs but on prometheus, elasticsearch exporter target was still reporting as up. The elastic exporter target was still healthy and because of that we missed the alert.

We are using elasticsearch exporter version 1.3.0. Please help in this

[sysadmind]

The elasticsearch_scrape_success metric should report whether or not a scrape from prometheus was successful. If my understanding is correct, this is the metric that would have alerted you to the problem with the master node.

[priyanshurohilla]

hi @sysadmind Thanks for the heads up. I have one doubt regarding this . How this metric evaluates whethere a scrape was successful?

I will give you one edge case. Our elasticsearch. cluster is having authentication which we need to pass while running the docker container for elasticsearch exporter. lets suppose someone changed the password at the elastic search level. Now since the password is reset, we will start getting authentication issues in the logs of exporter but there are certain heartbeat metrics which it will still be able to scrape but not all.

So my question is will this metric report this scenario as failure? If no, how can we solve this edge case?