Custom Metric on Elasticsearch Integration

[vynhart]

Hi, I tried to collect a custom query from Elasticsearch to be sent to Datadog. However, I got trouble that custom query requiring that data_path must exists. Maybe I misunderstood it, but let me give my case.

I need to collect count metric from Elasticsearch endpoint: /my-index-name/_count. The results of the Rest API is like the following:

{
    "count": 119602245,
    "_shards": {
        "total": 39,
        "successful": 39,
        "skipped": 0,
        "failed": 0
    }
}

From the conf.yaml.example, the definition of data_path is:

The JSON path up to, but not including, the metric.

So in the case of /my-index-name/_count, the metric that I want to collect is on the first level of JSON, the value of key count.

This is my configuration for custom query in conf.yaml:

    custom_queries:
    - endpoint:  "/my-index-name/_count"
      data_path: ""
      payload:
        query:
          range:
            "@timestamp":
              gte: "now-1h"
      columns:
      - value_path: count
        name: my_elasticsearch.indexed_doc.count
      tags:
      - index_name:my_index_name

Results I Received

When I fill the data_path with empty string data_path: "", the datadog-agent status shows me error:

Value error, Each custom query must have an `endpoint`, `data_path`, and `columns` values

Results I expected

Data path can be empty.

---

Maybe it's just that I misunderstood how it works, so I appreciate anyone who can give me the correct configuration to collect the _count data as a custom query. But if my understanding about the data_path is correct, then I think it should not be a required field because the data that I want to collect could have no parent on the JSON results.