Experimental feature: Prometheus Metrics

[irevoire]

Hey everyone 👋

For the v1.1 of Meilisearch (which was released on 3rd April), we will finally introduce the metrics feature as an experimental feature.

Feature Abstract

The /metrics endpoint exposes metrics to be scraped by a Prometheus collector at regular intervals and stored for analysis. By monitoring these metrics over time, teams can gain valuable insights into the behavior and performance of Meilisearch and use that information to optimize their performance, troubleshoot issues, and improve the overall reliability of their systems.

How to enable the feature in Meilisearch?

2 ways

• Launch Meilisearch with the CLI flag --experimental-enable-metrics or the MEILI_EXPERIMENTAL_ENABLE_METRICS env var
• Since v1.5.0, and not before, call this HTTP route:

curl \
  -X PATCH 'http://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json'  \
--data-binary '{
    "metrics": true
  }'

You can check the feature is well enabled by calling GET /experimental-features

⚠️ Do NOT use the CLI flag (or environment variable) and the HTTP call at the same time to enable the feature: this could lead to unexpected behavior when restarting the Meilisearch instance.
⚠️ An API key that has the right to see the metrics is able to see all the HTTP calls that happened for all the routes that have been hit on the instance. This may leak all the index names, some document's primary keys, or even worse, the other API keys.
=> So, be sure to consider the key that can access the API-key as a super private API-key that you must not share publicly.

How to use the feature?

• The metrics are sent using the prometheus format
• This exposes a new route in the API endpoint GET /metrics
  • To poll this route, your API key needs the right to pull metrics through the metrics.get action and access to all indexes.
  • The scrape_config must contain an Authorization header using the bearer method with the Meilisearch API Key authorized to poll the /metrics endpoint. You can find a basic config in the repository under assets/prometheus-basic-scraper.yml.

Step-by-step basic tutorial

To read before starting the tutorial:

• You need to install Prometheus and Grafana. Refer to the documentation of your distribution / OS.
• I won't explain how to secure your instance. Refer to this documentation if you need to learn how it works, and adapt the following command to include your master-key / api-key.

1. Run meilisearch with the experimental feature enabled: ./meilisearch --experimental-enable-metrics.
2. Download the Prometheus assets file stored on the meilisearch repository assets/prometheus-basic-scraper.yml.
3. If you need to use an API key, add the following lines to the end of the config file (and replace with your API key):

authorization:
	credentials: <api-key>

4. On another terminal or server, run Prometheus with your config file: ./prometheus --config.file=path-to-your-config-file.yml
5. Download the default Grafana dashboard on the following URL: https://github.com/meilisearch/meilisearch/blob/main/assets/grafana-dashboard.json.
6. Connect to your Grafana instance (by default, it runs on http://localhost:3000). Create an account if needed.
7. Add a data source that points to your Prometheus server (by default, it runs on http://localhost:9090).
8. Go into the dashboard menu and import a dashboard file.
9. Import the json previously downloaded. You should be good to go 🎉

What's currently exposed?

Name	Description	Type
meilisearch_http_requests_total	Returns the number of times an API resource is accessed.	counter
meilisearch_http_response_time_seconds	Returns a time histogram showing the number of times an API resource call goes into a time bucket (expressed in second).	histogram
meilisearch_db_size_bytes	Returns the “real” size of the database on disk in bytes. It includes all the lmdb memory mapped files plus all the files contained in the data.ms directory (mainly the updates files that were not processed yet).	gauge
meilisearch_used_db_size_bytes	Returns the size of the database actually used by meilisearch in bytes. Include all the same files as meilisearch_db_size_bytes except that when it comes to an LMDB database, we only count the pages used by meilisearch. This means if you see a large gap between both metrics, adding documents will probably re-use freed pages instead of growing meilisearch_db_size_bytes.	gauge
meilisearch_index_docs_count	Returns the number of documents for an index.	gauge
meilisearch_index_count	Returns the total number of index for the Meilisearch instance.	gauge
meilisearch_nb_tasks	Returns the total number of tasks for the Meilisearch instance parametrized by the kind of task and its value (see the table below).	counter
meilisearch_last_update	Returns the timestamp of the last update.	gauge
meilisearch_is_indexing	Returns 1 if Meilisearch is indexing or 0 if not.	gauge

---

What is an experimental feature

• The API and the behavior of an experimental feature can break between two minor versions of Meilisearch.
• The whole experimental feature process can break between 2 minor versions of Meilisearch.
• The newly introduced experimental features or improvements can be removed in the future. Providing a feature as “experimental” does not guarantee it will be stable one day.
• The experimental features are supposed to be unstable regarding the API usage but not bugged.

Why is this feature not stable yet?

We have yet to determine which metrics we want to expose and how. Thus this may change a lot depending on our needs and your needs.

When will the feature potentially be stable?

We don't know yet when this feature could be stable since we need to iterate on it. As explained previously, we do not guarantee it will be stable one day yet.

Depending on the feedback we get, we plan to iterate on the feature every 1 or 2 releases (Currently, our release cycle is two months long).

X metrics are missing

We know the feature is not finished yet.

Please let us comment on which information you would like to see integrated into the metrics.
And do not hesitate to create a PR as well. Most of the code is available in meilisearch/src/metrics.rs.

🗣️ You are welcome to give feedback about the feature or ask any questions; we can guide you through this new feature usage!

---

cc @meilisearch/cloud-team

[gmourier]

Hello, everyone following this discussion 👋

We have just released the first RC (release candidate) of Meilisearch containing this experimental feature!

You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.1.0-rc.0

You are welcome to communicate any feedback about this new implementation in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on 3rd April 2023

⚠️ RC (release candidates) are not recommended for production

[gmourier]

Hey folks 👋

v1.1 has been released and features the Prometheus metrics as an experimental one 🧪

We are waiting for tests and feedback before considering any stabilization. Please, tell us if this feature is helpful or what is missing for your monitoring use-case!

[hspak]

Thank you for building this! I had a question on how the histogram bins are set for the http_response_time_seconds metrics. It would be nice if we had more bins over 1 second.

Example

http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0005"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0008"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00085"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0009"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00095"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.001"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00105"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0011"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00115"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0012"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0015"} 10
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.002"} 751
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.003"} 1590
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="1"} 2533
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="+Inf"} 2533
http_response_time_seconds_sum{method="PUT",path="/indexes/myindex/documents"} 20.185828975999993 

This is on GCP Monitoring, but the heatmap looks like this:

(GCP doesn't expose PromQL for this, but here's their bespoke DSL that represents the heatmap):

fetch prometheus_target
| metric 'prometheus.googleapis.com/http_response_time_seconds/histogram'
| filter (metric.path != '/health' && metric.path != '/metrics')
| align delta(1m)
| every 1m
| group_by [],
    [value_http_response_time_seconds_aggregate:
       aggregate(value.http_response_time_seconds)]

[gmourier]

Hey @hspak 👋

I'm not sure to understand; please correct me if I'm wrong! What I understand is that you lack precision for the sub-second histogram is that right?

[hspak]

Yes! I'm not sure how fast these HTTP calls (especially index requests) are in internal testing, but the default buckets are way too aggressive in my opinion. Almost all of our requests are in this 0.003s to 1s bucket and that's not very helpful in catching trends. Some additional buckets that are greater than 1s would be helpful as well so latency spikes are obvious from looking at the histogram metrics.

[miiton]

but the default buckets are way too aggressive in my opinion

I have tried it and I feel the same way.

My PromQL

sum(rate(http_response_time_seconds_bucket{path="/multi-search"}[$__rate_interval])) by (le)

[rohankmr414]

I feel the same about the current bucket boundaries, they are too granular and only track latencies below 1s, for someone who is using a managed prometheus service the cost is going to be high for this metric because we're generating too many metric samples due to this granularity. I think we should follow the opentelemetry spec for the http duration metric, it's mentioned here that the bucket boundaries for should be set to 5ms, 10ms, 25ms, 50ms, 75ms, 100ms, 250ms, 500ms, 750ms, 1s, 2.5s, 5s, 7.5s, 10s. I can open a PR for this, @irevoire wdyt?

[rohankmr414]

PR for the same -> meilisearch/meilisearch#4530

[bannarisoftwares]

Should i need MEILI_EXPERIMENTAL_ENABLE_METRICS as true ?
like this ?

export MEILI_EXPERIMENTAL_ENABLE_METRICS=true

[curquiza]

Hello @bannarisoftwares!
Yes 😊
Once you test it, it would be awesome if you take the time to let us know if the feature fits your needs.

[miiton]

How about having a metric, like MEILISEARCH_TASK_QUEUES, to monitor the number and increase of task queues? I believe it would be useful for tuning the application, what do you think?

[curquiza]

@irevoire do you know how complex it is to implement?

[irevoire]

I don’t think it’s that hard, and I would like to work on it; that’s actually what I was working on before the index-scheduler refactor, but we were missing a lot of helpers to access the content of the task queue.
Now everything has been introduced, and getting the info should be pretty straightforward.

[curquiza]

@macraig for your information

[macraig]

@curquiza I think it makes sense to add it!

[curquiza]

Let us know (with @gillian-meilisearch as well) if it's something you want (and we can) add to the next sprint 😊

[hspak]

Would it be possible to include a metric for indexing latency per index? It would be helpful to get insights on end-to-end indexing latency.

[curquiza]

Hello @hspak, sorry for the delay
what does it mean indexing latency per index for you?

[hspak]

I would expect it to be the time from receiving the index request to seeing the indexed document available through search.

The metrics here only represents the time it took to enqueue the document to be indexed.

Sorry if this is out-of-date! I have not run meilisearch in some time.

[macraig]

Cross-posting feedback from @gabrieltz in #669 (comment)

Hello,

It would be nice to have some additional prometheus metrics that already exist in /stats endpoint:

1. lastUpdate exposed as a gauge with seconds since last update, or unix timestamp to detect if something is not "feeding meilisearch with new data" so we can define alert rules to prevent stale data.
2. isIndexing exposed as 0/1 would be helpful to correlate slower http_response_time_seconds_bucket with indexing

and from the /tasks endpoint

1. exposing the number of tasks with queued status ( gauge )
2. exposing the number of tasks with processing status ( gauge )

thank you

[irevoire]

FYI:

and from the /tasks endpoint

exposing the number of tasks with queued status ( gauge )
exposing the number of tasks with processing status ( gauge )

This as already pretty much been implemented and will come out in the next release.
I didn’t use a gauge but a graph for the first version though, you’ll see once it comes out.

[konstantin83]

Hi! Thank you for such a great app!

Currenlty experimental prometheus metrics are pretty poor. Is there a way to get more statictics? Like what queries were sent, and how much hits did they return. Or how much empty results and by which queries?

Or should I do my own middlware for that metrics?

[gillian-meilisearch]

Cross-posting from here to make sure we capture the suggestion details in the right place for @macraig to consider in future iterations

• Request count by each index
• Successful / Failed requests
• Which aliases were used and how much and how much results retrieved
• Average load by index
• RPS by index
• And so on.

[macraig]

Hi @konstantin83 , thank you for sharing your feedback. We appreciate your input. At the moment, the metrics you mentioned are not available, but we are actively considering adding them to our product in the future. However, I want to be transparent and let you know that our team is currently focused on other priorities, which may impact the timeline for implementation. I apologize for any inconvenience caused by the delay, and we will do our best to keep you updated on our progress.

[madalinignisca]

Sent queries with hits returned and similar stuff look more like in the area of LOGS and not time series metrics. I think this would add a lot of CPU load extra and extremely detailed metrics should be enabled by environment variables, so everybody else should have essentials only. For me as an SRE, time series metrics are critical in essentials and almost zero performance penalty. Anything that would enforce me to have a system that spends even 1% of CPU time to expose metrics is already having a poor performance hit.

[miiton]

I see that the Prometheus process is consuming a lot of CPU/Memory due to the large number of values related to TASKs.
I think this is a point for improvement.

http_response_time_seconds_count{method="GET",path="/tasks/1"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.0005"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.0008"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.00085"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.0009"} 1
...
...
...
http_response_time_seconds_count{method="GET",path="/tasks/9977"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.0005"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.0008"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.00085"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.0009"} 1
...

with scrape meilisearch metrics

without scrape meilisearch metrics

• Meilisearch v1.2.0
• Prometheus v2.45.0

[irevoire]

Thanks for the details; I opened an issue to track this change: meilisearch/meilisearch#3983

[Hades32]

No, the right way to handle this is to not use HTTP paths, but the routes. So e.g. instead of /tasks/1 you'd get /task/{task_id}. This would also be highly needed for the get document endpoint

[irevoire]

Oooh, ok, Prometheus can handle many parameters but not many routes, basically?

But do you think it really brings something to the end user to be able to see how fast ms answered on each specific document/task?

[Hades32]

It's a cardinality problem. You don't want to explode the number of things in the system basically. It's metrics and not logs after all.

And no, I don't think there is much value in getting metrics for a specific task/doc. The route metrics would be enough imho

[gh2k]

This is pretty critical from my perspective. For document deletes, this has a very very high cardinality problem

[gabrieltz]

I upgraded to version 1.3.0 yesterday and used some of the new metrics, however i noticed that the metric meilisearch_nb_tasks{instance="XXXXXX:7700", job="meilisearch", kind="statuses", value="processing"} is zero all the time, even while importing a few million documents in over 700 batches. The /tasks endpoint values were correct though

[curquiza]

Hello @gabrieltz
thanks for the report, we have opened an issue and we are working on it here

[heka1024]

Hi. I'm using version 1.3.4 in meilisearch cloud and I didn't see that option. Is this not available in the cloud?

[curquiza]

Hello @heka1024
So far no, it's not available, but we plan to do it if needed by the users.

In the meantime, feel free to contact the support to see if it's possible to activate if for your project 😊

[Schtil]

Greetings!
At the beginning I would like to express rays of support to the development team for the implementation of metrics as such.
However, are there any plans to do away with the standard, reserved fields within label names?
I mean the label with the name "value".
In our metrics collection system (frost scrapper) this leads to errors.
And in principle, it is not customary in the community to introduce some reserved names into label names. The value is associated with the value of metrics, and this leads to collisions when collecting and aggregating data in the future.

[Schtil]

I'll give it a try)

The point is, we have these metrics.

meilisearch_nb_tasks{kind="types",value="indexUpdate"} 0
meilisearch_nb_tasks{kind="types",value="settingsUpdate"} 0
meilisearch_nb_tasks{kind="types",value="snapshotCreation"} 0
meilisearch_nb_tasks{kind="types",value="taskCancelation"} 0
meilisearch_nb_tasks{kind="types",value="taskDeletion"} 0

These use two labels name, "kind" and "value".
These metrics are then scraped (collected) by monitoring systems
And then we are going to search by these metrics in conditional kibana.

It is common in the community to use a field value (in this case 0) in the "value" field of a metric.
However, here the "value" field is occupied in labels. And there is a collision.
Therefore, it is not common in the community to use some fixed names in labels.
Now we use crutches to rename label "value" to "argument" when scraping (we just took the name from the ceiling).

But we would like to get away from this crutch and solve the problem at the root.

[irevoire]

It is common in the community to use a field value (in this case 0) in the "value" field of a metric.

You mean that instead of sending something like that:
meilisearch_nb_tasks{kind="types",value="indexUpdate"} 0

People send something like this:
meilisearch_nb_tasks{kind="types",whatever="indexUpdate",value=0} 0

And if the value was 2 would they update both fields or only the one in the value thingy?

---

And so what you're asking for is basically that we rename the value thingy we use here to something else?

[Schtil]

Almost.

I'm talking about replacing it with

meilisearch_nb_tasks{kind="types",whatever="indexUpdate"} 0

In the search engine the query will look like (Prom QL) meilisearch_nb_tasks{kind="types",whatever="indexUpdate"} == 0
In grafana, the query might look like this
meilisearch_nb_tasks{kind="types",whatever="indexUpdate"}
And grafana will build a graph based on the value of the "value" field - and it will only have a number in it

[irevoire]

Ah, grafana will autogenerate the value field on its side?

[Schtil]

Rather Victoria Metrics (metrics storage system)
Grafana only displays the results from it

[capJavert]

Anyone else getting no data on the dashboard, I am on 1.5.x, checking manually the prometheus and grafana data is in there which points me that dashboard might be out of date?

I can also see the requests for /metrics on my meilisearch instance.

[capJavert]

@irevoire first time setting it up. I can see that indexe and instance is loaded correctly with data source and if I try to create my own dashboard the data is there.

And if yes, can you try to edit the « database size in bytes » chart to re-set the input source?

What do you mean by this, just edit something in the chart and save dashboard? (I tried it and still "No data")

[capJavert]

Ok, so basically the issue was I had a custom job_name for prometheus scrape config. This has to be meilisearch as it is hardcoded in the dashboard.

[capJavert]

@irevoire if you would accept I added the PR for job variable so this does not happen if user sets the custom job_name. Default would still be meilisearch as mentioned in the docs. meilisearch/meilisearch#4330

[irevoire]

Hey, yes I merged it thanks for your contribution 😁

[capJavert]

Yep, nice, tnx for the quick turnaround 🤘

[rohankmr414]

folks, I've opened a PR to add the http status code label to the meilisearch_http_requests_total metric meilisearch/meilisearch#4373, would love to have someone's review on this, also I'm not sure if this is the right way to do it as I'm not very good with rust.

[irevoire]

Thanks, I merged it

[productdevbook]

https://github.com/meilisearch/meilisearch/blob/main/assets/grafana-dashboard.json

Can you also make a separate json to cover the whole index?

[irevoire]

Hey, what do you mean by the whole index?
Which information is missing currently?