vmauth per-user metrics can cause high memory usage in the long term

[ebensom]

Describe the bug

The vmauth support dynamic config reloading. Over a long time period, many users can be created and deleted dynamically from the config, followed by reload. This is the scenario for us, where we use vmauth together with vm-operator, and we dynamically manage clients of the VictoriaMetrics cluster for each connected system using VMUser CRs, and there is a higher rate of change of VMUsers (30-100 creation or deletion per day).

vmauth exposes a bunch of very useful per-user metrics (the highest cardinality being of course the ones with type histogram/summary), however when a user gets removed from a config, the metrics corresponing to the user keep being exposed until restart. In the long term, this causes vmauth to consume more and more memory for metrics in the long term, potentially leading to OOM situations. Attached pprof and web graph for reference to troubleshoot the issue (see in Screenshots field)

Is there a way to change the authconfig reloading behavior (in a backards compatible, i.e. configurable way with a compatibility flag) to be able to detect config differences between the current loaded and the new config, and prune metrics for users not present in the new config using either DeleteLabelValues or DeletePartialMatch method the corresponding metrics?

To Reproduce

• start vmauth
• in an infinite loop:
  • add and remove several users
  • trigger config reload

Version

1.100.1

Logs

No response

Screenshots

vmauth_mem.pprof.tar.gz
vmauth_mem_inuse_web.pdf

Used command-line flags

No response

Additional information

No response

[f41gh7]

Thanks for reporting, It's a bug with metrics package usage. It doesn't properly unregistered summary metrics.

[hagen1778]

#6252 has been merged and will be included into the next release.