Out of memory after using Schedulers.enableMetrics()

[Novery]

Out of memory after using Schedulers.enableMetrics()

Actual Behavior

When metrics are enabled, the metrics do not converge.

The data for accessing /actuator/prometheus is as follows
name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-{$number} the number do not converge

executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-5939",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-4381",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-7102",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} 0.0
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-741",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-7235",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} 0.0
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-7368",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-1759",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-874",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-2823",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-2956",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-3051",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-7106",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} 0.0
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-3184",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-7239",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} 0.0
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-1625",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-5809",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-4382",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-5580",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-740",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-873",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-1758",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-2955",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-3185",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-7238",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-3052",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-7105",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-2822",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-1624",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-5808",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-4383",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN
executor_active_threads{application="web-service",name="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)-4250",reactor_scheduler_id="boundedElastic(\"boundedElastic\",maxThreads=40,maxTaskQueuedPerThread=100000,ttl=60s)",} NaN

Steps to Reproduce

reactor.core.scheduler.SchedulerMetricDecorator will generate an executorId incrementally for each execution for the same Scheduler

//we now want an executorId unique to a given scheduler
		String executorId = schedulerId + "-" +
				executorDifferentiator.computeIfAbsent(scheduler, key -> new AtomicInteger(0))
				                      .getAndIncrement();

At the same time executorId as metrics name

MetricsRemovingScheduledExecutorService() {
				super(ExecutorServiceMetrics.monitor(globalRegistry, service, executorId, tags));
			}

BoundedElasticScheduler will periodically recycle idle resources and recreate resources when needed later, and the executor Id is incremented when new resources are created

This leads to an uncontrollable number of Metric and eventually leads to memory overflow

Your Environment

• Reactor version(s) used: 3.3.11.RELEASE

[simonbasle]

we'd need to somehow detect schedulers with potential for high/unbounded cardinality and figure out how to cater to that use case... but can we really figure out a generic satisfying solution?

the only thing that gets instrumented in Schedulers is their backing ScheduledExecutorService(s), if any. If we didn't differentiate between these, we'd potentially get inconsistent data when a single Scheduler is backed by multiple ScheduledExecutorServices with each their own queue of tasks 🤦

on the other hand we get Schedulers.elastic() and Schedulers.boundedElastic() which create a lot of such services. Where to put the needle here? 🤔

[simonbasle]

@Novery what kind of metrics are you interested in, in Schedulers? (as a minimum)
it looks like decorating our plumbing (the ScheduledExecutorServices behind most Scheduler workers) was not a great choice in the first place. With the upcoming 3.5.0, maybe there's an opportunity to fix that and focus on data easily accessible for all Schedulers (not only the ones that can have a pool of workers).

[Novery]

@Novery what kind of metrics are you interested in, in Schedulers? (as a minimum) it looks like decorating our plumbing (the ScheduledExecutorServices behind most Scheduler workers) was not a great choice in the first place. With the upcoming 3.5.0, maybe there's an opportunity to fix that and focus on data easily accessible for all Schedulers (not only the ones that can have a pool of workers).

I mainly focus on executor_queued_tasks, executor_seconds_max, executor_queued_tasks, executor_completed_tasks_total to understand the load of the parallel scheduler

[simonbasle]

there's an ongoing work to add TimedScheduler wrapper in a new reactor-core-micrometer module in 3.5.0, see #3109 which could provide a viable alternative

[Novery]

ok,thank you. I will keep watching

[simonbasle]

@Novery any opinion on the direction of the aforementioned PR ?
it's bound to change a bit still, especially since there's a failing test, but I do need feedback

see #3109

[Novery]

@Novery any opinion on the direction of the aforementioned PR ? it's bound to change a bit still, especially since there's a failing test, but I do need feedback

see #3109

The tags now seem to be clean and controllable. The "executorId" is no longer works as the "executorServiceName". I think it can fix the current problems.