MeterFilters configured after a Meter has been registered

[shakuzen]

In #4917 we took a step toward warning users about this situation with the following log message:

A MeterFilter is being configured after a Meter has been registered to this registry.

Historically, we have allowed configuring MeterFilters after meters have been registered. Depending on how meters are used, the negative effects of doing this can be minimal. It is ideal to configure all desired MeterFilters before registering any Meters so that all registered Meters will have all configured MeterFilters applied.

If users have a use case for configuring MeterFilters "late" (after a Meter has been registered), we would be interested in understanding that so we can make informed decisions about what we allow in the future. It may also be the case that something out of a user's direct control is causing this situation to happen - a framework or library the user is using may be configuring MeterFilters or registering Meters at timing not controlled by the user. We would like to hear about these cases so we can work with the libraries or frameworks. The above log message includes a stack trace when DEBUG level logging is enabled on the meter registry implementation in question. This stack trace will help track down what is configuring the MeterFilter late.

[sanyarnd]

Hi, I have updated to 1.13.0, everything is ok, but I'm having this debug log popping up.

I'm using Vert.x with Guice and I am configuring some defaults meters before creating Vert.x object

stacktrace

0:54:09.416 DEBUG [main @coroutine#21            ] s.PrometheusMeterRegistry -- A MeterFilter is being configured after a Meter has been registered to this registry. All MeterFilters should be configured before any Meters are registered. If that is not possible or you have a use case where it should be allowed, let the Micrometer maintainers know at https://github.com/micrometer-metrics/micrometer/issues/4920.
java.base/java.lang.Thread.getStackTrace(Thread.java:2450)
	at io.micrometer.core.instrument.MeterRegistry$Config.logWarningAboutLateFilter(MeterRegistry.java:844)
	at io.micrometer.core.instrument.MeterRegistry$Config.meterFilter(MeterRegistry.java:830)
	at io.vertx.micrometer.backends.BackendRegistries.registerMatchers(BackendRegistries.java:127)
	at io.vertx.micrometer.backends.BackendRegistries.lambda$setupBackend$0(BackendRegistries.java:82)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at io.vertx.micrometer.backends.BackendRegistries.setupBackend(BackendRegistries.java:62)
	at io.vertx.micrometer.impl.VertxMetricsFactoryImpl.metrics(VertxMetricsFactoryImpl.java:54)
	at io.vertx.core.spi.VertxMetricsFactory.init(VertxMetricsFactory.java:50)
	at io.vertx.core.impl.VertxBuilder.initProviders(VertxBuilder.java:284)
	at io.vertx.core.impl.VertxBuilder.init(VertxBuilder.java:275)
	at io.vertx.core.Vertx$1.internalBuilder(Vertx.java:121)
	at io.vertx.core.Vertx$1.build(Vertx.java:125)
	at io.vertx.core.Vertx.vertx(Vertx.java:150)
	at VertxModule.vertx(VertxModule.kt:22)

code

    @Provides
    @Singleton
    fun prometheusMeterRegistry(): PrometheusMeterRegistry {
        fun registerDefaultJvmMetrics(registry: PrometheusMeterRegistry) {
            JvmGcMetrics().bindTo(registry)
            JvmHeapPressureMetrics().bindTo(registry)
            JvmInfoMetrics().bindTo(registry)
            JvmMemoryMetrics().bindTo(registry)
            JvmThreadMetrics().bindTo(registry)
            Log4j2Metrics().bindTo(registry)
            ProcessorMetrics().bindTo(registry)
            UptimeMetrics().bindTo(registry)
        }

        fun registerNettyAllocatorMetrics(registry: PrometheusMeterRegistry) {
            val nettyPooled = PooledByteBufAllocator.DEFAULT
            val nettyUnpooled = UnpooledByteBufAllocator.DEFAULT
            val vertxPooled = VertxByteBufAllocator.POOLED_ALLOCATOR as PooledByteBufAllocator
            val vertxUnpooled = VertxByteBufAllocator.UNPOOLED_ALLOCATOR as UnpooledByteBufAllocator

            val nettyName = "netty"
            val vertxName = "vertx"

            NettyAllocatorBinder
                .bindPooled(nettyName, nettyPooled.metric(), registry)
            NettyAllocatorBinder
                .bindPooled(vertxName, vertxPooled.metric(), registry)
            NettyAllocatorBinder
                .bindUnpooled(nettyName, nettyUnpooled.metric(), registry)
            NettyAllocatorBinder
                .bindUnpooled(vertxName, vertxUnpooled.metric(), registry)
        }

        val registry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT)
        registerDefaultJvmMetrics(registry)        // <--- bind here
        registerNettyAllocatorMetrics(registry)    // <--- bind here
        return registry
    }

class VertxModule : AbstractModule() {
    @Provides
    @Singleton
    fun vertx(vertxOptions: VertxOptions, customizer: ObjectMapperCustomizer): Vertx {
        val vertx = Vertx.vertx(vertxOptions)     // <--- create vert.x object here
        return vertx
    }

    @Provides
    @Singleton
    fun vertxOptions(
        config: ApplicationConfig,
        tracer: Tracer,
        propagator: TextMapPropagator,
        meterRegistry: PrometheusMeterRegistry,
        httpClientTagProvider: HttpClientTagProvider,
    ): VertxOptions = VertxOptions().apply {
        // ...

        metricsOptions = MicrometerMetricsOptions().apply {
            isEnabled = true
            micrometerRegistry = meterRegistry        // <--- provide registry here
            clientRequestTagsProvider = httpClientTagProvider
        }
        tracingOptions = OpenTelemetryOptions(tracer, propagator)
    }
}

NettyAllocatorBinder is my own binder, since I need some low-lever metrics, it looks like this

    public static void bindUnpooled(
        @NotNull String name,
        @NotNull ByteBufAllocatorMetric metric,
        @NotNull MeterRegistry registry
    ) {
        Tags unpooled = Tags.of(NAME, name, ALLOC, UNPOOLED);
        memoryUsage(metric, registry, unpooled);
    }

    private static void memoryUsage(
        @NotNull ByteBufAllocatorMetric metric,
        @NotNull MeterRegistry registry,
        @NotNull Tags tags
    ) {
        Gauge.builder(dot(NETTY, ALLOC, MEMORY, USED), metric, ByteBufAllocatorMetric::usedHeapMemory)
            .description("The number of the bytes of the heap memory")
            .tags(tags.and(MEMORY, HEAP))
            .register(registry);
        Gauge.builder(dot(NETTY, ALLOC, MEMORY, USED), metric, ByteBufAllocatorMetric::usedDirectMemory)
            .description("The number of the bytes of the direct memory")
            .tags(tags.and(MEMORY, DIRECT))
            .register(registry);
    }

[jonatan-ivanov]

As the warning and #4917 say, all MeterFilters should be configured before any Meters are registered. In your case it seems someone (Vert.x?) is registering MeterFilters after you registered your Meters.

Is there any way to register them at a later point in the lifecycle?
Maybe this would worth an issue for Vert.x so that when you get the registry instance, it is fully configured (MeterFilters are registered).

[sanyarnd]

Is there any way to register them at a later point in the lifecycle?

Guice doesn't have lifecycle support.

You can cheat it by registering default binders after injector creation (or by creating a fake bean):

val injector = ...
val registry = injector.getInstance<PrometheusMeterRegistry>()
bindDefaultMetrics(registry)

and by adding Vertx bean as a dependency to all other providers which depend on registry to register their own metrics

@Provides
@Singleton
fun foo(fake: Vertx, registry: PrometheusMeterRegistry): Foo = ...

It's not really a common case, I guess (using Vert.x with Guice). The message asked if there is a use case, so I provided one.

[perracodex]

I started to get this warning:

A MeterFilter is being configured after a Meter has been registered to this registry. All MeterFilters should be configured before any Meters are registered. If that is not possible or you have a use case where it should be allowed, let the Micrometer maintainers know at #4920.

I am using micro meters in a Ktor server. Not sure if the issue should be reported here or in the Ktor repository. The way I use it is as next:

fun Application.configureMicroMeterMetrics() {

    install(plugin = MicrometerMetrics) {
        registry = appMicrometerRegistry

        meterBinders = listOf(
            ClassLoaderMetrics(),
            JvmMemoryMetrics(),
            JvmGcMetrics(),
            ProcessorMetrics(),
            JvmThreadMetrics(),
            FileDescriptorMetrics(),
            UptimeMetrics()
        )
    }
}

val appMicrometerRegistry: PrometheusMeterRegistry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT).apply {
    config()
        .meterFilter(MeterFilter.deny { id ->
            id.name == "ktor.http.server.requests" && id.getTag("route") == "/rbac"
        })
}