Circuit breaker in DISABLED state sill moving back to HALF_OPEN/OPEN

[jmereaux]

Resilience4j version: 2.2.0

Java version: 17

After putting a circuit breaker instance to DISABLED state (using the transitionToDisabledState method), it is automatically moving to HALF_OPEN/OPEN when reaching the failure rate threshold.

Is it expected?

Is there any other way to disable a circuit breaker instance programmatically at runtime ?

Thanks 😊

[RobWin]

No, this is not expected. Could you please show some logs and code?

[jmereaux]

Hi RobWin.

Sure, let me try to give some more details.
I'm using the resilience4j-spring-boot2 maven artifact

I annotate the method on which I want to put the CircuitBreaker as follows :

@CircuitBreaker(name = "MS3CircuitBreaker")
@RateLimiter(name = "MS3RateLimiter")
@Retry(name = "MS3Retry")
public CardDTO debitCard(String transactionId, String cardId, BigDecimal amount, boolean chaosTesting) {

Here is the config I put in my application.yml

resilience4j:
  retry:
    instances:
      MS3Retry:
        maxAttempts: 5
        waitDuration: 20000
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 1
        ignoreExceptions:
        - com.test.exceptions.CardNotFoundException
        - com.test.exceptions.CardDisabledException
        - io.github.resilience4j.circuitbreaker.CallNotPermittedException
        - io.github.resilience4j.ratelimiter.RequestNotPermitted
        - com.test.exceptions.CardThrottlingException
  ratelimiter:
    instances:
      MS3RateLimiter:
        limitForPeriod: 100
        limitRefreshPeriod: 1s
        timeoutDuration: 0s
  circuitbreaker:
    instances:
      MS3CircuitBreaker:
        registerHealthIndicator: true
        failure-rate-threshold: 100
        ignoreExceptions:
        - com.test.exceptions.CardDisabledException
        - com.test.exceptions.CardNotFoundException
        - com.test.exceptions.CardThrottlingException
        - io.github.resilience4j.ratelimiter.RequestNotPermitted
        slowCallRateThreshold: 50
        slowCallDurationThreshold: 2000
        minimum-number-of-calls: 5
        automatic-transition-from-open-to-half-open-enabled: true
        wait-duration-in-open-state: 10000
        permitted-number-of-calls-in-half-open-state: 5
        sliding-window-size: 5
        sliding-window-type: COUNT_BASED

Here is the code I use to toggle the CircuitBreaker off

@PutMapping("/circuitbreaker/off")
public void disableCircuitBreaker() {
    log.info("Disabling Circuit Breaker");
    var ms3CircuitBreaker = cbRegistry.circuitBreaker("MS3CircuitBreaker");
    ms3CircuitBreaker.transitionToDisabledState();
}

And finally some logs that show the transitions of the CircuitBreaker

Mar 11 20:27:37 my-sample-app-5f947c487f-xvmxj my-sample-app WARN 1 --- [io-8080-exec-14] c.i.c.b.a.m.config.CircuitBreakerConfig  : Circuit breaker MS3CircuitBreaker - State transition from CLOSED to DISABLED
Mar 11 20:28:27 my-sample-app-5f947c487f-btcxm my-sample-app WARN 1 --- [io-8080-exec-28] c.i.c.b.a.m.config.CircuitBreakerConfig  : Circuit breaker MS3CircuitBreaker - State transition from CLOSED to OPEN
Mar 11 20:28:27 my-sample-app-5f947c487f-65dvj my-sample-app WARN 1 --- [io-8080-exec-73] c.i.c.b.a.m.config.CircuitBreakerConfig  : Circuit breaker MS3CircuitBreaker - State transition from CLOSED to OPEN
Mar 11 20:28:37 my-sample-app-5f947c487f-btcxm my-sample-app WARN 1 --- [ransitionThread] c.i.c.b.a.m.config.CircuitBreakerConfig  : Circuit breaker MS3CircuitBreaker - State transition from OPEN to HALF_OPEN
Mar 11 20:28:37 my-sample-app-5f947c487f-65dvj my-sample-app WARN 1 --- [ransitionThread] c.i.c.b.a.m.config.CircuitBreakerConfig  : Circuit breaker MS3CircuitBreaker - State transition from OPEN to HALF_OPEN
Mar 11 20:28:38 my-sample-app-5f947c487f-btcxm my-sample-app WARN 1 --- [io-8080-exec-24] c.i.c.b.a.m.config.CircuitBreakerConfig  : Circuit breaker MS3CircuitBreaker - State transition from HALF_OPEN to CLOSED
Mar 11 20:28:38 my-sample-app-5f947c487f-65dvj my-sample-app WARN 1 --- [io-8080-exec-87] c.i.c.b.a.m.config.CircuitBreakerConfig  : Circuit breaker MS3CircuitBreaker - State transition from HALF_OPEN to CLOSED

[RobWin]

Can you please show my your CircuitBreakerConfig?
I wonder why everything is logged twice.

[jmereaux]

Sure, here it is

@Configuration
@Slf4j
public class CircuitBreakerConfig {

    @Bean
    public RegistryEventConsumer<CircuitBreaker> circuitBreakerEventConsumer() {

        return new RegistryEventConsumer<CircuitBreaker>() {
            @Override
            public void onEntryAddedEvent(EntryAddedEvent<CircuitBreaker> entryAddedEvent) {
                entryAddedEvent.getAddedEntry().getEventPublisher()
                        .onFailureRateExceeded(event -> log.error("Circuit breaker {} - Failure rate : {}%",
                                event.getCircuitBreakerName(), event.getFailureRate()))
                        .onSlowCallRateExceeded(event -> log.error("Circuit breaker {} - Slow call rate : {}%",
                                event.getCircuitBreakerName(), event.getSlowCallRate()))
                        .onStateTransition(event -> log.warn("Circuit breaker {} - State transition from {} to {}",
                                event.getCircuitBreakerName(), event.getStateTransition().getFromState(),
                                event.getStateTransition().getToState()));
            }

            @Override
            public void onEntryRemovedEvent(EntryRemovedEvent<CircuitBreaker> entryRemoveEvent) {
                // Not needed
            }

            @Override
            public void onEntryReplacedEvent(EntryReplacedEvent<CircuitBreaker> entryReplacedEvent) {
                // Not needed
            }
        };
    }
}

[RobWin]

That's strange. In the disabled state nothing should happen -> https://github.com/resilience4j/resilience4j/blob/master/resilience4j-circuitbreaker/src/main/java/io/github/resilience4j/circuitbreaker/internal/CircuitBreakerStateMachine.java#L825

I don't see any transition from DISABLED to CLOSED.
Do you see multiple object instances of the same CircuitBreaker? Could you add it to the log statement or debug it?

[jmereaux]

Oh, you're right !

I had not realized I did have multiple instances of the circuit breaker. My app is running on Kubernetes and is spread on 3 pods. So when I send the "disable" request to the CB, actually, I'm only disabling one single instance. The two other ones are still active and can have their state changed.

I guess, I will have to find another solution to globally disable the circuit breaker

[jmereaux]

Is there any way to conditionally activate the @CircuitBreaker annotation on my debitCard method ?

[RobWin]

No, unfortunately not.
You could implement a ignoreExceptionPredicate and this predicate could ignore any exception when you want the CircuitBreaker to be disabled.

A custom Predicate which evaluates if an exception should be ignored and neither count as a failure nor success.

If activated == true then ignore only a specific list of exceptions (com.test.exceptions.CardDisabledException, com.test.exceptions.CardNotFoundException, ...)
If activated == false then ignore all exceptions.

[jmereaux]

That sounds like a really good idea, thanks.

Do you think this trick would also ignore slow requests that also trigger the opening of the circuit ?
Indeed for my use case, when the CB disabled, it should ignore failures but also slow requests above the defined threshold.

[RobWin]

Yes, it would ignore any exception. That means even TimeoutException would not increase the failure rate or the slow call rate.

[jmereaux]

Thanks @RobWin. I will try that

[jmereaux]

@RobWin Would you have any sample explaining how to implement the ignoreExceptionPredicate please ? I could not find anything in the doc.
Thanks

[RobWin]

Similar to the failure rate predicate.
Just implement a Java Predicate and configure it in our config file.

[jmereaux]

Hi @RobWin
I've come up to the following implementation :

public class CircuitBreakerIgnoreExceptionPredicate implements Predicate<Exception> {

    private static final String MS3_CIRCUITBREAKER_ENABLED_ENV_VAR_NAME = "MS3_CIRCUITBREAKER_ENABLED";

    @Override
    public boolean test(Exception e) {
        var ms3CircuitBreakerEnabled = Boolean.parseBoolean(System.getenv(MS3_CIRCUITBREAKER_ENABLED_ENV_VAR_NAME));
        log.debug("Testing exception {} - CircuitBreaker enabled : {}", e.getClass(), ms3CircuitBreakerEnabled);
        if (ms3CircuitBreakerEnabled) {
            return e instanceof CardDisabledException || e instanceof CardNotFoundException
                    || e instanceof CardThrottlingException || e instanceof RequestNotPermitted;
        }
        return true;
    }
}

with the following configuration

resilience4j:
  circuitbreaker:
    instances:
      MS3CircuitBreaker:
        registerHealthIndicator: true
        failure-rate-threshold: 100
        ignoreExceptionPredicate: com.test.CircuitBreakerIgnoreExceptionPredicate
        slowCallRateThreshold: 50
        slowCallDurationThreshold: 2000
        minimum-number-of-calls: 5
        automatic-transition-from-open-to-half-open-enabled: true
        wait-duration-in-open-state: 10000
        permitted-number-of-calls-in-half-open-state: 5
        sliding-window-size: 5
        sliding-window-type: COUNT_BASED

When my env var MS3_CIRCUITBREAKER_ENABLED is set to false, the circuit still opens when I send 5 slow requests above the threshold of 2000ms.
I would also like to ignore these slow requests when I disable my CircuitBreaker.

Did I miss anything ? Thanks

[RobWin]

Are you sure that the slow requests are taken into account?

https://github.com/resilience4j/resilience4j/blob/master/resilience4j-circuitbreaker/src/main/java/io/github/resilience4j/circuitbreaker/internal/CircuitBreakerStateMachine.java#L226-L231

This should not happen. Coud you debug or log it?