How to properly build a resilient event handler for Kubernetes events

[rivamarco]

Hi everyone,

I would like to build an event handler (i.e. something that will constantly receive some add, update and delete events) inside a Scala microservice, in order to watch for kubernetes resources that is resilient and in particular that:

• is "self healing" in some sort of way
• works with one or more replicas so that only one replica have the control of the events

So I've tried to build something using LeaderElection + SharedInformers (this is a very little example):

object Fabric8Controller extends App {

  val client                = new KubernetesClientBuilder().build()
  val sharedInformerFactory = client.informers
  val podInformer           = sharedInformerFactory.sharedIndexInformerFor(classOf[Pod], 0L)
  podInformer.addEventHandler(new SimpleHandler)
  val lockIdentity = UUID.randomUUID.toString

  val elector = client.leaderElector
    .withConfig(
      new LeaderElectionConfigBuilder()
        .withName("Sample Leader Election configuration")
        .withLeaseDuration(Duration.ofSeconds(15L))
        .withLock(new LeaseLock("namespace", "lease-name", lockIdentity))
        .withRenewDeadline(Duration.ofSeconds(10L))
        .withRetryPeriod(Duration.ofSeconds(2L))
        .withLeaderCallbacks(
          new LeaderCallbacks(
            () => {
              System.out.println("STARTED LEADERSHIP")
              sharedInformerFactory.startAllRegisteredInformers()
            },
            () => {
              System.out.println("STOPPED LEADERSHIP")
              sharedInformerFactory.stopAllRegisteredInformers()
            },
            newLeader => System.out.printf("New leader elected %s\n", newLeader)
          )
        )
        .build
    )
    .build

  elector.run()

}

class SimpleHandler extends ResourceEventHandler[Pod] {
  override def onAdd(obj: Pod): Unit                                       = println("ADD")
  override def onUpdate(oldObj: Pod, newObj: Pod): Unit                    = println("UPDATE")
  override def onDelete(obj: Pod, deletedFinalStateUnknown: Boolean): Unit = println("DELETE")
}

What I noticed is that this piece of code may fail (because of network error or also because of Kuberetes API errors) and is not able to restart itself if the leadership is lost.

I've also tried to put the elector.run() inside a while loop but it seems is not working correctly.

I don't know if i have to build an external structure to perform this kind of self healing process or if there is something out of the box.

I think I'm doing something wrong but I can't figure out what.

Thank you for your help

[andreaTP]

Hi @rivamarco ! Thanks for this discussion!

@csviri implemented basically all of this logic in the JOSDK: https://github.com/java-operator-sdk/java-operator-sdk

Additionally, the JOSDK should be pretty usable even from a Scala project (please open issues if you find rough edges) despite the need for Annotations 😝 .

[rivamarco]

Thank you I will try this!

[csviri]

@rivamarco I assume you don't write an operator, if you do, pls consider using JOSDK, where it is implemented.

If leadership is lost what we do is just exit the process (System.exit). Since we don't want to have parallel processing, I guess since you do leader election might be a good idea for you. In this case we can assume that other instance is already the leader.

Regarding self healing and events, in Informer that is already built int, if you mean that.

[csviri]

About self healing I noticed (in the above snippet) that if some communication issues happens (loss of network, unresponsive kubernetes api) it will fail and never recover. Is this what you mean?

Yes, in informers should be retried indefinitely (except some corner cases - usually you use multiple version of a custom resource, and a deserialization issue happen because of a wrong setup)

But @shawkins can tell more about this.

[andreaTP]

Yes my goal is to have a piece of code inside my microservice that can receive kubernetes events and react in some way, not a "proper" Kubernetes operator.

this is actually the definition of an operator 🙂 but "outside" the service.

[csviri]

probably it depends, so if just watching a configMap and based on that do some runtime config changes, but otherwise your application is a classic webservice, I would not call it an operator.

Also Operator pattern is mainly described in relation with custom resources and related controllers. Usually pod controllers are not replaces. Event built-in resource controllers are rarely replaces (Ingress is a nice exception however :) )

[rivamarco]

So this is Pod because of little fast example, but then I should watch a custom CRD.

It's barely first time I implement something like this so I may use wrong names 😄

I will also try more with "plain" Informers because they stop in some cases 😞

The goal is basically a bullet proof event handler.

Thanks again for the very quick help. 😄

[andreaTP]

@rivamarco from what you are saying my first feeling is that you will have more problems solved than headaches starting straight away with the Operator Framework, but, anyhow, enjoy the journey!