Will informer resync enqueue the older event?

[pan3793]

This is another question about the informer, will the informer "resync" enqueue the older event?

Let's say I have a Pod informer, and one POD produces events: CREATE, UPDATE, DELETE. For some reason, CREATE and DELETE events were consumed success, but the UPDATE event is not, after resync period, will the UPDATE event be re-enqueued and handled by the ResourceEventHandler?

new ResourceEventHandler<Pod>() {
  @Override
  public void onAdd(Pod pod) {
    ...
  }

  @Override
  public void onUpdate(Pod oldPod, Pod newPod) {
    // should I cache each Pod latest resourceVersion and do a comparison here?
    ...
  }

  @Override
  public void onDelete(Pod pod, boolean deletedFinalStateUnknown) {
    ...
  }
}

Another question is, I know the resync is totally a client-side behavior, I find some blogs discussed the Go client implementation but not Java's, so, in which cases(exceeding event rate limit or queue quota?), the informer think the event should be re-enqueued during the resync?

[shawkins]

will the UPDATE event be re-enqueued and handled by the ResourceEventHandler

No, the informer will make only a single attempt at the event processing. It's entirely up to the handler if some retry or secondary queue is maintained.

See a framework like the java operator sdk https://github.com/java-operator-sdk/java-operator-sdk that maintains a retriable event buffer above the informer level.

// should I cache each Pod latest resourceVersion and do a comparison here?

They will only be the same for a resync event.

informer think the event should be re-enqueued during the resync

The internals of the event queue and processing are completely different between the go and java implementations. As of now there is no circumstance under which the java implementation will reprocess a given event. At the next resync it will generate all applicable resync updates. Usage of the built-in resync is entirely optional. You can just as easily schedule a similar job only when you fail to process an event.

[shawkins]

Just this -

kubernetes-client/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/cache/ProcessorStore.java

Line 132 in f64529c

public void resync() {

- an update event for each item in the cache where old and new are the same.

[pan3793]

Thanks @shawkins, here is my understanding from reading the code, please correct me if I'm wrong.

All events will be sent to the SerialExecutor, but the ReflectorWatcher and the scheduled resync task are running in different threads, so there may have a chance that

t1 - ReflectorWatcher receives Pod-A UPDATE event e1, updates the ProcessorStore and sends e1 to the SerialExecutor
t2 - resync task is scheduled in a different thread, reads e1 from ProcessorStore and sends e1 to SerialExecutor
t3 - nearly t2, ReflectorWatcher receives Pod-A UPDATE event e2, updates the ProcessorStore and sends e2 to the SerialExecutor

t2 and t3 almost happen at the same times(t3 after t2), considering ReflectorWatcher and the scheduled resync task are running in the different threads, there is a chance that e2 is submitted before e1.

Then, the ResourceEventHandler receives

• onUpdate e0,e1 (e0 is the previous state)
• onUpdate e1,e2
• onUpdate e1,e1

Am I missing or misunderstanding something?

[shawkins]

Am I missing or misunderstanding something?

No. The resync currently works off of a snapshot of the cache. It does not attempt to lock against other events being processed before all of the resync events are sent out. I'm not sure if there was an earlier intention behind this, but given the queues in play now it could easily lock.

Generally I would avoid using resync in favor of more intentional retry processing and/or when it runs.

[pan3793]

It's clear to me now. Many thanks, @shawkins

[shawkins]

#5018 was opened to address the concern about resync event ordering.