[enhancement] ignore the update events of heartbeat time changed when reconcile node events

[microyahoo]

Is this a bug report or feature request?

• Bug Report

I consider this an enhancement more than a bug report.

I deployed a new cluster and found ceph cluster predicate for node watcher periodically prints networkfences.csiaddons.openshift.io CRD not found, skip creating networkFence. The predicate function watch the node events to trigger reconcile, then controller-runtime client handles node failure whenROOK_WATCH_FOR_NODE_FAILURE env set to true. The log will be pop up if networkfences csiaddons CRDs not found.

In order to figure out why the update event keeps being triggered and the handleNodeFailure method is called periodically, I hacked the code to print out the diff of ObjectNew and ObjectOld, and found that when the cluster is stable, the update is only the resource version and heartbeat time. Therefore, my question is, can we ignore this kind of event in the update event and only handle the real node failure or other changed events.

{
  "metadata": {
    "resourceVersion": "14127"
  },
  "status": {
    "$setElementOrder/conditions": [
      {
        "type": "NetworkUnavailable"
      },
      {
        "type": "MemoryPressure"
      },
      {
        "type": "DiskPressure"
      },
      {
        "type": "PIDPressure"
      },
      {
        "type": "Ready"
      }
    ],
    "conditions": [
      {
        "lastHeartbeatTime": "2024-04-12T10:36:54Z",
        "type": "MemoryPressure"
      },
      {
        "lastHeartbeatTime": "2024-04-12T10:36:54Z",
        "type": "DiskPressure"
      },
      {
        "lastHeartbeatTime": "2024-04-12T10:36:54Z",
        "type": "PIDPressure"
      },
      {
        "lastHeartbeatTime": "2024-04-12T10:36:54Z",
        "type": "Ready"
      }
    ]
  }
}

Deviation from expected behavior:

Expected behavior:

How to reproduce it (minimal and precise):

File(s) to submit:

• Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:

• Operator's logs, if necessary

• Crashing pod(s) logs, if necessary

  To get logs, use kubectl -n <namespace> logs <pod name>
  When pasting logs, always surround them with backticks or use the insert code button from the Github UI.
  Read GitHub documentation if you need help.

Cluster Status to submit:

• Output of kubectl commands, if necessary

  To get the health of the cluster, use kubectl rook-ceph health
  To get the status of the cluster, use kubectl rook-ceph ceph status
  For more details, see the Rook kubectl Plugin

Environment:

• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Cloud provider or hardware configuration:
• Rook version (use rook version inside of a Rook Pod):
• Storage backend version (e.g. for ceph do ceph -v):
• Kubernetes version (use kubectl version):
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

[parth-gr]

@microyahoo I also see the type is changing with time, which seems right to watch for

[microyahoo]

hi @parth-gr, I compared the current and the modified object. The diff is as follows

~ cat current | python3 -m json.tool > current.json
~ cat modified | python3 -m json.tool > modified.json
~ vimdiff current.json modified.json

[parth-gr]

Okay then we need to create a custom resourceComparater that avoids looking at the difference for this

[parth-gr]

/assign

[github-actions]

Thanks for taking this issue! Let us know if you have any questions!