Gitjob complains that Operation cannot be fulfilled on gitrepos.fleet.cattle.io

[thehejik]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

In rancher:v2.9-head-c9be13b with preinstalled fleet:104.0.0+up0.10.0-rc.11 there are new suspicious info log messages from gitjob pod that were not present in previous versions. Not sure if it has some impact on functionality yet.

I was testing with private azure https gitrepo but it is appearing also for public github gitrepo.

$ kubectl logs -n cattle-fleet-system -lapp=gitjob -f
{
  "level": "info",
  "ts": "2024-04-19T15:51:17Z",
  "logger": "gitjob-reconciler",
  "msg": "conflict updating status",
  "message": "Operation cannot be fulfilled on gitrepos.fleet.cattle.io \"gh-https-2\": the object has been modified; please apply your changes to the latest version and try again"
}

Reproducer:

Install rancher v2.9-head and add a gitrepo to fleet-local namespace, then observe the gitjob logs.

[thehejik]

During daily we discussed it is maybe related to another loglevel set or due to "zapr" logger.

[bigkevmcd]

The code around this

fleet/internal/cmd/controller/gitops/reconciler/gitjob_controller.go

Line 169 in b996ca8

return r.Status().Update(ctx, &gitRepoFromCluster)

could maybe be updated to .Patch() rather than .Update() and probably reload the status from the kube-apiserver before calculating the patch to apply.

[Tommy12789]

Additional QA

Issue:

2343

Problem

When applying a GitRepo, gitjob complaining about "Operation cannot be fulfilled on gitrepos.fleet.cattle.io" because gitrepo controller conflicting with gitjob controller.

Solution

Added the use of retryonconflict method within the updateStatusMethod of the gitjob reconciler to retry when a conflict happens

Testing

Repro steps are valid

Engineering Testing

Manual Testing

Repro steps from the issue

Automated Testing

N/A

QA Testing Considerations

• Apply multiple gitrepo in multiple namespace to see if any error appears

Regressions Considerations

N/A

[thehejik]

This problem is verified fixed in rancher:v2.9-head-de693b5 with fleet:104.0.0+up0.10.0-rc.14. The message is not appearing in gitjob logs anymore after adding a gitrepo.