Gitjob is not logging webhook payload anymore

[thehejik]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

In rancher:v2.9-head-c9be13b with preinstalled fleet:104.0.0+up0.10.0-rc.11 there are no logs entries for webhook payload anymore.

I was testing with private azure https gitrepo with created secret.

The webhook payload messages are not appearing even when --debug is specified in gitjob deployment.

Reproducer:

• Install rancher v2.9-head
• add a azure gitrepo with spec.pollingInterval: 24h to disable polling into fleet-local namespace
• enable webhook on dev.azure.com for that repo
• Then trigger the webhook by pushing a commit to repository (you may also use Test button in azure webhook settings)
• At the same time observe kubectl logs -n cattle-fleet-system -lapp=gitjob -f output. There wont be anything appearing but the resources will be updated by fleet.

[thehejik]

@manno I tried to test the webhook payload visibility on gitjob pod and still it is not appearing in logs, the fix should be in already but not sure if anything else is needed.

Tested on versions:

rancher:v2.9-98d8b8975948ce5929e38e8603950ceb223349de-head
fleet:104.0.0+up0.10.0-rc.13

Also there is some logger error/stacktrace in the logs (last line no matches for kind \"GitRepo\" in version \"fleet.cattle.io/v1alpha1\"):

> kubectl logs -n cattle-fleet-system -lapp=gitjob -f
{"level":"info","ts":"2024-05-02T13:30:54Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8081","secure":false}
{"level":"info","ts":"2024-05-02T13:30:54Z","logger":"setup","msg":"Setting up webhook listener"}
{"level":"info","ts":"2024-05-02T13:30:55Z","msg":"Starting EventSource","controller":"gitrepo","controllerGroup":"fleet.cattle.io","controllerKind":"GitRepo","source":"kind source: *v1alpha1.GitRepo"}
{"level":"info","ts":"2024-05-02T13:30:55Z","msg":"Starting EventSource","controller":"gitrepo","controllerGroup":"fleet.cattle.io","controllerKind":"GitRepo","source":"kind source: *v1.Job"}
{"level":"info","ts":"2024-05-02T13:30:55Z","msg":"Starting Controller","controller":"gitrepo","controllerGroup":"fleet.cattle.io","controllerKind":"GitRepo"}
{"level":"error","ts":"2024-05-02T13:30:55Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"GitRepo.fleet.cattle.io","error":"no matches for kind \"GitRepo\" in version \"fleet.cattle.io/v1alpha1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/loop.go:53\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/loop.go:54\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/source/kind.go:56"}

[rubhanazeem]

@thehejik can you please double check if this is still the case?

I am currently on fleet v0.10.0-rc.13 and I can see webhook payload in gitjob logs.

webhook    Webhook payload    {"payload": {"ref":"refs/heads/master","before":"","after":"7b30bce854ffcd27b976d5673aa2ef8d66fe08ed","created":false,"del │
│ eted":false,"forced":false,"base_ref":null,"compare":"","commits":null,"head_commit":{"id":"","node_id":"" ...

[thehejik]

I still don't the payload in the logs, I used spec.disablePolling: true this time in gitrepo and github webhook - the webhook is working fine as the resources were provisioned on push to my github repo. Maybe it is somehow related to rancher.

Tested on local rancher 2.9-head with fleet:104.0.0+up0.10.0-rc.13

[thehejik]

Payload still now appearing in rancher:v2.9-head with fleet:104.0.0+up0.10.0-rc.14

@rubhanazeem can we unblock this please?

[thehejik]

Got it ... problem is that the standalone dev/setup-fleet script is using extra helm options --set debug=true --set debugLevel=1.

Without them the payload is not visible, when I use them the payload is appearing in gitjob logs.

helm repo add fleet https://rancher.github.io/fleet-helm-charts/
helm -n cattle-fleet-system install --create-namespace --wait fleet-crd fleet/fleet-crd --devel --version 0.10.0-rc.14
helm -n cattle-fleet-system upgrade --install --create-namespace --wait fleet fleet/fleet --devel --version 0.10.0-rc.14 --set debug=true --set debugLevel=1

When I add --debug flag into gitjob deployment when fleet is installed by rancher the payload started to appear.

...
    spec:                            
      containers:                    
      - args:                        
        - fleetcontroller            
        - gitjob                          
        - --gitjob-image                                                                                                             
        - rancher/fleet:v0.10.0-rc.14      
        - --debug         
...

So it looks we need either:

• to add --debug flag in fleetcontroller gitjob ... --debug gitjob's args/cmd to be able see the wh payload
• or somehow change the loglevel for the payload message to be visible by default even without the --debug flag

This is how the messages looks like in kubectl logs -n cattle-fleet-system -lapp=gitjob output:

2024-05-30T14:02:32Z	INFO	Starting Controller	{"controller": "gitrepo", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo"}
2024-05-30T14:02:32Z	INFO	Starting workers	{"controller": "gitrepo", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "worker count": 1}
2024-05-30T14:03:32Z	DEBUG	webhook	Webhook payload	{"payload": {"ref":"refs/heads/main",...