Debugging Kubebuilder tests

[ArpanSriv]

Is there a way to debug the tests while developing a Kubernetes Operator?

For instance, I have the following files with the same package controller:

• internal/controller/component_a.go
• internal/controller/component_b.go
• and other files up till component_f.go

And then, as usual, I have the suite_test.go & component_test.go files in the same directory as the components (with the same package name controller).

Running make test works perfectly fine, but I'm wondering if there's a way to attach a debugger and add breakpoints somehow.

I've currently tried a couple of ways to attach a debugger:

• Tried replicating the exact same configuration in GoLand:

  • Added a run configuration with a Before Launch configuration to execute a Shell script with (Script Text selected) and the contents as the contents of the make targets manifests generate fmt vet envtest (which are run automatically when we run make test). Didn't work, since the shell script execution runs in one window, and then the go test execution runs in a fresh window inside GoLand. Ends up not being able to even start the control plane since the make envtest is crucial to set up the envtest binaries.

  • Added a run configuration with Before Launch configuration as Make targets. Didn't work. Complains about the same as above.

  • Tried running the make test in the terminal, and then running Attach to process inside GoLand, to attach to the go process of go test ./.... But that fails with an error:

    could not attach to pid 84628: stub exited while waiting for connection: exit status 0
    

    Meanwhile, attaching to other processes (like kube-apiserver and etcd) works like a charm, but I have no use attaching to them :).

• Tried to run the tests via dlv test:
  • Tried running dlv test ./... -output /dev/null but that fails with:

    could not launch process: not an executable file
    could not remove /dev/null: remove /dev/null: operation not permitted
    

I'm happy to share any more details.

Any ideas as to how I can debug this? Thanks in advance!

[camilamacedo86]

For you to debug with GolangIDEA, you can just run the make targets manually to ensure that all is generated and execute the controller suite test (suite_test.go) from the IDEA in debug mode.

You can also update the ENV TEST config to run it against an existent cluster instead of use the binaries, see: https://github.com/kubernetes-sigs/controller-runtime/blob/6747c42ce33966b0e77cac278c6b9087cc2f51c7/pkg/envtest/server.go#L149-L152

[ArpanSriv]

Thanks for answering, @camilamacedo86!

Sure, I tried doing that. What happens is that I run the make targets manually in the terminal and then when I run the suite_test.go from within the IDE, it fails to locate the etcd binary IIRC - and so it's not even able to start the control plane.

And that was the reason I tried configuring a 'Before Launch' configuration, hoping it'd execute the 'Before Launch' and the suite_test.go together, in one debugger window - which it doesn't :/

I'll share the exact error in a few hours when I'm back at my desk!

You can also update the ENV TEST config

Unfortunatlely, I am required to run it in the envtest cluster itself.

[camilamacedo86]

Unfortunatlely, I am required to run it in the envtest cluster itself.

Not sure if I follow it.
EnvTest is what we call the framework/lib implementation from controller-runtime to do the tests
So, you can run the EnvTest tests against any cluster running in your cluster.

When you run make test, you are installing the binaries (setup-invest) and using them instead of a real cluster.

However, you might either use the GolangIDEA within the binaries.
But you probably need to setup the ENV VAR: KUBEBUILDER_ASSETS to point out the bin directory
(See https://github.com/kubernetes-sigs/controller-runtime/blob/main/hack/check-everything.sh#L40)

[ArpanSriv]

Sorry - I mean to say I want to run the tests locally, preferably in the test cluster that is created when we use UseExistingCluster: false.

Thanks for sharing that, let me try setting the ENV variable inside the IDE.

[camilamacedo86]

Hi @ArpanSriv

Did it worked for you? Would you like to add a summary of how you achieve it and close this one?

[ArpanSriv]

Thanks for all the suggestions @camilamacedo86. I tried setting the env variable (KUBEBUILDER_ASSETS) but it doesn't seem to work.

Unfortunately, I was not able to make it work with UseExistingCluster = false.

Currently I am occupied with higher-priority projects, so I am out of time to spend on this. We can close it for now :)

[camilamacedo86]

You need to set true UseExistingCluster so that you can run the tests against a pre-existent cluster

[ArpanSriv]

Sure, however I wanted to test / debug without an existing cluster and hence I set the value of useExistingCluster to false.