container toolkit daemon only cleans up containerd runtime config, and doesn't clean up the nvidia container runtime binaries on host /user/local/nvidia

[wyike]

I also see code here https://gitlab.com/nvidia/container-toolkit/container-toolkit/-/blob/main/tools/container/nvidia-toolkit/run.go?ref_type=heads#L131-140. It only reset the container runtime config.

After gpu operator or container toolkit removal, the nvidia/toolkit will be left on host:

worker/d3067738-6590-433d-8dd5-2568f7bccbb9:~# ls /usr/local/nvidia/toolkit/
libnvidia-container-go.so.1       nvidia-container-cli          nvidia-container-runtime.cdi.real   nvidia-container-runtime.legacy.real  nvidia-ctk.real
libnvidia-container-go.so.1.15.0  nvidia-container-cli.real     nvidia-container-runtime-hook       nvidia-container-runtime.real
libnvidia-container.so.1          nvidia-container-runtime      nvidia-container-runtime-hook.real  nvidia-container-toolkit
libnvidia-container.so.1.15.0     nvidia-container-runtime.cdi  nvidia-container-runtime.legacy     nvidia-ctk

Asking here if it is an intentional design or some cleanup step we have missed. Thanks!