@jsharpe I am not quite remember why it was designed as previous. What is the concern to disallow non-root module to call local_toolchain? I'd like to verify it a little bit before adding tests.

A non-root module can currently do this:

cuda = use_extension("@rules_cuda//cuda:extensions.bzl", "toolchain")
use_repo(cuda, "local_cuda")

This makes local_cuda visible in the scope of the non-root module but leaves the configuration of local_cuda still to be the root module as only the root module can tell you the path to where the cuda toolkit is installed.
I think the real problem though is the need for non-root modules to reach into local_cuda - in the future a hermetic toolchain would not be called local_cuda and so really we need another location that is independent of the configured toolchain to provide e.g. the cublas targets.

It is not about cublas only, but rules instead. if you think abount it a bit further, say cukernels target is wrapped in a clibrary target and then is exposed as a public interface/target then all clibrary consumers will be required to configure rules_cuda explicitly. This will be quite annoying from the users' PoV. And it is contagious that makes it even worse.

@local_cuda is a bit misleading if we introduce hermetic toolchain, but we still need sth like @cuda_toolkit to the host toolchains so that it can do the compilation (headers) and linking (libs) with the necessary files.

To me, it is not the configuration being a problem, someone need to do it at some level/point anyway. It is the endless propagation to downstream module/repo transitively being a problem.