Create .so symlinks for driver libraries in container #326

elezar · 2024-02-01T09:26:38Z

When NVIDIA_DOT_SO_SYMLINKS is not equal to enabled:

elezar@dgx0126:~/src/container-toolkit$ docker run --rm -ti --runtime=nvidia-dev -e NVIDIA_DOT_SO_SYMLINKS=not-enabled -e NVIDIA_VI
SIBLE_DEVICES=all ubuntu bash -c "ls -al /usr/lib/x86_64-linux-gnu/libnvidia-ml.*"
lrwxrwxrwx 1 root root      26 Nov 27 14:19 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 -> libnvidia-ml.so.515.105.01
-rw-r--r-- 1 root root 1683960 Feb 27  2023 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.515.105.01

When NVIDIA_DOT_SO_SYMLINKS=enabled:

elezar@dgx0126:~/src/container-toolkit$ docker run --rm -ti --runtime=nvidia-dev -e NVIDIA_DOT_SO_SYMLINKS=enabled -e NVIDIA_VISIBL
E_DEVICES=all ubuntu bash -c "ls -al /usr/lib/x86_64-linux-gnu/libnvidia-ml.*"
lrwxrwxrwx 1 root root      17 Nov 27 14:19 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so -> libnvidia-ml.so.1
lrwxrwxrwx 1 root root      26 Nov 27 14:19 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 -> libnvidia-ml.so.515.105.01
-rw-r--r-- 1 root root 1683960 Feb 27  2023 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.515.105.01

This also works with CDI:

docker run --rm -ti --device elezar.nvidia.com/gpu=all ubuntu
root@9d74d452dc2c:/# ls -al /usr/lib/x86_64-linux-gnu/libnvidia-ml.*
lrwxrwxrwx 1 root root      17 Apr  3 14:35 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so -> libnvidia-ml.so.1
lrwxrwxrwx 1 root root      25 Apr  3 14:35 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 -> libnvidia-ml.so.550.54.15
-rw-r--r-- 1 root root 2078360 Mar  5 22:01 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.54.15
root@9d74d452dc2c:/# ls -al /usr/lib/x86_64-linux-gnu/libcuda.*
lrwxrwxrwx 1 root root       12 Apr  3 14:35 /usr/lib/x86_64-linux-gnu/libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root       20 Apr  3 14:35 /usr/lib/x86_64-linux-gnu/libcuda.so.1 -> libcuda.so.550.54.15
-rw-r--r-- 1 root root 28392536 Mar  5 22:26 /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15

Note that a --no-dot-so-symlinks option is added to the nvidia-ctk cdi generate command.

Migrated from https://gitlab.com/nvidia/container-toolkit/container-toolkit/-/merge_requests/510

Depends on #327

klueska

Initial pass. I lost some steam near the end, but wanted to drop the comments I had for now.

klueska · 2024-04-15T11:47:43Z

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

+	// Create the '' command
+	c := cli.Command{
+		Name:  "create-dot-so-symlinks",
+		Usage: "A hook to create symlinks in the container. This can be used to process CSV mount specs",


Is this specific to CSV mount specs?

No. This is not correct. Will update.

klueska · 2024-04-15T11:48:52Z

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

+	libs, err := lookup.NewLibraryLocator(
+		lookup.WithLogger(m.logger),
+		lookup.WithRoot(containerRoot),
+		lookup.WithOptional(true),
+	).Locate("*.so." + cfg.driverVersion)


Can you separate this into two calls? Chaining on separate lines like this always looks awkward to me in go code.

Sure. Will update.

klueska · 2024-04-15T11:50:31Z

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

+			continue
+		}
+		libSoPath := strings.TrimSuffix(lib, "."+cfg.driverVersion)
+		libSoXPaths, err := filepath.Glob(libSoPath + ".[0-9]")


It's probably fine for our purposes, but should this actually be ".[0-9]+" t include multi-digit versions?

Yes, it should. I was thinking one should probably rather extract the SONAME from the target library and do it that way, but thought this heuristic was enough.

One issue is that a Glob pattern is not a regex and the + is not available there.

I suppose instead of including this logic, we chould just symlink .so -> .so.driverVersion directly and bypass the .so.[0-9] symlink here. Thoughts?

klueska · 2024-04-15T11:53:31Z

internal/config/features.go

+	// DotSoSymlinks allows for the creation of .so symlinks to .so.1 driver
+	// files to be opted out of.


Technically it allows them to opt-in, correct? We just happen to set the default to true so that opting-out is the "opposite" action, but the flag still exists so that one may opt-in (meaning true==create them, false==don't create them).

You're right. I initially had this as "on-by-default" with opt-out logic, but rolled that back and didn't update the comment. Will update.

klueska · 2024-04-15T11:56:25Z

internal/config/features.go

+	// featureNotControlledByEnvvar is used for features that have no envvar to
+	// allow per-container opt-in.
+	featureNotControlledByEnvvar = ""


It's not immediately obvious reading this top-to-bottom how this will be used. From what I can tell, we will allow the feature being added here to be controlled by an envvar (namely NVIDIA_DOT_SO_SYMLINKS), so I'm not sure where this is going to apply yet (maybe it will be come clear later on).

Edit: I see now -- this is just giving a name to what was previously a hard coded "".

That said -- my gut says that leaving it as a hard-coded "" with a comment above the one place it is checked is clearer than adding this constant.

Yes, I initially had this as "" in the code and thought a constant read a little bit better. Let me have another look at it.

klueska · 2024-04-15T12:00:56Z

internal/lookup/root/root.go

+	libCudaPaths, err := cuda.New(
+		r.Libraries(),
+	).Locate(".*.*")


Again, this chaining looks awkward to me, but I don't feel too strongly against it if that is your preference.

elezar

I think some of the issues were because this was out of date.

elezar · 2024-04-15T12:42:28Z

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

+	// Create the '' command
+	c := cli.Command{
+		Name:  "create-dot-so-symlinks",
+		Usage: "A hook to create symlinks in the container. This can be used to process CSV mount specs",


No. This is not correct. Will update.

elezar · 2024-04-15T12:43:14Z

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

+	libs, err := lookup.NewLibraryLocator(
+		lookup.WithLogger(m.logger),
+		lookup.WithRoot(containerRoot),
+		lookup.WithOptional(true),
+	).Locate("*.so." + cfg.driverVersion)


Sure. Will update.

elezar · 2024-04-15T12:47:44Z

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

+			continue
+		}
+		libSoPath := strings.TrimSuffix(lib, "."+cfg.driverVersion)
+		libSoXPaths, err := filepath.Glob(libSoPath + ".[0-9]")


Yes, it should. I was thinking one should probably rather extract the SONAME from the target library and do it that way, but thought this heuristic was enough.

One issue is that a Glob pattern is not a regex and the + is not available there.

I suppose instead of including this logic, we chould just symlink .so -> .so.driverVersion directly and bypass the .so.[0-9] symlink here. Thoughts?

elezar · 2024-04-15T12:49:15Z

internal/config/features.go

+	// featureNotControlledByEnvvar is used for features that have no envvar to
+	// allow per-container opt-in.
+	featureNotControlledByEnvvar = ""


Yes, I initially had this as "" in the code and thought a constant read a little bit better. Let me have another look at it.

This change adds an opt-in feature for creating .so symlinks to all injected driver files in a contianer. If features.dot-so-symlinks = true is set in the config.toml, the creation of symlinks for driver files is enabled. This can also be triggered on a per-container basis using the envvar NVIDIA_DOT_SO_SYMLINKS=enabled. Signed-off-by: Evan Lezar <elezar@nvidia.com>

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar self-assigned this Feb 13, 2024

elezar mentioned this pull request Feb 14, 2024

Add support for feature flags #362

Merged

elezar force-pushed the CNT-4766/create-so-symlinks branch 4 times, most recently from e20826f to 7b5f6b3 Compare April 3, 2024 15:10

elezar marked this pull request as ready for review April 3, 2024 15:10

elezar requested review from klueska and cdesiniotis April 3, 2024 15:11

klueska reviewed Apr 15, 2024

View reviewed changes

elezar commented Apr 15, 2024

View reviewed changes

elezar force-pushed the CNT-4766/create-so-symlinks branch 2 times, most recently from df99dc3 to 93753a9 Compare April 15, 2024 14:28

elezar added 2 commits April 15, 2024 16:31

Generate hook to create .so symlinks in CDI

020f598

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar force-pushed the CNT-4766/create-so-symlinks branch from 93753a9 to 020f598 Compare April 15, 2024 14:31

elezar requested a review from klueska May 15, 2024 12:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create .so symlinks for driver libraries in container #326

Create .so symlinks for driver libraries in container #326

elezar commented Feb 1, 2024 •

edited

klueska left a comment

klueska Apr 15, 2024

elezar Apr 15, 2024

klueska Apr 15, 2024

elezar Apr 15, 2024

klueska Apr 15, 2024

elezar Apr 15, 2024

klueska Apr 15, 2024

elezar Apr 15, 2024

klueska Apr 15, 2024

klueska Apr 15, 2024

klueska Apr 15, 2024

elezar Apr 15, 2024

klueska Apr 15, 2024

elezar left a comment

elezar Apr 15, 2024

elezar Apr 15, 2024

elezar Apr 15, 2024

elezar Apr 15, 2024

		// DotSoSymlinks allows for the creation of .so symlinks to .so.1 driver
		// files to be opted out of.

Create .so symlinks for driver libraries in container #326

Are you sure you want to change the base?

Create .so symlinks for driver libraries in container #326

Conversation

elezar commented Feb 1, 2024 • edited

klueska left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elezar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elezar commented Feb 1, 2024 •

edited