Support for system-provided libzstd

[jirutka]

Can you please add support for loading system-provided libzstd when it’s available?

[luben]

I don't plan to add support for that. What is the problem that you try to solve?

For context, the JAR includes parts of libzstd statically linked with the JNI functions. Zstd-JNI cannot function against the upstream libzstd as it is missing the JNI bindings. If we ship just the bindings, it will introduce risks to version mismatches, e.g. bindings are against v.1.5 but the system provides v1.4 that lacks some symbols.

[luben]

Closing as I didn't hear back about the motivation for such a change.

[jirutka]

I’m sorry for the late response, I forgot about it.

The motivation is to have a single libzstd on the system managed by the system package manager. When some security vulnerability is found in libzstd, the user and package maintainers) just update libzstd. Bundling native binaries in JARs is not a good practice, because it’s “hidden” and tends to be outdated.

Another reason is support for various CPU architectures – Linux distros typically support many CPU architectures. For example, you don’t precompile zstd-jni for s390x or armv7, both supported by Alpine Linux. And again, it’s a “hidden” constraint – when you download some Java application with zstd-jni among its bundled JARs, it’s not apparent that some contain native code that won’t run on the given platform.

I’ve packaged zstd-jni for Alpine Linux. There’s java-zstd-jni with portable JAR (without native code) and java-zstd-jni-native with the JNI library. The latter should be ideally linked against the system-provided libzstd instead of bundling it. This is currently used for the neo4j package.

[luben]

Reopening. @jirutka, let's discuss.

Topic: features and backward compatibility

Some of the symbols in zstd are available only for statically linking, so if we dynamically link we will lose some features, e.g. magicless frames, multiple dicts, etc. This means that if we change to dynamic linking we will break customers that use these features.

We can do some king of static/dynamic linking mix (hoping that LTO can strip unused parts) but this will make the result even more fragile. The statically linked symbols are experimental and there is no guarantee by zstd that they will not change across versions.

Topic: security patching

Due to use of experimental features libzstd-jni-1.X.Y-Z.so depends on libzstd.so.1.X.Y - and if there is a mismatch it may lead to runtime failures. This means that if Zstd is patched for security issue, there will be libzstd.so.1.X.(Y+1), so we will need anyway to produce libzstd-jni-1.X.(Y+1)-1.so, and if there is a security issue on the Java side, we will need to produce anyway libzstd-jni-1.X.Y-(Z+1).so.

So, currently if there is security vulnerability in libzstd, packagers should also re-compile libzstd-jni. But dynamically linking libzstd.so we will not change that.

Regarding security response: I think the right approach here is to ensure the right process is in place, e.g. make sure I and few other people with permissions are notified about security releases before making them public so that we can synchronize releases.

Topic: architecture support

Support for CPU architectures: yes both x390x and armv7 are supported by the bundle [1], and we also provide qualified jars with individual architectures, e.g. [2], [3]

Topic: packaging

Yes, it makes sense to split the portable/native parts in different distro packages as you have done - we don't have to extract the .so on each execution, and also this allows to harden the JVM security settings. This is already supported. Though I don't see reason why the native part will depend/link to the system libzstd.so.

Links:

1. https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.5.5-4/zstd-jni-1.5.5-4.jar
2. https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.5.5-4/zstd-jni-1.5.5-4-linux_s390x.jar
3. https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.5.5-4/zstd-jni-1.5.5-4-linux_arm.jar