Unprivileged podman and buildah and reproducibility?

[bureado]

Thank you for putting together this example, it has been tremendous learning for me and I'm sure many others.

Is cargo build expected to happen in the container? If so, at least in my system the Makefile needs a backslash to include the line in the docker run, see bureado@d1a49ff

Also I just wanted to note that in my system (with podman and buildah) I didn't need to specify a uidmap or use sudo. In fact the uidmap was getting me in trouble with cargo.

I hope that is good news (unprivileged and rootless by default!) but I'm also getting a different hash for asdf and I wonder if that's a result of those changes? I have tried reproducing with two different setups and I get the same hash in both systems, but it's different from the one in README.md today, see bureado@bec0cfe

Again, thank you so much for putting this together!

PS: I also used git-notes to store the hashes of the binary and container images with the commit for ease of reproducing, see bureado@cad165f, here's an example of how I use it for manual reproducibility purposes.

[SantiagoTorres]

Would you be able to produce a diffoscope of asdf as produced by @kpcyrd ?

[bureado]

@SantiagoTorres yes, I'll paste one shortly (just learning to use diffoscope)

But this might be related to the (missing?) backslash in Makefile; it might mean that in some systems asdf is built locally and not in the container. A readelf -a on the binary in the v0.1.0 release vs. the one I build locally shows the one in the release has this (not a Rust or cargo person, but I guess this shows a linker took a pass at the binary?):

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 6e16becbaa8538c2e8b0dde344b095f1607188e6

[bureado]

@SantiagoTorres just an update with the diffoscope JSON output, it does mention the notes section as one of the differences: https://gist.github.com/bureado/8b2ee46df581b86f6b82b72f903e5ee2

[SantiagoTorres]

Yeha, I think you're right. I did a little bit of fiddling and it seems it's mostly:

1. There's a new section added .gnu_build-id
2. There's a corresponding offset re-location

The good news is that then it's really not a "repro" issue (because semantically the code is the same), but rather figuring out why this thing was added here and not before (or vice-versa)...

[kpcyrd]

hi, thanks for looking into this!

This is indeed due to the missing backslash, it was probably lost when updating the hashes before the initial release. abdac109cfe060fdf59e7196b11b39be623da148cb03fe47973edab5f28a708ab2bfe02a1785c0a907d0cb3d1cb4d7baf66d155b317af9b0452fb9cb1de895a9 is the correct binary.

Effectively what was executed is:

docker run --rm -v "$(PWD):/app" -w /app -u "$(shell id -u):$(shell id -g)" rust@sha256:8463cc29a3187a10fc8bf5200619aadf78091b997b0c3941345332a931c40a64
cargo build --release --locked --target=x86_64-unknown-linux-musl

The entry point of that docker image is /bin/sh, because docker run was used without -i this is effectively like running sh < /dev/null, which just exits immediately. The 2nd command was then executed outside of the container, on an Arch Linux environment.

The binaries are very similar because both systems use rustc 1.54.0 from rustup and the build was configured to be static, but rust seems to also use the system linker and binutils, which causes the binary to differ. At the moment it's still possible to reproduce the binary that I've published:

% docker run --rm -v "$(pwd):/app" -w /app -it archlinux
[root@6c555a416296 app]# pacman -Sy rustup
:: Synchronizing package databases...
 core                                                                                                             136.2 KiB   324 KiB/s 00:00 [######################################################################################] 100%
 extra                                                                                                           1574.1 KiB  4.96 MiB/s 00:00 [######################################################################################] 100%
 community                                                                                                          5.7 MiB  3.86 MiB/s 00:01 [######################################################################################] 100%
resolving dependencies...
looking for conflicting packages...

Packages (1) rustup-1.24.3-1

Total Download Size:   2.67 MiB
Total Installed Size:  8.08 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 rustup-1.24.3-1-x86_64                                                                                             2.7 MiB  3.22 MiB/s 00:01 [######################################################################################] 100%
(1/1) checking keys in keyring                                                                                                                [######################################################################################] 100%
(1/1) checking package integrity                                                                                                              [######################################################################################] 100%
(1/1) loading package files                                                                                                                   [######################################################################################] 100%
(1/1) checking for file conflicts                                                                                                             [######################################################################################] 100%
:: Processing package changes...
(1/1) installing rustup                                                                                                                       [######################################################################################] 100%
You may need to run rustup update stable
and possibly also rustup self upgrade-data
Optional dependencies for rustup
    lldb: rust-lldb script
    gdb: rust-gdb script
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[root@6c555a416296 app]# rustup install stable && rustup target add x86_64-unknown-linux-musl
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2021-07-29, rust version 1.54.0 (a178d0322 2021-07-26)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
 21.9 MiB /  21.9 MiB (100 %)  20.2 MiB/s in  1s ETA:  0s
info: downloading component 'rustc'
 50.1 MiB /  50.1 MiB (100 %)   6.9 MiB/s in  8s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
 16.7 MiB /  16.7 MiB (100 %)   3.3 MiB/s in  5s ETA:  0s
info: installing component 'rust-std'
 21.9 MiB /  21.9 MiB (100 %)   7.5 MiB/s in  2s ETA:  0s
info: installing component 'rustc'
 50.1 MiB /  50.1 MiB (100 %)   9.5 MiB/s in  5s ETA:  0s
info: installing component 'rustfmt'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.54.0 (a178d0322 2021-07-26)

info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl'
info: installing component 'rust-std' for 'x86_64-unknown-linux-musl'
 30.4 MiB /  30.4 MiB (100 %)   8.1 MiB/s in  3s ETA:  0s
[root@6c555a416296 app]# cargo build --release --locked --target=x86_64-unknown-linux-musl
   Compiling asdf v0.1.0 (/app)
error: linker `cc` not found
  |
  = note: No such file or directory (os error 2)

error: aborting due to previous error

error: could not compile `asdf`

To learn more, run the command again with --verbose.
[root@6c555a416296 app]# pacman -S gcc
resolving dependencies...
looking for conflicting packages...

Packages (4) binutils-2.36.1-3  elfutils-0.185-1  libmpc-1.2.1-1  gcc-11.1.0-1

Total Download Size:    40.80 MiB
Total Installed Size:  199.45 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 elfutils-0.185-1-x86_64                                                                                          552.6 KiB  1168 KiB/s 00:00 [######################################################################################] 100%
 binutils-2.36.1-3-x86_64                                                                                           6.2 MiB  2.37 MiB/s 00:03 [######################################################################################] 100%
 libmpc-1.2.1-1-x86_64                                                                                             70.5 KiB   705 KiB/s 00:00 [######################################################################################] 100%
 gcc-11.1.0-1-x86_64                                                                                               34.0 MiB   896 KiB/s 00:39 [######################################################################################] 100%
 Total (4/4)                                                                                                       40.8 MiB   989 KiB/s 00:42 [######################################################################################] 100%
(4/4) checking keys in keyring                                                                                                                [######################################################################################] 100%
(4/4) checking package integrity                                                                                                              [######################################################################################] 100%
(4/4) loading package files                                                                                                                   [######################################################################################] 100%
(4/4) checking for file conflicts                                                                                                             [######################################################################################] 100%
:: Processing package changes...
(1/4) installing elfutils                                                                                                                     [######################################################################################] 100%
(2/4) installing binutils                                                                                                                     [######################################################################################] 100%
(3/4) installing libmpc                                                                                                                       [######################################################################################] 100%
(4/4) installing gcc                                                                                                                          [######################################################################################] 100%
Optional dependencies for gcc
    lib32-gcc-libs: for generating code for 32-bit ABI
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[root@6c555a416296 app]# cargo build --release --locked --target=x86_64-unknown-linux-musl
   Compiling asdf v0.1.0 (/app)
    Finished release [optimized] target(s) in 0.92s
[root@6c555a416296 app]# b2sum target/x86_64-unknown-linux-musl/release/asdf
35578eb5fbd13fe27bbc9f799488de2a196acfdb00886d7a5b88e13e0a73e8197fded1afdc9eb6b886864cd39bdeaa17910351da971710056630b1cb3a31a8cd  target/x86_64-unknown-linux-musl/release/asdf
[root@6c555a416296 app]# 

For posterity, list of Arch Linux packages used:

[root@6c555a416296 app]# pacman -Q
acl 2.3.1-1
archlinux-keyring 20210802-1
argon2 20190702-3
attr 2.5.1-1
audit 3.0.3-1
base 2-2
bash 5.1.008-1
binutils 2.36.1-3
bzip2 1.0.8-4
ca-certificates 20210603-1
ca-certificates-mozilla 3.69-1
ca-certificates-utils 20210603-1
coreutils 8.32-1
cryptsetup 2.3.6-1
curl 7.78.0-1
dbus 1.12.20-1
device-mapper 2.03.12-1
e2fsprogs 1.46.3-3
elfutils 0.185-1
expat 2.4.1-1
file 5.40-5
filesystem 2021.05.31-1
findutils 4.8.0-1
gawk 5.1.0-1
gcc 11.1.0-1
gcc-libs 11.1.0-1
gettext 0.21-1
glib2 2.68.3-1
glibc 2.33-5
gmp 6.2.1-1
gnupg 2.2.29-1
gnutls 3.7.2-2
gpgme 1.16.0-1
grep 3.6-1
gzip 1.10-3
hwids 20210613-1
iana-etc 20210728-1
icu 69.1-1
iproute2 5.13.0-1
iptables 1:1.8.7-1
iputils 20210722-1
json-c 0.15-1
kbd 2.4.0-2
keyutils 1.6.3-1
kmod 29-1
krb5 1.19.1-1
less 1:590-1
libarchive 3.5.1-1
libassuan 2.5.5-1
libcap 2.51-1
libcap-ng 0.8.2-3
libcroco 0.6.13-2
libelf 0.185-1
libffi 3.3-4
libgcrypt 1.9.3-1
libgpg-error 1.42-1
libidn2 2.3.2-1
libksba 1.6.0-1
libldap 2.4.59-2
libmnl 1.0.4-3
libmpc 1.2.1-1
libnetfilter_conntrack 1.0.8-1
libnfnetlink 1.0.1-4
libnftnl 1.2.0-1
libnghttp2 1.44.0-1
libnl 3.5.0-3
libp11-kit 0.24.0-1
libpcap 1.10.1-1
libpsl 0.21.1-1
libsasl 2.1.27-3
libseccomp 2.5.1-2
libsecret 0.20.4-1
libssh2 1.9.0-3
libtasn1 4.17.0-1
libtirpc 1.3.2-1
libunistring 0.9.10-3
libxcrypt 4.4.25-1
libxml2 2.9.10-9
licenses 20200427-1
linux-api-headers 5.12.3-1
lz4 1:1.9.3-2
mpfr 4.1.0.p13-1
ncurses 6.2-2
nettle 3.7.3-1
npth 1.6-3
openssl 1.1.1.k-1
p11-kit 0.24.0-1
pacman 6.0.0-5
pacman-mirrorlist 20210808-1
pam 1.5.1-1
pambase 20210605-2
pciutils 3.7.0-1
pcre 8.45-1
pcre2 10.37-1
pinentry 1.1.1-1
popt 1.18-1
procps-ng 3.3.17-1
psmisc 23.4-1
readline 8.1.001-1
rustup 1.24.3-1
sed 4.8-1
shadow 4.8.1-4
sqlite 3.36.0-1
systemd 249.3-1
systemd-libs 249.3-1
systemd-sysvcompat 249.3-1
tar 1.34-1
tzdata 2021a-1
util-linux 2.37.1-3
util-linux-libs 2.37.1-3
xz 5.2.5-1
zlib 1:1.2.11-4
zstd 1.5.0-1

I'm going to fix this and publish a new release.

Thanks!

[kpcyrd]

I've fixed this in the 0.1.1 release (and cherry-picked some of your commits), thanks again for reporting!

git-notes is a neat trick that I didn't know about, since I'm using the git commit hash to ensure integrity of all instructions I'd prefer to avoid referring to data that isn't tracked by the commit hash.

The podman/buildah approach is interesting, I've linked to the repo in the readme!

Thanks! :)