Fails to start after update

[davidak]

I updated my system to NixOS 22.05.4270.b68a6a27adb and trustix from 32ee78a to c852d3c.

After the nixos-rebuild switch, trustix-nix-cache.service and trustix.service do not start.

warning: the following units failed: trustix-nix-cache.service, trustix.service

Here, for some reason, it can't read the private key anymore.

Dec 03 06:58:03 gaming systemd[1]: Stopping Trustix daemon...
Dec 03 06:58:23 gaming trustix[1106]: time="2022-12-03T06:58:23+01:00" level=error msg="Could not update STH" error="rpc error: code = Unavailable desc = timed out waiting for server handshake" logI>
Dec 03 06:58:23 gaming systemd[1]: trustix.service: Deactivated successfully.
Dec 03 06:58:23 gaming systemd[1]: Stopped Trustix daemon.
Dec 03 06:58:23 gaming systemd[1]: trustix.service: Consumed 1d 47min 58.636s CPU time, no IP traffic.
Dec 03 06:58:25 gaming systemd[1]: Started Trustix daemon.
Dec 03 06:58:25 gaming trustix[2956960]: time="2022-12-03T06:58:25+01:00" level=info msg="Creating state directory" directory=.
Dec 03 06:58:25 gaming trustix[2956960]: time="2022-12-03T06:58:25+01:00" level=info msg="Creating signer" name=davidak type=ed25519
Dec 03 06:58:25 gaming trustix[2956960]: Error: open /var/trustix/keys/private: permission denied
Dec 03 06:58:25 gaming trustix[2956960]: Usage:
Dec 03 06:58:25 gaming trustix[2956960]:   trustix daemon [flags]
Dec 03 06:58:25 gaming trustix[2956960]: Flags:
Dec 03 06:58:25 gaming trustix[2956960]:       --config string    Path to config.toml/json
Dec 03 06:58:25 gaming trustix[2956960]:   -h, --help             help for daemon
Dec 03 06:58:25 gaming trustix[2956960]:       --interval float   Log poll interval in seconds (default 1800)
Dec 03 06:58:25 gaming trustix[2956960]:       --listen strings   Listen to address
Dec 03 06:58:25 gaming trustix[2956960]:       --state string     State directory (default ".local/share/trustix")
Dec 03 06:58:25 gaming trustix[2956960]: Global Flags:
Dec 03 06:58:25 gaming trustix[2956960]:       --address string   Connect to address (default "unix:///tmp/trustix.sock")
Dec 03 06:58:25 gaming trustix[2956960]:       --log-id string    Log ID
Dec 03 06:58:25 gaming trustix[2956960]:       --timeout int      Timeout in seconds (default 20)
Dec 03 06:58:25 gaming trustix[2956960]: open /var/trustix/keys/private: permission denied
Dec 03 06:58:25 gaming systemd[1]: trustix.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 06:58:25 gaming systemd[1]: trustix.service: Failed with result 'exit-code'.

Maybe this crashes, because the socket does not exist:

Dec 03 06:58:25 gaming systemd[1]: trustix-nix-cache.service: Failed with result 'exit-code'.
Dec 03 06:58:26 gaming systemd[1]: Started Trustix Nix binary cache daemon.
Dec 03 06:58:26 gaming trustix-nix[2957284]: time="2022-12-03T06:58:26+01:00" level=debug msg="Creating client for remote" address="unix:///run/trustix-daemon.socket"
Dec 03 06:58:26 gaming trustix-nix[2957284]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 03 06:58:26 gaming trustix-nix[2957284]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7b55be]
Dec 03 06:58:26 gaming trustix-nix[2957284]: goroutine 1 [running]:
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/bufbuild/connect-go.newClientConfig({0xc000140100?, 0xc000140100?}, {0xc000197ab0, 0x1, 0x90d240?})
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/bufbuild/connect-go/client.go:198 +0x1de
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/bufbuild/connect-go.NewClient[...]({0xad43a0, 0xc000126570?}, {0xc000140100, 0x3f}, {0xc000197ab0, 0x1, 0x1})
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/bufbuild/connect-go/client.go:41 +0xa9
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/nix-community/trustix/packages/trustix-proto/api/apiconnect.NewLogAPIClient({0xad43a0, 0xc000126570}, {0x7fff73150b72?, 0x7?}, {0xc000197ab0, >
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/nix-community/trustix/packages/trustix-proto/api/apiconnect/api.connect.go:140 +0xcd
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/nix-community/trustix/packages/trustix/client.newLogAPIConnectClient(...)
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/nix-community/trustix/packages/trustix/client/connect_logapi.go:23
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/nix-community/trustix/packages/trustix/client.CreateClient({0x7fff73150b72, 0x21}, {0xc000197ab0, 0x1, 0x1})
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/nix-community/trustix/packages/trustix/client/conn.go:35 +0x1d5
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/nix-community/trustix/packages/trustix-nix/cmd.glob..func1(0xe4d5a0?, {0x9f4c1f?, 0x6?, 0x6?})
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/nix-community/trustix/packages/trustix-nix/cmd/binary-cache-proxy.go:155 +0x205
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/spf13/cobra.(*Command).execute(0xe4d5a0, {0xc00010e120, 0x6, 0x6})
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/spf13/cobra/command.go:872 +0x694
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/spf13/cobra.(*Command).ExecuteC(0xe4daa0)
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/spf13/cobra/command.go:990 +0x3bd
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/spf13/cobra.(*Command).Execute(...)
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/spf13/cobra/command.go:918
Dec 03 06:58:26 gaming trustix-nix[2957284]: github.com/nix-community/trustix/packages/trustix-nix/cmd.Execute()
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/nix-community/trustix/packages/trustix-nix/cmd/root.go:57 +0x45
Dec 03 06:58:26 gaming trustix-nix[2957284]: main.main()
Dec 03 06:58:26 gaming trustix-nix[2957284]:         github.com/nix-community/trustix/packages/trustix-nix/main.go:11 +0x17
Dec 03 06:58:26 gaming systemd[1]: trustix-nix-cache.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 03 06:58:26 gaming systemd[1]: trustix-nix-cache.service: Failed with result 'exit-code'.

I debug it further...

[davidak]

The private key is now owned by a user and group that does not exist:

[root@gaming:~]# ll /var/trustix/keys/private
-rw------- 1 997 991 88 May  7  2022 /var/trustix/keys/private

[davidak]

I think this problem was introduced with dc9a4ec.

Changing ownership to root does not help.

The service might also not have access to /var/trustix which is provided as an example here https://nix-community.github.io/trustix/howto-nix/binarycache.html.

Changing the location of the files fixes it!

mv /var/trustix/keys /var/lib/trustix/

And updating config.

  services.trustix = {
    enable = true;

    signers.davidak = {
      type = "ed25519";
      ed25519.private-key-path = "/var/lib/trustix/keys/private";
    };

...

[davidak]

Now trustix-nix-cache.service has a different error:

Dec 03 08:20:39 gaming systemd[1]: trustix-nix-cache.service: Failed with result 'exit-code'.
Dec 03 08:38:46 gaming systemd[1]: Started Trustix Nix binary cache daemon.
Dec 03 08:38:46 gaming trustix-nix[3002147]: panic: open /var/lib/trustix/keys/cache-private-key.pem: permission denied
Dec 03 08:38:46 gaming trustix-nix[3002147]: goroutine 1 [running]:
Dec 03 08:38:46 gaming trustix-nix[3002147]: github.com/tweag/trustix/packages/trustix-nix/cmd.readKey({0x7ffdbd88bbb6, 0x1})
Dec 03 08:38:46 gaming trustix-nix[3002147]:         /build/source/cmd/binary-cache-proxy.go:70 +0x26d
Dec 03 08:38:46 gaming trustix-nix[3002147]: github.com/tweag/trustix/packages/trustix-nix/cmd.glob..func1(0xe4ff40, {0x9c58c8, 0x6, 0x6})
Dec 03 08:38:46 gaming trustix-nix[3002147]:         /build/source/cmd/binary-cache-proxy.go:101 +0x85
Dec 03 08:38:46 gaming trustix-nix[3002147]: github.com/spf13/cobra.(*Command).execute(0xe4ff40, {0xc0001348a0, 0x6, 0x6})
Dec 03 08:38:46 gaming trustix-nix[3002147]:         /build/source/vendor/github.com/spf13/cobra/command.go:850 +0x60e
Dec 03 08:38:46 gaming trustix-nix[3002147]: github.com/spf13/cobra.(*Command).ExecuteC(0xe50480)
Dec 03 08:38:46 gaming trustix-nix[3002147]:         /build/source/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
Dec 03 08:38:46 gaming trustix-nix[3002147]: github.com/spf13/cobra.(*Command).Execute(...)
Dec 03 08:38:46 gaming trustix-nix[3002147]:         /build/source/vendor/github.com/spf13/cobra/command.go:895
Dec 03 08:38:46 gaming trustix-nix[3002147]: github.com/tweag/trustix/packages/trustix-nix/cmd.Execute()
Dec 03 08:38:46 gaming trustix-nix[3002147]:         /build/source/cmd/root.go:61 +0x45
Dec 03 08:38:46 gaming trustix-nix[3002147]: main.main()
Dec 03 08:38:46 gaming trustix-nix[3002147]:         /build/source/main.go:14 +0x17
Dec 03 08:38:46 gaming systemd[1]: trustix-nix-cache.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 03 08:38:46 gaming systemd[1]: trustix-nix-cache.service: Failed with result 'exit-code'.

But we know how to fix it!

Add StateDirectory to trustix-nix-cache.

[root@gaming:~]# ll /var/lib/trustix-nix-cache
lrwxrwxrwx 1 root root 25 Dec  3 09:17 /var/lib/trustix-nix-cache -> private/trustix-nix-cache

[root@gaming:~]# ll /var/lib/trustix-nix-cache/
total 0

[root@gaming:~]# mkdir /var/lib/trustix-nix-cache/keys

[root@gaming:~]# mv /var/lib/trustix/keys/cache-* /var/lib/trustix-nix-cache/keys/

[root@gaming:~]# ll /var/lib/trustix-nix-cache/keys/
total 8
-rw------- 1 62583 62583 112 May  8  2022 cache-private-key.pem
-rw------- 1 62583 62583  68 May  8  2022 cache-public-key.pem

[root@gaming:~]# chown root:root /var/lib/trustix-nix-cache/keys/*

[root@gaming:~]# ll /var/lib/trustix-nix-cache/keys/
total 8
-rw------- 1 root root 112 May  8  2022 cache-private-key.pem
-rw------- 1 root root  68 May  8  2022 cache-public-key.pem

It does not work!

Even at panic: open /root/cache-private-key.pem: permission denied without StateDirectory.

Also /var/trustix-nix-cache/keys/cache-private-key.pem and /var/trustix-nix-cache/cache-private-key.pem does not work.

[davidak]

The state is now:

trustix.service runs, but the hook crash:

[root@gaming:~]# nix-build -E '(import <nixpkgs> {}).writeText "hello" "Hello World!"' --no-out-link --check
checking outputs of '/nix/store/fmvm1z6l9k78s6xcgbj8qkrhkbiz5ar4-hello.drv'...
running post-build-hook '/nix/store/7nzwfm3plmajjyvjvvry6vl2gyz3kvhd-trustix-hook'...
post-build-hook: time="2022-12-03T13:20:09+01:00" level=debug msg="Submitting mapping" storePath=/nix/store/cswi7m0frxljix1l6ixsjdryg5z75i3y-hello
post-build-hook: time="2022-12-03T13:20:09+01:00" level=debug msg="Creating client for remote" address="unix:///run/trustix-daemon.socket"
post-build-hook: panic: runtime error: invalid memory address or nil pointer dereference
post-build-hook: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7b55be]
post-build-hook: 
post-build-hook: goroutine 1 [running]:
post-build-hook: github.com/bufbuild/connect-go.newClientConfig({0xc0002b4200?, 0xc0002b4200?}, {0xc0001a9d68, 0x1, 0x90d240?})
post-build-hook:        github.com/bufbuild/connect-go/client.go:198 +0x1de
post-build-hook: github.com/bufbuild/connect-go.NewClient[...]({0xad43a0, 0xc00029aa20?}, {0xc0002b4200, 0x3f}, {0xc0001a9d68, 0x1, 0x1})
post-build-hook:        github.com/bufbuild/connect-go/client.go:41 +0xa9
post-build-hook: github.com/nix-community/trustix/packages/trustix-proto/api/apiconnect.NewLogAPIClient({0xad43a0, 0xc00029aa20}, {0x7ffd10f965b1?, 0x7?}, {0xc0001a9d68, 0x1, 0x1})
post-build-hook:        github.com/nix-community/trustix/packages/trustix-proto/api/apiconnect/api.connect.go:140 +0xcd
post-build-hook: github.com/nix-community/trustix/packages/trustix/client.newLogAPIConnectClient(...)
post-build-hook:        github.com/nix-community/trustix/packages/trustix/client/connect_logapi.go:23
post-build-hook: github.com/nix-community/trustix/packages/trustix/client.CreateClient({0x7ffd10f965b1, 0x21}, {0xc0001a9d68, 0x1, 0x1})
post-build-hook:        github.com/nix-community/trustix/packages/trustix/client/conn.go:35 +0x1d5
post-build-hook: github.com/nix-community/trustix/packages/trustix-nix/cmd.glob..func2(0xe4d820?, {0x9f4c1f?, 0x4?, 0x4?})
post-build-hook:        github.com/nix-community/trustix/packages/trustix-nix/cmd/post-build-hook.go:87 +0x35d
post-build-hook: github.com/spf13/cobra.(*Command).execute(0xe4d820, {0xc0002c20c0, 0x4, 0x4})
post-build-hook:        github.com/spf13/cobra/command.go:872 +0x694
post-build-hook: github.com/spf13/cobra.(*Command).ExecuteC(0xe4daa0)
post-build-hook:        github.com/spf13/cobra/command.go:990 +0x3bd
post-build-hook: github.com/spf13/cobra.(*Command).Execute(...)
post-build-hook:        github.com/spf13/cobra/command.go:918
post-build-hook: github.com/nix-community/trustix/packages/trustix-nix/cmd.Execute()
post-build-hook:        github.com/nix-community/trustix/packages/trustix-nix/cmd/root.go:57 +0x45
post-build-hook: main.main()
post-build-hook:        github.com/nix-community/trustix/packages/trustix-nix/main.go:11 +0x17
error: program '/nix/store/7nzwfm3plmajjyvjvvry6vl2gyz3kvhd-trustix-hook' failed with exit code 2

So i think it's just broken at the latest commit.

[davidak]

Let's see where this issue was introduced.

Commit's and errors:

28e221f works (i fixed it)
8c934e2 one commit later, it's broken

[root@gaming:~]# nix-build -E '(import <nixpkgs> {}).writeText "hello" "Hello World!"' --no-out-link --check
checking outputs of '/nix/store/fmvm1z6l9k78s6xcgbj8qkrhkbiz5ar4-hello.drv'...
running post-build-hook '/nix/store/aqickhvgszqg6zh2505cr5z9kipfq7dd-trustix-hook'...
post-build-hook: time="2022-12-04T17:01:56+01:00" level=debug msg="Submitting mapping" storePath=/nix/store/cswi7m0frxljix1l6ixsjdryg5z75i3y-hello
post-build-hook: time="2022-12-04T17:01:56+01:00" level=debug msg="Dialing remote" address="unix:///run/trustix-daemon.socket"
post-build-hook: time="2022-12-04T17:01:56+01:00" level=fatal msg="could not submit: rpc error: code = Unknown desc = Denied peer creds"
error: program '/nix/store/aqickhvgszqg6zh2505cr5z9kipfq7dd-trustix-hook' failed with exit code 1

5577950 same error
ad6617f error: getting status of '/home/davidak/code/trustix/nix/overlays.nix': No such file or directory
d01f602 same
e5fe848 same
ad69674 same
5af3da0 same
b3b86d1 same
d00a43d same
399759d same
d4b4109 same
207627c same
576953a error: getting status of '/home/davidak/code/trustix/packages/trustix-nix-r13y/nixos': No such file or directory
e688127 nixos-rebuild works, trustix compiles, trustix runs, but hook crashes: error: program '/nix/store/npgr9cb93xm55xf4m4lgw8ky246kx1g4-trustix-hook' failed with exit code 2

So this project is broken since august 2022.

[davidak]

To fix this issue, it needs to be clarified where the keys need to be stored and which permissions they need.

For trustix, it works for me to have them in

/var/lib/trustix/keys/private
/var/lib/trustix/keys/public

I haven't figured it out for trustix-nix-cache (#36).

The solution needs to be added to the documentation.