Transition to simpler key management

[pjbgf]

Move away from key hierarchy keys, into simpler data slicing instead.

The use of key hierarchy, saving the DEKs alongside the data, is very
effective when there is a need for long-term storage. As Lasso's KEK is
not persisted externally, once the instance restarts the encrypted data
can no longer be used as the DEKs can't be decrypted.

Due to that, a more performant approach would be do away with the KEK, and
keep all the DEKs in memory. The stored data now has an uint32 keyID, as opposed
to having an encrypted DEK and its nonce - saving 40 bytes per row.

As a result, the encryption process had a throughput increase of ~260MB/s
and a 22% decrease in allocations. Conversely, the decryption process had
throughput increase of ~400MB/s and a 50% decrease in allocations.

goos: linux
goarch: amd64
pkg: github.com/rancher/lasso/pkg/cache/sql/encryption
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
                           │ /tmp/before  │             /tmp/after              │
                           │    sec/op    │   sec/op     vs base                │
Encryption/encrypt-1024-16    1.416µ ± 0%   1.013µ ± 1%  -28.46% (p=0.000 n=10)
Encryption/encrypt-4096-16    2.240µ ± 0%   1.853µ ± 0%  -17.32% (p=0.000 n=10)
Encryption/encrypt-8192-16    3.279µ ± 1%   2.844µ ± 0%  -13.28% (p=0.000 n=10)
Decryption/decrypt-1024-16   1044.5n ± 1%   662.8n ± 4%  -36.54% (p=0.000 n=10)
Decryption/decrypt-4096-16    1.910µ ± 1%   1.541µ ± 3%  -19.34% (p=0.000 n=10)
Decryption/decrypt-8192-16    3.101µ ± 3%   2.617µ ± 1%  -15.61% (p=0.000 n=10)
geomean                       2.002µ        1.557µ       -22.21%

                           │ /tmp/before  │              /tmp/after               │
                           │     B/s      │      B/s       vs base                │
Encryption/encrypt-1024-16   689.7Mi ± 0%    964.2Mi ± 1%  +39.79% (p=0.000 n=10)
Encryption/encrypt-4096-16   1.702Gi ± 0%    2.059Gi ± 0%  +20.97% (p=0.000 n=10)
Encryption/encrypt-8192-16   2.327Gi ± 1%    2.683Gi ± 0%  +15.32% (p=0.000 n=10)
Decryption/decrypt-1024-16   934.6Mi ± 1%   1473.4Mi ± 4%  +57.64% (p=0.000 n=10)
Decryption/decrypt-4096-16   1.997Gi ± 1%    2.475Gi ± 3%  +23.97% (p=0.000 n=10)
Decryption/decrypt-8192-16   2.460Gi ± 2%    2.916Gi ± 1%  +18.49% (p=0.000 n=10)
geomean                      1.512Gi         1.944Gi       +28.56%

                           │ /tmp/before  │              /tmp/after              │
                           │     B/op     │     B/op      vs base                │
Encryption/encrypt-1024-16   2.078Ki ± 0%   2.016Ki ± 0%   -3.01% (p=0.000 n=10)
Encryption/encrypt-4096-16   5.703Ki ± 0%   5.641Ki ± 0%   -1.10% (p=0.000 n=10)
Encryption/encrypt-8192-16   10.20Ki ± 0%   10.14Ki ± 0%   -0.61% (p=0.000 n=10)
Decryption/decrypt-1024-16   2.781Ki ± 0%   1.875Ki ± 0%  -32.58% (p=0.000 n=10)
Decryption/decrypt-4096-16   5.781Ki ± 0%   4.875Ki ± 0%  -15.68% (p=0.000 n=10)
Decryption/decrypt-8192-16   9.781Ki ± 0%   8.875Ki ± 0%   -9.27% (p=0.000 n=10)
geomean                      5.166Ki        4.590Ki       -11.16%

                           │ /tmp/before │             /tmp/after             │
                           │  allocs/op  │ allocs/op   vs base                │
Encryption/encrypt-1024-16    9.000 ± 0%   7.000 ± 0%  -22.22% (p=0.000 n=10)
Encryption/encrypt-4096-16    9.000 ± 0%   7.000 ± 0%  -22.22% (p=0.000 n=10)
Encryption/encrypt-8192-16    9.000 ± 0%   7.000 ± 0%  -22.22% (p=0.000 n=10)
Decryption/decrypt-1024-16   12.000 ± 0%   6.000 ± 0%  -50.00% (p=0.000 n=10)
Decryption/decrypt-4096-16   12.000 ± 0%   6.000 ± 0%  -50.00% (p=0.000 n=10)
Decryption/decrypt-8192-16   12.000 ± 0%   6.000 ± 0%  -50.00% (p=0.000 n=10)
geomean                       10.39        6.481       -37.64%

[moio]

Great work @pjbgf , thank you very much for it!

Simpler code, better perf, better test coverage, better security... is there anything more I can really ask for? 😉 really appreciated!

There are a few nitpicks/cosmetic suggestions and only one nontrivial question on pkg/cache/sql/store/store.go - although that's definitely not a deal breaker in any case.

Once that is answered this is ready for my approval and will go to the Frameworks team for another round of review.

[macedogm]

Indeed excellent work, @pjbgf!

Apparently some small updates are still being worked out, so I'll give a general LGTM and avoid approving, as @moio has the fine say.

[pjbgf]

I believe I worked through all the feedback on the new commit. The PR description is now updated with uint32 keyID and the latest benchstat. PTAL

[moio]

All good, green light from my side!

(I took the liberty to directly add a few mostly cosmetic commits on top)

Will merge as soon as we get a 👍 from the frameworks team

Thank you so much for the collaboration

[tomleb]

Overall looks good.

One concern on memory leaks and one missing lock call.

I'm not super confident with the mocks tests (not your fault) since in the past we've had passing unit tests but the code would fail to run because the integration of sql queries & encryptor was missing some parameters in the SQL. (We've already identified we'd like to have more integration tests for this). So, was this change compiled + tested by running Rancher? That'd make me more confident about the changes here.

[moio]

Thanks for the persistence @pjbgf - latest fixes address all my concerns now

FWIW I have also run the integration tests and they are all green rancher/rancher#46015