[backend] kubernetes: fix secret size limitation #29678

k0da · 2021-09-30T15:43:58Z

By now kubernetes backend could hold up to defaultETCDSize gzipped data
(which is 1-1.5Mb). This doesn't scale for larger states.

This commit implements spliting data across multiple secrets bound by
the same Secret labels. This practically removes etcd value size
limitation and allows backend to scale across multiple secrets.

This also takes care of cases when state needs to be shrinked. In such
case code will cleanup unneeded secrets.

Signed-off-by: Dinar Valeev dinar.valeev@absa.africa

Skaronator · 2021-12-07T10:59:44Z

This is basically a similar fix as #28838, but for the kubernetes state file.

hashicorp-cla · 2022-03-12T16:53:26Z

All committers have signed the CLA.

crw · 2022-05-24T00:16:54Z

Thanks for this submission! I will raise this with the internal maintainers of the kubernetes backend. Although I cannot commit to having this PR reviewed at this time, we acknowledge your contribution and appreciate it! Thanks again for the submission!

Bernix01 · 2022-08-30T18:33:56Z

Just making noise to this PR, is this a ready to go?

alexsomesan

It was a bit difficult to comprehend certain parts of this change so I left some spot questions that might hopefully clarify it.

Also, a more general question: is there potential for conflict if more than one terraform workspace is stored in the same K8s namespace?

internal/backend/remote-state/kubernetes/client.go

alexsomesan · 2023-01-27T00:12:38Z

internal/backend/remote-state/kubernetes/client.go

 				},
-			},
+			}
+			secret, err = c.kubernetesSecretClient.Create(ctx, secret, metav1.CreateOptions{})


AFAIK the Create call translates into a POST HTTP method. How does this behave when the Secret for the chunk already exists?

The logic above this checks if the secret already exists or not and only calls create if it does not already exist. We could however reformulate this as a single patch call instead. This is just logic that was already present for the single secret backend, it has just been shuffled around in this PR.

internal/backend/remote-state/kubernetes/client.go

jrhouston · 2023-02-02T04:28:41Z

Also, a more general question: is there potential for conflict if more than one terraform workspace is stored in the same K8s namespace?

The purpose of the secret_suffix attribute here is so you can store multiple terraform states in the same namespace and the secrets will get a different name. You would cause problems if you deliberately used the same prefix across more than one terraform config.

alexsomesan

Thanks for the fixes @jrhouston!
Looks good!

Bobonium · 2023-08-14T09:33:24Z

Was there a specific reason why this was not merged and released after @alexsomesan approved it in february? This feature would still be very much appreciated.

crw · 2023-08-24T21:55:55Z

Looks like it got missed. I will add it to the triage queue. Thanks!

By now kubernetes backend could hold up to defaultETCDSize gzipped data (which is 1-1.5Mb). This doesn't scale for larger states. This commit implements spliting data across multiple secrets bound by the same Secret labels. This practically removes etcd value size limitation and allows backend to scale across multiple secrets. This also takes care of cases when state needs to be shrinked. In such case code will cleanup unneeded secrets Signed-off-by: Dinar Valeev <dinar.valeev@absa.africa>

- Remove unneeded function - Consistently use "chunk" as the unit - Sort the secret response by the index - rename etcdDefaultSize to defaultChunkSize

kmoe · 2023-08-25T16:29:12Z

Thanks for your patience, everyone!

github-actions · 2023-08-25T16:29:31Z

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

sftim · 2023-09-28T10:04:42Z

When the data are gzip encoded, what I'd expect to find is that each Secret holds a valid gzip stream. Concatenating these also produces a gzip stream that you can uncompress and deserialize.

Is that how it works?

sftim · 2023-09-28T10:05:26Z

💭 I wonder whether and how we provide atomic updates when there are multiple Secrets

github-actions · 2023-12-08T02:09:53Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

k0da mentioned this pull request Sep 30, 2021

kubernetes backend size limit #29679

Open

apparentlymart added backend/k8s enhancement labels Sep 30, 2021

kuritka approved these changes Oct 1, 2021

View reviewed changes

k0da force-pushed the kubernetes_multipart branch 2 times, most recently from 9930bf6 to cfa8df9 Compare October 8, 2021 07:01

k0da force-pushed the kubernetes_multipart branch from cfa8df9 to fdd5ecd Compare February 11, 2022 11:50

k0da requested review from jrhouston and alexsomesan as code owners February 11, 2022 11:50

k0da force-pushed the kubernetes_multipart branch from fdd5ecd to 948bb18 Compare July 24, 2022 11:02

vercel bot deployed to Preview July 24, 2022 11:04 View deployment

k0da force-pushed the kubernetes_multipart branch from 948bb18 to 807c5d1 Compare July 24, 2022 11:12

vercel bot deployed to Preview July 24, 2022 11:15 View deployment

alexsomesan reviewed Jan 27, 2023

View reviewed changes

jrhouston force-pushed the kubernetes_multipart branch from bad6c66 to 7f1188e Compare February 2, 2023 04:24

alexsomesan approved these changes Feb 2, 2023

View reviewed changes

k0da and others added 7 commits August 25, 2023 11:54

Refactor

03e9dc9

- Remove unneeded function - Consistently use "chunk" as the unit - Sort the secret response by the index - rename etcdDefaultSize to defaultChunkSize

Add acceptance test to create a very large state

Verified

This commit was signed with the committer’s verified signature.

bflad Brian Flad

GPG key ID: AC40A9A6632E4888

Verified
Learn about vigilant mode

5e8d2c1

Remove documentation warning about secret size limit

Verified

This commit was signed with the committer’s verified signature.

bflad Brian Flad

GPG key ID: AC40A9A6632E4888

Verified
Learn about vigilant mode

a054043

fix broken test

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

ed5ec2c

remove unneeded code

4374692

add missing proxy_url attribute

c3d518e

update k8s.io dependencies to v1.25.5

119a8c0

jrhouston force-pushed the kubernetes_multipart branch from dd84fa3 to 119a8c0 Compare August 25, 2023 16:02

kmoe merged commit 200ff93 into hashicorp:main Aug 25, 2023

dbanck mentioned this pull request Sep 26, 2023

Reflect Terraform 1.6 kubernetes backend changes hashicorp/terraform-schema#259

Closed

greenkiwi mentioned this pull request Oct 6, 2023

kubernets backend has error when secret_suffix ends with -<number> #33995

Closed

github-actions bot locked as resolved and limited conversation to collaborators Dec 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[backend] kubernetes: fix secret size limitation #29678

[backend] kubernetes: fix secret size limitation #29678

k0da commented Sep 30, 2021 •

edited

Loading

Skaronator commented Dec 7, 2021

hashicorp-cla commented Mar 12, 2022 •

edited

Loading

crw commented May 24, 2022

Bernix01 commented Aug 30, 2022

alexsomesan left a comment

alexsomesan Jan 27, 2023

jrhouston Feb 2, 2023 •

edited

Loading

jrhouston commented Feb 2, 2023 •

edited

Loading

alexsomesan left a comment •

edited

Loading

Bobonium commented Aug 14, 2023

crw commented Aug 24, 2023

kmoe commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

sftim commented Sep 28, 2023

sftim commented Sep 28, 2023

github-actions bot commented Dec 8, 2023

[backend] kubernetes: fix secret size limitation #29678

[backend] kubernetes: fix secret size limitation #29678

Conversation

k0da commented Sep 30, 2021 • edited Loading

Skaronator commented Dec 7, 2021

hashicorp-cla commented Mar 12, 2022 • edited Loading

crw commented May 24, 2022

Bernix01 commented Aug 30, 2022

alexsomesan left a comment

Choose a reason for hiding this comment

alexsomesan Jan 27, 2023

Choose a reason for hiding this comment

jrhouston Feb 2, 2023 • edited Loading

Choose a reason for hiding this comment

jrhouston commented Feb 2, 2023 • edited Loading

alexsomesan left a comment • edited Loading

Choose a reason for hiding this comment

Bobonium commented Aug 14, 2023

crw commented Aug 24, 2023

kmoe commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

sftim commented Sep 28, 2023

sftim commented Sep 28, 2023

github-actions bot commented Dec 8, 2023

k0da commented Sep 30, 2021 •

edited

Loading

hashicorp-cla commented Mar 12, 2022 •

edited

Loading

jrhouston Feb 2, 2023 •

edited

Loading

jrhouston commented Feb 2, 2023 •

edited

Loading

alexsomesan left a comment •

edited

Loading