[WIP] Ephemeral Values prototype #35078

apparentlymart · 2024-04-24T19:35:26Z

This is another attempt at introducing to Terraform the idea of objects and values being "ephemeral", which means something like "lives only for the duration of one Terraform phase".

Terraform already has at least two concepts that meet this definition, despite us not previously naming it:

Provider configurations (provider blocks): Terraform re-evaluates the arguments in a provider block separately during the plan and apply phases, and doesn't mind if the configuration is different between the two as long as the apply-time configuration allows performing the actions that were proposed during the plan phase.
Provisioners (provisioner and connection blocks): Terraform fully evaluates these only during the apply phase, so they aren't really considered during the plan phase at all, aside from basic static validation.

However, because the idea of "ephemeral" is not available in the rest of the language, it's tough to actually benefit from this ephemerality. This prototype aims to introduce "ephemeral" as a cross-cutting concern supported broadly across the language.

Ephemeral Values

The most fundamental idea is that values used in expressions can either be ephemeral or non-ephemeral. This is an idea similar to "sensitive" in that Terraform will perform dynamic analysis such that any value derived from an ephemeral value is itself ephemeral. Ephemeral values can then be used only in parts of the language which would not require persisting the value either between the plan phase and the apply phase, or from one plan/apply round to the next.

Considering only pre-existing language features, ephemeral values can be freely used in provider blocks, provisioner blocks, connection blocks, and in local values. The following sections describe some new additions that either accept or produce ephemeral values.

resource blocks (aside from special nested parts like the aforementioned provisioner blocks) do not accept ephemeral values, because preserving resource configuration unchanged between the plan and apply phases is a fundamental part of how Terraform works to keep its promise of either doing what the plan described or returning an error explaining why that's not possible.

Because ephemeral values are not expected to persist from plan to apply or between plan/apply rounds, there is no need to save them in saved plan files or state snapshots, thus finally giving a plausible answer for what to do about #516, which has been on my mind since long before I worked at HashiCorp.

Ephemeral Input Variables

An ephemeral input variable is, in the most general terms, just an input variable that is declared as accepting ephemeral values. A non-ephemeral input variable cannot accept ephemeral values, while an ephemeral value will accept both ephemeral and non-ephemeral values but the value will always be treated as ephemeral when used inside the declaring module.

The main interesting case is when a root module declares an ephemeral input variable. In that case, Terraform will no longer remember the value for the variable provided during planning and will instead expect any ephemeral variable set during the plan step to be provided again -- possibly with a different value -- during the apply step.

The primary goal of this is to be able to use input variables to set arguments in ephemeral contexts. For example, an input variable that's both ephemeral and sensitive could provide a JSON Web Token to be used when configuring a specific provider, and then automation around Terraform could provide separate JSON Web Tokens across the plan and apply phases so that the apply phase isn't subject to the expiration time for the plan-time JWT, and so that the plan-time JWT doesn't get persisted to disk as part of a saved plan.

Ephemeral Output Values

An ephemeral output value is essentially the opposite of an ephemeral input variable, allowing a module to expose an ephemeral value to its caller. As with input variables, a non-ephemeral output value will reject having an ephemeral value assigned to it. An ephemeral output value can have both ephemeral and non-ephemeral values assigned to it, but the calling module will always see it as ephemeral.

To start the utility of this is limited just to echoing back values derived from ephemeral input variables, since nothing else I've described so far actually produces ephemeral values. However, allowing this is important to ensure that ephemeral values are supported symmetrically and will cooperate well with all other language features.

Ephemeral Resources

The final idea in this prototype -- one which this prototype probably won't explore fully just yet, and introduce only just enough to validate that it fits in well with everything else -- is a new resource mode for representing remote objects that are ephemeral themselves.

Terraform currently has two "resource modes": managed resources (resource blocks) describe objects that Terraform is directly managing, while data resources (data blocks) describe objects that are managed elsewhere that the current configuration depends on. But in both cases the assumption is that those objects persist in some sense from plan to apply and from one plan/apply round to the next, and that Terraform is supposed to detect and react to any changes to those objects and therefore needs to persist information about them itself.

Ephemeral resources, (ephemeral blocks) on the other hand, represent objects that -- at least, as far as Terraform is concerned -- exist only briefly during a single Terraform phase, and then get cleaned up once the phase is complete. This idea is an evolution of some much earlier design work I did before I even worked at HashiCorp 😀 in relation to #8367, which was about establishing temporary SSH tunnels, and the HashiCorp Vault provider I wrote in #9158 (which evolved into today's official hashicorp/vault).

The general idea of ephemeral resources, then, is that their lifecycle includes three events:

OpenEphemeral: Prepares the object for use. For some kinds of objects this would represent a "create" action, but for others it might just open a temporary session to something that already exists, such as in the SSH tunnel use-case.

This operation is the one that establishes the result attributes that can be accessed from other parts of the module where the resource is declared. All of these results would be ephemeral values, so that they can vary from plan to apply. For example, opening an SSH tunnel is likely to cause a different local TCP port number to be allocated each time, and so consistency between plan and apply phases is not expected.
RenewEphemeral: Some ephemeral remote objects need to be periodically refreshed in order to stay "live", such as leases for Vault secrets.

This optional operation is therefore opted into by the provider's OpenEphemeral response, by providing a private set of data that should be sent back to the provider's RenewEphemeral implementation and a deadline before which Terraform must renew it. The provider can then do whatever is needed to keep the object from expiring, and optionally return another renew request with a new deadline in order to repeat this renewal process.
CloseEphemeral: Once Terraform has completed work for all objects that refer to the ephemeral resource, this operation gives the provider an explicit signal that the object is not longer required so that it can be promptly destroyed or invalidated.

This detail is particularly helpful for the Vault provider and fixes a limitation I ran into immediately back in 2016: a dynamic secret fetched using a data block can never have its lease explicitly terminated, because data resources were intended only to read information about an object someone else is managing, not to directly manage an object (a Vault lease).

Because the results from ephemeral resources are ephemeral values, they're primarily useful in configuration for other ephemeral objects: provider blocks, provisioner/connection blocks, and of course other ephemeral blocks.

Actually changing the provider protocol and implementing real providers is not in scope for my initial prototyping work here, and so I intend to prototype this in a more limited way that just emulates how this mechanism might behave, so we can see how well it interacts with the rest of the language and the other ephemeral values discussed here.

I've also been considering a mechanism to allow managed resource types to declare individual arguments as being "write-only", such as for an RDS database password that only needs to be provided during creation and should not be provided again unless the operator actually intends to reset it. I don't intend to prototype that in here, but I intend to lay the foundations for it by having a convention that ephemeral input values and write-only arguments both treat null as meaning "don't set or change" and non-null as "set or change", thereby creating a small imperative-shaped niche in the otherwise-declarative Terraform Language to allow for using Terraform to manage objects that have write-only (typically, sensitive) arguments without needing to persist them in plan and state.

I'm still working on this, so not everything described above is in here yet, but the foundations for ephemeral values themselves are already in. I've opened this draft largely just because I need to put this work down for a while for a team offsite and don't want to lose the context.

This isn't actually used anywhere yet. We'll introduce uses of it in subsequent commits.

This doesn't actually do anything yet, but in future commits it will enable some new language features for marking input variables and output values as "ephemeral", meaning that they don't get saved as part of state snapshots or saved plan files.

When the ephemeral_values experiment is active, a module author can designate individual input variables and output values as being "ephemeral", which doesn't currently do anything but in future commits will represent that the values can be used only in locations that don't require Terraform to persist the value as part of state snapshots or saved plan files.

This is a checkpoint commit on the path to supporting ephemeral values as a cross-cutting concern in the Terraform language. An ephemeral value is one that lives only in RAM during a single phase and so cannot persist from the plan phase to the apply phase, or between consecutive plan/apply rounds. Terraform tracks whether each value is ephemeral using the cty "marks" concept, thus achieving the same dynamic analysis as we already employ for the concept of "sensitive values" that prevents displaying a value in the user interface. This commit is just a starting point which gets some of the basics into place: - input variables and output values can both be statically declared as being ephemeral. Only ephemeral inputs/outputs can have ephemeral values assigned to them, and the recipient of the value sees it as ephemeral even if the assigned value was not already ephemeral. This creates a dynamic analysis cutoff point at module boundaries so that it's possible to determine in isolation whether a single module is using ephemeral values correctly, without having to test it in every possible calling context. - Managed and data resources cannot have ephemeral values assigned into their configurations because Terraform and providers both expect the resource attributes to persist between phases. - Ephemeral values _can_ be used in provider and provisioner configurations, because both of those effectively meet the definition of the new "ephemeral" concept despite it not previously having a name. - Ephemeral markings propagate through all of the built-in language features for dynamic analysis purposes, largely relying on cty's efforts to do that in similar vein as for sensitive marks. In particular, it's possible to define an ephemeral value whose expression produces an ephemeral result, and passing ephemeral values to functions should propagate the ephemeral mark to the results when appropriate. (I've not yet thoroughly reviewed all of the built-in functions for correct marks handling though, so there may be some gaps to address in later commits.) The next step for this work will be to change the modules runtime to have support for a workflow involving ephemeral _root_ input variables, where their values must be re-supplied during the apply phase. That will then achieve (in experimental capacity) the first goal of ephemeral values: to be able to provide non-persistent settings such as time-limited API tokens to use in provider configuration blocks.

We'll use this to track what subset of the ephemeral input variables were set in the planning options, and which ones must therefore be re-supplied in the apply phase options. This new plan field is not yet populated anywhere. The uses of this will follow in subsequent commits.

The new concept of "ephemeral input variables" creates the possibility of needing to provide an input variable value during the apply phase, rather than just retaining it in the plan. Now we'll remember in the plan the names of the variables that need to be re-supplied during apply -- that is, any ephemeral values whose plan-time values were non-null -- and then check at the start of the apply phase whether those variables (and _only_ those variables) were provided in the planning options. This doesn't yet update Terraform CLI to actually populate this stuff, so as of this commit any plan with apply-time variables is effectively unapplyable. We'll deal with that in future commits.

When the ephemeral variables experiment is active we can potentially have input variables whose values need to be provided separately in both the plan and apply phases, as a compromise to avoid writing those values as part of a saved plan file and to allow the given value to vary between the two phases if necessary. The CLI layer must therefore re-process the given input variable values during the apply phase whenever this experiment is active for the root module and the plan recorded at least one apply-time variable name. To reduce the risk of this new logic accidentally impacting non-experimental usage, the whole call is guarded by whether the root module is participating in the experiment. Checking just the root module is sufficient here because only the root input variables are directly handled by the CLI layer; input variables for descendent modules are handled entirely within the modules runtime.

This is the new resource mode for ephemeral resources.

Ephemeral resources, declared using "ephemeral" blocks, represent objects that are instantiated only for the duration of a single Terraform phase, and are intended for uses such as temporary network tunnels or time-limited leases of sensitive values from stores such as HashiCorp Vault.

Similar to terraform_data, this is really just here to use as a placeholder when one needs an ephemeral resource for some reason but doesn't need any specific one. This might get removed before the ephemeral_values experiment gets stabilized. For now it's here to use as an initial testing vehicle since we don't have any mechanism for offering experimental features in the provider plugin protocol, whereas this provider is not a plugin.

This change is not shippable as-is because it changes the interpretation of any reference starting with "ephemeral.", which would previously have referred to a managed resource type belonging to a provider whose local name is "ephemeral". Therefore this initial attempt is only for prototyping purposes and would need to be modified in some way in order to be shippable. It will presumably need some sort of opt-in within the calling module so that the old interpretation can be preserved by default.

We don't yet do anything useful when we get there, but we do at least fail in a vaguely-graceful way.

For now these graph nodes don't actually do anything, but the graph shape is at least plausible for what we'll need.

We now need to clean up any straggling ephemeral resource instances before we complete each graph walk, and ephemeral resource instances are ultimately owned by the graph walker, so the graph walker now has a Close method that's responsible for cleaning up anything that the walker owns which needs to be explicitly closed at the end of a walk.

Because ephemeralResourceCloseTransformer runs very late in the transform sequence, it's too late to get provider open and close nodes associated with it automatically. We don't actually need to worry about the provider _open_ dependency because our close node always depends on all of our open nodes and they will in turn depend on the provider open they need. But for close we need to delay closing the provider until all of the associated ephemeral resources have been closed, so we need to do a little fixup: If any of particular ephemeral resource's open nodes have provider close nodes depending on them, those provider close nodes should also depend on the ephemeral resource close node. That then describes that the provider should remain open for as long as at least one ephemeral resource instance owned by that provider remains live, which makes it okay for us to do our periodic background renew requests and our final close requests.

…alysis Previously we had a special interface graphNodeEphemeralResourceConsumer and a helper for implementing it in terms of GraphNodeReferencer, but for the moment we'll just use GraphNodeReferencer directly with that helper because that gives us broad coverage across many node types without having to make such sprawling changes just to support a prototype. The separated interface design might return later if we discover a need for a node to report that it uses an ephemeral resource without actually including any expression references for it, but we'll wait to see if that additional complexity is actually needed.

Ephemeral resources work quite differently than managed or data resources in that their instances live only in memory and are never persisted, and in that we need to handle the possibility of the object having become invalid by the time we're evaluating a reference expression. Since we're just prototyping ephemeral resources for now, this works as a totally separate codepath in the evaluator. The resource reference handling in the evaluator is long overdue for being reworked so that it doesn't depend so directly on the implementation details of how we keep track of resources, and the new ephemeral codepath is perhaps a simplified example of what that might look like in future, but for now it's used only for ephemeral resources to limit the invasiveness of this prototype.

If a saved plan is being applied using a Terraform CLI/Core build that allows use of language experiments then that should be allowed when the configuration is being loaded from the snapshot saved in a plan file, in the same way as it would've been handled when preparing to generate that plan file. The local backend doesn't get told directly whether it should allow experiments, and instead its operation comes with a pre-configured configuration loader that may have the experiment flag set internally. When we load from snapshot we can't use the operation's own config loader because that loads from the real filesystem, and so we'll copy over the experiments-allowed flag from the operation's config loader to achieve consistency while still loading the configuration from the snapshot in the plan file. Without this a plan successfully created with experiments would fail with confusing errors during apply, because the snapshot config loader would not accept the same experiments that the plan-time config loader did.

I'm honestly not really sure yet how to explain _why_ ephemeral resource nodes are getting pruned when they shouldn't; for the sake of prototyping this is just a hard-coded special exception to just not consider them at all in the pruneUnusedNodesTransformer. The later ephemeralResourceCloseTransformer has its own logic for deciding that an ephemeral resource isn't actually needed in the current graph and pruning both their open and close nodes, so these will still get pruned but it will happen in different circumstances and based on a later form of the graph with more nodes and edges already present, thus preventing some cases of ephemeral resources being pruned when they shouldn't be.

The modules runtime should always use a different strategy to keep track of live ephemeral resource instances, and should never persist them in the plan or state. These checks are here just to reduce the risk that a bug in the modules runtime could inadvertently result in an ephemeral resource instance being persisted. This is a bit of a "defense-in-depth" strategy, because the state and plan types all have most of their fields exported and so we can't be sure that all modifications will go through the mutation methods.

This is just enough to skip writing and reading ephemeral resources and their instances in the plan and state, so that we can reach the code that manages them in their own separate data structure. This relies on the new idea of some resource modes not being persisted between rounds and not being persisted from plan to apply, although for now EphemeralResourceMode is the only mode that doesn't do both of those things.

To support ephemeral values we need a more complicated set of rules about what input variables can and must be set when applying a saved plan. The command itself does not have enough information to implement those rules itself, so we'll let them pass through and check this in the local backend's apply phase instead. The local backend's apply phase already has basic support for dealing with apply-time variable values, so this just removes the blocker that was preventing values from reaching that logic.

We need to remember which ephemeral values were set during planning so that we can require them to be set again (possibly to a different value) during the apply step.

apparentlymart added enhancement config labels Apr 24, 2024

apparentlymart self-assigned this Apr 24, 2024

apparentlymart mentioned this pull request Apr 24, 2024

[WIP] Ephemeral Values prototype #35077

Closed

vercel bot deployed to Preview April 24, 2024 19:36 View deployment

apparentlymart added 7 commits April 29, 2024 11:11

lang: A new mark type for ephemeral values

034baa2

This isn't actually used anywhere yet. We'll introduce uses of it in subsequent commits.

apparentlymart force-pushed the f-ephemeral-values branch from f5841fc to 2ce8606 Compare April 29, 2024 18:52

apparentlymart mentioned this pull request Apr 29, 2024

A way to refresh provider credentials #29182

Open

vercel bot deployed to Preview April 29, 2024 18:57 View deployment

providers: New Interface methods for ephemeral resource types

5c7deeb

apparentlymart force-pushed the f-ephemeral-values branch from 2ce8606 to 5c7deeb Compare April 29, 2024 18:58

vercel bot deployed to Preview April 29, 2024 19:03 View deployment

addrs: EphemeralResourceMode

60365f6

This is the new resource mode for ephemeral resources.

vercel bot deployed to Preview April 29, 2024 21:18 View deployment

vercel bot deployed to Preview April 29, 2024 22:54 View deployment

vercel bot deployed to Preview April 29, 2024 23:20 View deployment

vercel bot deployed to Preview April 30, 2024 16:47 View deployment

apparentlymart force-pushed the f-ephemeral-values branch from be64383 to aaa345f Compare April 30, 2024 18:09

vercel bot deployed to Preview April 30, 2024 18:13 View deployment

vercel bot deployed to Preview April 30, 2024 18:30 View deployment

vercel bot deployed to Preview May 1, 2024 00:05 View deployment

apparentlymart added 3 commits May 1, 2024 15:26

apparentlymart added 2 commits May 1, 2024 15:26

terraform: Don't panic when visiting ephemeral resource nodes

f186d9a

We don't yet do anything useful when we get there, but we do at least fail in a vaguely-graceful way.

terraform: Graph nodes for closing ephemeral resource instances

2ab8071

For now these graph nodes don't actually do anything, but the graph shape is at least plausible for what we'll need.

apparentlymart force-pushed the f-ephemeral-values branch from 25072c8 to 2ab8071 Compare May 1, 2024 22:35

vercel bot deployed to Preview May 1, 2024 22:39 View deployment

vercel bot deployed to Preview May 2, 2024 00:05 View deployment

resources/ephemeral: A place to track ephemeral resource instances

59ea75b

apparentlymart force-pushed the f-ephemeral-values branch from 95e1621 to 59ea75b Compare May 3, 2024 19:00

vercel bot deployed to Preview May 3, 2024 19:04 View deployment

apparentlymart added 2 commits May 3, 2024 14:29

terraform: Plumb in an ephemeral.Resources object for each graph walk

b586811

vercel bot deployed to Preview May 3, 2024 21:49 View deployment

vercel bot deployed to Preview May 3, 2024 23:25 View deployment

apparentlymart added 2 commits May 6, 2024 08:45

terraform: Open and close ephemeral resource instances during graph walk

432571e

apparentlymart force-pushed the f-ephemeral-values branch from 059c2f9 to e6c8430 Compare May 6, 2024 18:22

vercel bot deployed to Preview May 6, 2024 18:26 View deployment

apparentlymart added 2 commits May 6, 2024 11:49

vercel bot deployed to Preview May 6, 2024 18:56 View deployment

vercel bot deployed to Preview May 7, 2024 18:13 View deployment

vercel bot deployed to Preview May 7, 2024 19:07 View deployment

vercel bot deployed to Preview May 7, 2024 21:51 View deployment

vercel bot deployed to Preview May 7, 2024 22:59 View deployment

apparentlymart added 2 commits May 7, 2024 16:33

terraform: Propagate apply-time variables from plan to apply

bbd6dce

We need to remember which ephemeral values were set during planning so that we can require them to be set again (possibly to a different value) during the apply step.

vercel bot deployed to Preview May 7, 2024 23:38 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Ephemeral Values prototype #35078

[WIP] Ephemeral Values prototype #35078

apparentlymart commented Apr 24, 2024 •

edited

[WIP] Ephemeral Values prototype #35078

Are you sure you want to change the base?

[WIP] Ephemeral Values prototype #35078

Conversation

apparentlymart commented Apr 24, 2024 • edited

Ephemeral Values

Ephemeral Input Variables

Ephemeral Output Values

Ephemeral Resources

apparentlymart commented Apr 24, 2024 •

edited