Improve function secrets ergonomics #4130
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ator (#4012) Today, Functions Emulator parses function trigger from source as follows: 1) Spin up an instance of Functions runtime process 2) Invoke a "function" which triggers a path that parses the trigger by calling out to `extractTriggers.js` 3) Send parsed triggers via IPC from runtime to emulator. Emulator now knows about the triggers. This has the advantage of running the trigger parsing in the emulated runtime (which properly mocks out calls to the DB, applies network filtering, uses the same node version when possible, etc.) but has the disadvantage of complicating the runtime implementation as well as diverging from how the triggers are parsed in `firebase deploy`. Using runtime delegate, we have: 1) Use runtime delegate to discover the delegate appropriate for function source (i.e. Node, but in the future can be some other runtime) 2) Spin up a node subprocess to parse trigger. Emulator now knows about the triggers. i.e. the same procedure used during `firebase deploy` By using runtime delegate, we align the function deploy to production and to the emulated environment and simplify the runtime code a bit. This also puts us into a good position in the future when we make the function deploy process a little more complex, e.g. params and secrets support.
… functions) (#4105) Now that we've simplified the Functions Emulator to separate out process for loading triggers from the one running the trigger, we can slim down [`FunctionsRuntimeBundle`](https://github.com/firebase/firebase-tools/blob/2e68803f994dbe4f72eb0965dd6a12e7a043b597/src/emulator/functionsEmulatorShared.ts#L53-L88) that is passed between the Functions Emulator and the Functions Runtime process. This change removes almost all payload attributes in the Functions Runtime Bundle except `proto`. This is nice - we are getting very close to the payload that's passed to a production function instance. We have to leave couple of things like socketpath and debug features - this will probably be removed when we move over to pure-http based protocol (socket) and SDK based debug feature enablement. This could be worked later when I have a little more time! One more change - we pass around the whole trigger definition in the Functions Emulator instead of pieces of it. This makes it easier to do something else I'm doing... (secret emulator) in the subsequent PR.
Support deploying secret environment variables on a function. Prior to deploying functions with secret configuration, the CLI will run somewhat comprehensive validation to ensure that secret config will work when deployed, e.g. 1. Secret version exists. 2. Secret version is in ENABLED state. 3. Secret version can be access by the runtime service account. We do this since the GCF doesn't do the same level of validation and instead repeatedly fail to spin up a new instance with an invalid secret config. This often results on super long deploys (probably until some master timeout is met for function instance deploy). I took the opportunity to refactor the code a little to group various "ensure" and "validate" used in function deploys in their own files. Emulator support for secrets will come in a separate PR.
…ed for CF3. (#4021) One of several family of commands to be implemented for managing secrets for CF3. `functions:secrets:set` command is used to create a new secret version in Secret Manager. If a secret doesn't exist, a secret will be created before adding a new version. To guide users to our recommended best practices, we will only allow users to create secrets in `UPPER_SNAKE_CASE` - this makes it more obvious how these secrets can be accessed at runtime (via environment variable of the same name). Usage: ``` $ echo SHHHH > SECRET_FILE $ firebase functions:secrets:set MY_SECRET --data-file=SECRET_FILE ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/0 i Please deploy your functions for the change to take effect by running: firebase deploy --only functions // Calling set on existing secret name will create a new version. $ echo SHHHHHHHH > SECRET_FILEE $ firebase functions:secrets:set MY_SECRET --data-file=SECRET_FILE ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/1 i Please deploy your functions for the change to take effect by running firebase deploy --only functions // "-" as STDIN is supported but discouraged since it will leave the secret in shell history $ echo SHHHHHHHHHH | firebase functions:secrets:set --data-file=- MY_SECRET ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/2 i Please deploy your functions for the change to take effect by running firebase deploy --only functions // Without --data-file flag, begin interactive prompt to take user input $ firebase functions:secrets:set MY_SECRET ? Enter a value for MY_SECRET [input is hidden]: ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/3 i Please deploy your functions for the change to take effect by running: firebase deploy --only functions ```
Follow up #4021 to add other management commands for CF3 secrets. Note that `destroy` commands can be improved by making sure we don't accidentally delete secrets versions currently in use (which would immediately break the function!). I'll add these feature in a follow up PR when we finish reviewing the PR w/ `prune` command.
Each active secret version cost money. To help save cost on using Secret Manager, we add `functions:secrets:prune` command which: 1) Looks up all secret versions from secrets marked with label "firebase-managed". All secrets created using the Firebase CLI will have this label. 2) Look up all secret bindings for CF3 function instance. 3) Figure out which secret version isn't currently being used. Since destroying a secret version is irrevocable and immediately breaking for clients that depend on it, we will always ask for a confirmation for the destroy operations (and not support -f flag). Note that we now query `v1` of Secret Manager since `v1beta` does not offer filtering by labels.
Emulator will now recognize function triggers with secret environments and ensure that secret environment variables are populated in the emulated runtime. Secrets in Functions Emulator can come from 2 sources: 1) From local override file (`.secret.local`). 2) From Google Cloud Secret Manager. In this case, default application credentials (i.e. credentials used in Firebase CLI) will be used to fetch the secret from GCP. As suspected, (1) take precedence over (2). If accessing secret from GCP fails for any reason, the Emulator logs, but does not throw, the failed attempt and proceeds to execute the functions code. Some refactoring changes needed to be in the Emulator: * Some functions turned into async. * We pass around the whole trigger in more places.
…nto cf3-secrets
Preparing for launching dotenv support for CF3. At launch, CF3 environment variables support will default to picking up dotenv file if any, without need for preview flag.
colerogers
reviewed
Mar 8, 2022
inlined
reviewed
Apr 5, 2022
| .filter((e) => secrets.inUse({ projectId, projectNumber }, sv.secret, e)); | ||
| if (boundEndpoints.length > 0) { | ||
| const endpointsMsg = boundEndpoints | ||
| .map((e) => `${e.id}[${e.platform}](${e.region})`) |
There was a problem hiding this comment.
A function can only be gcf v1 or v2, so we can probably drop this disambiguation. It'll also help us avoid leaking gcfv2's existence/support when we release secrets support.
| const projectId = needProjectId(options); | ||
| const projectNumber = await needProjectNumber(options); | ||
| // Re-load backend with all endpoints, not just the ones deployed. | ||
| const reloadedBackend = await backend.existingBackend({ projectId } as args.Context); |
There was a problem hiding this comment.
Isn't there a "force" option you can use to keep context?
|
|
||
| type PruneResult = { | ||
| destroyed: backend.SecretEnvVar[]; | ||
| erred: { message: string }[]; |
There was a problem hiding this comment.
nit: IIRC google style guide says that non-trivial types should use Array<>
| entryPoint: fn.entryPoint, | ||
| secretEnvironmentVariables: updatedSevs, | ||
| }); | ||
| // Using fabricator.gcfV1PollerOptions doesn't work - apiVersion is empty on that object. |
There was a problem hiding this comment.
I'm a bit worried that long term we're going to invest in making Fabricator more resilient to large deploys and we'll not benefit here. I wonder if we should just toss all the functions to update into Fabricator.
|
I link the issue #4408 as I believe this is link to this change. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
functions:secrets:setto redeploy affected function triggers and delete now stale secret versions post redeploy.functions:secrets:destroyto block if the secret version is in use.Also sneaking in a change to improve
prunelogic by finding all secret versions that isn't destroyed (DISABLED secret versions still cost $).