Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve function secrets ergonomics #4130

Merged
merged 65 commits into from
Mar 31, 2022
Merged

Commits on Jan 27, 2022

  1. Use runtime delegate to parse function triggers in the Functions Emul…

    …ator (#4012)
    
    Today, Functions Emulator parses function trigger from source as follows:
    
    1) Spin up an instance of Functions runtime process
    2) Invoke a "function" which triggers a path that parses the trigger by calling out to `extractTriggers.js`
    3) Send parsed triggers via IPC from runtime to emulator. Emulator now knows about the triggers.
    
    This has the advantage of running the trigger parsing in the emulated runtime (which properly mocks out calls to the DB, applies network filtering, uses the same node version when possible, etc.) but has the disadvantage of complicating the runtime implementation as well as diverging from how the triggers are parsed in `firebase deploy`.
    
    Using runtime delegate, we have:
    
    1) Use runtime delegate to discover the delegate appropriate for function source (i.e. Node, but in the future can be some other runtime)
    2) Spin up a node subprocess to parse trigger. Emulator now knows about the triggers.
    
    i.e. the same procedure used during `firebase deploy`
    
    By using runtime delegate, we align the function deploy to production and to the emulated environment and simplify the runtime code a bit. This also puts us into a good position in the future when we make the function deploy process a little more complex, e.g. params and secrets support.
    taeold committed Jan 27, 2022
    Configuration menu
    Copy the full SHA
    41696b3 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2022

  1. Slim down Functions Emulator Runtime (i.e. args sent over to emulated…

    … functions) (#4105)
    
    Now that we've simplified the Functions Emulator to separate out process for loading triggers from the one running the trigger, we can slim down [`FunctionsRuntimeBundle`](https://github.com/firebase/firebase-tools/blob/2e68803f994dbe4f72eb0965dd6a12e7a043b597/src/emulator/functionsEmulatorShared.ts#L53-L88) that is passed between the Functions Emulator and the Functions Runtime process.
    
    This change removes almost all payload attributes in the Functions Runtime Bundle except `proto`. This is nice - we are getting very close to the payload that's passed to a production function instance. We have to leave couple of things like socketpath and debug features - this will probably be removed when we move over to pure-http based protocol (socket) and SDK based debug feature enablement. This could be worked later when I have a little more time!
    
    One more change - we pass around the whole trigger definition in the Functions Emulator instead of pieces of it. This makes it easier to do something else I'm doing... (secret emulator) in the subsequent PR.
    taeold committed Jan 31, 2022
    Configuration menu
    Copy the full SHA
    ac72bf8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    dd8cef8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8b1a590 View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2022

  1. CF3 Secrets Support (#3959)

    Support deploying secret environment variables on a function.
    
    Prior to deploying functions with secret configuration, the CLI will run somewhat comprehensive validation to ensure that secret config will work when deployed, e.g.
    
    1. Secret version exists.
    2. Secret version is in ENABLED state.
    3. Secret version can be access by the runtime service account.
    
    We do this since the GCF doesn't do the same level of validation and instead repeatedly fail to spin up a new instance with an invalid secret config. This often results on super long deploys (probably until some master timeout is met for function instance deploy).
    
    I took the opportunity to refactor the code a little to group various "ensure" and "validate" used in function deploys in their own files.
    
    Emulator support for secrets will come in a separate PR.
    taeold committed Feb 3, 2022
    Configuration menu
    Copy the full SHA
    39f2ac7 View commit details
    Browse the repository at this point in the history
  2. Add new command (functions:secrets:set) for creating secrets to be us…

    …ed for CF3. (#4021)
    
    One of several family of commands to be implemented for managing secrets for CF3.
    
    `functions:secrets:set` command is used to create a new secret version in Secret Manager. If a secret doesn't exist, a secret will be created before adding a new version.
    
    To guide users to our recommended best practices, we will only allow users to create secrets in `UPPER_SNAKE_CASE` - this makes it more obvious how these secrets can be accessed at runtime (via environment variable of the same name).
    
    Usage:
    
    ```
    $ echo SHHHH > SECRET_FILE
    $ firebase functions:secrets:set MY_SECRET --data-file=SECRET_FILE
    ✔  Created a new secret version projects/my-project/secrets/MY_SECRET/versions/0
    i  Please deploy your functions for the change to take effect by running:
        firebase deploy --only functions
    
    // Calling set on existing secret name will create a new version.
    $ echo SHHHHHHHH > SECRET_FILEE
    $ firebase functions:secrets:set MY_SECRET --data-file=SECRET_FILE
    ✔  Created a new secret version projects/my-project/secrets/MY_SECRET/versions/1
    i  Please deploy your functions for the change to take effect by running
        firebase deploy --only functions
    
    // "-" as STDIN is supported but discouraged since it will leave the secret in shell history
    $ echo SHHHHHHHHHH | firebase functions:secrets:set --data-file=- MY_SECRET
    ✔  Created a new secret version projects/my-project/secrets/MY_SECRET/versions/2
    i  Please deploy your functions for the change to take effect by running
        firebase deploy --only functions
    
    
    // Without --data-file flag, begin interactive prompt to take user input
    $ firebase functions:secrets:set MY_SECRET
    ? Enter a value for MY_SECRET [input is hidden]:
    ✔  Created a new secret version projects/my-project/secrets/MY_SECRET/versions/3
    i  Please deploy your functions for the change to take effect by running:
        firebase deploy --only functions
    
    ```
    taeold committed Feb 3, 2022
    Configuration menu
    Copy the full SHA
    3293186 View commit details
    Browse the repository at this point in the history
  3. Add functions:secrets:{access, destroy, get} commands. (#4026)

    Follow up #4021 to add other management commands for CF3 secrets.
    
    Note that `destroy` commands can be improved by making sure we don't accidentally delete secrets versions currently in use (which would immediately break the function!). I'll add these feature in a follow up PR when we finish reviewing the PR w/ `prune` command.
    taeold committed Feb 3, 2022
    Configuration menu
    Copy the full SHA
    08f2236 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2022

  1. Add command to prune unused secrets (#4108)

    Each active secret version cost money. To help save cost on using Secret Manager, we add `functions:secrets:prune` command which:
    
    1) Looks up all secret versions from secrets marked with label "firebase-managed". All secrets created using the Firebase CLI will have this label.
    
    2) Look up all secret bindings for CF3 function instance.
    
    3) Figure out which secret version isn't currently being used.
    
    Since destroying a secret version is irrevocable and immediately breaking for clients that depend on it, we will always ask for a confirmation for the destroy operations (and not support -f flag).
    
    Note that we now query `v1` of Secret Manager since `v1beta` does not offer filtering by labels.
    taeold committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    3310fdc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5731022 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ea194e1 View commit details
    Browse the repository at this point in the history
  4. Update log messages.

    taeold committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    ca8a09e View commit details
    Browse the repository at this point in the history
  5. Use utility fn instead.

    taeold committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    87534f7 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8144038 View commit details
    Browse the repository at this point in the history
  7. Improve log messages.

    taeold committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    3b45d53 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    7906346 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2022

  1. Better types, better docs.

    taeold committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    1853e7f View commit details
    Browse the repository at this point in the history
  2. Eslints.

    taeold committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    5bb029e View commit details
    Browse the repository at this point in the history
  3. Add support for secrets in the Functions Emulator (#4106)

    Emulator will now recognize function triggers with secret environments and ensure that secret environment variables are populated in the emulated runtime.
    
    Secrets in Functions Emulator can come from 2 sources:
    
    1) From local override file (`.secret.local`).
    
    2) From Google Cloud Secret Manager. In this case, default application credentials (i.e. credentials used in Firebase CLI) will be used to fetch the secret from GCP.
     
    As suspected, (1) take precedence over (2). If accessing secret from GCP fails for any reason, the Emulator logs, but does not throw, the failed attempt and proceeds to execute the functions code.
    
    Some refactoring changes needed to be in the Emulator:
    
    * Some functions turned into async.
    * We pass around the whole trigger in more places.
    taeold committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    bb51612 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7e1d70d View commit details
    Browse the repository at this point in the history
  5. Add test cases.

    taeold committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    b784fb5 View commit details
    Browse the repository at this point in the history
  6. Make better code comments.

    taeold committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    4c1dc1f View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2022

  1. Configuration menu
    Copy the full SHA
    c459b23 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    108beea View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ef86cce View commit details
    Browse the repository at this point in the history
  4. Remove preview flag, add option to disable dotenv support. (#4022)

    Preparing for launching dotenv support for CF3.
    
    At launch, CF3 environment variables support will default to picking up dotenv file if any, without need for preview flag.
    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    e799549 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    ad8be43 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    f271e69 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    21ae5fb View commit details
    Browse the repository at this point in the history
  8. Update log messages.

    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    c9e7a2e View commit details
    Browse the repository at this point in the history
  9. Use utility fn instead.

    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    09670c1 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    704003e View commit details
    Browse the repository at this point in the history
  11. Improve log messages.

    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    38d5b1a View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    91a358d View commit details
    Browse the repository at this point in the history
  13. Better types, better docs.

    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    64bb335 View commit details
    Browse the repository at this point in the history
  14. Eslints.

    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    8cd70c6 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    3c6ddb4 View commit details
    Browse the repository at this point in the history
  16. Add test cases.

    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    a342ba3 View commit details
    Browse the repository at this point in the history
  17. Make better code comments.

    taeold committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    f6a52d0 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    432159a View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    2ada4f3 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2022

  1. Configuration menu
    Copy the full SHA
    fd16fb8 View commit details
    Browse the repository at this point in the history
  2. Add -f option to prune.

    taeold committed Feb 17, 2022
    Configuration menu
    Copy the full SHA
    fc0a888 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2022

  1. Configuration menu
    Copy the full SHA
    31d0fe6 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2022

  1. Merge branch 'master' of https://github.com/firebase/firebase-tools i…

    …nto dl-cf3-secret-cmds-ergonomics
    taeold committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    6cd5c16 View commit details
    Browse the repository at this point in the history
  2. Nits.

    taeold committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    31b9c9c View commit details
    Browse the repository at this point in the history
  3. Add changelog.

    taeold committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    2cfbcce View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2022

  1. Configuration menu
    Copy the full SHA
    d0ca620 View commit details
    Browse the repository at this point in the history
  2. Use === instead of ==.

    taeold committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    207f202 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2022

  1. Configuration menu
    Copy the full SHA
    50020b6 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    502af3d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a6cf8c7 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2022

  1. Configuration menu
    Copy the full SHA
    f8a3b01 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2022

  1. Configuration menu
    Copy the full SHA
    af43869 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2022

  1. Configuration menu
    Copy the full SHA
    2a66e7d View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2022

  1. Configuration menu
    Copy the full SHA
    cf14e46 View commit details
    Browse the repository at this point in the history
  2. Update CHANGELOG.md

    taeold committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    7bf6214 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2022

  1. Configuration menu
    Copy the full SHA
    09d93ae View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2022

  1. Configuration menu
    Copy the full SHA
    8f2f309 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2022

  1. Configuration menu
    Copy the full SHA
    28897a4 View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2022

  1. Prettier.

    taeold committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    74067b9 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2022

  1. Configuration menu
    Copy the full SHA
    8c557c0 View commit details
    Browse the repository at this point in the history
  2. Merge conflict.

    taeold committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    02fbd31 View commit details
    Browse the repository at this point in the history
  3. Fix merge gone wrong.

    taeold committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    05068df View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2022

  1. Pretty.

    taeold committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    6abe03a View commit details
    Browse the repository at this point in the history
  2. Pretty.

    taeold committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    d679d0d View commit details
    Browse the repository at this point in the history