Experience Report: Automate deployments with Flux CD, Google Cloud Source Repositories, Google Cloud Build and Google Artifact Registry

[aran]

I recently set up Flux for GitOps-style Kubernetes configuration deployments with Google Cloud Source Repositories, Google Cloud Build and Google Artifact Registry. Notably I did not use the ssh access method for Cloud Source Repositories. I am posting here because maybe my notes will help someone in the future, and I'm also going to note a few issues which might inform future Flux work.

The short overview of the final system is as follows: Code is stored in a subdirectory of a Google Cloud Source Repository. A Cloud Build trigger invokes a flux push artifact (using a customized flux-cli image) to send it to Google Artifact Registry. An OCIRepository source picks it up to deploy. A Kustomization sets up the Kubernetes side of Workload Identity for source-controller. Personally I was delighted with the conceptual clarity of the Flux tool and its clean architecture—if any contributors read this, thank you for making such a great tool.

Cloud Source Repository issue and solution.

The documentation (https://fluxcd.io/flux/use-cases/gcp-source-repository/) suggests the following configuration:

The above command will prompt you to add a deploy key to your repository, but Cloud Source Repository does not support repository or org-specific deploy keys. You may add the deploy key to a user’s personal SSH keys, but take note that revoking the user’s access to the repository will also revoke Flux’s access. The better alternative is to create a machine-user whose sole purpose is to store credentials for automation. Using a machine-user also has the benefit of being able to be read-only or restricted to specific repositories if this is needed.

You can also use an ssh key that was already added to Cloud Source Repository by adding the --private-key-file and --password flags.

I ran into two issues with this.

1. The more fundamental issue is that this contradicts the Google Cloud security model and creates new key management problems. I wanted a solution that followed Google Cloud best practices (e.g. this) , which is to use service accounts with Google-managed key pairs for automation. Google Cloud does not allow service accounts to use ssh keys to authenticate to Cloud Source Repositories.
2. This section was very confusing. I didn't know what is a "deploy key" or "machine-user". Does "deploy key" mean an ssh key or something else? Is a "machine-user" a fake Google Account, a GCP Service Account, or something else? Is this "a" way to use CSR or "the only" way to use CSR?

Because I wasn't sure, I tried hacking to see if I could get flux to use native Google Cloud authentication for its git operations. I went through https://fluxcd.io/flux/components/source/gitrepositories/, tried forcing libgit2, looked through the code and more. This was a big time sink partly because the documentation didn't clearly state that it is not supported, and partly because flux does support Google Cloud authentication in other areas, just not for git.

Now I'm more convinced that Flux doesn't support it. Maybe one day CSR will support more authentication types (doubt it), maybe go-git can be extended to support gitcredentials, or maybe Flux can be directly extended to allow "real" git porcelain with git credential helper support as an optional git implementation, or something else.

One way or the other, without flux able to run a git clone, that ruled out flux bootstrap and using GitRepository source. That meant I needed to re-implement bootstrap myself. For this purpose, it would have been nice to have more tooling around flux bootstrap: Maybe an '--export', '--dry-run', and more verbosity especially in logging along the authentication code path.

Manual Flux bootstrap

Outside of Flux directly, I set up a Cloud Source Repository, a dedicated Service Account with Workload Identity enabled, a Cloud Build trigger, and a Google Artifact Registry, all with a messy tangle of required permissions. I happened to use Terraform for configuring these.

I also performed my own Flux bootstrap:
flux install --export
flux create source oci flux-system --url oci://{registry}/{kubernetes_name} --provider gcp --tag latest --export
flux create kustomization flux-system --source OCIRepository/flux-system --path {relative_path/from_repository_root/to_root_of_folder_flux_is_managing} --prune true --interval 60m --wait --export

I put a {relative_path/from_repository_root/to_root_of_folder_flux_is_managing}/kustomization.yaml, flux-system folder under {relative_path/from_repository_root/to_root_of_folder_flux_is_managing}, put the flux install output in that flux-system/gotk-components.yaml, put the output of the other two, concatenated, in flux-system/gotk-sync.yaml, and added a flux-system/kustomization.yaml like so:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - patch: |-
      - op: add
        path: /metadata/annotations
        value:
            iam.gke.io/gcp-service-account: {my-service-account}
    target:
      kind: ServiceAccount
      name: source-controller

I committed these files to git and then ran kubectl apply -k on the root of them repeatedly. Twice usually does the trick. The first apply would fail because the CRDs weren't ready before kubectl apply started trying to use them. This is a known issue with kubectl.

Note this Kustomization for Workload Identity is slightly different from what was in the Flux docs. I wasn't able to get the suggested Kustomization from the docs to work. Maybe there has been some drift.

flux push

Next I set up a Cloud Build trigger. Here is the minimal build script:

timeout: 300s
options:
  logging: CLOUD_LOGGING_ONLY
steps:
 - name: "${_GAR}/flux"
   args:
    - push
    - artifact
    - oci://${_GAR}/${_KUBERNETES_NAME}:latest
    - --path
    - ${_FLUX_PATH}
    - --source
    - "Cloud Build ${TRIGGER_NAME}"
    - --revision
    - ${REVISION_ID}
    - --provider
    - gcp

A few notes here:

1. ${_GAR} is my Google Artifact Registry. This is using a customized flux-cli container. I tried using ghcr.io/fluxcd/flux-cli directly at first. It sets the Docker user to nobody, so it can't write to the file system, which means it isn't able to flux push. I ran into mysterious permission-denied errors until figuring this out. Since this is a CLI tool not a daemon, my customization just resets the user to root.
2. In reading docs about setting up the OCI artifact workflow, I came across https://fluxcd.io/flux/guides/sortable-image-tags/ which was very confusing. I still don't know what that page is for.
3. I noticed that flux push artifact --help says --revision should be "the source revision in the format '<branch|tag>/<commit-sha>'" and that my build script got this wrong - but it didn't seem to matter. It wasn't clear to me if --source and --revision could be given sane defaults rather than being required.
4. I ran into multiple issues finding the right value for $_FLUX_PATH.
   1. I think I saw a surprising behavior difference between specifying an absolute path and a relative path. An absolute --path starting with '/' seemed to act as a stripped prefix in the resulting artifact directory structure, whereas a relative path resulted in the entire path being included. I could be wrong, I didn't dwell on it.
   2. When I got it wrong, repeatedly, the way I found the error was that in flux get all, I saw the following error: kustomize-controller kustomization path not found: stat /tmp/[…]: no such file or directory It took a while to trace that back to an artifact construction misconfiguration relative to the consuming kustomization --path. It would be nice to see documented guidance on aligning those two --paths that must work together and a special cased error message in kustomization-controller such as "kustomization path not found: expected to find [...], try checking your artifact with flux pull artifact [...] to ensure files are placed where expected. Note also that kustomization-controller reveals that it uses /tmp whereas source-controller uses /data which also hindered debugging a bit.
   3. https://fluxcd.io/flux/components/kustomize/kustomization/#recommended-settings recommends setting an ignore in the source kustomization, but for an OCIRepository I don't think this is necessary or valuable. An ignore entry is another source of potential errors as its path also needs to be synchronized, syntax errors are possible, and it also can result in an empty artifact.

Deployment

After a couple round trips where my noob errors resulted in Flux deleting its own controllers, I got it working. I tested it out by using a hello world deployment found at https://cloud.google.com/kubernetes-engine/docs/samples/container-helloapp-deployment and was surprised to see an error: “Deployment/helloweb namespace not specified, error: the server could not find the requested resource”. Similar to #2433. I found it surprising that Flux has a behavior difference from native kubernetes/kubectl where a missing namespace implicitly uses the "default" namespace. Thankfully the workaround is minor but it was an issue worth mentioning.

Deployment Bliss

I mentioned earlier I was using Terraform for provisioning Google Cloud infrastructure. Looking back at build logs, a typical Terraform deploy with Google-provided scripts on a bare-bones GKE set up was taking about 1m20s to 2m30s, with occasional spikes way beyond. By comparison, with Flux the Cloud Builds are taking 10s–20s. Nice. More importantly Flux offers a much more Kubernetes-native model at very low complexity which I'm hoping will pay dividends over time.

As I mentioned in the beginning I have high hopes this experience report is useful to anyone considering Flux on GCP and to the Flux team. Thanks again. Please reach out if you have any questions or comments.

[stefanprodan]

Thank you very much @aran for the detailed guide and great feedback. We'll create dedicated issues for the things you reported. I do think we need to promote more OCI vs Git, I really like how you've setup OCI with Google Artifact Registry instead of Git/SSH with Google Cloud Source. Thanks again ❤️

[stefanprodan]

Did some small changes to the docs here fluxcd/website#1311 more to come

[stefanprodan]

I think I saw a surprising behavior difference between specifying an absolute path and a relative path. An absolute --path starting with '/' seemed to act as a stripped prefix in the resulting artifact directory structure, whereas a relative path resulted in the entire path being included.

We're fixing this in fluxcd/pkg#418

[srgvg]

I had an issue pushing the oci manifests from Google Cloud Run (using the --provider gcp option). Problem is that fluxcd/flux-cli image runs as user nobody, and Cloud build checks out the git repository at /workspace with 755 root:root permissions. flux push artifact wants then to create a tmp file at /workspacexxxxxx but has no permission for that.
As a workaround i added a build step with a bash container (which runs as root as well) and added this snippet

mkdir -v /workspace/target/
mv /workspace/* /workspace/target/
chown nobody:nobody -R /workspace

and updated the flux push artifact --path to --path /workspace/target/