Flux for temporary review environments #831
-
|
Vi are planning to use flux for our dev/stage/prod where it seems a perfect fit. Would flux still be a good fit, for this type of workflow, or should I look at more |
Beta Was this translation helpful? Give feedback.
Replies: 13 comments 19 replies
-
|
There are two types of ephemeral environments that we wish to address in Flux v2:
To deploy applications from feature branches onto ephemeral namespaces, we could introduce a new toolkit component (e.g. preview-controller) that reacts to events such as PR/branch create/push/delete. The preview-controller could manage ephemeral namespaces and app deployments by generating GitRepository, Kustomization, and HelmRelease objects on the fly. When the PR is merged and the branch deleted, the controller will react by removing the generated objects and thus triggering garbage collection of the whole preview environment. The preview-controller could leverage the kustomize-controller var substitutions feature and helm-controller values override to inject values for vars like: PR_ID and GIT_BRANCH. These variables could be used to prefix/suffix the ephemeral namespace, ingress sub-domain/url, certs, configmap data, etc. |
Beta Was this translation helpful? Give feedback.
-
An abstract, technical point: I think it's more in the spirit of Kubernetes API to reflect PRs as objects in the cluster, rather than to respond directly to events (e.g., webhooks). This distinction would matter if the controller or event receiver is unavailable when a PR is opened, for instance. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @stefanprodan & @squaremo, I'm really interested to work on this feature and I am wondering what is the process? Do you have a resource to define what should be in the RFC issue? |
Beta Was this translation helpful? Give feedback.
-
|
This would be a phenomenal addition to the GOTK suite! |
Beta Was this translation helpful? Give feedback.
-
|
Any update? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
what could be the timeline for this ? considering flux GA is available now |
Beta Was this translation helpful? Give feedback.
-
|
Just wanted to share that we are also in need of a more dynamic environment. What we want:
We now have to see if we can keep the current GitOps approach or if we have to find something new. |
Beta Was this translation helpful? Give feedback.
-
|
@paprickar did you guys find any solution with fluxcd? or have you guys moved to a different solution altogether. We have implemented fluxcd for our static environments, and currently looking for solutions to implement dynamic environments with fluxcd, but this seems like its going to take some time.. Appreciate your comments on how you decided to proceed. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW we use Flux for all static deployments but implemented this outside of Flux using GitHub Actions and Helm upgrade/uninstall on PR open/close events. It works pretty well, you could also use comments on PRs if you wanted to get fancy. |
Beta Was this translation helpful? Give feedback.
-
|
Nothing new on that yet was to busy with other topics. |
Beta Was this translation helpful? Give feedback.
-
|
@mrknmc thank you for the response, even we are now looking at Github Actions for our preview requirements. I found one solution which is available at Github Actions Marketplace : https://github.com/marketplace/actions/preview-pull-requests-in-azure-kubernetes-service |
Beta Was this translation helpful? Give feedback.
-
|
@paprickar yes there are different solution, but what I was focusing on is using fluxcd itself instead of going with two different solution for static environments and dynamic environment. |
Beta Was this translation helpful? Give feedback.
-
|
I've done feature branch releases like this before: This script acts as a simple branch controller that produces GitRepositories, Kustomizations, and a target Namespace. You can put this into a CronJob or sleep-loop within a Deployment like so. You'll need to have an image that has parse_branches() {
tr '/' ' ' | awk '{print $3}'
}
sanitize_branches() {
# "--branch_TEST-@@@.^-name--" --> "branch-test-name"
tr '[:upper:]' '[:lower:]' \
| tr -c '[:alnum:]' - \
| tr -s - \
| sed -e 's/^-*//' -e 's/-*$//' -e 's/--*/-/'
}
git fetch origin
git branch -a --list 'origin/*feature*' | parse_branches | sort -u > ./branches
kubectl get gitrepository -oname | grep feature | cut -d/ -f2 | sort -u > ./in_cluster
comm -2 <(cat ./branches | sanitize_branches) ./in_cluster > ./to_delete
for raw_br in $(cat ./branches); do
kube_br="$(echo "${raw_br}" | sanitize_branches)"
kubectl apply -f- << EOF
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: ${kube_br}
namespace: flux-system
spec:
interval: 1m0s
ref:
branch: ${raw_br}
url: ssh://git@github.com/myorg/myrepo
secretRef: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: ${kube_br}
namespace: flux-system
spec:
interval: 1m0s
path: ./
prune: true
sourceRef:
kind: GitRepository
name: ${kube_br}
targetNamespace: ${kube_br}
---
apiVersion: v1
kind: Namespace
metadata:
name: ${kube_br}
EOF
done
for kube_br in $(cat ./to_delete); do
kubectl delete --wait=false gitrepository "${kube_br}"
kubectl delete --wait=false kustomization "${kube_br}"
kubectl delete --wait=false namespace "${kube_br}"
done |
Beta Was this translation helpful? Give feedback.
-
|
I would also like to add that this would be an extremely valuable feature to have and while @stealthybox's solution could provide an interim solution, I am looking forward to a |
Beta Was this translation helpful? Give feedback.
-
|
definitely keen on this feature - I might try to PoC this out using @stealthybox 's suggestion. |
Beta Was this translation helpful? Give feedback.
-
|
have started spiking this out as I am interested in this pattern for a platform I am building for multiple product teams https://github.com/alam0rt/ephemeral-environment-controller |
Beta Was this translation helpful? Give feedback.
-
|
@alam0rt thanks for publishing that! It's looking really cool. |
Beta Was this translation helpful? Give feedback.
-
|
any status on this? |
Beta Was this translation helpful? Give feedback.
-
|
Hope to see this on the roadmap soon. |
Beta Was this translation helpful? Give feedback.
-
|
Here, I will focus on ephemeral environment == "I got an application, I want to deploy multiple preview of my app on a single cluster." Ephemeral environment is a challenge by itself. The first step is to know if it is a valid approach on your app. On some case, I think we can assume this can be a good and an easy thing to manage (front application for instance). On some other case, I do not really see a value to ephemeral environment vs a testcontainer based test on CI ( app like data streaming service for instance). For application in between, an ephemeral environment may be complex to generate ( duplicate data storage ? use a single data storage but then how to ensure no conflict between data definition ? What if I need a configuration change because of my PR ? So yes, not impossible, but can be really tricky) and may have more or less value to build it. All in all, having a process to do that is something I will vote for. Earlier comment suggest an approach to create a "preview-controller" driven by PR webhook. I'm not so found of this approach. If the trigger is webhook it also means the controller will have to create an update policy because it will be triggered before the image is built (or the failure to build one :D ). Doing that raises the question as why do we need this controller? if we need to call something (with a secret) to create something in a cluster, why not simplify the process and call directly (with a secret) a job on the repository to create the needed file. Ok it will be provider dependent, but the controller would also need support on each provider for simplicity. The other solution I have in mind is to extends the image automation process to duplicate environment. From image tag, we can extract multiple data (for instance PR number) and from that, duplicate some kubernetes objects. |
Beta Was this translation helpful? Give feedback.
-
|
Writing this make me think a solution inside flux is perhaps not the best answer. On GitHub creating a well crafted GitHub action may be beneficial for instance. Will try to think a little bit more about this. |
Beta Was this translation helpful? Give feedback.
-
|
I'd expect that the webook would only notify the Your GitHub action shouldn't need a secret key or auth token in order to poke the webhook, since the poke itself doesn't result in any changes applied to your cluster if there are no corresponding changes in the source repo. But I'm speculating a bit here and don't know for sure if @stefanprodan might be thinking differently. |
Beta Was this translation helpful? Give feedback.
-
|
@sbernheim This hook partially already exist. It listens gitrepository event : https://fluxcd.io/docs/guides/webhook-receivers/ What is difficult here is ephemeral / temporary environment may not be a copy/pasted from something. You may need to update something. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have created a very basic pr-ephemeral-env-controller controller. This currently works with Github and creates (updates/ deletes) new Flux HelmReleases for each PRs, I have based it on the Argo CD ApplicationSet Pull Request Generator. The Flux HelmRelease refers a Helm Chart through a Flux Source Repo. The values passed to the chart by the controller are the PR Number and PR Head SHA. You can design your Helm Chart to create a new environment in separate Namespace for each PR, or totally isolated (including isolated cloud services) environments for each PR. In the sample linked on the Controller Repository, for each PR to the sample application repository (The sample app is a simple todo API which needs a backend postgres database), a new isolated environment is created for each PR, with several isolated resources (The Helm Chart uses Crossplane Azure Jet Provider and Crossplane Helm Provider resources), including a new resource group, a new AKS cluster to which the application deployment and service (corresponding to the PR SHA commit of the application) are applied , and a new Azure Postgres backend database to which the application points to read and persist data. The Helm chart repository which the Flux HelmRelease points to is ephemeral-env-infra Thought I will share it here. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Any update on adding new resources like "preview namespaces (Kustomize+Helm)" "preview clusters (ClusterAPI)" to devops toolkit to support ephemeral envs? |
Beta Was this translation helpful? Give feedback.
-
|
The Template Controller might be a viable solution for some of you. It allows to create arbitrary templated objects (e.g. HelmRelease or Kustomizatin) from "matrix inputs", which in turn can be arbitrary objects as well. The controller also provides a See the Introducing the Template Controller and building GitOps Preview Environments blog post for an introduction and examples. The example are based on Kluctl, but can easily be changed to work with Kustomization/HelmRelease. |
Beta Was this translation helpful? Give feedback.
-
|
Just thought I'd ask, though no pressure to answer - is there a desire to get this functionality or workflow formalised before flux v2 hits GA/1.0.0 status across the board? Would be good to know if there's a timeline/priority for it, or if it's just a loosely aspirational goal to have as and when the time is right. |
Beta Was this translation helpful? Give feedback.
-
The answer is 99% likely to be "no" because we're all committed to bringing Flux v2 to GA first and foremost. |
Beta Was this translation helpful? Give feedback.
-
|
Hi thanks for your answer 👍 . Now that Flux v2 has reached GA, is there a chance that this issue will be considered? This feature seems to be the most requested feature for flux (upvotes in discussion), which would be awesome. |
Beta Was this translation helpful? Give feedback.
-
|
our developers have simply written the commit automation to place the manifests into the directories/branches for dev deploys. Basically a really convoluted way to run We just push temp deploys into a shared namespace because apps don't typically need custom ServiceAccounts and it would be commit automation into an admin repo to get ephemeral SA's+Namespaces, which is a no-go for us. |
Beta Was this translation helpful? Give feedback.
-
|
looking at the Template Controller and various "create HR's from pull requests" mentioned above, I wonder how those will stitch together fleets of microservices in different repos? e.g. repo-a's service makes calls to repo-b's service, how do we declare the intent to connect these 2 ephemeral/temporary releases? In the end, repo-a would need to know repo-b's ephemeral service name, so do we just put that data in the PR where it will most likely get merged into master? |
Beta Was this translation helpful? Give feedback.
-
|
not sure if this helps but here is an example how we achieved it in combination with gitlab ci. this might not be a replacement for a more "controller" like approach but that's what we/devs needed atm -> https://6uellerbpanda.gitlab.io/posts/Feature-branch-deployments-flux-gitlab-ci/ |
Beta Was this translation helpful? Give feedback.
and others
There are two types of ephemeral environments that we wish to address in Flux v2:
To deploy applications from feature branches onto ephemeral namespaces, we could introduce a new toolkit component (e.g. preview-controller) that reacts to events such as PR/branch create/push/delete.
The preview-controller could manage ephemeral namespaces and app deployments by generating GitRepository, Kustomization, and HelmRelease objects on the fly. When the PR is merged and the branch deleted, the controller will react by removing the generated objects and thus triggering garbage collection of the whole preview environment.
The previe…