){kind=icon}
){kind=icon}
){kind=icon}
){kind=icon}
){kind=icon}
Example project showing how to use the crossplane together with ArgoCD
This project is based on the crossplane only repository https://github.com/jonashackt/crossplane-aws-azure, where the basics about crossplane.io are explained in detail - incl. how to provision to AWS and Azure.
The idea is "simple": Why not treat infrastructure deployments/provisioning the same way as application deployment?! An ideal combination would be crossplane as control plane framework, which manages infrastructure through the Kubernetes api together with ArgoCD as GitOps framework to have everything in sync with our version control system.
If you don't want to read much text, do the following steps:
# fire up kind
kind create cluster --image kindest/node:v1.29.2 --wait 5m
# Install ArgoCD
kubectl create namespace argocd
kubectl apply -k argocd/install
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server --namespace argocd --timeout=300s
# Create Secret with Doppler Service Token
kubectl create secret generic doppler-token-auth-api --from-literal dopplerToken="$DOPPLER_SERVICE_TOKEN"
# Prepare Secret with ArgoCD API Token for Crossplane ArgoCD Provider (port forward can be run in subshell appending ' &' + Ctrl-C and beeing deleted after running create-argocd-api-token-secret.sh via 'fg 1%' + Ctrl-C)
kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443
./create-argocd-api-token-secret.sh
# Bootstrap Crossplane via ArgoCD
kubectl apply -n argocd -f argocd/crossplane-eso-bootstrap.yaml
# Install Crossplane EKS APIs/Composition
kubectl apply -f argocd/crossplane-apis/crossplane-apis.yaml
# Create actual EKS cluster via Crossplane & register it in ArgoCD via argocd-provider
kubectl apply -f argocd/infrastructure/aws-eks.yaml
# Optional: If you want, have a look onto the new cluster
kubectl get secret eks-cluster-kubeconfig -o jsonpath='{.data.kubeconfig}' | base64 --decode > ekskubeconfig
# integrate the contents of `ekskubeconfig` into your `~/.kube/config` (better w/ VSCode!) & switch over to the new kube context
# Run Application on EKS cluster using Argo
kubectl apply -f argocd/applications/microservice-api-spring-boot.yamlNow you should see both clusters (kind & EKS) running and the app beeing deployed:
First we need a simple management cluster for our ArgoCD and crossplane deployments. As in the base project we simply use kind here:
Be sure to have some packages installed. On a Mac:
brew install kind helm kubectl kustomize argocdOr on Arch/Manjaro:
pamac install kind-bin helm kubectl-bin kustomize argocdhttps://docs.crossplane.io/latest/cli/
Also we should install the crossplane CLI
curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" | sh
sudo mv crossplane /usr/local/binNow the kubectl crossplane --help command should be ready to use.
Now spin up a local kind cluster
kind create cluster --image kindest/node:v1.29.2 --wait 5mhttps://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/
https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/
"There are different ways to configure how Argo CD tracks resources. With Crossplane, you need to configure Argo CD to use Annotation based resource tracking."
You may already used ArgoCD with resource tracking via the well-known label app.kubernetes.io/instance, which is the default resource tracking mode in Argo. But from ArgoCD 2.2 on there are additional ways of tracking resources. One of them is the annotation based resource tracking. This has some advantages:
"The advantages of using the tracking id annotation is that there are no clashes any more with other Kubernetes tools and Argo CD is never confused about the owner of a resource. The annotation+label can also be used if you want other tools to understand resources managed by Argo CD."
The resource tracking method has to be configured inside the argocd-cm ConfigMap using the application.resourceTrackingMethod field:
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
data:
application.resourceTrackingMethod: annotationhttps://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-exclusion
Crossplane providers generates a
ProviderConfigUsagefor each of the managed resource (MR) it handles. This resource enable representing the relationship between MR and a ProviderConfig so that the controller can use it as finalizer when a ProviderConfig is deleted. End-users of Crossplane are not expected to interact with this resource.
What this means is that if we have a lot of Crossplane Resources that we work with like in the following image, the ArgoCD UI reactivity can be impacted:
And because these resources don't give us anymore insights, we can savely remove them from ArgoCD UI. Therefore we also configure this in the argocd-cm ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
data:
...
# Set Resource Exclusion (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-exclusion)
resource.exclusions: |
- apiGroups:
- "*"
kinds:
- ProviderConfigUsage But how do we install ArgoCD and change the ConfigMap in a flexible, GitOps-style and renovatebot-enabled way?
There's a great repo https://github.com/jonashackt/tekton-argocd-eks where also a great way on how to configure and install ArgoCD is described https://stackoverflow.com/a/71692892/4964553 (and also in the Argo docs).
First we create a directory argocd/install and a file kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/argoproj/argo-cd//manifests/cluster-install?ref=v2.9.3
## changes to config maps
patches:
- path: argocd-cm-patch.yml
namespace: argocdWe also need to create a file argocd-cm-patch.yml, where we can implement our needed annotation based resource tracking mode in the Argo ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
labels:
app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd
data:
server.insecure: "true"
application.resourceTrackingMethod: annotationCreating the Argo namespace and installing it via kubectl apply -k is all we need to do:
kubectl create namespace argocd
kubectl apply -k argocd/installTODO: Configure Argo configmaps to handle Crossplane correctly
https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/
First we need to obtain the initial password for the admin user via:
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echoNow we have multiple options here - where the simplest might be a port-forward:
kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8080:80Now access the ArgoCD UI inside your Browser at http://localhost:8080 using admin user and the obtained password.
https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli
In order to be able to add applications to Argo, we should login our ArgoCD CLI into our argocd-server Pod installed in kind:
argocd login localhost:8080 --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo) --insecureRemember to change the initial password in production environments!
Is it possible to already use the GitOps approach right from here on to install crossplane? Let's try it.
As already used from https://github.com/jonashackt/crossplane-aws-azure and explained in https://stackoverflow.com/a/71765472/4964553 we have a simple Helm chart, which is able to be managed by RenovateBot - and thus kept up-to-date. Our Chart lives in crossplane/Chart.yaml:
apiVersion: v2
type: application
name: crossplane-argocd
version: 0.0.0 # unused
appVersion: 0.0.0 # unused
dependencies:
- name: crossplane
repository: https://charts.crossplane.io/stable
version: 1.14.5This Helm chart needs to be picked up by Argo in a declarative GitOps way (not through the UI).
But as this is a non-standard Helm Chart, we need to define a Secret first as the docs state:
"Non standard Helm Chart repositories have to be registered explicitly. Each repository must have url, type and name fields."
So we first create crossplane-helm-secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: crossplane-helm-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
name: crossplane
url: https://charts.crossplane.io/stable
type: helm We need to apply it via:
kubectl apply -f argocd/crossplane-bootstrap/crossplane-helm-secret.yamlNow telling ArgoCD where to find our simple Crossplane Helm Chart, we use Argo's Application manifest in argocd/crossplane-bootstrap/crossplane.yaml:
# The ArgoCD Application for crossplane core components themselves
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
path: crossplane
destination:
server: https://kubernetes.default.svc
namespace: crossplane-system
syncPolicy:
automated:
prune: true
syncOptions:
- CreateNamespace=true
retry:
limit: 1
backoff:
duration: 5s
factor: 2
maxDuration: 1mAs the docs state https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#crossplane-bootstrap
"Without the
resources-finalizer.argocd.argoproj.io finalizer, deleting an application will not delete the resources it manages. To perform a cascading delete, you must add the finalizer. See App Deletion."
In other words, if we would run kubectl delete -n argocd -f argocd/crossplane-bootstrap/crossplane.yaml, Crossplane wouldn't be undeployed as we may think. Only the ArgoCD Application would be deleted, but Crossplane Pods etc. would be still running.
Our Application configures Crossplane core componentes to be automatically pruned https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/#automatic-pruning via automated: prune: true.
We also use syncOptions: - CreateNamespace=true here to let Argo create the crossplane crossplane-system namespace for us automatically.
kubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane.yamlNow ArgoCD deploys our core crossplane components for us :)
Just have a look into Argo UI:
We can double check everything is there on the command line via:
kubectl get all -n crossplane-systemhttps://docs.crossplane.io/latest/getting-started/provider-aws/#generate-an-aws-key-pair-file
I assume here that you have aws CLI installed and configured. So that the command aws configure should work on your system. With this prepared we can create an aws-creds.conf file:
echo "[default]
aws_access_key_id = $(aws configure get aws_access_key_id)
aws_secret_access_key = $(aws configure get aws_secret_access_key)
" > aws-creds.confDon't ever check this file into source control - it holds your AWS credentials! For this repository I added
*-creds.confto the .gitignore file.
Now we need to use the aws-creds.conf file to create the Crossplane AWS Provider secret:
kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.confTODO: create Secret as via manifest (in GitHub Actions)?!
Our crossplane AWS provider reside in upbound/provider-aws-s3/config/provider-aws-s3.yaml:
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: upbound-provider-aws-s3
spec:
package: xpkg.upbound.io/upbound/provider-aws-s3:v0.46.0
packagePullPolicy: Always
revisionActivationPolicy: Automatic
revisionHistoryLimit: 1How do we let ArgoCD manage and deploy this to our cluster? The simple way of defining a directory containing k8s manifests is what we're looking for. Therefore we create a new ArgoCD Application CRD at argocd/crossplane-bootstrap/crossplane-provider-aws.yaml, which tells Argo to look in the directory path upbound/provider-aws-s3/config:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane-provider-aws-s3
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
path: upbound/provider-aws/config
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
destination:
namespace: default
server: https://kubernetes.default.svc
# Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with
# 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'
syncPolicy:
automated:
prune: true The crucial point here is to use the syncPolicy.automated flag as described in the docs: https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/. Otherwise the deployment of the Crossplane provider-aws-s3 will give the following error:
Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3The automated syncPolicy makes sure that child apps are automatically created, synced, and deleted when the manifest is changed.
This flag enables ArgoCD's "true" GitOps feature, where the CI/CD pipeline doesn't deploy themselfes (Push-based GitOps) but only makes a git commit. Then the GitOps operator inside the Kubernetes cluster (here ArgoCD) recognizes the change in the Git repository and deploys the changes to match the state of the repository in the cluster.
We also use the finalizer resources-finalizer.argocd.argoproj.io finalizer like we did with the Crossplane core components so that a kubectl delete -f would also undeploy all components of our Provider provider-aws-s3.
Let's apply this Application to our cluster also:
kubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws.yaml We run into the following error while syncing in Argo:
The Kubernetes API could not find aws.upbound.io/ProviderConfig for requested resource default/default. Make sure the "ProviderConfig" CRD is installed on the destination cluster.
To get our Provider finally working we also need to create a ProviderConfig accordingly that will tell the Provider where to find it's AWS credentials. Therefore we create a upbound/provider-aws/config/provider-aws-config.yaml:
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: aws-creds
key: credsCrossplane resources use the
ProviderConfignameddefaultif no specific ProviderConfig is specified, so this ProviderConfig will be the default for all AWS resources.
The secretRef.name and secretRef.key has to match the fields of the already created Secret.
To let ArgoCD manage and deploy our ProviderConfig we again create a new ArgoCD Application CRD at argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml defining a directory containing k8s manifests, which tells Argo to look in the directory path upbound/provider-aws/config:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: provider-aws-config
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
path: upbound/provider-aws/config
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
destination:
namespace: default
server: https://kubernetes.default.svc
# Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with
# 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'
syncPolicy:
automated:
prune: true kubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml We finally managed to let Argo deploy the Crossplane core components together with the AWS Provider and ProviderConfig correctly:
While our setup works now and also fully implements the GitOps way, we have a lot of Application files, that need to be applied in a specific order. Furthermore the Provider and ProviderConfig manifests (which simply configure the AWS Crossplane Provider) need to reside in different directories - upbound/provider-aws-s3/provider and upbound/provider-aws-s3/config.
Our goal should be a single manifest defining the whole Crossplane setup incl. core, Provider, ProviderConfig etc. in ArgoCD
If we would use an Application that points to a directory with multiple manifests, we'll run into errors like this:
The Kubernetes API could not find aws.upbound.io/ProviderConfig for requested resource default/default. Make sure the "ProviderConfig" CRD is installed on the destination cluster.Since deployment order wouldn't be clear and the Provider manifests need to be fully deployed before the ProviderConfig - otherwise the deployment fails because of missing CRDs.
Wouldn't be Argo's SyncWaves feature a great match for that issue?
The ArgoCD docs have a great video explaining SyncWaves and Hooks: https://www.youtube.com/watch?v=zIHe3EVp528
Another great SyncWave tutorial can be found here https://redhat-scholars.github.io/argocd-tutorial/argocd-tutorial/04-syncwaves-hooks.html
Sadly using Argo's SyncWaves feature alone doesn't really help here, if we use them at the Application level. I had a hard time figuring that one out, but to really use the SyncWaves feature, we would need to use the annotations like metadata: annotations: argocd.argoproj.io/sync-wave: "2" on every of the Crossplane Provider's Kubernetes objects (and thus alter the manifests to add the annotation).
Now there are multiple patterns you can use to manage multiple ArgoCD application. You can for example go with the App of Apps Pattern or with ApplicationSets, which moved into the ArgoCD main project around version 2.6.
You'd might say: ApplicationSets is the way to go today. But App of Apps is not deprecated argoproj/argo-cd#11892 (comment) The exact same GitHub issue shows our discussion. From it I would extract the following TLDR: If you want to bootstrap a cluster (e.g. installing tools like Crossplane), the App of Apps feature together with it's support for SyncWaves is pretty handsome. That might be the reason, the feature is described inside the operator-manual/cluster-bootstrapping part of the docs: https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern
If you want to get your teams enabled to deploy their apps in a GitOps fashion (incl. self-service) and want a great way to use multiple manifests in apps also from within monorepos (e.g. backend, frontend, db), then the ApplicationSet feature is match for you. It also generates the Application manifests automatically leveraging it's many generators, like Git Generator: Directories, Git Generator: Files and so on. My colleague Daniel Häcker wrote a great post about that topic.
As we're focussing on bootstrapping our cluster with ArgoCD and Crossplane, let's go with the App of Apps Pattern here.
ArgoCD Applications can be used in ArgoCD Applications - since they are normal Kubernetes CRDs.
Therefore let's define a new top level Application that manages the whole Crossplane setup incl. core, Provider, ProviderConfig etc.
I created my App of Apps definition in argocd/crossplane-bootstrap.yaml:
# The ArgoCD App of Apps for all Crossplane components
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
path: argocd/crossplane-bootstrap
destination:
server: https://kubernetes.default.svc
namespace: crossplane-system
syncPolicy:
automated:
prune: true
syncOptions:
- CreateNamespace=true
retry:
limit: 1
backoff:
duration: 5s
factor: 2
maxDuration: 1mThis Application will look for manifests at argocd/crossplane-bootstrap in our repository https://github.com/jonashackt/crossplane-argocd. And there all our Crossplane components are already defined as ArgoCD Application manifests.
Also don't forget to define the finalizers finalizers: - resources-finalizer.argocd.argoproj.io. Otherwise the Applications managed by this App of Apps won't be deleted and will still be running, if you delete just the App of Apps!
Voilá. Now we need to use Argo's SyncWaves feature as already mentioned to define, which ArgoCD Application (representing a Crossplane component each) needs to be deployed by Argo in which exact order.
First we need to deploy the Crossplane Helm Secret, so we add the annotations: argocd.argoproj.io/sync-wave configuration to it's metadata:
metadata:
annotations:
argocd.argoproj.io/sync-wave: "0"We use sync-wave: "0" here, to define it as the earliest stage of Argo deployment (you could use negative numbers though, but for simplicity we start at zero).
Then we need to deploy the Crossplane core components, defined in argocd/crossplane-bootstrap/crossplane.yaml. There we add the next SyncWave as sync-wave: "1":
metadata:
annotations:
argocd.argoproj.io/sync-wave: "1"You get the point! We also add the sync-wave annotation to the AWS Provider in argocd/crossplane-bootstrap/crossplane-provider-aws.yaml and the ProviderConfig at argocd/crossplane-bootstrap/crossplane-provider-config-aws.yaml.
Now we should be able to finally apply our Crossplane App of Apps in Argo:
kubectl apply -n argocd -f argocd/crossplane-bootstrap.yaml And like magic all our Crossplane components get deployed step by step in correct order:
Now if we have a look into crossplane App of Apps we see all the needed components to deploy a running Crossplane installation using ArgoCD (which I found is super nice):
Ok, enough theory :)) Let's create a pipeline that shows stuff works. Let's introduce a .github/workflows/crossplane-argocd.yml:
name: crossplane-argocd
on: [push]
env:
KIND_NODE_VERSION: v1.29.0
# AWS
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: 'eu-central-1'
jobs:
provision:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@master
- name: Spin up kind via brew
run: |
echo "--- Create kind cluster"
kind create cluster --image "kindest/node:$KIND_NODE_VERSION" --wait 5m
echo "--- Let's try to access our kind cluster via kubectl"
kubectl get nodes
- name: Install ArgoCD into kind
run: |
echo "--- Create argo namespace and install it"
kubectl create namespace argocd
echo " Install & configure ArgoCD via Kustomize - see https://stackoverflow.com/a/71692892/4964553"
kubectl apply -k argocd/install
echo "--- Wait for Argo to become ready"
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server --namespace argocd --timeout=300s
- name: Prepare crossplane AWS Secret
run: |
echo "--- Create aws-creds.conf file"
echo "[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
" > aws-creds.conf
echo "--- Create a namespace for crossplane"
kubectl create namespace crossplane-system
echo "--- Create AWS Provider secret"
kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf
- name: Use ArgoCD's AppOfApps pattern to deploy all Crossplane components
run: |
echo "--- Let Argo do it's magic installing all Crossplane components"
kubectl apply -n argocd -f argocd/crossplane-bootstrap.yaml
- name: Check crossplane status
run: |
echo "--- Wait for crossplane to become ready (now prefaced with until as described in https://stackoverflow.com/questions/68226288/kubectl-wait-not-working-for-creation-of-resources)"
until kubectl wait --for=condition=PodScheduled pod -l app=crossplane --namespace crossplane-system --timeout=120s > /dev/null 2>&1; do : ; done
kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s
echo "--- Wait until AWS Provider is up and running (now prefaced with until to prevent Error from server (NotFound): providers.pkg.crossplane.io 'provider-aws-s3' not found)"
until kubectl get provider/provider-aws-s3 > /dev/null 2>&1; do : ; done
kubectl wait --for=condition=healthy --timeout=180s provider/provider-aws-s3
kubectl get all -n crossplane-systemBe sure to create both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY configured as GitHub Repository Secrets:
Also make sure to have your Default region configured as a env: variable.
Let's create a simple S3 Bucket in AWS. The docs tell us, which config we need. infrastructure/bucket.yaml features a super simply example:
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
name: crossplane-argocd-s3-bucket
spec:
forProvider:
region: eu-central-1
providerConfigRef:
name: defaultSince we're using Argo, we should deploy our Bucket as Argo Application too. I created a new folder argocd/infrastructure
here, since the Crossplane provisioned infrastructure may not automatically be part of the bootstrap App of Apps.
So here's our Argo Application for all the Crossplane managed infrastructure that may come: argocd/infrastructure/aws-s3.yaml:
# The ArgoCD Application for all Crossplane Managed Resources
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane-s3
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
path: infrastructure
destination:
namespace: default
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mApply it with:
kubectl apply -f argocd/infrastructure/aws-s3.yamlIf everything went fine, the Argo app should look Healthy like this:
And inside the AWS console, there should be a new S3 Bucket provisioned:
CI pushes Secrets into the cluster via kubectl apply...
This is an anti-GitOps pattern - so let's do something different!
TODO: Insert why here :) GitOps Pull instead of Push...
After reading through lot's of "How to manage Secrets with GitOps articles" (like this, this and this to name a few), I found that there's currently no widly accepted way of doing it. But there are some recommendations. E.g. checking Secrets into Git (although encrypted) using Sealed Secrets or SOPS/KSOPS might seem like the kind of easiest solution in the first place. But they have their own caveats in the long therm. Think of multiple secrets defined in multiple projects used by multiple teams all over your Git repositories - and now do a secret or key rotation...
The TLDR; of most (recent) articles and GitHub discussions I distilled for me is: Use an external secret store and connect that one to your ArgoCD managed cluster. With an external secret store you get key rotation, support for serving secrets as symbolic references, usage audits and so on. Even in the case of secret or key compromisation you mostly get proven mitigations paths.
There is a huge list of possible plugins or operators helping to integrate your ArgoCD managed cluster with an external secret store. You can for example have a look onto the list featured in the Argo docs. I had a look on some promising candidates:
A lightweight solution could be https://github.com/argoproj-labs/argocd-vault-plugin / https://argocd-vault-plugin.readthedocs.io/en/stable/. It supports multiple backends like AWS Secrets Manager, Azure Key Vault, Hashicorp Vault etc. But the installation isn't that lightweight, because we need to download the Argo Vault Plugin in a volume and inject it into the argocd-repo-server (although there are pre-build Kustomize manifests available) by creating a custom argocd-repo-server image with the plugin and supporting tools pre-installed... Also a newer sidecar option is available, which nevertheless has it's own caveats.
There's also Hashicorps own Vault Agent and the Secrets Store CSI Driver, who both handle secrets without the need for Kubernetes Secrets. The first works with a per-pod based sidecar approach to connect to Vault via the agent and the latter uses the Container Storage Interface.
Both look nice, but I found the following the most promising solution right now: The External Secrets Operator (ESO). Featuring also a lot of GitHub stars External Secrets simply creates a Kubernetes Secret for each external secret. According to the docs:
"ExternalSecret, SecretStore and ClusterSecretStore that provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you."
And what's also promising, the community seems to be growing rapidly:
"Multiple people and organizations are joining efforts to create a single External Secrets solution based on existing projects."
The External Secrets Operator supports a multitude of tools for secret management! Just have a look at the docs & you'll see more than 20 tools supported, featuring the well known AWS Secretes Manager, Azure Key Vault, Hashicorp Vault, Akeyless and so on.
And as I like to show solutions that are fully cromprehensible - ideally without a creditcard - I was on the lookout for a tool, that had a small free plan. But without the need to selfhost the solution, since that would be out of scope for this project. At first glance I thought that Hashicorp's Vault Secrets as part of the Hashicorp Cloud Platform (HCP) would be a great choice since so many projects love and use Vault. But sadly External Secrets Operator currently doesn't support HCP Vault Secrets and I would have been forced to switch to Hashicorp Vault Secrets Operator (VSO), which is for sure also an interesting project. But I wanted to stick with the External Secrets Operator since it's wide support for providers and it looks as it could develop into the defacto standard in external secrets integration in Kubernetes.
So I thought the exact secret management tool I use in this case is not that important and I trust my readers that they will choose the provider that suites them the most. That beeing said I chose Doppler with their generous free Developer plan.
As External-Secrets introduce more complexity to our setup, I decided to divide the crossplane only solution from the more advanced using External Secrets Operator. Therefore the argocd directory now looks like this:
$ tree
.
├── crossplane-bootstrap
│ ├── crossplane.yaml
│ ├── crossplane-helm-secret.yaml
│ └── crossplane-provider-aws.yaml
├── crossplane-eso-bootstrap
│ ├── crossplane.yaml
│ ├── crossplane-helm-secret.yaml
│ ├── crossplane-provider-aws.yaml
│ ├── external-secrets-config.yaml
│ └── external-secrets-operator.yaml
├── crossplane-bootstrap.yaml
├── crossplane-eso-app-of-apps.yaml
...Where crossplane-bootstrap and the corresponding crossplane-bootstrap.yaml feature the crossplane only solution - and crossplane-eso-bootstrap with it's crossplane-eso-app-of-apps.yaml App-of-Apps counterpart feature the more advanced ESO solution.
So let's create our first secret in Doppler. If you haven't already done so sign up at https://dashboard.doppler.com (e.g. with your GitHub account). Then click on Projects on the left navigation bar and on the + to create a new project. In this example I named it according to this example project: crossplane-argocd.
Doppler automatically creates well known environments for us: development, staging and production. To create a new Secret, choose a environment and click on Add First Secret. Now give it the key CREDS. The value will be a multiline value. Just like it is stated in the crossplane docs, we should have an aws-creds.conf file created already (that we don't want to check into source control):
echo "[default]
aws_access_key_id = $(aws configure get aws_access_key_id)
aws_secret_access_key = $(aws configure get aws_secret_access_key)
" > aws-creds.confCopy the contents of the aws-creds.conf into the value field in Doppler. The Crossplane AWS Provider or rather it's ProviderConfig will later consume the secret just like it is as multiline text:
Don't forget so click on save.
As stated in the External Secrets docs, we need to create a Doppler Service Token in order to be albe to connect to Doppler later on.
In Doppler Service Tokens are created on project level - inside a specific environment, where we already created our secrets. As I created my secrets in the dev environment, I create the Service Token also there. Simply head over to your Doppler project, select the environment you created your secrets in and click on Access. Here you should find a button called + Generate to create a new Service Token. Click the button and create a Service Token with read access and no expiration and copy it somewhere locally.
In order to be able to let the External Secrets Operator access Doppler, we need to create a Kubernetes Secret containing the Doppler Service Token:
kubectl create secret generic \
doppler-token-auth-api \
--from-literal dopplerToken="dp.st.xxxx"https://external-secrets.io/latest/introduction/getting-started/
Installing External Secrets Operator in a GitOps fashion & have updates managed by Renovate, we can use the method already applied to Crossplane and explained in https://stackoverflow.com/a/71765472/4964553. Therefore we create a simple Helm chart at external-secrets/Chart.yaml:
apiVersion: v2
type: application
name: external-secrets
version: 0.0.0 # unused
appVersion: 0.0.0 # unused
dependencies:
- name: external-secrets
repository: https://charts.external-secrets.io
version: 0.9.11Now telling ArgoCD where to find our simple external-secrets Helm Chart, we again use Argo's Application manifest in argocd/crossplane-eso-bootstrap/external-secrets-operator.yaml:
# The ArgoCD Application for external-secrets-operator
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: external-secrets-operator
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
path: external-secrets/install
destination:
server: https://kubernetes.default.svc
namespace: external-secrets
syncPolicy:
automated:
prune: true
syncOptions:
- CreateNamespace=true
retry:
limit: 1
backoff:
duration: 5s
factor: 2
maxDuration: 1mWe define the SyncWave to deploy external-secrets before every other Crossplane component via annotations: argocd.argoproj.io/sync-wave: "-1".
Just for checking if it works, we can use a kubectl apply -f argocd/crossplane-bootstrap/external-secrets.yaml to apply it to our cluster. If everything went correctly, there should be a new ArgoCD Application featuring a bunch of CRDs, some roles and three Pods: external-secrets, external-secrets-webhook and external-secrets-cert-controller:
https://external-secrets.io/latest/provider/doppler/#authentication
https://external-secrets.io/latest/introduction/overview/#secretstore
The idea behind the
SecretStoreresource is to separate concerns of authentication/access and the actual Secret and configuration needed for workloads. The ExternalSecret specifies what to fetch, the SecretStore specifies how to access.
In this project I opted for the similar ClusterSecretStore. As the docs state:
"The ClusterSecretStore is a global, cluster-wide SecretStore that can be referenced from all namespaces. You can use it to provide a central gateway to your secret provider."
Sounds like a good fit for our setup. But you can also opt for the namespaced SecretStore too. Our ClusterSecretStore reside here external-secrets/config/cluster-secret-store.yaml:
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: doppler-auth-api
spec:
provider:
doppler:
auth:
secretRef:
dopplerToken:
name: doppler-token-auth-api
key: dopplerToken
namespace: defaultDon't forget to configure a namespace for the doppler-token-auth-api Secret we created earlier. Otherwise we'll run into errors like:
admission webhook "validate.clustersecretstore.external-secrets.io" denied the request: invalid store: cluster scope requires namespace (retried 1 times).The External Secrets Operator will create a Secret that's similar to the one mentioned in the Crossplane docs (if you decode it), but with the uppercase CREDS key we used in Doppler:
CREDS: |+
[default]
aws_access_key_id = yourAccessKeyIdHere
aws_secret_access_key = yourSecretAccessKeyHerehttps://external-secrets.io/latest/introduction/overview/#externalsecret
https://external-secrets.io/latest/provider/doppler/#use-cases
As we already defined how the external secret store (Doppler) could be accessed (using our ClusterSecretStore CRD) should now specify which secrets to fetch using the ExternalSecret CRD. Therefore let's create a external-secrets/config/external-secret.yaml:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: auth-api-db-url
spec:
secretStoreRef:
kind: ClusterSecretStore
name: doppler-auth-api
# access our 'CREDS' key in Doppler
dataFrom:
- find:
path: CREDS
# Create a Kubernetes Secret just as we're used to without External Secrets Operator
target:
name: aws-secrets-from-dopplerWe created a CREDS secret in Doppler, so the ExternalSecret looks for this exact path.
We also need to create a ArgoCD Application so that Argo will deploy both ClusterSecretStore and ExternalSecret for us :) Therefore I created argocd/crossplane-eso-bootstrap/external-secrets-config.yaml:
# The ArgoCD Application for external-secrets-operator
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: external-secrets-config
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
path: external-secrets
destination:
server: https://kubernetes.default.svc
namespace: external-secrets
syncPolicy:
automated:
prune: true
syncOptions:
- CreateNamespace=true
retry:
limit: 1
backoff:
duration: 5s
factor: 2
maxDuration: 1mOur ClusterSecretStore and ExternalSecrets deployment in Argo looks like this:
But the deployment doesn't run flawless, although configured as argocd.argoproj.io/sync-wave: "-1" right AFTER the external-secrets Argo Application, which deployes the External Secrets components:
Failed sync attempt to 603cce3949c2a916f51f3917e87aa814698e5f92: one or more objects failed to apply, reason: Internal error occurred: failed calling webhook "validate.externalsecret.external-secrets.io": failed to call webhook: Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-externalsecret?timeout=5s": dial tcp 10.96.42.44:443: connect: connection refused,Internal error occurred: failed calling webhook "validate.clustersecretstore.external-secrets.io": failed to call webhook: Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-clustersecretstore?timeout=5s": dial tcp 10.96.42.44:443: connect: connection refused (retried 1 times).It seems that our external-secrets-webhook isn't healthy already, but the ClusterSecretStore & the ExternalSecret already want to access the webhook. So we may need to wait for the external-secrets-webhook to be really available before we deploy our external-secrets-config?!
Therefore let's give our external-secrets-config more syncPolicy.retry.limit:
syncPolicy:
...
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mWe need to change our ProviderConfig at upbound/provider-aws/provider-eos/provider-config-aws.yaml to use another Secret name and namespace:
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
namespace: external-secrets
name: aws-secrets-from-doppler
key: CREDSWith this final piece our setup should be complete to be able to provision some infrastructure with ArgoCD and Crossplane!
Here are all components together we deployed so far using Argo:
Deploying our argocd/infrastructure/aws-s3.yaml should also work as expected:
kubectl apply -f argocd/infrastructure/aws-s3.yamlIf everything went fine, the Argo app should look Healthy like this:
And inside the AWS console, there should be a new S3 Bucket provisioned:
Let's create another pipeline that shows what differences are to the deployment without External Secrets Operator. Let's introduce a .github/workflows/crossplane-argocd-external-secrets:
name: crossplane-argocd-external-secrets
on: [push]
env:
KIND_NODE_VERSION: v1.29.0
# Doppler
DOPPLER_SERVICE_TOKEN: ${{ secrets.DOPPLER_SERVICE_TOKEN }}
jobs:
provision:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@master
- name: Spin up kind via brew
run: |
echo "--- Create kind cluster"
kind create cluster --image "kindest/node:$KIND_NODE_VERSION" --wait 5m
echo "--- Let's try to access our kind cluster via kubectl"
kubectl get nodes
- name: Install ArgoCD into kind
run: |
echo "--- Create argo namespace and install it"
kubectl create namespace argocd
echo " Install & configure ArgoCD via Kustomize - see https://stackoverflow.com/a/71692892/4964553"
kubectl apply -k argocd/install
echo "--- Wait for Argo to become ready"
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server --namespace argocd --timeout=300s
- name: Create Secret with the Doppler Service Token for External Secrets Operator
run: kubectl create secret generic doppler-token-auth-api --from-literal dopplerToken="$DOPPLER_SERVICE_TOKEN"
- name: Use ArgoCD's AppOfApps pattern to deploy all Crossplane components
run: |
echo "--- Let Argo do it's magic installing all Crossplane components"
kubectl apply -n argocd -f argocd/crossplane-eso-app-of-apps.yaml
- name: Check crossplane status
run: |
echo "--- Wait for crossplane to become ready (now prefaced with until as described in https://stackoverflow.com/questions/68226288/kubectl-wait-not-working-for-creation-of-resources)"
until kubectl wait --for=condition=PodScheduled pod -l app=crossplane --namespace crossplane-system --timeout=120s > /dev/null 2>&1; do : ; done
kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s
echo "--- Wait until AWS Provider is up and running (now prefaced with until to prevent Error from server (NotFound): providers.pkg.crossplane.io 'provider-aws-s3' not found)"
until kubectl get provider/provider-aws-s3 > /dev/null 2>&1; do : ; done
kubectl wait --for=condition=healthy --timeout=180s provider/provider-aws-s3
kubectl get all -n crossplane-systemBe sure to create DOPPLER_SERVICE_TOKEN as GitHub Repository Secrets.
Let's create a publicly accessible S3 bucket in our infrastructure/bucket.yaml:
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
name: crossplane-argocd-s3-bucket
spec:
forProvider:
region: eu-central-1
providerConfigRef:
name: default
---
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketPublicAccessBlock
metadata:
name: crossplane-argocd-s3-bucket-pab
spec:
forProvider:
blockPublicAcls: false
blockPublicPolicy: false
ignorePublicAcls: false
restrictPublicBuckets: false
bucketRef:
name: crossplane-argocd-s3-bucket
region: eu-central-1
---
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketOwnershipControls
metadata:
name: crossplane-argocd-s3-bucket-osc
spec:
forProvider:
rule:
- objectOwnership: ObjectWriter
bucketRef:
name: crossplane-argocd-s3-bucket
region: eu-central-1
---
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketACL
metadata:
name: crossplane-argocd-s3-bucket-acl
spec:
forProvider:
acl: "public-read"
bucketRef:
name: crossplane-argocd-s3-bucket
region: eu-central-1
---
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketWebsiteConfiguration
metadata:
name: crossplane-argocd-s3-bucket-websiteconf
spec:
forProvider:
indexDocument:
- suffix: index.html
bucketRef:
name: crossplane-argocd-s3-bucket
region: eu-central-1Also let's sync the Nuxt.js project https://github.com/jonashackt/microservice-ui-nuxt-js via the used aws s3 sync:
aws s3 sync .output/public/ s3://crossplane-argocd-s3-bucket --acl public-readAnd we should be able to access our via http://crossplane-argocd-s3-bucket.s3-website.eu-central-1.amazonaws.com
Application sources are generally Kubernetes manifests in Argo https://argo-cd.readthedocs.io/en/stable/user-guide/application_sources/
So how do we actually deploy our static website to S3?
According to argoproj/argo-cd#5052 there's the way to use custom Config Management Plugins https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/
Proposal for Parameterized Configuration Management Plugins in Argo: https://argo-cd.readthedocs.io/en/latest/proposals/parameterized-config-management-plugins/
But maybe we should simply deploy our static website to K8s as well? https://gimlet.io/blog/hosting-static-sites-on-kubernetes
https://thenewstack.io/gitops-as-an-evolution-of-kubernetes/
To be able to deploy a nested Composition like this for EKS we need to install multiple Crossplane Providers: provider-aws-ec2, provider-aws-eks, provider-aws-iam additionally to our already installed provider-aws-s3. Therefore we should enhance our concept on how to install a Provider with ArgoCD!
Since every Upbound provider family has one ProviderConfig to access the credentials, but multiple providers, it would make sense to enhance the Argo Application argocd/crossplane-bootstrap/crossplane-provider-aws.yaml to support multiple providers:
# The ArgoCD Application for all Crossplane AWS providers incl. it's ProviderConfig
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane-provider-aws
namespace: argocd
labels:
crossplane.jonashackt.io: crossplane
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: HEAD
path: upbound/provider-aws/provider
destination:
namespace: default
server: https://kubernetes.default.svc
# Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with
# 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mThus this Application simply references the folder upbound/provider-aws/provider, where all the Provider manifests can be stored:
└── provider-aws
...
├── config
│ └── provider-config-aws.yaml
...
└── provider
├── provider-aws-ec2.yaml
├── provider-aws-eks.yaml
├── provider-aws-iam.yaml
└── provider-aws-s3.yamlNow in Argo, the Application shows all available Crossplane providers:
If new Provider versions get released, you can watch Argo trying to deploy the old version vs. Crossplane deploying the new one, which leads to a degraded status of the Providers:
The problem is this error: Only one reference can have Controller set to true. Found "true" in references for Provider/provider-aws-ec2 and Provider/provider-aws-ec2:
cannot apply package revision: cannot create object: ProviderRevision.pkg.crossplane.io "provider-aws-ec2-150095bdd614" is invalid: metadata.ownerReferences: Invalid value: []v1.OwnerReference{v1.OwnerReference{APIVersion:"pkg.crossplane.io/v1", Kind:"Provider", Name:"provider-aws-ec2", UID:"30bda236-6c12-412c-a647-b96368eff8b6", Controller:(*bool)(0xc02afeb38c), BlockOwnerDeletion:(*bool)(0xc02afeb38d)}, v1.OwnerReference{APIVersion:"pkg.crossplane.io/v1", Kind:"Provider", Name:"provider-aws-ec2", UID:"ee890f53-7590-4957-8f81-e92b931c4e8d", Controller:(*bool)(0xc02afeb38e), BlockOwnerDeletion:(*bool)(0xc02afeb38f)}}: Only one reference can have Controller set to true. Found "true" in references for Provider/provider-aws-ec2 and Provider/provider-aws-ec2Therefore we should change some options regarding the Provider upgrades in our Provider configurations:
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: upbound-provider-aws-ec2
spec:
package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.1.1
packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache.
revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate & healthy
revisionHistoryLimit: 1As we're doing GitOpsified Crossplane with ArgoCD, we should configure the packagePullPolicy to IfNotPresent instead of Always (which means " Check for new packages every minute and download any matching package that isn’t in the cache", see https://docs.crossplane.io/master/concepts/packages/#configuration-package-pull-policy) - BUT leave the revisionActivationPolicy to Automatic! Since otherwise, the Provider will never get active and healty! See https://docs.crossplane.io/master/concepts/packages/#revision-activation-policy), but I didn't find it documented that way!
See also https://stackoverflow.com/a/78230499/4964553
Now with packagePullPolicy: IfNotPresent & revisionActivationPolicy: Automatic to do a Provider version upgrade, we simply need to upgrade the spec.package version number:
spec:
package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1 # --> Upgraded to 1.2.1
packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache.
revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate & healthy
revisionHistoryLimit: 1We need to commit the change as always, but also be a bit patient here with Argo and Crossplane to initiate and do the update for us. Look at a kubectl get providerrevisions. Even after the update commited and registered by Argo, Crossplane will take it's time. First it looks like this:
k get providerrevisions
NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
provider-aws-ec2-3d66ea2d7903 True 1 xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1 Active 1 1 5m31s
provider-aws-eks-5021e69b327c True 2 xpkg.upbound.io/upbound/provider-aws-eks:v1.2.1 Inactive 1 1 4m11s
provider-aws-eks-fbb6768e46c0 True 3 xpkg.upbound.io/upbound/provider-aws-eks:v1.1.1 Active 1 1 30m
provider-aws-iam-9565c6312cd0 True 1 xpkg.upbound.io/upbound/provider-aws-iam:v1.1.1 Active 1 1 30m
provider-aws-s3-6ca829a5198b True 1 xpkg.upbound.io/upbound/provider-aws-s3:v1.1.1 Active 1 1 30m
upbound-provider-family-aws-7cc64a779806 True 1 xpkg.upbound.io/upbound/provider-family-aws:v1.2.1 Active 30mNow after a while and some events (look at them in k9s for example):
Some time later the new Provider version should be the Active one:
k get providerrevisions
NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
provider-aws-ec2-3d66ea2d7903 True 1 xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1 Active 1 1 6m52s
provider-aws-eks-5021e69b327c True 4 xpkg.upbound.io/upbound/provider-aws-eks:v1.2.1 Active 1 1 5m32s
provider-aws-eks-fbb6768e46c0 True 3 xpkg.upbound.io/upbound/provider-aws-eks:v1.1.1 Inactive 1 1 31m
provider-aws-iam-9565c6312cd0 True 1 xpkg.upbound.io/upbound/provider-aws-iam:v1.1.1 Active 1 1 31m
provider-aws-s3-6ca829a5198b True 1 xpkg.upbound.io/upbound/provider-aws-s3:v1.1.1 Active 1 1 31m
upbound-provider-family-aws-7cc64a779806 True 1 xpkg.upbound.io/upbound/provider-family-aws:v1.2.1 Active 31mAnd luckily without any errors like mentioned above!
I offloaded all the EKS Nested Composition as a separate repository, which publishes a Crossplane Configuration Package as OCI image: https://github.com/jonashackt/crossplane-eks-cluster
We should be able to use it via the following Configuration:
apiVersion: pkg.crossplane.io/v1
kind: Configuration
metadata:
name: crossplane-eks-cluster
spec:
package: ghcr.io/jonashackt/crossplane-eks-cluster:v0.0.2Let's try to apply it to our cluster and use it:
kubectl apply -f upbound/provider-aws/apis/crossplane-eks-cluster.yamlWe should make our newly created EKS Configuration package viewable in Argo!
Therefore let's create a new folder argocd/crossplane-apis and a new Application argocd/crossplane-apis/crossplane-apis.yaml:
# The ArgoCD Application for all Crossplane Managed Resources
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane-apis
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: app-deployment
path: upbound/provider-aws/apis
destination:
namespace: default
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mNow we can apply this crossplane-apis Application to our ArgoCD:
kubectl apply -f argocd/crossplane-apis/crossplane-apis.yamlThat's pretty cool: Now we see all of our installed APIs as Argo Apps:
Now we use our installed APIs to create a Claim in infrastructure/eks/deploy-target-eks.yaml:
# Use the spec.group/spec.versions[0].name defined in the XRD
apiVersion: k8s.crossplane.jonashackt.io/v1alpha1
# Use the spec.claimName or spec.name specified in the XRD
kind: KubernetesCluster
metadata:
namespace: default
name: deploy-target-eks
spec:
id: deploy-target-eks
parameters:
region: eu-central-1
nodes:
count: 3
# Crossplane creates the secret object in the same namespace as the Claim
# see https://docs.crossplane.io/latest/concepts/claims/#claim-connection-secrets
writeConnectionSecretToRef:
name: eks-cluster-kubeconfigDon't apply it directly, we'll create a Argo App in a second.
We should also create a Argo App for our EKS cluster Composite Resource Claim to see our infrastructure beeing deployed visually :)
Therefore we create the Application argocd/infrastructure/aws-eks.yaml:
# The ArgoCD Application for all Crossplane Managed Resources
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane-eks
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: app-deployment
path: infrastructure/eks
destination:
namespace: default
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mNow this will deploy our EKS cluster using ArgoCD and our EKS Configuration Package based Nested EKS Composition https://github.com/jonashackt/crossplane-eks-cluster:
kubectl apply -f argocd/infrastructure/aws-eks.yaml
https://dev.to/thenjdevopsguy/registering-a-new-cluster-with-argocd-12mn
https://www.padok.fr/en/blog/argocd-eks
https://itnext.io/argocd-setup-external-clusters-by-name-d3d58a53acb0
Before using argocd CLI, be sure to have logged the CLI into the current argocd-server instance. Therefore have a port forward ready
$ kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8080:80
$ argocd login localhost:8080 --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo) --insecure
'admin:login' logged in successfully
Context 'localhost:8080' updatedhttps://argo-cd.readthedocs.io/en/stable/user-guide/commands/argocd_cluster_add/
argocd cluster add deploy-target-eksThis will add a few resources to the Target cluster like ServiceAccount, ClusterRole and ClusterRoleBinding:
$ argocd cluster add deploy-target-eks
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `deploy-target-eks` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0002] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0002] ClusterRole "argocd-manager-role" updated
INFO[0002] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://736F91649BD7B7A70846AD9F8363EDA8.yl4.eu-central-1.eks.amazonaws.com' addedThe new cluster becomes visible in the Argo web ui also:
Is there only the argocd cluster add command or could we achieve that using a manifest?
Maybe the Crossplane ArgoCD Provider has the crucial Manifest for us? See crossplane-contrib/provider-argocd#18 and https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd/v0.6.0/resources/cluster.argocd.crossplane.io/Cluster/v1alpha1
You might already wondered, what the Crossplane ArgoCD provider is about: https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd
Thats what the project README says https://github.com/crossplane-contrib/provider-argocd about it's purpose:
Custom Resource Definitions (CRDs) that model Argo CD resources
With this we can create a Cluster which is able to represent the EKS cluster we just created. This Cluster itself can be referenced again by an ArgoCD Application managing for example our Spring Boot application we finally want to deploy.
The whole process might become more straightforward in the future: crossplane-contrib/provider-argocd#14 (comment)
So let's install the Crossplane ArgoCD provider, which is a community contribution project. Thus we create the crossplane-contrib folder containing a provider-argocd folder, where the new Provider should reside as provider-argocd.yaml in the provider dir:
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-argocd
spec:
package: xpkg.upbound.io/crossplane-contrib/provider-argocd:v0.6.0
packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache.
revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate & healthy
revisionHistoryLimit: 1As we want to manage the Provider also using Argo, we need to create a new Argo Application. It get's the same argocd.argoproj.io/sync-wave: "4" as the other providers in our setup:
# The ArgoCD Application for all Crossplane Community contribution Providers needed in the setup
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane-provider-contrib
namespace: argocd
labels:
crossplane.jonashackt.io: crossplane
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "4"
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: app-deployment
path: crossplane-contrib
destination:
namespace: default
server: https://kubernetes.default.svc
# Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with
# 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mApply it via the ususal bootstrap setup:
kubectl apply -f argocd/crossplane-eso-bootstrap.yamlArgo should now list our new Provider:
As stated in the docs https://github.com/crossplane-contrib/provider-argocd?tab=readme-ov-file#create-a-new-argo-cd-user we need to create an API token for the ProviderConfig of the Crossplane ArgoCD provider to use. To create the API token, we first need to create a new ArgoCD user.
Therefore we enhance the ConfigMap argocd-cm again:
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
data:
...
# add an additional local user with apiKey capabilities for provider-argocd
# see https://github.com/crossplane-contrib/provider-argocd?tab=readme-ov-file#getting-started-and-documentation
accounts.provider-argocd: apiKey As the ArgoCD docs about user management state this is not enough:
"each of those users will need additional RBAC rules set up, otherwise they will fall back to the default policy specified by policy.default field of the
argocd-rbac-cmConfigMap."
So we need to create another Kustomization patch for the argocd-rbac-cm ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-rbac-cm
data:
# For the provider-argocd user we need to add an additional rbac-rule
# see https://github.com/crossplane-contrib/provider-argocd?tab=readme-ov-file#create-a-new-argo-cd-user
policy.csv: "g, provider-argocd, role:admin" Don't forget to add this patch into the []kustomization.yaml](argocd/install/kustomization.yaml)!
First we need to access the argocd-server Service somehow. In the simplest manner we create a port forward:
kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443We also need to have the ArgoCD password ready:
ARGOCD_ADMIN_SECRET=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo)Now we create a temporary JWT token for the provider-argocd user we just created (we need to have jq installed for this command to work):
# be sure to have jq installed via 'brew install jq' or 'pamac install jq' etc.
ARGOCD_ADMIN_TOKEN=$(curl -s -X POST -k -H "Content-Type: application/json" --data '{"username":"admin","password":"'$ARGOCD_ADMIN_SECRET'"}' https://localhost:8443/api/v1/session | jq -r .token)Now we finally create an API token without expiration that can be used by provider-argocd:
ARGOCD_API_TOKEN=$(curl -s -X POST -k -H "Authorization: Bearer $ARGOCD_ADMIN_TOKEN" -H "Content-Type: application/json" https://localhost:8443/api/v1/account/provider-argocd/token | jq -r .token)You can double check in the ArgoCD UI at Settings/Accounts if the Token got created:
The ARGOCD_API_TOKEN can be used to create a Kubernetes Secret for the Crossplane ArgoCD Provider:
kubectl create secret generic argocd-credentials -n crossplane-system --from-literal=authToken="$ARGOCD_API_TOKEN"I also added all these steps to a script create-argocd-api-token-secret.sh so that we're able to run all the steps without much thinking:
#!/usr/bin/env bash
set -euo pipefail
echo "### This Script will prepare a K8s Secret with a ArgoCD API Token for Crossplane ArgoCD Provider (be sure to have a service/argocd-server 8443:443 running before)"
echo "--- Extract ArgoCD password"
ARGOCD_ADMIN_SECRET=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo)
echo "--- Create temporary JWT token for the provider-argocd user"
ARGOCD_ADMIN_TOKEN=$(curl -s -X POST -k -H "Content-Type: application/json" --data '{"username":"admin","password":"'$ARGOCD_ADMIN_SECRET'"}' https://localhost:8443/api/v1/session | jq -r .token)
echo "--- Create ArgoCD API Token"
ARGOCD_API_TOKEN=$(curl -s -X POST -k -H "Authorization: Bearer $ARGOCD_ADMIN_TOKEN" -H "Content-Type: application/json" https://localhost:8443/api/v1/account/provider-argocd/token | jq -r .token)
echo "--- Already create a namespace for crossplane for the Secret (if not already exist, see https://stackoverflow.com/a/65411733/4964553)"
kubectl create namespace crossplane-system --dry-run=client -o yaml | kubectl apply -f -
echo "--- Create Secret containing the ARGOCD_API_TOKEN for Crossplane ArgoCD Provider"
kubectl create secret generic argocd-credentials -n crossplane-system --from-literal=authToken="$ARGOCD_API_TOKEN"Now all the steps to create the Secret for the Crossplane argocd-provider can be run via a simple:
kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443
./create-argocd-api-token-secret.shThe
kubectl port-forwardcommand can be run in subshell appending&+Ctrl-Cand beeing deleted after running create-argocd-api-token-secret.sh viafg 1%(where 1 is the subshell id, obtain viajobscommand) +Ctrl-C(see https://stackoverflow.com/a/72983554/4964553 & https://www.baeldung.com/linux/foreground-background-process).
Our GitHub Actions workflow now also integrates the Secret creation:
- name: Prepare Secret with ArgoCD API Token for Crossplane ArgoCD Provider
run: |
echo "--- Access the ArgoCD server with a port-forward in the background, see https://stackoverflow.com/a/72983554/4964553"
kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443 &
echo "--- Extract ArgoCD password"
ARGOCD_ADMIN_SECRET=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo)
echo "--- Create temporary JWT token for the `provider-argocd` user"
ARGOCD_ADMIN_TOKEN=$(curl -s -X POST -k -H "Content-Type: application/json" --data '{"username":"admin","password":"'$ARGOCD_ADMIN_SECRET'"}' https://localhost:8443/api/v1/session | jq -r .token)
echo "--- Create ArgoCD API Token"
ARGOCD_API_TOKEN=$(curl -s -X POST -k -H "Authorization: Bearer $ARGOCD_ADMIN_TOKEN" -H "Content-Type: application/json" https://localhost:8443/api/v1/account/provider-argocd/token | jq -r .token)
echo "--- Already create a namespace for crossplane for the Secret"
kubectl create namespace crossplane-system
echo "--- Create Secret containing the ARGOCD_API_TOKEN for Crossplane ArgoCD Provider"
kubectl create secret generic argocd-credentials -n crossplane-system --from-literal=authToken="$ARGOCD_API_TOKEN"Now finally we're able to tell our Crossplane ArgoCD Provider where it should obtain the ArgoCD API Token from. Let's create a ProviderConfig at crossplane-contrib/provider-argocd/config/provider-config-argocd.yaml:
apiVersion: argocd.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
name: argocd-provider
spec:
credentials:
secretRef:
key: authToken
name: argocd-credentials
namespace: crossplane-system
source: Secret
insecure: true
plainText: false
serverAddr: argocd-server.argocd.svc:443We should also create a ArgoCD Application for the ProviderConfig:
# The ArgoCD Application for the Crossplane ArgoCD providers ProviderConfig
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: crossplane-provider-argocd-config
namespace: argocd
labels:
crossplane.jonashackt.io: crossplane
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "5"
spec:
project: default
source:
repoURL: https://github.com/jonashackt/crossplane-argocd
targetRevision: app-deployment
path: crossplane-contrib/provider-argocd/config
destination:
namespace: default
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mNow we're where we wanted to be: We can finally create a Cluster in ArgoCD referencing the Crossplane created EKS cluster. Therefore we make use of the Crossplane ArgoCD Providers Cluster CRD in our infrastructure/eks/cluster.yaml:
apiVersion: cluster.argocd.crossplane.io/v1alpha1
kind: Cluster
metadata:
name: argo-reference-deploy-target-eks
labels:
purpose: dev
spec:
forProvider:
config:
kubeconfigSecretRef:
key: kubeconfig
name: eks-cluster-kubeconfig # Secret containing our kubeconfig to access the Crossplane created EKS cluster
namespace: default
name: deploy-target-eks # name of the Cluster registered in ArgoCD
providerConfigRef:
name: argocd-providerBe sure to provide the
forProvider.nameAFTER theforProvider.config, otherwise the name of the Cluster will we overwritten by the EKS server address from the kubeconfig!
The providerConfigRef.name.argocd-provider references our ProviderConfig, which gives the Crossplane ArgoCD Provider the rights (via our API Token) to change the ArgoCD Server configuration (and thus add a new Cluster).
As the docs state https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd/v0.6.0/resources/cluster.argocd.crossplane.io/Cluster/v1alpha1
`kubeconfigSecretRef' is described at what we need:
KubeconfigSecretRef contains a reference to a Kubernetes secret entry that contains a raw kubeconfig in YAML or JSON.
The Secret containing the exact EKS kubeconfig is named eks-cluster-kubeconfig by our EKS Configuration and resides in the default namespace.
Let's create the Cluster manually for now:
kubectl apply -f infrastructure/eks/cluster.yamlIf everything went correctly, a kubectl get cluster should state READY and SYNCED as True:
kubectl get cluster
NAME READY SYNCED AGE
argo-reference-deploy-target-eks True True 21sAnd also in the ArgoCD UI you should find the newly registerd Cluster now at Settings/Clusters:
To also have the ArgoCD Cluster configuration available as Argo Application, it's enough to have the cluster.yaml be placed together with the deploy-target-eks.yaml in infrastructure/eks directory. The Argo Application argocd/infrastructure/aws-eks.yaml will pick it up:
It won't be available until the EKS cluster is fully deployed, thus producing some CannotCreateExternalResource events:
Now we finally finally have the cluster dynamically referencable via the Crossplane ArgoCD Provider created Cluster object with the name deploy-target-eks! Let's try to use that in an Application deployment.
In order to deploy our example app https://github.com/jonashackt/microservice-api-spring-boot
we need the corresponding Kubernetes deployment manifests, provided by https://github.com/jonashackt/microservice-api-spring-boot-config
Having both in place, we can craft a matching ArgoCD Application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: microservice-api-spring-boot
namespace: argocd
labels:
crossplane.jonashackt.io: application
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/jonashackt/microservice-api-spring-boot-config
targetRevision: HEAD
path: deployment
destination:
namespace: default
server: deploy-target-eks
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mAs you can see we use our Cluster name deploy-target-eks as spec.destination.server.
Now let's finally deploy our app via:
kubectl apply -f argocd/applications/microservice-api-spring-boot.yamlBut we get the following error in Argo:
cluster 'deploy-target-eks' has not been configuredLooking into the docs we get the point we're missing:
destinationreference to the target cluster and namespace. For the cluster one of server or name can be used, [...] Under the hood when the server is missing, it is calculated based on the name and used for any operations.
Thus we need to use spec.destination.name instead of spec.destination.server. This will then look into Argo's Cluster list and should find our deploy-target-eks.
Now the working manifest looks like this:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: microservice-api-spring-boot
namespace: argocd
labels:
crossplane.jonashackt.io: application
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/jonashackt/microservice-api-spring-boot-config
targetRevision: HEAD
path: deployment
destination:
namespace: default
name: deploy-target-eks
syncPolicy:
automated:
prune: true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 1mkubectl apply -f argocd/applications/microservice-api-spring-boot.yamlIf everything went fine, our App should be deployed by ArgoCD:
Finally a full cycle is possible - from full bootstrap of ArgoCD & Crossplane Managed cluster to target EKS cluster creation in AWS via Crossplane to configuring that one in Argo and finally deploying an App dynamically referencing this Cluster!
https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/
https://blog.upbound.io/argo-crossplane-managing-application-stack
https://docs.upbound.io/concepts/mcp/control-plane-connector/
https://blog.upbound.io/2023-09-26-product-updates
https://morningspace.medium.com/using-crossplane-in-gitops-what-to-check-in-git-76c08a5ff0c4
Infrastructure-as-Apps https://codefresh.io/blog/infrastructure-as-apps-the-gitops-future-of-infra-as-code/
https://docs.upbound.io/spaces/git-integration/
https://codefresh.io/blog/using-gitops-infrastructure-applications-crossplane-argo-cd/
Configuration drift in Tf: Terraform horror stories about incomplete/invalid state https://www.youtube.com/watch?v=ix0Tw8uinWs
BADGES :
https://argo-cd.readthedocs.io/en/stable/user-guide/status-badge/
https://codefresh.io/blog/argo-cd-application-dependencies/
https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern
https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/
https://github.com/christianh814/golist
https://docs.crossplane.io/knowledge-base/integrations/vault-as-secret-store/
External Secret Stores are an alpha feature. They’re not recommended for production use. Crossplane disables External Secret Stores by default.
https://github.com/crossplane/crossplane/blob/master/design/design-doc-external-secret-stores.md
storing sensitive information in external secret stores is a common practice. Since applications running on K8S need this information as well, it is also quite common to sync data from external secret stores to K8S. There are quite a few tools out there that are trying to resolve this exact same problem. However, Crossplane, as a producer of infrastructure credentials, needs the opposite, which is storing sensitive information to external secret stores.
--> So this feature is NOT for retrieving secrets FROM external secret providers, BUT for storing secrets IN external secret providers!
But the External Secrets Operator has also PushSecrets https://external-secrets.io/latest/api/pushsecret/ which seem to do the same