source-controller: S3 Compatible Storage

[stefanprodan]

Besides GitRepositories and HelmRepositories we could extend the artifacts acquisition to S3 compatible storage such as Minio, AWS S3, Google Cloud Storage with S3 compatible access enabled, Alibaba Cloud Object Storage Service and others.

API Proposal

Introduce a new API kind called Bucket that defines the bucket name, endpoint and authentication credentials (read-only access). Based on a bucket definition, source-controller will download all files from the bucket and create an artifact inside the cluster in the same way it does for git or helm sources. As a first iteration, we can start by implementing a generic S3 connector that requires a static accesskey/secretkey for authentication.

Spec

// BucketSpec defines the desired state of an S3 compatible bucket
type BucketSpec struct {
	// The S3 compatible storage provider name, default ('generic').
	// +kubebuilder:validation:Enum=generic;aws
	// +optional
	Provider string `json:"provider,omitempty"`

	// The bucket name.
	// +required
	BucketName string `json:"bucketName"`

	// The bucket endpoint address.
	// +required
	Endpoint string `json:"endpoint"`

	// Insecure allows connecting to a non-TLS S3 HTTP endpoint.
	// +optional
	Insecure bool `json:"insecure,omitempty"`

	// The bucket region.
	// +optional
	Region string `json:"region,omitempty"`

	// The secret name containing the bucket accesskey and secretkey.
	// +optional
	SecretRef *corev1.LocalObjectReference `json:"secretRef,omitempty"`

	// The interval at which to check for bucket updates.
	// +required
	Interval metav1.Duration `json:"interval"`

	// The timeout for download operations, default ('20s').
	// +optional
	Timeout *metav1.Duration `json:"timeout,omitempty"`

	// Ignore overrides the set of excluded patterns in the .sourceignore
	// format (which is the same as .gitignore).
	// +optional
	Ignore *string `json:"ignore,omitempty"`
}

// BucketStatus defines the observed state of a bucket
type BucketStatus struct {
	// ObservedGeneration is the last observed generation.
	// +optional
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`

	// Conditions holds the conditions for the Bucket.
	// +optional
	Conditions []SourceCondition `json:"conditions,omitempty"`

	// URL is the download link for the artifact output of the last Bucket sync.
	// +optional
	URL string `json:"url,omitempty"`

	// Artifact represents the output of the last successful Bucket sync.
	// +optional
	Artifact *Artifact `json:"artifact,omitempty"`
}

Examples

Minio or any other S3 compatible storage with static auth:

apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: Bucket
metadata:
  name: podinfo
  namespace: gitops-system
spec:
  interval: 1m
  provider: generic
  bucketName: podinfo
  endpoint: minio.minio.svc.cluster.local:9000
  insecure: true
  secretRef:
    name: minio-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: minio-credentials
  namespace: gitops-system
type: Opaque
data:
  accesskey: <BASE64> 
  secretkey: <BASE64> 

EKS with S3 IAM role for service accounts:

apiVersion: source.toolkit.fluccd.io/v1alpha1
kind: Bucket
metadata:
  name: podinfo
  namespace: gitops-system
spec:
  interval: 1m
  provider: aws
  bucketName: podinfo-deploy
  endpoint: s3.amazonaws.com
  region: us-east-1

When the provider is aws and the secret credentials is not specified, source-controller will retrieve credentials from the EC2 service.

[stefanprodan]

Implemented in:

• Implement S3 bucket source source-controller#146
• Add support for S3 bucket sources kustomize-controller#117
• Support Helm charts from Bucket sources source-controller#151
• Add support for S3 buckets to alerts and receivers notification-controller#51
• Support Helm charts from Bucket sources helm-controller#89
• Implement bucket CRUD commands #267
• Add support for Bucket sources to Kustomizations/HRs #270