[PoC] feat: allow flexible customizing for task/web/postgres pod definition

[kurokobo]

SUMMARY

This PR introduces task_pod_template, web_pod_template, and postgres_pod_template in AWX CR.

Related to:

• Add podSpec/podTemplate/statefuSet field to override pod settings #479
• Implement extra sidecar support #1236
• Added PodSpec for DNS config,policy and Hostnetwork for rsyslog/redis/ee/task/init and web containers. #1471
• [PoC] using kustomize to provide flexible override to awx deployments #1695

ISSUE TYPE

• New or Enhanced Feature

ADDITIONAL INFORMATION

DEMO CR:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  ...

  task_pod_template:
    metadata:
      annotations:
        custom_annotation: task-annotation
      labels:
        custom_label: task-label
    spec:
      initContainers:
        - name: init
          env:
            - name: CUSTOM_ENV_FOR_INIT
              value: custom_env_for_init
      containers:
        - name: awx-task
          env:
            - name: CUSTOM_ENV_FOR_TASK
              value: custom_env_for_TASK
          volumeMounts:
            - name: demo-task-emptydir
              mountPath: "/tmp/task/emptydir"
        - name: redis
          securityContext:
            capabilities:
              add:
                - CHOWN
                - SETUID
                - SETGID
        - name: sidecar
          image: quay.io/ansible/awx-ee:latest
          args:
          - /bin/sh
          - -c
          - |
            echo "Hello from sidecar"
            tail -f
      hostAliases:
        - ip: 10.0.0.1
          hostnames:
            - demo.example.com
      volumes:
        - name: demo-task-emptydir
          emptyDir: {}

  web_pod_template:
    metadata:
      annotations:
        custom_annotation: web-annotation
      labels:
        custom_label: web-label

  postgres_pod_template:
    metadata:
      annotations:
        custom_annotation: postgres-annotation
      labels:
        custom_label: postgres-label
    spec:
      containers:
        - name: postgres
          resources:
            requests:
              cpu: 10m
              memory: 128Mi
          volumeMounts:
            - name: demo-postgres-emptydir
              mountPath: "/tmp/postgres/emptydir"
      volumes:
        - name: demo-postgres-emptydir
          emptyDir: {}

RESULT:

$ kubectl -n awx get deployment/awx-task -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  ...
  name: awx-task
  namespace: awx
  ...
spec:
  ...
  template:
    metadata:
      annotations:
        ...
        custom_annotation: task-annotation ✅
        kubectl.kubernetes.io/default-container: awx-task
      creationTimestamp: null
      labels:
        ...
        app.kubernetes.io/version: 23.6.0
        custom_label: task-label ✅
    spec:
      containers:
      - args:
        ...
        name: awx-ee
        ...
      - args:
        ...
        name: awx-rsyslog
        ...
      - args:
        ...
        env:
        ...
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CUSTOM_ENV_FOR_TASK ✅
          value: custom_env_for_TASK ✅
        ...
        name: awx-task
        ...
        volumeMounts:
        ...
        - mountPath: /var/lib/awx/projects
          name: awx-projects
        - mountPath: /tmp/task/emptydir ✅
          name: demo-task-emptydir ✅
      - args:
        ...
        name: redis
        ...
        securityContext: ✅
          capabilities: ✅
            add: ✅
            - CHOWN ✅
            - SETUID ✅
            - SETGID ✅
        ...
      - args:
        - /bin/sh ✅
        - -c ✅
        - | ✅
          echo "Hello from sidecar" ✅
          tail -f ✅
        image: quay.io/ansible/awx-ee:latest ✅
        ...
        name: sidecar ✅
        ...
      ...
      hostAliases: ✅
      - hostnames: ✅
        - demo.example.com ✅
        ip: 10.0.0.1 ✅
      initContainers:
      - command:
        ...
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: CUSTOM_ENV_FOR_INIT ✅
          value: custom_env_for_init ✅
        ...
        name: init
        ...
      - command:
        ...
        name: init-projects
        ...
      ...
      volumes:
      ...
      - name: awx-projects
        persistentVolumeClaim:
          claimName: awx-projects-claim
      - emptyDir: {} ✅
        name: demo-task-emptydir ✅
...

$ kubectl -n awx get deployment/awx-web -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  ...
  name: awx-web
  namespace: awx
spec:
  ...
  template:
    metadata:
      annotations:
        ...
        custom_annotation: web-annotation ✅
        kubectl.kubernetes.io/default-container: awx-web
      labels:
        ...
        app.kubernetes.io/version: latest
        custom_label: web-label ✅
    ...
...

$ kubectl -n awx get statefulset/postgres-13 -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  ...
  name: awx-postgres-13
  namespace: awx
spec:
  ...
  template:
    metadata:
      annotations:
        custom_annotation: postgres-annotation ✅
      labels:
        ...
        app.kubernetes.io/part-of: awx
        custom_label: postgres-label ✅
    spec:
      containers:
      - env:
        ...
        name: postgres
        ...
        resources: ✅
          requests: ✅
            cpu: 10m ✅
            memory: 128Mi ✅
        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: postgres-13
          subPath: data
        - mountPath: /tmp/postgres/emptydir ✅
          name: demo-postgres-emptydir ✅
      ...
      volumes:
      - emptyDir: {} ✅
        name: demo-postgres-emptydir ✅
  ...

TODO

• Comment some debug code out
• Add comments in tasks
• Documentation
• Deprecate replacable parameters
• Deprecation warning
• Add feature flag
• Tests
• Update CSV
• Update TODO

[kurokobo]

CC: @TheRealHaoLiu @rooftopcellist @shanemcd

[kurokobo]

I think this allows for flexible templating. I believe we can replace some of the existing CR parameters, so please give it a try.

Note that my implementation will template only under spec.template of deployment and statefulset, since it is pod template instead of deployment template nor statefulset template.
So some CR params (e.g. replicas) can't be replaced with any *_pod_template.

Of course, it can also be a deployment template, but in this case deeper nesting is required to change the pod definition:

spec:
  ...
  task_deployment_template:
    spec:
      replicas: 2
      template:
        spec:
          containers:
            - name: awx-task
              ...

I would appreciate feedback on whether to proceed with #1697 (this) or #1695, and whether to implement pod template or deployment template in the case of this PR.