AWS Elastic Kubernetes Service Kubernetes 1.28 Update #237

4141done · 2023-11-14T20:31:21Z

Updates for EKS 1.28

This change brings MARA up to date for execution in Amazon EKS running Kubernetes 1.28.
The additional changes in this PR are:

Changes to python versions (3.11.6 is suggested, 3.11 required) to support building our combination of dependencies
It is now required to install the aws tool yourself

The docker-based runs have not been updated and will be addressed in a future PR.

Run Instructions

Prerequisites

An AWS account (be aware that this stand up is big and requires t2.large instances with the accompanying costs)
A (free) Pulumi account
Python 3.11 (3.11.6 is tested) installed. Consider using a tool like pyenv or asdf
The aws cli tool installed and configured
(optional) The kubectl tool installed
(optional) The k9s tool installed

All of the other suggested tooling can be managed with asdf if you want.

NOTE FOR containerd USERS ON ARM64 ARCHITECTURE: this will not work using containerd for the moment due to how it handles manifests. If you are using Docker for Mac on an M1/M2 system you'll need to disable the containerd mode before continuing

Steps

git clone --recurse-submodules https://github.com/nginxinc/kic-reference-architectures to check out this repo and the demo app.
git checkout pulumi_2023 to get this branch
./bin/setup_venv.sh to set up python and install all the dependencies
In ./config/pulumi create a file like this with the name Pulumi.your_stack_name.yaml

config:
  aws:profile: 369313531325_Users
  aws:region: us-west-2
  eks:desired_capacity: 3
  eks:instance_type: t2.large
  eks:k8s_version: '1.28'
  eks:max_size: 12
  eks:min_size: 3
  kubernetes:infra_type: AWS
  vpc:azs:
  - us-west-2a
  - us-west-2b
  - us-west-2c
  - us-west-2d
  kic:image_name: nginx/nginx-ingress:3.3.2
  kic:image_origin: registry
  kic-helm:chart_name: nginx-ingress
  kic-helm:chart_version: 1.0.2
  kic-helm:helm_timeout: 900
  # Chart name for the helm chart for the logagent
  logagent:chart_version: 7.17.3
  # Elasticsearch takes a LOOONG time to deploy
  logstore:helm_timeout: 900
  logstore:chart_version: 19.13.9
  logstore:chart_name: elasticsearch
  logstore:helm_repo_name: bitnami
  logstore:helm_repo_url: https://charts.bitnami.com/bitnami
  # Cert Manager Configuration
  certmgr:chart_name: cert-manager
  # Chart hame for the helm chart for certmanager
  certmgr:chart_version: v1.12.6
  # Chart version for the helm chart for certmanager
  certmgr:certmgr_helm_repo_name: cert-manager
  # Name of the repo to pull the certmanager chart from
  certmgr:certmgr_helm_repo_url: https://charts.jetstack.io
  # URL of the chart repo to pull certmanager from
  certmgr:helm_timeout: 300

  # Prometheus Configuration
  prometheus:chart_name: kube-prometheus-stack
  # Chart name for the helm chart for prometheus
  prometheus:chart_version: 54.0.1
  # Name of the statsd chart (uses the same repo as the prom chart)
  prometheus.statsd_chart_version: 0.10.1

  # Prometheus Configuration for Bank of Sirius Application
  # https://artifacthub.io/packages/helm/prometheus-community/prometheus-postgres-exporter
  sirius:chart_version: 5.2.0
  sirius:helm_repo_name: prometheus-community

Execute the standup script. The environment variable isn't strictly necessary if you are not on an linux/arm64 machine but it doesn't hurt to leave it there.

DOCKER_DEFAULT_PLATFORM=linux/amd64 pulumi/python/runner -d -p aws -s your_stack_name up

Exploring

Once the standup script is complete, you should see something that looks like this:

application_url     : "https://longstring-119454905.us-west-2.elb.amazonaws.com"

You can use that to visit the Bank of Sirius UI

To view other installed tooling, first make sure you have kubectl configured to have access to your cluster like this:

aws eks update-kubeconfig --region YOUR_REGION --name CLUSTER_NAME --profile PROFILE_NAME

The CLUSTER_NAME can be obtained either from the AWS UI or by looking at the output from this project in the "EKS" section at the beginning. It will look like this:

Outputs:
    cluster_name: "aws-eks-yourstackname-eksCluster-cf1f5e2"

Once that is done, run:

./bin/test-forward.sh

Which till create tunnels to the appropriate endpoints in the deployed MARA instance.

It is also recommended to install and deploy k9s (link) to explore the Kubernetes cluster. If you set up your kubeconfig in the earlier step then it should just work.

4141done · 2023-11-14T20:36:02Z

bin/setup_venv.sh

@@ -91,7 +93,7 @@ if ! command -v python3 >/dev/null; then

 	mkdir -p "${PYENV_ROOT}"
 	git_clone_log="$(mktemp -t pyenv_git_clone-XXXXXXX.log)"
-	if git clone --depth 1 --branch v2.0.3 https://github.com/pyenv/pyenv.git "${PYENV_ROOT}" 2>"${git_clone_log}"; then
+	if git clone --depth 1 --branch v2.3.31 https://github.com/pyenv/pyenv.git "${PYENV_ROOT}" 2>"${git_clone_log}"; then


annotation:
Older versions of pyenv don't know about the newer python version we are using.

4141done · 2023-11-15T04:45:46Z

pulumi/python/kubernetes/nginx/ingress-controller/__main__.py

 srv = Service.get(resource_name="nginx-ingress",
-                  id=Output.concat("nginx-ingress", "/", pstatus.name, "-nginx-ingress"),
+                  id=Output.concat("nginx-ingress", "/", pstatus.name, "-nginx-ingress-controller"),


Annotation:
Necessary to align with the new name of the controller in new helm chart

4141done · 2023-11-15T04:46:33Z

pulumi/python/utility/kic-image-build/ingress_controller_image_base_provider.py

@@ -47,7 +47,8 @@ def _docker_pull(self, image_name: str) -> str:
        :param image_name: full container image name in the format of repository:tag
        :return full image name with server name (e.g. docker.io/library/debian:buster-slim)
        """
-        cmd = f'docker pull --quiet "{image_name}"'
+
+        cmd = f'docker pull --platform linux/amd64 --quiet "{image_name}"'


annotation:
Necessary to make sure we get the version of the image that will be compatible with the deployment environment not the computer running it.

4141done · 2023-11-15T04:47:13Z

pulumi/python/utility/kic-pulumi-utils/setup.py

@@ -7,5 +7,5 @@
      version_config=True,
      packages=['kic_util'],
      install_requires=[
-          'pyyaml>=5.3.1,<6.0', 'passlib>=1.7.4,<2.0.0', 'GitPython>=3.1.18,<3.2.0'
+          'pyyaml', 'passlib>=1.7.4,<2.0.0', 'GitPython>=3.1.18,<3.2.0'


annotation:
Relaxing this requirement is ok and allows us to update the various pulumi packages which have moved to version >=6

4141done · 2023-11-15T04:48:51Z

pulumi/python/Pipfile

@@ -4,29 +4,25 @@ verify_ssl = true
 name = "pypi"

 [packages]
-awscli = "~=1.25.35"


annotation:
The AWS cli has some pretty strict pins on its dependencies. PyYaml specifically. The advice should be to install the v2 aws CLI tool separately.

4141done and others added 8 commits November 8, 2023 05:59

small m1 fixes

91ddd23

updated dependancies to run localy on a m1 mac and provision in AWS.

dd70919

small fixes and debug

b4944cd

working on m1 up until elastic

07b5990

working up to cert manager

9c9d0b9

update cert manager repo name and version

9df3b22

remove old hack

070304f

update otel collector configuration

6f1ec35

4141done commented Nov 15, 2023

View reviewed changes

4141done added 4 commits November 14, 2023 20:49

clean up

fad88de

update github actions workflow to enforce python versions

6dcb9db

small hack for gh actions setup python cleanup workflow

866f4eb

remove failing hack for macos tests

a4e2f65

4141done changed the title ~~[WIP] Pulumi 2023~~ AWS Elastic Kubernetes Service Kubernetes 1.28 Update Nov 15, 2023

4141done marked this pull request as ready for review November 15, 2023 08:35

4141done added 2 commits November 15, 2023 07:13

fix to simple otel collector and port forwarding script

8742d7b

hack to fix python venv cleanup

c15966b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS Elastic Kubernetes Service Kubernetes 1.28 Update #237

AWS Elastic Kubernetes Service Kubernetes 1.28 Update #237

4141done commented Nov 14, 2023 •

edited

4141done Nov 14, 2023

4141done Nov 15, 2023

4141done Nov 15, 2023

4141done Nov 15, 2023

4141done Nov 15, 2023

AWS Elastic Kubernetes Service Kubernetes 1.28 Update #237

Are you sure you want to change the base?

AWS Elastic Kubernetes Service Kubernetes 1.28 Update #237

Conversation

4141done commented Nov 14, 2023 • edited

Updates for EKS 1.28

Run Instructions

Prerequisites

Steps

Exploring

4141done Nov 14, 2023

Choose a reason for hiding this comment

4141done Nov 15, 2023

Choose a reason for hiding this comment

4141done Nov 15, 2023

Choose a reason for hiding this comment

4141done Nov 15, 2023

Choose a reason for hiding this comment

4141done Nov 15, 2023

Choose a reason for hiding this comment

4141done commented Nov 14, 2023 •

edited