infrastructure

The infrastructure configuration, written using Terraform, Packer and Ansible.

Image Modification Workflow

This is the general workflow for provisioning and deploying a new images. Reasons to deploy a new image include: new operating system packages, webserver modifications, filesystem changes, onboarding a new user with SSH access, etc.

1. Create a branch with your changes to the provisioners or image builders.
2. Push your changes up to Github, and open a pull request.
3. Navigate to "Build Images" Github Action and click "Run workflow", specifying your branch in the dropdown. Once you've verified that all of the images were built without issue, add a reviewer to your PR and get approval. If the change you've made is solely adding a new user's SSH key, you can skip building the images.
4. Once your PR is approved, merge it.
5. (complete steps 5-7 during a deployment window) Head over to our Terraform Workspaces and select the workspace where you would like to deploy the newly built AMIs.
6. Click "Queue plan" in the top right corner, and confirm once the workflow reaches the "Plan Finished" state.
7. Repeat steps 5-6 for each workspace once the new configuration has been rolled out across the entire fleet.

Infrastructure Modification Worflow

The instances directory contains the Terraform configuration. You would want to make changes to Terraform to modify the infrastructure layout or hardware (e.g. new autoscaling groups, larger or more EC2 instances, etc). Our Terraform configuration currently manages all Permanent AMIs, EC2 instances, Autoscaling Groups, Launch Configurations, and Target Groups for the Load Balacers. Terraform does NOT manage: the Load Balancer itself, S3 buckets, SQS queues, VPCs, Security Groups, subnets, IAM, etc. To make changes to the infrastructure, push your changes to a branch, open a PR, and get a code review. Once the PR is merged, the change (depending on the directory they are in), should queue a terraform plan job our Terraform Cloud organization. Review the plan, and approve the changes if they look good. Terraform will then roll-out the new infrastructure.

Local development

Install required software

Install Terraform, Packer and Ansible to get started with infrastructure provisioning.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update
sudo apt install terraform packer
sudo pip3 install ansible
ansible-galaxy install willshersystems.sshd

Ansible can also be installed with your preferred local package manager (e.g. apt).

Create Images

The easiest way to create an image is to use the "Build Images" Github Action. Image creation can also be done manually on your local machine. The following example illustrates the steps for building the cron image.

cp .env.template .env # add your AWS access credentials
source .env
cd images
packer build -var-file=cron.json image.json

For Permanent employees: use the AWS access keys associated with the build IAM user, not the keys associated with your personal AWS account.

If a new category of image is being established by a Pull Request then it is possible that automated checks will fail without first manually invoking the Build Images Github Action. This is because Terraform expects the images to exist in the AMI registry in order to succeed.

Deploy Images

Images should generally only be deployed using Terraform Cloud. If you'd like to bring up a single image for a small test, do so using the AWS console.

Deploy Code

Code deployment can be initiated by manually triggering the "Deploy code" Github Action.

Manage SSH access

To onboard a new user with ssh access, create a file in the ssh directory. The file name should be the new user's Linux username, and the file contents should be their public key(s).

To offboard a user, remove their ssh file.

In both cases, the AMIs needs to be rebuilt and deployed. See Workflow for more details.

Debian Version

We use the latest Debian base image prescribed in our Debian verison policy.

Autoscaling

There was an initial attempt to configure autoscaling for the staging and production deployments. That attempt can be found in the staging Terraform configuration in the "autoscaling" branch. There are several open issues that are prerequisites for the rollout of autoscaling.

The current configuration simply brings up regular EC2 instances directly.

Quirks

Q: Why is ANSIBLE_PIPELINING=True for the deploy provisioner? A: Because aws s3 cp must be run as the appropriately-credentialed deployer user. Running an Ansible command as a non-root user requires using Ansible's become_user. Ansible normally writes files to a temporary filesystem initially, which requires root, but with pipelining enabled, there is no need to write to this temporary filesystem. If this sounds confusing, that's because it is. See here for more info: https://docs.ansible.com/ansible/latest/user_guide/become.html#risks-of-becoming-an-unprivileged-user