[SPIKE] Run smoke tests and user flow tests as ECS task or Lambda in VAEC

[ffafara-tw]

Goal

The goal of the spike is to evaluate whether ECS task or Lambda would be better suited for us to run smoke tests and user flow tests in VAEC after deployment .

Since we cannot hit these endpoints from within VAEC, we need to call the user flows/smoke tests from a different system—ECS or Lambda. Then, we can have all the testing included in prod in VAEC.

Checklist:

• Create an ECS task that will run the tests
• Create a Lambda that will run the tests
• Compare costs of both solutions - lambda cheaper
• Compare execution time - lambda faster (800 ms versus 69 seconds)
• Compare ease of pipeline integration (how easy to trigger, how easy to extract results)
• Prepare recommendation in the form of ADR

Timebox:

Suggested 2 days (1 day for Lambda, 1 day for ECS)

Comparison

ECS Implementation:

• ECS task definition (1 for each env), refactor run_ci_migration script for our use case, put into pipeline- use same ECS steps from deployment-tw to deploy, set up logs - cloudwatch groups, new execution role, new security group, build docker container in pipeline & push to ECR (separate work flow), ECR lifecycle policy, new repository in ECR
• ECS pipeline steps: building docker container, render, register, and run task definition

Lambda Implementation:

• separate pipeline include AWS CLI command (fail pipeline if fail, succeed otherwise), terraform for lambda function, automate the zipping (maybe by terraform?), new role, new security group, pass in secrets referencing param store (pass name of env and then lambda retrieve secret using boto?), set up logs - cloudwatch groups
• Lambda pipeline steps: pip install virtualenv, zip, execute lamdbda function (AWS CLI)

Overlap:

• set up logs, separate workflow, new security group and role, probably the same for extracting results

Overall:
ECS: more resource intensive, longer deploy time
Lambda: quicker run, cheaper, more lightweight

[ffafara-tw]

Some resource for building and deploying python lambda:

• https://medium.com/analytics-vidhya/deploying-aws-lambda-function-with-terraform-custom-dependencies-7874407cd4fc
• https://www.serverless.com/framework/docs/providers/aws/examples/hello-world/python/
• https://www.scalyr.com/blog/aws-lambda-with-python/
• https://sanderknape.com/2018/02/comparing-aws-sam-with-serverless-framework/
• https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html
• https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html
• SAM vs Serverless: switch to Serverless again? brefphp/bref#320
• https://docs.google.com/spreadsheets/d/1DP2uVbg6TLfBa56XJ1GwlSSKQM1sM87-S3LQ2Pl9sPI/edit#gid=0
• https://peterkovari.blog/aws-lambda-in-python-with-sam/

[marisahoenig]

ADR added to vanotify-team repo here & our work on ECS and Lambda is on branch 179-ecs-lambda-spike

[marisahoenig]

Notes/Resources for Lambda:

• For zipping file, follow instructions "With a virtual environment" here. You will pip install the requirements_for_user_flows.txt, and this gets zipped along with steps.py and lambda.py (all the tests...we moved the contents of test_retrieve_everything.py into the lambda.py with the new handler() method).
• Terraform resource for aws_lamdba_function(will need to add it to a VPC when we implement)
• AWS Lambda with Python Getting Started Guide
• Lambda Function Handler instructions/docs

[marisahoenig]

ADR accepted by team, created implementation card #228. Moving this to closed.