Update dependencies in all components

[binarymist]

SUT

• NodeGoat
  • Update fork https://github.com/purpleteam-labs/purpleteam-iac-sut
  • docker-compose build

System

• terraform (Terraform itself should be handled by the systems package manager)
  • https://releases.hashicorp.com/terraform/
  • https://www.terraform.io/downloads.html
  • https://www.hashicorp.com/security
  • https://www.terraform.io/upgrade-guides/0-13.html
  • https://github.com/hashicorp/terraform/blob/v0.13.5/CHANGELOG.md
  • purpleteam-iac
    • Update Terraform providers and versions.tf
      • terragrunt init -upgrade
      • terragrunt apply
    • Update AWS AMIs
    • Update Stage Two images
      • selenium (https://github.com/SeleniumHQ/docker-selenium/):
        • Update versions and publish dates in variables.tf of contOrc root
      • zap (https://www.zaproxy.org/download/, https://www.zaproxy.org/docs/desktop/releases/2.11.0/)
        • Update versions and publish dates in variables.tf of contOrc root
  • purpleteam-iac-sut
    • Update Terraform providers and versions.tf
      • terragrunt init -upgrade
      • terragrunt apply
    • Update AWS AMIs
    • build and push nodegoat image to ECR
• terragrunt
  • https://github.com/gruntwork-io/terragrunt/releases
• nvm (just run nvm -v to check current version, then nvm upgrade)
• Node
  • Updated shell to 16
  • The main constrain is what AWS Lambda supports (current support is only 14). Also the Terraform AWS provider may need updating
• docker-compose (https://github.com/docker/compose/releases)

Project

Update CommonJS to ESM

• purpleteam-build-test-cli
  • convert to ESM
• purpleteam-logger
  • convert to ESM
  • gitlab-ci node version
  • Upgrade as many dependencies as possible and convert to ESM
  • The best way to reference the logger package is by pushing the changes to a git feature branch, then just keep git commit --amending, --force pushing. In the consuming projects:
    • In the dependencies object of package.json of projects that consume purpleteam-logger set "purpleteam-logger": "purpleteam-labs/purpleteam-logger.git#binarymist/upgrade"
    • On each change to logger:
      • Delete the package-lock.json and node_modules and npm install again
  • test via purpleteam
  • test via purpleteam-orchestraiter
  • test via purpleteam-app-scanner
  • test via purpleteam-tls-scanner
  • publish to NPM and update orchestrator and app-scanner deps
  • test locally via cloud SUT
• mocksse
  • Change test lib as lab doesn't support ESM
  • Convert to ESM
• purpleteam (CLI)
  • convert to ESM
  • gitlab-ci node version
  • Upgrade as many dependencies as possible
  • Change test lib as lab doesn't support ESM
  • got is going to require some work
  • test local
  • test local with cloud sut
  • test cloud
• purpleteam-lambda
  • Upgrade as many dependencies as possible
  • test local
  • test local with cloud sut
  • test cloud
  • got in Lambda functions can not currently be upgraded past 11.8.2 due to lack of AWS support of ESM
• purpleteam-orchestrator
  • convert to ESM
  • github ci action - node version
  • Dockerfile node version
  • Upgrade as many dependencies as possible
  • test local
  • test local with cloud sut
  • test cloud
• purpleteam-app-scanner
  • convert to ESM
  • github ci action - node version
  • Dockerfile node version
  • Upgrade as many dependencies as possible
  • test local
  • test local with cloud sut
  • test cloud
• purpleteam-tls-scanner
  • convert to ESM
  • github ci action - node version
  • Dockerfile node version
  • Upgrade as many dependencies as possible
  • Upgrade testssl.sh
• purpleteam-server-scanner
  • github ci action - node version
  • Update all dependencies
• purpleteam-s2-containers
  • Update Stage Two images
    • selenium (https://github.com/SeleniumHQ/docker-selenium/):
      • Update versions and publish dates in variables.tf of contOrc root
    • zap (https://www.zaproxy.org/download/, https://www.zaproxy.org/docs/desktop/releases/2.11.0/)
      • Update versions and publish dates in variables.tf of contOrc root
  • docker-compose 3.6 - 3.8 (dependant on docker-compose-ui). Can now be updated as we have a new maintainer/fork and it appears that 3.8 is now supported:
    • Upstream (archived): https://github.com/francescou/docker-compose-ui
    • https://github.com/nsano-rururu/docker-compose-ui Discuss issues in Github Discussions
  • test local

Zaproxy Upgrade URLs

• https://github.com/purpleteam-labs/purpleteam-s2-containers
• https://www.zaproxy.org/download/
• https://github.com/zaproxy/zaproxy/releases/tag/v2.11.1
• https://hub.docker.com/r/owasp/zap2docker-stable/tags
• https://www.zaproxy.org/docs/desktop/releases/2.11.0/
• https://www.zaproxy.org/docs/desktop/addons/report-generation/

[binarymist]

Unfortunatly due to the fact that we consume a number of sindresorhus packages, and @sindresorhus decided to move their packages to (ESM only), we were forced to also make this move. This move had a good number of unanticipated side effects.

Waiting on:

Sywac

We were forced to fork sywac to make some changes so that our CLI which uses sywac (which doesn't yet support ESM) to consume our now ESM code would still work. The Pull Request is still waiting on feedback from maintainers (@nexdrew, @elliot-nelson)

docker-compose-ui

To be forked and support the latest docker-compose version. Currently we're locked to versions of docker-compose before v2. @paularah is working on replacing the docker-compose-ui functionality we use to support stage two containers in the local environment with kubernetes jobs.

URLs currently looking at:

• https://www.google.com/search?q=docker-compose+versions
• https://docs.docker.com/compose/compose-file/compose-versioning/
• https://www.google.com/search?q=docker+versions
• https://docs.docker.com/compose/install/
• https://github.com/docker/compose/releases?page=1
• https://github.com/docker/compose#where-to-get-docker-compose
• https://docs.docker.com/compose/cli-command/
• https://www.google.com/search?q=docker+compose+ui
• https://github.com/francescou/docker-compose-ui/pull/119/commits
• https://github.com/nsano-rururu/docker-compose-ui

Another option instead of forking and maintaining docker-compose-ui could be to move from docker-compose-ui to k8s, something like Minikube and k8s jobs for stage two containers.

If we went the k8s route we'd have to make sure that the stage two containers can be brought up and down during a test run based on the number of Test Sessions in a given Job file.

Further thoughts on the GSoC ideas page

Fixed:

redis

Issue that covers the problem and fix

Redis is now fixed, rebased and pushed

node-redis was currently broken: redis/node-redis#1870 (comment)

For our Testers upgraded to 4.0.2, but orchestrator is currently stuck on 3.1.2. Anything later is currently producing errors.

URLs we're currently watching:

• https://github.com/redis/node-redis
• https://github.com/redis/node-redis/blob/HEAD/docs/v3-to-v4.md
• https://github.com/redis/node-redis/blob/be51abe3470285548a8af8b02a83a870f2094df5/CHANGELOG.md

We currently have a branch in orchestrator "binarymist/upgrade-incl-redis" that we will be rebasing on main and continuing to work on until we can move to a later version of redis that is fixed. If this doesn't happen, we'll consider changing to ioredis.

Cucumber

We have been depending on undocumented API features for years because we needed the functionality.
In version 8 many undocumented API features were removed, the Cucumber CLI's getConfiguration was one of these.
This means that retreiving the testplan is now broken until Cucumber reinstates the functionality.

Issues tracking this are at:

• api: add runCucumber function internally cucumber/cucumber-js#1849 which in contradiction to the "no breaking changes" statements broke our testplan functionality
• Programmatic API for running cucumber-js cucumber/cucumber-js#1711 (comment)
• Changelog

The above is now fixed. Now the colours are broken. Issue specifically for broken colours is here. As a temporary measure we've added the environment variable (FORCE_COLOR=1) to the app-scanner and tls-scanner Dockerfiles. This needs to be removed once cucumber has provided a fix.

Tested and all good.