Architecture: rebase testkube ontop of argo-workflows

[gberche-orange]

Is your feature request related to a problem? Please describe.

As a testkube user,
In order to benefit from the features available in argo workflow that are missing in test kube,
I need testkube to leverage argo workflow while keeping differentiating testkube features

Currently, testkube is implementing its features on top of plain k8s api, generating Job and CronJob resources, with init container fetching git sources and custom triggers.

Describe the solution you'd like

TestKube to leverage argo workflows for most of the facilities, and focus on missing and test-specific features instead

Additional context

Following are some thoughts on requirements in my team for a test facade, and how testkube could leverage argo workflows to support them.

Actors:

• Test author (~google TE, SWE) : authors new tests to assert some behavior
• Test template author (~google SET): maintains templates that factor out behavior common to multiple tests
• Test reader (~google TE, SWE ): invoke tests in specific environments. Read test status. Diagnose root cause of test failures

Key desired features of an off the shelf K8S test framework

• Gitops and K8S native: authoring via CRD and gitops
• Catalog of maintained templates for running off-the-shelf test tools
  • e.g. kuttl Kuttl executor #1142
  • e.g. goss https://github.com/goss-org/goss
  • client test tool default sensible configuration and execution
    • container image
    • default options (output archival formatting, debug verbose flags)
  • sample dogfooding/community usage and benefits: test source code, diagnosed issues, outputs (possibly sampled)
  • Server-side installation if required
  • Service subscription for commercial managed services
• Git-ops ready:
  • Trigger when changes to workflow/script content
  • Trigger on external conditions: SUT (system-under-test) changes
• Web ui for "Test reader" actor
  • auth
  • trigger tests/scripts through custom input web forms
  • logs archiving
  • artefacts archiving
  • observability
    • prometheus metrics
    • notifications

features	sub-feature	Argo-workflow	TestKube
K8S native	authoring via CRD	~OK	OK
Git-ops ready	Mount content from Gitops repo	To-test	OK
Git-ops ready	Trigger when changes to workflow/script content	To-test (gh webhook)	#4007
Git-ops ready	Trigger on external conditions: SUT (system-under-test) changes	To-test (gh webhook)	#3712 #4017
Web ui	ease of use, user friendly	Weak	Good
Web ui	auth	OK	OK
Web ui	trigger tests/scripts through custom input web forms	~ OK (basic enums)	#4030
Web ui	logs archiving	OK	OK
Web ui	artefacts archiving	OK	OK
Web ui	artefacts visualization	To-test	#4029
Web ui	prometheus metrics	To-test	OK
Web ui	notifications	To-test	To-test
Community	size, maturity	Large	Small
Community	support	To-test	OSS or enterprise/cloud subscription
Community	reuseable templates	To-test	Builtin Test executors

argo workflow details

trigger when changes to script/workflow content

argoproj/argo-workflows#8415 Conditionally Run Workflows Based on Git Artifact Files Changed

• Use Argo Events to setup a GitHub event source: https://github.com/argoproj/argo-events/blob/master/docs/eventsources/setup/github.md

• Also https://github.com/argoproj/argo-events/blob/master/docs/eventsources/setup/gitlab.md

webui

Use cases

• List templates
• Run templates
  • With params forms
• List executions

reuseable templates

https://github.com/argoproj-labs/argo-workflows-catalog/tree/master/templates few unmaintained templates

• https://equinor.github.io/flowify-documentation/components/
  • https://github.com/equinor/?q=flowify-component&type=all&language=&sort= jq, http, gcp, azure blob, array

resulting test artifacts

https://argoproj.github.io/argo-workflows/artifact-visualization/

Prometheus metrics

https://argoproj.github.io/argo-workflows/metrics/

Trigger on git change

Likely through a webhook from the git repo provider (e.g. github or gitlab)
https://argoproj.github.io/argo-workflows/events/#workflow-template-triggered-by-the-event
https://argoproj.github.io/argo-workflows/webhooks/

The secret argo-workflows-webhook-clients tells Argo:
What type of webhook the account can be used for, e.g. github.
What "secret" that webhook is configured for, e.g. in your Github settings page.

https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/

Argo CD polls Git repositories every three minutes to detect changes to the manifests. To eliminate this delay from polling, the API server can be configured to receive webhook events. Argo CD supports Git webhook notifications from GitHub, GitLab, Bitbucket, Bitbucket Server and Gogs. The following explains how to configure a Git webhook for GitHub, but the same process should be applicable to other providers.

CRD schema

Argo workflow CRD schema are currently empty because the schema size exceeds the K8S supported size. See argoproj/argo-workflows#8190 and argoproj/argo-helm#2105

Mount content from gitops repo

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: osb-smoke-test-template
spec:
  entrypoint: osb-smoke-test
  arguments:
    artifacts:
      - name: myrepo
        git:
          repo: https://
          branch: main
          depth: 1
          disableSubmodules: true
          usernameSecret:
            name: name
            key: key
          passwordSecret:
            name: name
            key: key 

• https://argoproj.github.io/argo-workflows/artifact-repository-ref/
• https://argoproj.github.io/argo-workflows/fields/#gitartifact

Related issues:

• Ability to do sparse checkout for git artifact argoproj/argo-workflows#10375
• azure devops git repo unable to use git artifact  argoproj/argo-workflows#10600
• Ability to do sparse checkout for git artifact argoproj/argo-workflows#10375 parse checkout RFE
• move away from go-git argoproj/argo-workflows#10091 move away from go-git (being discussed)

[gberche-orange]

thanks to @dejanzele and @TheBrunoLopes for our 2nd sync up meeting today on this topic and your sharing of current testkube dogfooding:

TestKube helm release is tested using testkube, covering test kube api using postman executor, and playright executor (+ executor smoke tests)

• github action source at https://github.com/kubeshop/helm-charts/blob/75ea82c618d01a708aa3758992560965a8bb9a33/.github/workflows/helm-releaser-testkube-charts.yaml#L380-L424
• github action logs https://github.com/kubeshop/helm-charts/actions/workflows/helm-releaser-testkube-charts.yaml
• postman tests
  • source at

    testkube/test/postman/TestKube-Sanity.postman_collection.json

    Lines 61 to 451 in bc32651

    	"request": {
    	"method": "POST",
    	"header": [],
    	"body": {
    	"mode": "raw",
    	"raw": "{\n \"name\": \"{{test_name}}\",\n \"type\": \"{{test_type}}\",\n \"labels\": {\n \"toDelete\": \"yes\"\n },\n \"namespace\": \"testkube\",\n \"content\": {\n \"type\": \"string\",\n \"data\": \"{\\r\\n\\t\\\"info\\\": {\\r\\n\\t\\t\\\"_postman_id\\\": \\\"3d9a6be2-bd3e-4cf7-89ca-354103aab4a7\\\",\\r\\n\\t\\t\\\"name\\\": \\\"testkube\\\",\\r\\n\\t\\t\\\"schema\\\": \\\"https:\\/\\/schema.getpostman.com\\/json\\/collection\\/v2.1.0\\/collection.json\\\"\\r\\n\\t},\\r\\n\\t\\\"item\\\": [\\r\\n\\t\\t{\\r\\n\\t\\t\\t\\\"name\\\": \\\"Health\\\",\\r\\n\\t\\t\\t\\\"event\\\": [\\r\\n\\t\\t\\t\\t{\\r\\n\\t\\t\\t\\t\\t\\\"listen\\\": \\\"test\\\",\\r\\n\\t\\t\\t\\t\\t\\\"script\\\": {\\r\\n\\t\\t\\t\\t\\t\\t\\\"exec\\\": [\\r\\n\\t\\t\\t\\t\\t\\t\\t\\\"pm.test(\\\\\\\"Status code is 200\\\\\\\", function () {\\\",\\r\\n\\t\\t\\t\\t\\t\\t\\t\\\" pm.response.to.have.status(200);\\\",\\r\\n\\t\\t\\t\\t\\t\\t\\t\\\"});\\\"\\r\\n\\t\\t\\t\\t\\t\\t],\\r\\n\\t\\t\\t\\t\\t\\t\\\"type\\\": \\\"text\\/javascript\\\"\\r\\n\\t\\t\\t\\t\\t}\\r\\n\\t\\t\\t\\t}\\r\\n\\t\\t\\t],\\r\\n\\t\\t\\t\\\"request\\\": {\\r\\n\\t\\t\\t\\t\\\"method\\\": \\\"GET\\\",\\r\\n\\t\\t\\t\\t\\\"header\\\": [],\\r\\n\\t\\t\\t\\t\\\"url\\\": {\\r\\n\\t\\t\\t\\t\\t\\\"raw\\\": \\\"{{test_api_uri}}\\/health\\\",\\r\\n\\t\\t\\t\\t\\t\\\"host\\\": [\\r\\n\\t\\t\\t\\t\\t\\t\\\"{{test_api_uri}}\\\"\\r\\n\\t\\t\\t\\t\\t],\\r\\n\\t\\t\\t\\t\\t\\\"path\\\": [\\r\\n\\t\\t\\t\\t\\t\\t\\\"health\\\"\\r\\n\\t\\t\\t\\t\\t]\\r\\n\\t\\t\\t\\t}\\r\\n\\t\\t\\t},\\r\\n\\t\\t\\t\\\"response\\\": []\\r\\n\\t\\t}\\r\\n\\t]\\r\\n}\"\n }\n}",
    	"options": {
    	"raw": {
    	"language": "json"
    	}
    	}
    	},
    	"url": {
    	"raw": "{{api_uri}}/v1/tests",
    	"host": [
    	"{{api_uri}}"
    	],
    	"path": [
    	"v1",
    	"tests"
    	]
    	}
    	},
    	"response": []
    	},
    	{
    	"name": "List tests",
    	"event": [
    	{
    	"listen": "test",
    	"script": {
    	"exec": [
    	"pm.test(\"Status code is 200\", function () {",
    	" pm.response.to.have.status(200);",
    	"});",
    	"",
    	"pm.test(\"Test is on the list\", function () {",
    	" let jsonData = pm.response.json();",
    	" let contains = false;",
    	" for (let i=0; i<jsonData.length; i++) {",
    	" if (jsonData[i].name == pm.environment.get(\"test_name\")) {",
    	" contains = true;",
    	" }",
    	" }",
    	"",
    	" pm.expect(contains).to.be.true",
    	"});"
    	],
    	"type": "text/javascript"
    	}
    	},
    	{
    	"listen": "prerequest",
    	"script": {
    	"exec": [
    	"console.log(\"uri\", pm.environment.get(\"api_uri\"));",
    	"console.log(\"test name\", pm.environment.get(\"test_name\"))",
    	"console.log(\"test type\", pm.environment.get(\"test_type\"))",
    	"console.log(\"exec name\", pm.environment.get(\"execution_name\"))",
    	"",
    	""
    	],
    	"type": "text/javascript"
    	}
    	}
    	],
    	"request": {
    	"method": "GET",
    	"header": [],
    	"url": {
    	"raw": "{{api_uri}}/v1/tests",
    	"host": [
    	"{{api_uri}}"
    	],
    	"path": [
    	"v1",
    	"tests"
    	]
    	}
    	},
    	"response": []
    	},
    	{
    	"name": "Get test",
    	"event": [
    	{
    	"listen": "test",
    	"script": {
    	"exec": [
    	"pm.test(\"Check response data\", function () {",
    	" var jsonData = pm.response.json();",
    	" pm.expect(jsonData.name).to.eql(pm.environment.get(\"test_name\"));",
    	"});",
    	"",
    	""
    	],
    	"type": "text/javascript"
    	}
    	},
    	{
    	"listen": "prerequest",
    	"script": {
    	"exec": [
    	"console.log(\"uri\", pm.environment.get(\"api_uri\"));",
    	"console.log(\"test name\", pm.environment.get(\"test_name\"))",
    	"console.log(\"test type\", pm.environment.get(\"test_type\"))",
    	"console.log(\"exec name\", pm.environment.get(\"execution_name\"))",
    	"",
    	""
    	],
    	"type": "text/javascript"
    	}
    	}
    	],
    	"request": {
    	"method": "GET",
    	"header": [],
    	"url": {
    	"raw": "{{api_uri}}/v1/tests/{{test_name}}",
    	"host": [
    	"{{api_uri}}"
    	],
    	"path": [
    	"v1",
    	"tests",
    	"{{test_name}}"
    	]
    	}
    	},
    	"response": []
    	},
    	{
    	"name": "API Start Test",
    	"event": [
    	{
    	"listen": "test",
    	"script": {
    	"exec": [
    	"pm.test(\"Status code is 201 CREATED\", function () {",
    	" pm.response.to.have.status(201);",
    	"});",
    	"",
    	"pm.test(\"Check execution is created\", function () {",
    	" let jsonData = pm.response.json();",
    	" console.log(\"create response\", jsonData);",
    	"",
    	" let executionName = jsonData.name ",
    	" let executionID = jsonData.id ",
    	" pm.expect(executionName).is.not.empty;",
    	" pm.environment.set(\"execution_name\", executionName)",
    	" pm.environment.set(\"execution_id\", executionID)",
    	"});"
    	],
    	"type": "text/javascript"
    	}
    	},
    	{
    	"listen": "prerequest",
    	"script": {
    	"exec": [
    	"console.log(\"test name\", pm.environment.get(\"test_name\"))",
    	""
    	],
    	"type": "text/javascript"
    	}
    	}
    	],
    	"request": {
    	"method": "POST",
    	"header": [],
    	"body": {
    	"mode": "raw",
    	"raw": "{\"namespace\":\"testkube\"}",
    	"options": {
    	"raw": {
    	"language": "json"
    	}
    	}
    	},
    	"url": {
    	"raw": "{{api_uri}}/v1/tests/{{test_name}}/executions",
    	"host": [
    	"{{api_uri}}"
    	],
    	"path": [
    	"v1",
    	"tests",
    	"{{test_name}}",
    	"executions"
    	]
    	}
    	},
    	"response": []
    	},
    	{
    	"name": "Get created test execution by ID",
    	"event": [
    	{
    	"listen": "test",
    	"script": {
    	"exec": [
    	"",
    	"pm.test(\"Check successfull test execution\", function () {",
    	" let jsonData = pm.response.json();",
    	" console.log(\"response\", jsonData);",
    	" let status = jsonData.executionResult.status;",
    	" console.log(\"execution status\", status);",
    	" ",
    	" pm.expect(status).to.not.eq(\"failed\");",
    	" if(status != \"passed\") { ",
    	" setTimeout(() => {}, 1000); // wait for 1 second before retrying",
    	" postman.setNextRequest(pm.info.requestId);",
    	" return;",
    	" } ",
    	"",
    	" pm.expect(jsonData.executionResult.status).to.eql(\"passed\");",
    	" pm.expect(jsonData[\"name\"]).to.eql(pm.environment.get(\"execution_name\"));",
    	" pm.expect(jsonData.executionResult[\"output\"]).contains(\"Health\");",
    	" pm.expect(jsonData.executionResult[\"output\"]).contains(\"200 OK\");",
    	"",
    	"});"
    	],
    	"type": "text/javascript"
    	}
    	},
    	{
    	"listen": "prerequest",
    	"script": {
    	"exec": [
    	"console.log(\"uri\", pm.environment.get(\"api_uri\"));",
    	"console.log(\"test name\", pm.environment.get(\"test_name\"))",
    	"console.log(\"test type\", pm.environment.get(\"test_type\"))",
    	"console.log(\"exec name\", pm.environment.get(\"execution_name\"))",
    	"console.log(\"exec id\", pm.environment.get(\"execution_id\"))",
    	"",
    	""
    	],
    	"type": "text/javascript"
    	}
    	}
    	],
    	"request": {
    	"method": "GET",
    	"header": [],
    	"url": {
    	"raw": "{{api_uri}}/v1/executions/{{execution_id}}",
    	"host": [
    	"{{api_uri}}"
    	],
    	"path": [
    	"v1",
    	"executions",
    	"{{execution_id}}"
    	]
    	}
    	},
    	"response": []
    	},
    	{
    	"name": "List test executions",
    	"event": [
    	{
    	"listen": "test",
    	"script": {
    	"exec": [
    	"pm.test(\"Successfull test execution\", function () {",
    	" let json = pm.response.json();",
    	" jsonData = json.results;",
    	"",
    	" console.log(\"results\", jsonData);",
    	" let contains = false;",
    	" for (let i=0; i<jsonData.length; i++) {",
    	" if (jsonData[i].name == pm.environment.get(\"execution_name\")) {",
    	" contains = true;",
    	" }",
    	" }",
    	"",
    	" pm.expect(contains).to.be.true",
    	"});"
    	],
    	"type": "text/javascript"
    	}
    	},
    	{
    	"listen": "prerequest",
    	"script": {
    	"exec": [
    	"console.log(\"uri\", pm.environment.get(\"api_uri\"));",
    	"console.log(\"test name\", pm.environment.get(\"test_name\"))",
    	"console.log(\"test type\", pm.environment.get(\"test_type\"))",
    	"console.log(\"exec name\", pm.environment.get(\"execution_name\"))",
    	"console.log(\"exec id\", pm.environment.get(\"execution_id\"))",
    	""
    	],
    	"type": "text/javascript"
    	}
    	}
    	],
    	"request": {
    	"method": "GET",
    	"header": [],
    	"url": {
    	"raw": "{{api_uri}}/v1/tests/{{test_name}}/executions",
    	"host": [
    	"{{api_uri}}"
    	],
    	"path": [
    	"v1",
    	"tests",
    	"{{test_name}}",
    	"executions"
    	]
    	}
    	},
    	"response": []
    	},
    	{
    	"name": "List recent test executions",
    	"event": [
    	{
    	"listen": "test",
    	"script": {
    	"exec": [
    	"pm.test(\"Execution is among the recent ones\", function () {",
    	" console.log(\"response\", pm.response.json());",
    	" let jsonArray = pm.response.json();",
    	" let jsonData = jsonArray.results",
    	"",
    	" let contains = false;",
    	" for (let i=0; i<jsonData.length; i++) {",
    	" if (jsonData[i].name == pm.environment.get(\"execution_name\")) {",
    	" contains = true;",
    	" }",
    	" }",
    	"",
    	" pm.expect(contains).to.be.true",
    	"});"
    	],
    	"type": "text/javascript"
    	}
    	},
    	{
    	"listen": "prerequest",
    	"script": {
    	"exec": [
    	"console.log(\"uri\", pm.environment.get(\"api_uri\"));",
    	"console.log(\"test name\", pm.environment.get(\"test_name\"))",
    	"console.log(\"test type\", pm.environment.get(\"test_type\"))",
    	"console.log(\"exec name\", pm.environment.get(\"execution_name\"))",
    	"console.log(\"exec id\", pm.environment.get(\"execution_id\"))",
    	""
    	],
    	"type": "text/javascript"
    	}
    	}
    	],
    	"request": {
    	"method": "GET",
    	"header": [],
    	"url": {
    	"raw": "{{api_uri}}/v1/executions?limit=2",
    	"host": [
    	"{{api_uri}}"
    	],
    	"path": [
    	"v1",
    	"executions"
    	],
    	"query": [
    	{
    	"key": "limit",
    	"value": "2"
    	}
    	]
    	}
    	},
    	"response": []
    	},
    	{
    	"name": "Delete test",
    	"event": [
    	{
    	"listen": "test",
    	"script": {
    	"exec": [
    	"pm.test(\"Most recent execution is that one recently run\", function () {",
    	" pm.response.to.have.status(204);",
    	"});"
    	],
    	"type": "text/javascript"
    	}

  • executions logs within https://demo.testkube.io/tests/executions/sanity
• playright tests
• test source code
  • https://github.com/kubeshop/testkube-dashboard/blob/7bdd0c5ae9804f5d4afcf5cb2440b1d9690cbac2/test/dashboard-e2e/tests/create-test.spec.ts#L13-L31
  • https://github.com/kubeshop/testkube-dashboard/blob/7bdd0c5ae9804f5d4afcf5cb2440b1d9690cbac2/test/dashboard-e2e/tests/run-test.spec.ts#L9-L33
• execution logs https://demo.testkube.io/tests/executions/dashboard-e2e-tests

[vsukhin]

#4017 is done
#4007 is in progress

[senare]

Looks promising !

I got 2 questions (or suggestions maybe?)

First when it comes to auth and authn ... would it be possible to stand on the shoulders of ArgoCD here ?
I just configured ArgoCD <= SAML => Authentik and then added ArgoWorkflow <= OIDC => DEX => ArgoCD (I think thats the flow atleast)

Then we also have Argo Events, which I was looking to add in to the mix for triggering the right flow and coordination etc ... but when you describe it above it is not mentioned, would you care to elaborate as to why a bit ?

LINKS

• Argo Events
• Use Argo CD Dex for authentication
• Authentik