feat: Dynatrace scaler

[cyrilico]

Add new scaler for interacting with Dynatrace and its Get Metric Data Points API

Checklist

• When introducing a new scaler, I agree with the scaling governance policy
• I have verified that my change is according to the deprecations & breaking changes policy
• Tests have been added
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update the documentation on (repo) (if applicable)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Closes #2411

Relates to kedacore/keda-docs#1360

[zroubalik]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

Please, rebase your branch @cyrilico , the e2e issue was related with some breaking changes in otel-collector which are already fixed in main branch (your tests hasn't been executed yet indeed)

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[cyrilico]

new changes, please rerun

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

@cyrilico I've merged 2 fixes to the e2e to trigger them again (you'll need to pull the changes :) )

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

It seems that the query isn't correct. Checking the logs I see lines like this:

2024-04-28T22:27:24Z	ERROR	dynatrace_scaler	error executing Dynatrace query	{"type": "ScaledObject", "namespace": "dynatrace-test-ns", "name": "dynatrace-test-so", "error": "dynatrace response does not contain any results"}

[cyrilico]

I misnamed a response property name which must have caused the inability to correctly parse it, not sure where I got the original from, probably misread; the query itself is pretty generic, should always return something

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

2024-04-29T19:24:29.2272752Z === RUN   TestDynatraceScaler
2024-04-29T19:24:29.2276455Z     dynatrace_test.go:***9: --- setting up ---
2024-04-29T19:24:29.2280365Z     helper.go:***48: deleting namespace dynatrace-test-ns
2024-04-29T19:24:29.2299743Z     helper.go:***0***: waiting for namespace dynatrace-test-ns deletion
2024-04-29T19:24:29.2304187Z     helper.go:***5: Creating namespace - dynatrace-test-ns
2024-04-29T19:24:29.2308107Z     helper.go:550: Applying template: secretTemplate
2024-04-29T19:24:29.2312194Z     helper.go:550: Applying template: triggerAuthenticationTemplate
2024-04-29T19:24:29.2316268Z     helper.go:550: Applying template: serviceTemplate
2024-04-29T19:24:29.2320314Z     helper.go:550: Applying template: monitoredDeploymentTemplate
2024-04-29T19:24:29.2323148Z     helper.go:550: Applying template: deploymentTemplate
2024-04-29T19:24:29.2325160Z     helper.go:550: Applying template: scaledObjectTemplate
2024-04-29T19:24:29.2327143Z     helper.go:550: Applying template: dynakubeTemplate
2024-04-29T19:24:29.2328861Z     helper.go:564: 
2024-04-29T19:24:29.2330976Z         	Error Trace:	/__w/keda/keda/tests/helper/helper.go:564
2024-04-29T19:24:29.2333355Z         	            				/__w/keda/keda/tests/helper/helper.go:59***
2024-04-29T19:24:29.2335864Z         	            				/__w/keda/keda/tests/scalers/dynatrace/dynatrace_test.go:***46
2024-04-29T19:24:29.2338330Z         	Error:      	Received unexpected error:
2024-04-29T19:24:29.2340662Z         	            	error: error when retrieving current configuration of:
2024-04-29T19:24:29.2343670Z         	            	Resource: "dynatrace.com/v***beta***, Resource=dynakubes", GroupVersionKind: "dynatrace.com/v***beta***, Kind=DynaKube"
2024-04-29T19:24:29.2346580Z         	            	Name: "", Namespace: "default"
2024-04-29T19:24:29.2349146Z         	            	from server for: "/tmp/dynakubeTemplate***6***97***5***": resource name may not be empty
2024-04-29T19:24:29.2351676Z         	Test:       	TestDynatraceScaler
2024-04-29T19:24:29.2354391Z         	Messages:   	cannot apply file - error: error when retrieving current configuration of:
2024-04-29T19:24:29.2357696Z         	            	Resource: "dynatrace.com/v***beta***, Resource=dynakubes", GroupVersionKind: "dynatrace.com/v***beta***, Kind=DynaKube"
2024-04-29T19:24:29.2360534Z         	            	Name: "", Namespace: "default"
2024-04-29T19:24:29.2370269Z         	            	from server for: "/tmp/dynakubeTemplate***6***97***5***": resource name may not be empty
2024-04-29T19:24:29.2373656Z     helper.go:444: Waiting for deployment replicas to hit target. Deployment - dynatrace-test-deployment, Current  - 0, Target - 0
2024-04-29T19:24:29.2376385Z     dynatrace_test.go:***58: --- testing activation ---

[JorTurFer]

/run-e2e dynatrace
Update: You can check the progress here

[JorTurFer]

Same result:

2024-04-29T20:05:14Z	ERROR	dynatrace_scaler	error executing Dynatrace query	{"type": "ScaledObject", "namespace": "dynatrace-test-ns", "name": "dynatrace-test-so", "error": "dynatrace response does not contain any metric series"}

Is this metric correct? "builtin:service.requestCount.total:splitBy():fold"

[cyrilico]

@JorTurFer I am certain the query should return a valid result. Is the host variable still using the .apps sub domain instead of .live as we discussed in the issue? If so, I believe that might indeed be a problem, as testing locally against another Dynatrace instance I have access to, using that URL results in an error, thinking we're trying to use another kind of API: Unsupported authorization scheme 'Api-Token'. Dynatrace platform APIs require the authorization scheme 'Bearer'. Use the OAuth2.0 Client Credentials Flow to obtain a token.

[JorTurFer]

I've been on a business trip this week, but let me check out your branch and check the current code using the current secrets

[JorTurFer]

Hello,
I've been testing this locally and I found some things:

• The dynatrace operator fails because the token doesn't have enough permissions for it
• The job generators aren't generating the load because the values are overridden
  

For the dynatrace operator issue, I've created a new secret 'DYNATRACE_OPERATOR_TOKEN', please use it for the operator and the other one for metrics (including for reading them by KEDA)

After making these changes, I see that dynatrace operator has created more pods:

But the metric isn't still available:

I think that the issue can be in something related with the e2e test configuration because I can see that KEDA is querying the value and getting an empty response with status 200 (so from KEDA side seems that works)

[cyrilico]

Appreciate the help @JorTurFer 🙏 What metrics are available in that page? The one I used is supposed to be a built-in, i.e., supported OOTB, so I'm curious as to what exactly is available in that dynatrace environment