[tempo-distributed] Enable autoscaling for Compactor

[7onn]

related issue: #2820

This PR aims to enable autoscaling feature for the compactor workload in tempo-distrubuted Helm Chart.

It implements two autoscaling possibilities, via HPA and/or via ScaledObject.

My motivation came while I was oncall and one alert introduced me to this runbook. Even though I scaled out the workload manually, I couldn't help but to think that it would make sense to scale out this workload based on the metric that triggered the alert as a critical thing. The value commented in the example was copied from here.

How to locally test this code change?

• Checkout this branch
• In the values.yaml, enable the autoscaling + keda and/or hpa
  • Replace the empty array for the commented Prometheus trigger
• Use the commands below to output the manifests

$ cd charts/tempo-distributed
# Check for the Keda ScaledObject output
$ helm -n tempo template tempo . | yq '. | select(.kind == "ScaledObject")' | kubectl diff -f -
# Check for the HPA output
$ helm -n tempo template tempo . | yq '. | select(.kind == "HorizontalPodAutoscaler")' | kubectl diff -f -

[CLAassistant]

All committers have signed the CLA.