Label runner pod with the current job running on it

[fishpen0]

What would you like added?

Update the labels on the runner pod to include the current job name and id whenever a job is assigned to a pod and remove it when the job exits.

Why is this needed?

We would like better insights into which jobs are accurately sized for our different worker classes. If ARC could update the labels on a runner pod with the name or id of the job that is currently running on it, that would allow us to aggregate metrics based on the historical performance of jobs in different worker configurations

Additional context

Pods do not need to be restarted when the only change is the label. Prometheus and other tools can collect and filter metrics on pods based on the current labels

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[duhow]

NOTE: There's a manual workaround in here from @0xF4D3C0D3 : #2176 (comment)

• Adding a hook to the Runner image to update Pod itself upon picking a job.