Duration metrics for important reconciliation actions

[seaneagan]

#329 added a metric to determine the durations of full reconciliation cycles. This doesn't however say anything about the durations of impactful actions taken during the reconciliations. Most reconciliations will be noops and run very quickly. Thus it would be useful to have metric(s) to track each important action that the controllers take during reconciliation. For example:

gotk_action_duration_seconds_bucket{kind="Kustomization",name="podinfo",namespace="gotk-system", action="build" le="1e-08"} 0
gotk_action_duration_seconds_bucket{kind="Kustomization",name="podinfo",namespace="gotk-system", action="apply" le="1e-08"} 0
gotk_action_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="gotk-system", action="install" le="1e-08"} 0
gotk_action_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="gotk-system", action="upgrade" le="1e-08"} 0
gotk_action_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="gotk-system", action="test" le="1e-08"} 0
gotk_action_duration_seconds_bucket{kind="GitRepository",name="podinfo",namespace="gotk-system", action="clone" le="1e-08"} 0

It would also be possible to simply add reconcile as one of these actions, and subsume the gotk_reconcile_duration metric:

gotk_action_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="gotk-system", action="reconcile" le="1e-08"} 0

This does multiply the cardinality by whatever the average number of action types taken on each resource is. It also increases the API surface, so actions should be named and included carefully.