Investigate resource usage on hot clusters

[sebgoa]

When a cluster has e.g 30,000 secrets our informers could get quite busy .

Check the latest knative controller ways to reduce the load.

[sebgoa]

From Evan:

There are typically two cases:
We accidentally watch() a resource which normally has low cardinality, but some other application on the cluster pushes it to high cardinality. We’ve seen this for ConfigMaps and Secrets.
The Informer cache can get large in large clusters. In those cases, you need to increase memory limits on the controller, and you probably want to increase requests, too. You can observe this with standard telemetry, and can use the profiling port and profiling telemetry option to get detailed heap allocation statistics to pinpoint which objects are contributing.

and check: knative/serving#13442