Webhook Flake on Upgrade

[dprotaso]

I wonder if we are clearing certificates?

upgrade.go:98: Failed to create Service: Internal error occurred: failed calling webhook "webhook.serving.knative.dev": failed to call webhook: Post "https://webhook.db15bd17-dfe9-41c9-9dfb-dd8115ecfe22.svc:443/?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "webhook.db15bd17-dfe9-41c9-9dfb-dd8115ecfe22.svc")

Originally posted by @dprotaso in #15141 (comment)

[skonto]

@dprotaso is not true that the certificate reconciler fills in the secret with a certificate based on the service name of the webhook and during the upgrade we override the secret with empty content? I suspect the new webhook controller loads the new cert before it is filled in by the reconciler and thus the error. I think we need to keep the secret around and not update it or wait for the webhook or something? I am wondering if instead of just presenting the certificate with GetCertificate we should also link readiness with proper certificate content (it happens elsewhere too tbh cert-manager/cert-manager#3045)?