RFC2136 provider sending wrong domain to DNS server

[frankfil]

Describe the bug:
I have struck what I believe is the exact same issue as previously reported in #5650.

In my case details are as follows:

cert-manager was attempting to renew a certificate for example.com.au and www.example.com.au.

It had stalled for a couple of days due to bug #5917 so I restarted the cert-manager Pod as this sometimes fixes that problem.

However this time cert-manager started logging the following errors (repeated many times):

{"ts":1715752316870.572,"caller":"dns/dns.go:88","msg":"cert-manager/challenges/Present: presenting DNS01 challenge for domain","v":2,"resource_name":"example-com-au-1-2457021599-1243604751","resource_namespace":"example-namespace","resource_kind":"Challenge","resource_version":"v1","dnsName":"www.example.com.au","type":"DNS-01","resource_name":"example-com-au-1-2457021599-1243604751","resource_namespace":"example-namespace","resource_kind":"Challenge","resource_version":"v1","domain":"www.example.com.au"}
{"ts":1715752316878.8406,"caller":"controller/controller.go:167","msg":"cert-manager/challenges: re-queuing item due to error processing","key":"example-namespace/example-com-au-1-2457021599-1243604751","err":"DNS update failed: dns: bad authentication"}
{"ts":1715752336737.8887,"caller":"dns/dns.go:88","msg":"cert-manager/challenges/Present: presenting DNS01 challenge for domain","v":2,"resource_name":"example-com-au-1-2457021599-875669868","resource_namespace":"example-namespace","resource_kind":"Challenge","resource_version":"v1","dnsName":"example.com.au","type":"DNS-01","resource_name":"example-com-au-1-2457021599-875669868","resource_namespace":"example-namespace","resource_kind":"Challenge","resource_version":"v1","domain":"example.com.au"}
{"ts":1715752336742.7222,"caller":"controller/controller.go:167","msg":"cert-manager/challenges: re-queuing item due to error processing","key":"example-namespace/example-com-au-1-2457021599-875669868","err":"DNS update failed: dns: bad authentication"}

And on my DNS server running PowerDNS I see the following log entries (repeated many times):

May 15 16:02:16 pdns-hostname pdns_server[2813604]: UPDATE (51511) from 172.21.130.35 for com.au: TSIG is provided, but domain is not secured with TSIG. Processing continues
May 15 16:02:16 pdns-hostname pdns_server[2813604]: UPDATE (51511) from 172.21.130.35 for com.au: Can't determine backend for domain 'com.au' (or backend does not support DNS update operation)

Sure enough if I create a domain in the server called com.au then cert-manager promptly sets the ACME Challenge TXT record there rather than under the expected domain example.com.au.

I've tried deleting the Challenges, Order, Certificate Request, Certificate, all individually and at the same time, but the issue comes back again.

I've tried deleting those objects and restarting cert-manager whilst they are deleted, still the same problem.

I've verified the CSR in the Certificate Request and Order objects has the correct domain names, and that the spec.dnsName property in the Challenges are also correct.

Expected behaviour:
The DNS record is created in the correct DNS zone!

Steps to reproduce the bug:
Sorry I have no idea how to duplicate this at this time.

What is really confusing is that even after deleting the Certificate and restarting cert-manager this comes back.

Is there some data stored/cached somewhere else?

Anything else we need to know?:

Environment details::

• Kubernetes version: v1.23.7
• Cloud-provider/provisioner: Bare metal
• cert-manager version: v1.13.0
• Install method: helm

/kind bug

[frankfil]

Not a bug!

Turns out the domain registration had not been renewed (and obviously this had not been communicated to everyone who should know this) and somehow this has resulted in cert-manager thinking the domain it needed to update the DNS challenge records in was com.au and not example.com.au.

That might be considered a separate bug in of itself, or at least something that should log a more meaningful error, but given this is a client side configuration issue I'm going to close this and update my clients domain decommissioning procedure!