Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reconcile when managed annotations/ labels are out-of-sync #6176

Merged
merged 4 commits into from Jul 4, 2023
Merged

Reconcile when managed annotations/ labels are out-of-sync #6176

merged 4 commits into from Jul 4, 2023

Conversation

inteon
Copy link
Member

@inteon inteon commented Jun 23, 2023 •
edited

Previously, the cert-manager.io annotations were just being ignored while checking the annotations in the template.
This PR adds checks that make sure the correct cert-manager.io annotations are present and that they have the correct value.
Same goes for the controller.cert-manager.io/fao label.

eg. Previously, you could remove or overwrite a cert-manager.io/common-name annotation on a Secret. Now, this annotation will be automatically updated to match its expected value after a change is detected.

Kind

/kind cleanup

Release Note

The SecretPostIssuancePolicyChain now also makes sure that the `cert-manager.io/common-name`, `cert-manager.io/alt-names`, ... annotations on Secrets are kept at their correct value.

@jetstack-bot jetstack-bot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. labels Jun 23, 2023
@jetstack-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jetstack-bot jetstack-bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 23, 2023
@inteon
improve label and annotation checks
a933984 
Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
@inteon inteon force-pushed the reconcile_managed_annotations_and_labels branch from d5d887d to a933984 Compare June 23, 2023 15:08
@inteon inteon requested a review from wallrj June 23, 2023 15:53
wallrj
wallrj reviewed Jun 29, 2023
View reviewed changes
Copy link
Member

@wallrj wallrj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, please:

  • Add a release note to the PR
  • Add a self review, explaining why functions have been renamed or re-ordered.
  • Add an example in the PR description of how the existing code might allow the annotations and labels to get out of sync.

inteon and others added 3 commits June 29, 2023 12:54
@inteon @wallrj
Update internal/controller/certificates/policies/checks.go
1649730 
Co-authored-by: Richard Wall <wallrj@users.noreply.github.com>
Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
@inteon
use .Delete()
c16a34e 
Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
@inteon
add comments explaining what the label and annotation checks do
bfa61c7 
Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
inteon
inteon commented Jun 29, 2023
View reviewed changes
internal/controller/certificates/policies/policies.go
@@ -96,14 +96,16 @@ func NewReadinessPolicyChain(c clock.Clock) Chain {
// correctness of metadata and output formats of Certificate's Secrets.
func NewSecretPostIssuancePolicyChain(ownerRefEnabled bool, fieldManager string) Chain {
return Chain{
SecretTemplateMismatchesSecret,
SecretTemplateMismatchesSecretManagedFields(fieldManager),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, the cert-manager.io annotations were just being ignored while checking the annotations in the template.
This PR adds checks that make sure the correct cert-manager.io annotations are present and that they have the correct value.
Same goes for the controller.cert-manager.io/fao label.

@jetstack-bot jetstack-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jun 29, 2023
@inteon
Copy link
Member Author

inteon commented Jun 29, 2023

/retest

wallrj
wallrj reviewed Jun 30, 2023
View reviewed changes
Copy link
Member

@wallrj wallrj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@inteon Thanks for the extra comments, example and release note.
That helps me understand the problem being solved here.

But is it actually a problem?

If a user chooses to modify or delete labels and annotations which are clearly prefixed with the cert-manager.io/ namespace, they are doing so at their own risk.
When the certificate gets re-issued, the labels will be repopulated anyway.

I vaguely remember that the fao annotation is intended to be changed by the user.
By default cert-manager will add it to the Secrets that it creates, and it causes cert-manager to cache content (in addition to the metadata) of those Secrets.
But a user with thousands of cert-manager managed TLS Secrets can remove that annotation so as to reduce the memory usage of the cert-manager controller.
(conscious that it will increase the traffic to the K8S API ).
@irbekrm: Am I misremembering?

What happens if a Certificate resource is deleted?
Will all the cert-manager.io annotations and labels be removed from the orphaned Secret?

@irbekrm
Copy link
Collaborator

irbekrm commented Jun 30, 2023 •
edited

I vaguely remember that the fao annotation is intended to be changed by the user.
By default cert-manager will add it to the Secrets that it creates, and it causes cert-manager to cache content (in addition to the metadata) of those Secrets.
But a user with thousands of cert-manager managed TLS Secrets can remove that annotation so as to reduce the memory usage of the cert-manager controller.
(conscious that it will increase the traffic to the K8S API ).
@irbekrm: Am I misremembering?

The other way round with regards to user interactions. Secrets with fao label are cached in full, for all non-labellled ones metadata only is cached. Users are intended to add labels to secrets that cert-manager does not label, but accesses if they want to reduce the amount of traffic to kube apiserver. I haven't looked at the contents of this PR, but afaik the fao label was always added if it was found missing on issuance controller reconcile (not just when there was issuance in progress) and it would be good to preserve this behaviour to prevent extra requests to kube apiserver during first re-issuance.

@irbekrm
Copy link
Collaborator

irbekrm commented Jun 30, 2023 •
edited

an remove that annotation so as to reduce the memory usage of the cert-manager controller

To clarify, this was never an intended user action. The cert secrets are not that big, so I don't think it is needed and the cost of retrieving the secret multiple times from kube apiserver would be too high to make sense.

@wallrj
Copy link
Member

wallrj commented Jul 4, 2023

/lgtm

Discussed during standup and I now agree that the post-issuance checks are responsible for all non-re-issuance triggering changes to the Secret.

@jetstack-bot jetstack-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 4, 2023
@jetstack-bot jetstack-bot merged commit 914944c into cert-manager:master Jul 4, 2023
6 checks passed
nrdufour added a commit to nrdufour/home-ops that referenced this pull request Sep 12, 2023
@nrdufour
chore(deps): update helm release cert-manager to v1.13.0 (#84)
ea78c14 
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [cert-manager](https://github.com/cert-manager/cert-manager) | minor | `v1.12.4` -> `v1.13.0` |

---

### Release Notes

<details>
<summary>cert-manager/cert-manager (cert-manager)</summary>

### [`v1.13.0`](https://github.com/cert-manager/cert-manager/releases/tag/v1.13.0)

[Compare Source](cert-manager/cert-manager@v1.12.4...v1.13.0)

cert-manager is the easiest way to automatically manage certificates in Kubernetes and OpenShift clusters.

This is the 1.13 release of cert-manager!

#### Community

Welcome to these new cert-manager members (more info - cert-manager/cert-manager#6260):
[@&#8203;jsoref](https://github.com/jsoref)
[@&#8203;FlorianLiebhart](https://github.com/FlorianLiebhart)
[@&#8203;hawksight](https://github.com/hawksight)
[@&#8203;erikgb](https://github.com/erikgb)

Thanks again to all open-source contributors with commits in this release, including:
[@&#8203;AcidLeroy](https://github.com/AcidLeroy)
[@&#8203;FlorianLiebhart](https://github.com/FlorianLiebhart)
[@&#8203;lucacome](https://github.com/lucacome)
[@&#8203;cypres](https://github.com/cypres)
[@&#8203;erikgb](https://github.com/erikgb)
[@&#8203;ubergesundheit](https://github.com/ubergesundheit)
[@&#8203;jkroepke](https://github.com/jkroepke)
[@&#8203;jsoref](https://github.com/jsoref)
[@&#8203;gdvalle](https://github.com/gdvalle)
[@&#8203;rouke-broersma](https://github.com/rouke-broersma)
[@&#8203;schrodit](https://github.com/schrodit)
[@&#8203;zhangzhiqiangcs](https://github.com/zhangzhiqiangcs)
[@&#8203;arukiidou](https://github.com/arukiidou)
[@&#8203;hawksight](https://github.com/hawksight)
[@&#8203;Richardds](https://github.com/Richardds)
[@&#8203;kahirokunn](https://github.com/kahirokunn)

Thanks also to the following cert-manager maintainers for their contributions during this release:
[@&#8203;SgtCoDFish](https://github.com/SgtCoDFish)
[@&#8203;maelvls](https://github.com/maelvls)
[@&#8203;irbekrm](https://github.com/irbekrm)
[@&#8203;inteon](https://github.com/inteon)

Equally thanks to everyone who provided feedback, helped users and raised issues on Github and Slack and joined our meetings!

Special thanks to [@&#8203;AcidLeroy](https://github.com/AcidLeroy) for adding "load options from a versioned config file" support for the cert-manager controller! This has been on our wishlist for a very long time. (see cert-manager/cert-manager#5337)

Also, thanks a lot to [@&#8203;FlorianLiebhart](https://github.com/FlorianLiebhart) for adding support for DNS over HTTPS for the ACME DNS self-check. This is very useful in case all traffic must be HTTP(S) trafic, eg. when using a HTTPS_PROXY. (see cert-manager/cert-manager#5003)

Thanks also to the [CNCF](https://www.cncf.io/), which provides resources and support, and to the AWS open source team for being good community members and for their maintenance of the [PrivateCA Issuer](https://github.com/cert-manager/aws-privateca-issuer).

In addition, massive thanks to [Venafi](https://www.venafi.com/) for contributing developer time and resources towards the continued maintenance of cert-manager projects.

#### Changes since v1.12.0

##### Feature

-   Add support for logging options to webhook config file. ([#&#8203;6243](cert-manager/cert-manager#6243), [@&#8203;inteon](https://github.com/inteon))
-   Add view permissions to the well-known (Openshift) user-facing `cluster-reader` aggregated cluster role ([#&#8203;6241](cert-manager/cert-manager#6241), [@&#8203;erikgb](https://github.com/erikgb))
-   Certificate Shim: distinguish dns names and ip address in certificate ([#&#8203;6267](cert-manager/cert-manager#6267), [@&#8203;zhangzhiqiangcs](https://github.com/zhangzhiqiangcs))
-   Cmctl can now be imported by third parties. ([#&#8203;6049](cert-manager/cert-manager#6049), [@&#8203;SgtCoDFish](https://github.com/SgtCoDFish))
-   Make `enableServiceLinks` configurable for all Deployments and `startupapicheck` Job in Helm chart. ([#&#8203;6292](cert-manager/cert-manager#6292), [@&#8203;ubergesundheit](https://github.com/ubergesundheit))
-   Promoted the StableCertificateRequestName and SecretsFilteredCaching feature gates to Beta (enabled by default). ([#&#8203;6298](cert-manager/cert-manager#6298), [@&#8203;inteon](https://github.com/inteon))
-   The cert-manager controller options are now configurable using a configuration file. ([#&#8203;5337](cert-manager/cert-manager#5337), [@&#8203;AcidLeroy](https://github.com/AcidLeroy))
-   The pki CertificateTemplate functions now perform validation of the CSR blob, making sure we sign a Certificate that matches the IsCA and (Extended)KeyUsages that are defined in the CertificateRequest resource. ([#&#8203;6199](cert-manager/cert-manager#6199), [@&#8203;inteon](https://github.com/inteon))
-   \[helm] Add prometheus.servicemonitor.endpointAdditionalProperties to define additional properties on a ServiceMonitor endpoint, e.g. relabelings ([#&#8203;6110](cert-manager/cert-manager#6110), [@&#8203;jkroepke](https://github.com/jkroepke))

##### Design

-   DNS over HTTPS (DoH) is now possible for doing the self-checks during the ACME verification.
    The DNS check method to be used is controlled through the command line flag: `--dns01-recursive-nameservers-only=true` in combination with `--dns01-recursive-nameservers=https://<DoH-endpoint>` (e.g. `https://8.8.8.8/dns-query`). It keeps using DNS lookup as a default method. ([#&#8203;5003](cert-manager/cert-manager#5003), [@&#8203;FlorianLiebhart](https://github.com/FlorianLiebhart))

##### Bug or Regression

-   Allow overriding default pdb .minAvailable with .maxUnavailable without setting .minAvailable to null ([#&#8203;6087](cert-manager/cert-manager#6087), [@&#8203;rouke-broersma](https://github.com/rouke-broersma))
-   BUGFIX: `cmctl check api --wait 0` exited without output and exit code 1; we now make sure we perform the API check at least once and return with the correct error code ([#&#8203;6109](cert-manager/cert-manager#6109), [@&#8203;inteon](https://github.com/inteon))
-   BUGFIX: the issuer and certificate-name annotations on a Secret were incorrectly updated when other fields are changed. ([#&#8203;6147](cert-manager/cert-manager#6147), [@&#8203;inteon](https://github.com/inteon))
-   BUGFIX\[cainjector]: 1-character bug was causing invalid log messages and a memory leak ([#&#8203;6232](cert-manager/cert-manager#6232), [@&#8203;inteon](https://github.com/inteon))
-   Fix CloudDNS issuers stuck in propagation check, when multiple instances are issuing for the same FQDN ([#&#8203;6088](cert-manager/cert-manager#6088), [@&#8203;cypres](https://github.com/cypres))
-   Fix indentation of Webhook NetworkPolicy matchLabels in helm chart. ([#&#8203;6220](cert-manager/cert-manager#6220), [@&#8203;ubergesundheit](https://github.com/ubergesundheit))
-   Fixed Cloudflare DNS01 challenge provider race condition when validating multiple domains ([#&#8203;6191](cert-manager/cert-manager#6191), [@&#8203;Richardds](https://github.com/Richardds))
-   Fixes a bug where webhook was pulling in controller's feature gates.
    ⚠️  ⚠️ BREAKING ⚠️ ⚠️ : If you deploy cert-manager using helm and have `.featureGates` value set, the features defined there will no longer be passed to cert-manager webhook, only to cert-manager controller. Use `webhook.featureGates` field instead to define features to be enabled on webhook.
    **Potentially breaking**: If you were, for some reason, passing cert-manager controller's features to webhook's `--feature-gates` flag, this will now break (unless the webhook actually has a feature by that name). ([#&#8203;6093](cert-manager/cert-manager#6093), [@&#8203;irbekrm](https://github.com/irbekrm))
-   Fixes an issue where cert-manager would incorrectly reject two IP addresses as being unequal when they should have compared equal. This would be most noticeable when using an IPv6 address which doesn't match how Go's `net.IP.String()` function would have printed that address. ([#&#8203;6293](cert-manager/cert-manager#6293), [@&#8203;SgtCoDFish](https://github.com/SgtCoDFish))
-   We disabled the `enableServiceLinks` option for our ACME http solver pods, because the option caused the pod to be in a crash loop in a cluster with lot of services. ([#&#8203;6143](cert-manager/cert-manager#6143), [@&#8203;schrodit](https://github.com/schrodit))
-   ⚠️ possibly breaking: Webhook validation of CertificateRequest resources is stricter now: all KeyUsages and ExtendedKeyUsages must be defined directly in the CertificateRequest resource, the encoded CSR can never contain more usages that defined there. ([#&#8203;6182](cert-manager/cert-manager#6182), [@&#8203;inteon](https://github.com/inteon))

##### Other (Cleanup or Flake)

-   A subset of the klogs flags have been deprecated and will be removed in the future. ([#&#8203;5879](cert-manager/cert-manager#5879), [@&#8203;maelvls](https://github.com/maelvls))
-   All service links in helm chart deployments have been disabled. ([#&#8203;6144](cert-manager/cert-manager#6144), [@&#8203;schrodit](https://github.com/schrodit))
-   Cert-manager will now re-issue a certificate if the public key in the latest CertificateRequest resource linked to a Certificate resource does not match the public key of the key encoded in the Secret linked to that Certificate resource ([#&#8203;6168](cert-manager/cert-manager#6168), [@&#8203;inteon](https://github.com/inteon))
-   Chore: When hostNetwork is enabled, dnsPolicy is now set to ClusterFirstWithHostNet. ([#&#8203;6156](cert-manager/cert-manager#6156), [@&#8203;kahirokunn](https://github.com/kahirokunn))
-   Cleanup the controller configfile structure by introducing sub-structs. ([#&#8203;6242](cert-manager/cert-manager#6242), [@&#8203;inteon](https://github.com/inteon))
-   Don't run API Priority and Fairness controller in webhook's extension apiserver ([#&#8203;6085](cert-manager/cert-manager#6085), [@&#8203;irbekrm](https://github.com/irbekrm))
-   Helm: Add apache 2.0 license annotation ([#&#8203;6225](cert-manager/cert-manager#6225), [@&#8203;arukiidou](https://github.com/arukiidou))
-   Make apis/acme/v1/ACMEIssuer.PreferredChain optional in JSON serialization. ([#&#8203;6034](cert-manager/cert-manager#6034), [@&#8203;gdvalle](https://github.com/gdvalle))
-   The SecretPostIssuancePolicyChain now also makes sure that the `cert-manager.io/common-name`, `cert-manager.io/alt-names`, ... annotations on Secrets are kept at their correct value. ([#&#8203;6176](cert-manager/cert-manager#6176), [@&#8203;inteon](https://github.com/inteon))
-   The cmctl logging has been improved and support for json logging has been added. ([#&#8203;6247](cert-manager/cert-manager#6247), [@&#8203;inteon](https://github.com/inteon))
-   Updates Kubernetes libraries to `v0.27.2`. ([#&#8203;6077](cert-manager/cert-manager#6077), [@&#8203;lucacome](https://github.com/lucacome))
-   Updates Kubernetes libraries to `v0.27.4`. ([#&#8203;6227](cert-manager/cert-manager#6227), [@&#8203;lucacome](https://github.com/lucacome))
-   We now only check that the issuer name, kind and group annotations on a Secret match in case those annotations are set. ([#&#8203;6152](cert-manager/cert-manager#6152), [@&#8203;inteon](https://github.com/inteon))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi4yMy4yIiwidXBkYXRlZEluVmVyIjoiMzYuMjMuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Reviewed-on: https://git.home/nrdufour/home-ops/pulls/84
Co-authored-by: Renovate <renovate@ptinem.io>
Co-committed-by: Renovate <renovate@ptinem.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wallrj wallrj wallrj requested changes

Assignees

@wallrj wallrj

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@inteon @jetstack-bot @irbekrm @wallrj