[WIP] Feat: Add clusterTLSConfigs to the Alertmanager CRD

[mviswanathsai]

Description

This PR adds mTLS config to AlertManager, allowing user to configure gossip between alert managers using mTLS for authentication.
Closes: #4241

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• [ x] FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Verification

Please check the Prometheus-Operator testing guidelines for recommendations about automated tests.

Changelog entry

Please put a one-line changelog entry below. This will be copied to the changelog file during the release process.

add mTLS configuration to alertmanager CRD

[mviswanathsai]

Questions:

1. What about concurrency and timeouts, that we have in AlertmanagerWebConfig? Would that apply here?
2. I am still not sure about the naming, feedback is much needed.
3. What about "de-duplication" logic in [WIP] add MTLS config for alert manager #4593? would we need to have different naming conventions for server and client tls creds? clarity is needed in this area.
4. Would like suggestions if any, for the code yet to be written. For example, I am writing the code in alertmanager/statefulset.go to include the volumes, mounts and args for the clusterTLSConfig (mTLSConfig), on the condition that the version of prometheus is >=0.27.0. This is yet to be pushed.

[mviswanathsai]

Requesting review: @simonpasquier @ArthurSens @nicolastakashi @slashpai.

Some concepts are still not clear to me, especially stuff that is not directly related to the code itself but the cluster tls config in general. I think the review will help a lot. Kindly bear with me, if I am not up to speed!

[simonpasquier]

I don't think that the comment applies here because an empty mTLS config file isn't allowed. So we'll have to redeploy the pods if the spec changes from no-TLS to TLS (and vice-versa).

[mviswanathsai]

Right, so we cannot mount an empty volume like we do with web tls. So here, we only mount the config and redeploy the pod depending on whether there is mTLS config or not.

[mviswanathsai]

// If the am version >= 0.24.0, create a new secret for the mTLS configuration.
// The secret volume is always created and mounted, even if the secret is empty.
// But, empty volume cannot be passed as the path in the argument for the cluster TLS configuration.
// So, we only pass the argument `--cluster.tls-config` if the ClusterTLSConfig field is not nil.

[simonpasquier]

After c.Client.Validate(), we should check that the cert and key fields are valid (not empty) configmap/secret references.

[mviswanathsai]

I have added some checks. However, I do not quite understand why this does not apply for web TLS. Could you briefly touch upon that?

[mviswanathsai]

Background:
When configuring TLS credentials for both the server and the clients, there was an issue where the same names or parameters could be assigned to both, leading to duplication. To handle this, a "de-duplication" logic was implemented (Edit: in #4593). This logic ensured that secrets with the same name were not added more than once to the list of parameters.

Problem with De-duplication:
The "de-duplication" mechanism seems to assume that the secrets with the same name, regardless of being part of the client's tls creds or the server's are the same. I am unable to verify if this is right or not. So, to avoid any confusions, I adopted the following alternative approach.

Alternative Approach:
To avoid potential issues with de-duplication, a new approach was adopted. Unique suffixes are now assigned to the TLS credentials for the server and the clients. This ensures that each set of credentials is distinct, eliminating the risk of duplication without the need for additional logic. This logic can be seen in action here.