KEP-4322: Add cluster inventory definition

[ryanzhang-oss]

• One-line PR description:
  Add the definition of a cluster Inventory and some notes on how it should be consumed.

• Issue link:

• Other comments:

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ryanzhang-oss / name: Ryan Zhang (340745d)

[k8s-ci-robot]

Welcome @ryanzhang-oss!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @ryanzhang-oss. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ArangoGutierrez]

/cc

[ArangoGutierrez]

/ok-to-test

[ArangoGutierrez]

@ryanzhang-oss: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-enhancements-verify	04f03c3	link	true	/test pull-enhancements-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ryanzhang-oss please address the issue

[ArangoGutierrez]

Summary of the error

Table of content not up to date. Did you run 'make update-toc' ?

[ArangoGutierrez]

Fix build

Table of content not up to date. Did you run 'make update-toc' ?

[sftim]

(nit)
/retitle KEP-4322: Add cluster inventory definition

[ryanzhang-oss]

/test pull-enhancements-verify

[ArangoGutierrez]

please squash the commits :)

[mikeshng]

Suggested change

	- **Cluster Inventory**: A collection of clusters governed by a single cluster manager.
	- **Cluster Inventory**: A conceptual term referring to a collection of clusters managed by a single cluster manager.

[mikeshng]

Should we need to give a quick mention to ClusterSet? https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api#terminology

[corentone]

I'm confused why ClusterInventory is linked to a cluster Manager and not linked to "All cluster profiles in a given tuple {HubCluster, Namespace}" or "ClusterInventory is the List of ClusterProfile in a namespace". A hub cluster can hold multiple inventories.

Why do we need to talk about ClusterManager here? I feel that's adding stuff to an api not needing it at the moment.

Also, +1 to mikesng@'s "Conceptual term" edit.

[mikeshng]

Suggested change

	- **Member Cluster**: A kubernetes cluster that is part of the cluster Inventory.
	- **Member Cluster**: A kubernetes cluster that is part of the Cluster Inventory.

cluster Inventory should be either Cluster Inventory or cluster inventory. Same comment for the recurrences below.

[ryanzhang-oss]

I will use the term "cluster inventory"

[mikeshng]

Do we need to add:

#### What's the relationship between a ClusterProfile API and ClusterSet?

[ryanzhang-oss]

I thought it was transitive as we defined the relationship between ClusterProfile API <-> cluster Inventory and cluster Inventory <-> clusterSet but I think we can be more clear.

[dixudx]

A clusterset is normally defined as a bundle of clusters that governed by a single authority. They usually have some common or close characteristics/labels, such as the same region/cloud provider/business prupose, etc.

While for cluster inventory, we don't have such a strong meaning. Those clusters can be loosely tied together to a cluster inventory. I think we can think of cluster inventory as a superset of clustersets.

[corentone]

That still works here I think, given the definition of ClusterInventory to be a collection of ClusterProfile.
A clusterInventory could be taken at the hub-cluster level and still each namespace to be a clusterSet.

I think the ClusterProfile is within a Namespace, then one can consume the inventory at the namespace or hub-cluster level (sum of all namespaces).
For cluster-set, we're implying the namespace level is the way and adding an extra requirement (all clusters in a namespace-level inventory are part of the same clusterset)

[corentone]

Looking forward to finalizing tomorrow!

I think an inventory isClusterProfileList(Namespace).
Which means that it's not only a concept but it's API-tied to ClusterProfile and Namespace on a hub cluster.

Then let's clarify the ClusterSet rules. I think we're on the same page.
An inventory can be at most a single cluster set or there is no clusterset concept within the inventory.

[corentone]

I'm confused why ClusterInventory is linked to a cluster Manager and not linked to "All cluster profiles in a given tuple {HubCluster, Namespace}" or "ClusterInventory is the List of ClusterProfile in a namespace". A hub cluster can hold multiple inventories.

Why do we need to talk about ClusterManager here? I feel that's adding stuff to an api not needing it at the moment.

Also, +1 to mikesng@'s "Conceptual term" edit.

[corentone]

does it need to be specified? How else would a ClusterManager be able to do its job?

[corentone]

(new to kep writing here) Do kep usually refer to CRDs as API?

A ClusterProfile CR/Instance = a Cluster
A ClusterInventory = The list of Clusters in the same target hub Cluster, Namespace.

Is that similar to what you're saying?

[ryanzhang-oss]

Yes, it's the same. I think KEP usually lays out the concepts and rules instead of going to describe system design.

[corentone]

I'm not sure this is automatic like that. I think it should be a conscious decision, available to consumers for proper assumptions.

Suggested change

	A cluster inventory is considered a clusterSet if all its member clusters adhere to the
	A cluster inventory may or may not represent a ClusterSet. If it does, only one clusterset should be represented in the inventory. (In other terms, a ClusterInventory can map to 0 or 1 ClusterSet).
	As a result of representing a ClusterSet, member clusters would adhere to sameness rules among themselves.

[corentone]

with my definition above, they do since the inventory IS the {cluster, namespace} tuple, it's not a recommendation, it's by definition.

I think inventory is an actual ListClusterProfile(Namespace), not really a concept?

[ryanzhang-oss]

cluster inventory is a concept. I am not sure if we force the ClusterProfile to be CRDs in this KEP.

[corentone]

Suggested change

While there are no strict requirements, we suggest making the clusterProfile API a namespace-scoped object.

The ClusterProfile CRD is a namespace-scoped object, this approach allows to maintain separate cluster inventories on the same cluster, making for easier testing, a multi-inventory feature such multi-environment deployment support and other use cases that may require crossing inventory boundaries. It also allows RBAC to be set at an inventory level, providing some stronger isolation between inventories.

I think there is at this point. We should be less ambiguous.

Let's say that ClusterProfile CRD is made namespace-scoped

[corentone]

I don't think that's true.

I think clusterset would be a flag on the inventory at this point. So all clusters of the inventory are in the clusterset. I don't think it guarantees completeness of the clusterset. It means that consumers of the inventory can leverage sameness within that inventory safely.

[ryanzhang-oss]

I wonder what's the benefit to allow splitting one clusterSet into multiple namespaces? Also, what will be the name of the namespace?

[corentone]

The name of the namespace doesn't really matter if you have the label.

I think what's important with clusterset is guaranteeing that all clusters can be considered "having sameness" for a user, because that enables a lot of management simplicity. The requirement is that all clusters are part of the same cluster, not necessarily that they are all there.

use cases I can think of:

• regional management of clusterset; if one wanted localized management of clusters. Example: you have 3 clusters in europe and 3 clusters in us. They are part of the same clusterset but you want a regional hub-cluster. You would have 2 hub cluster that each manage 1 inventory of 3 clusters. The two regional clusters would have to be guaranteed by something global that they wont break sameness (maybe a global hub-cluster?).
• isolation of management: let's say we've given a inventory consumer the ability to manage N namespaces within some clusterProfiles, we could have an inventory of clusters thats a subset of the ClusterSet
• migration: if we needed to migrate a clusterset, we will need to move one cluster at a time to the new representation of the clusterset (but without breaking the ClusterSet rules)
• timing of ON/OFF: clusters may come and go and join/leave the set as they are managed, so the completeness of the set may vary

Obviously a consumer of the API would be limited by the set of profiles they see and it may impact behaviors if the clusterset is not complete, but I think the main rule that we care about here is that the consumer can treat the clusters as being part of the same clusterset.

[corentone]

+1 to the first sentence about labelling. It's a bit opiniated but has the advantage of being specific :)

I don't understand the second part RBAC, it sounds like another piece you have in mind. What is "native namespace-based RBAC" here? why is it different for a namespace with clusterset label?

[corentone]

it's not only about sameness, its about clusterset rules.
You can just say:

A Cluster can only be in one ClusterSet and therefore its ClusterProfiles can only be in namespaces of that same clusterset or no clusterset.

[ryanzhang-oss]

I think I am explaining the reason behind the rule

[mikeshng]

/lgtm

[k8s-ci-robot]

@mikeshng: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[qiujian16]

/lgtm with only a minor comment

[dixudx]

A clusterset is normally defined as a bundle of clusters that governed by a single authority. They usually have some common or close characteristics/labels, such as the same region/cloud provider/business prupose, etc.

While for cluster inventory, we don't have such a strong meaning. Those clusters can be loosely tied together to a cluster inventory. I think we can think of cluster inventory as a superset of clustersets.

[corentone]

almost there :)

I think it defines most of what we need and my comments are on some details at this point. Thank you!

[corentone]

"If they wish to restrict access to certain clusters within the inventory"
should be:
"If they wish to control access to certain cluster profiles"

Otherwise this defines the inventory as the full hub, which is not correct. An inventory is any group of cluster per the definition above. (if we do get into that kind of detail, I think ClusterInventory is a Namespace of ClusterProfiles; but I think it's okay for the KEP to be vague here)

[ryanzhang-oss]

I wonder how one can use namespace based RBAC to control access of certain cluster profiles if all the ClusterProfiles in a clusterInventory are in the same namespace?

[corentone]

I think the inventory was not as defined as it is now, my suggestion was around not binding this decision to the inventory, but I think its fine with the current definition of inventory.

[corentone]

"we recommend" instead of "we suggest"?

[ArangoGutierrez]

we strongly recommend

[corentone]

The name of the namespace doesn't really matter if you have the label.

I think what's important with clusterset is guaranteeing that all clusters can be considered "having sameness" for a user, because that enables a lot of management simplicity. The requirement is that all clusters are part of the same cluster, not necessarily that they are all there.

use cases I can think of:

• regional management of clusterset; if one wanted localized management of clusters. Example: you have 3 clusters in europe and 3 clusters in us. They are part of the same clusterset but you want a regional hub-cluster. You would have 2 hub cluster that each manage 1 inventory of 3 clusters. The two regional clusters would have to be guaranteed by something global that they wont break sameness (maybe a global hub-cluster?).
• isolation of management: let's say we've given a inventory consumer the ability to manage N namespaces within some clusterProfiles, we could have an inventory of clusters thats a subset of the ClusterSet
• migration: if we needed to migrate a clusterset, we will need to move one cluster at a time to the new representation of the clusterset (but without breaking the ClusterSet rules)
• timing of ON/OFF: clusters may come and go and join/leave the set as they are managed, so the completeness of the set may vary

Obviously a consumer of the API would be limited by the set of profiles they see and it may impact behaviors if the clusterset is not complete, but I think the main rule that we care about here is that the consumer can treat the clusters as being part of the same clusterset.

[corentone]

That still works here I think, given the definition of ClusterInventory to be a collection of ClusterProfile.
A clusterInventory could be taken at the hub-cluster level and still each namespace to be a clusterSet.

I think the ClusterProfile is within a Namespace, then one can consume the inventory at the namespace or hub-cluster level (sum of all namespaces).
For cluster-set, we're implying the namespace level is the way and adding an extra requirement (all clusters in a namespace-level inventory are part of the same clusterset)

[ArangoGutierrez]

minor comments in line with @corentone
Looks good we are almost there

[ArangoGutierrez]

we strongly recommend

[corentone]

I think this looks good now.

the namespacing and clusterSet rule were the points I cared about, I think they are reasonably described.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: corentone, mikeshng, ryanzhang-oss
Once this PR has been reviewed and has the lgtm label, please assign jeremyot for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-multicluster/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment