Multitenancy question: Is it possible to run multiple copies of Kargo on the same cluster?

[empath-nirvana]

I've got a control plane setup that has multiple argocds installed on it, each of which are managing multiple remote clusters. I did that for multitenancy purposes. What i'd like to do is also have a kargo per tenant, but I don't think that's possible because projects are cluster scoped. I definitely don't want to have to setup a whole cluster per tenant just to run kargo on it. I do understand that one kargo can manage multiple argocds and that's a great feature, but I don't really see a really nice way to isolate tenants from each other on a single kargo unless i'm missing something.

[krancour]

Hey @empath-nirvana, this is an excellent question.

I don't think that's possible because projects are cluster scoped

This is correct, and definitely a limiting factor for you... maybe...

You have what I usually describe as a "management cluster" that is hosting things shared by multiple other clusters -- in this case, multiple Argo CD control planes, each managing multiple clusters. We have seen this before, but it's not exactly ubiquitous, since there are known security and scalability issues with using Argo CD in a "hub and spoke" topology, and you've essentially got multiple hubs in a single cluster, which compounds those issues. You may want to reconsider this topology for reasons completely unrelated to Kargo.

With that being said, I still want to respond to how Kargo fits into that topology...

What i'd like to do is also have a kargo per tenant

I don't really see a really nice way to isolate tenants from each other on a single kargo

Projects are actually Kargo's built-in tenancy model. Projects are cluster-scoped, as you observe, but every Project reconciles as a namespace (with some default ServiceAccounts, Roles, and RoleBindings within it; you can also edit these or add your own). This means permissions for everything within a Project can be managed at the namespace-level, as with everything else in Kubernetes.

This means it is entirely conceivable to have a single Kargo control plane running in your "management cluster" along with just a single Kargo controller (i.e. no need for sharding) because all the Argo CD resources that the Kargo control plane will be interacting with are, in-fact, all located on just one cluster.

The implications of this are that users will end up with ability to list all cluster-scoped Projects, but doing anything at all with any one Project requires specific permissions within the Project. Although I would consider this an adequate separation of "tenants" (Projects), one usability issue with this is that although users won't be able to do anything with Projects in which they have no permissions, they will still see those Projects listed on the Projects screen, which can be a bit noisy if you have a lot of Projects. #1836 has already been opened to address this.

I have the sense, however, that you may be looking for something beyond this. If that is the case, it would require modifications to Kargo to support label-based filtering of Projects so that Kargo can run multiple control planes side-by-side. I will open an issue for that, but cannot guarantee it's a high priority. Likely it's something that will get done faster if someone from the community wants to champion that and do the work.

In the meantime, I can offer you one possible workaround, which is to run the Kargo control planes (i.e. API server, management controller, and garbage collector) on different clusters than the one that's hosting your multiple Argo CD control planes. You would then run n Kargo controllers on each management cluster, each connected to one of your n Kargo control planes.

[krancour]

I opened #2003