Scalability of Repo-per-app Deployment Model

[vijayansarathy]

Hi Everyone,
I am trying to move the new organization I joined to adopt Flux, migrating from Azure DevOps. There is a large number of application repositories, approaching ~1000. The easiest migration path would be to adopt the Repo per app model, storing store both the source code and deployment manifests for each application in the same repo.
I am concerned about how well the Flux Controllers will handle the load of sync'ing from such a large number of repos.
I am sure people here must have pushed the limits on this front and am looking for some guidance and/or first-hand experiences dealing with a similar scale, if you can call it that.
Thanks,
Viji

[kingdonb]

The scalable solution is OCIRepository, which is an ideal fit for the repo-per-app model. Because your apps already push a runtime (docker container) image to an OCI format repository, OCI repositories can use ambient credentials for authentication and authorization, keyless schemes for verification and signing (cosign) there's an obvious opportunity to use the same format repo to package the app deploy manifests and version them that way through your CI release process.

A CI process should publish with flux push artifact when all the other app artifacts are ready, it can be multiple app images, an umbrella or constellation, then on the last step when all the tests pass, push the manifests with a semantically versioned tag, and use spec.semver - or you can do some staging production latest type tagging thing - OCI tags are like branches, you can make them manage named environments or image streams if you want, (I prefer semver tags)

Right now we are still early I think defining how to use these, since the API isn't at GA yet, it will likely get some changes before it does; there are some good examples to follow, but the best one is probably this Cheatsheet: https://fluxcd.io/flux/cheatsheets/oci-artifacts/

Mixing your deploy artifacts in the git repository with your app code makes sense from a developer perspective, because why would you keep them anywhere else, but from the cluster perspective you know that repo can grow to unbounded size, and 99.9% of the files in it Flux doesn't need to deploy. But Git repository mechanisms for ignoring files when cloning out, shallow clones and sparse checkouts, are both very limiting, Flux cannot use them with like Image Update Automation for example, because you need a full clone to make a new valid commit. It's much easier for Flux to scale if you only make it do meaningful work, but the Git protocol doesn't lend itself perfectly to this problem. That's why we added OCI.

Anyway, I suggest you check out OCI Repositories for manifest distribution as tag lists are way cheaper to ship around than entire git repositories, and Flux can definitely scale to thousands of OCI repos much better than the thousands of Git repos.

Hope this helps!

Kingdon

[kingdonb]

PS. Forgot to mention the Flux benchmarks repo:
https://github.com/fluxcd/flux-benchmark

The results are quite favorable and I think responsive to your question's formulary, and show that you can indeed scale Flux to 1000 helmreleases, without sending the cycle times outside of >10 minutes, and Kustomization time for 1000 at around 5 minutes. I guess that was with a GitRepository source, but you should take these benchmarks for what they are worth and at face value, I think the test repo was probably one that was designed favorably, not one that was loaded with irrelevant junk - it is a Flux benchmark after all!

I don't know if these results show at all the difference between GitRepository and OCIRepository clearly in a way, I don't see that GitRepository is included in these benchmarks actually, could be mistaken. I think that you should check out RESULTS.md if you are interested in the details, welcome to provide feedback or suggest PRs if you want to see more different benchmarks in the future.