Git Proxy 2.x architecture

[coopernetes]

Tracking issue to revisit Git Proxy's internal architecture and figuring out how best to design the application going forward. I'm including my initial sketch and using this issue to bounce around ideas, flesh out the right interfaces & API boundaries as well as discuss overall what we want to change in Git Proxy internals.

Issues that are blocked on the current architecture or would greatly benefit from a new one:

• SSH support #27
• Patch oriented workflow #61
• Sign outgoing contributions #62
• Provide facility to manage multiple forks of the same upstream repository #66
• Synchronous Webhooks #10
• Asynchronous management of a push request  #428
• Scan for leaked credentials in PRs and code diff using TruffleHog OSS 🐷  #414

Related issues:

• TypeScript and ESM support #276
• Open Policy Agent checks #99

[coopernetes]

Here's first stab at a new architecture. Some initial thoughts:

• the idea of a "chain" of actions that Git Proxy executes in some sequence is extremely valuable. I don't think we need to rework this that much as we currently have a series of Step/Action types that the application cycles through for every git push.
• @JamieSlome brought up some interesting points in Open Policy Agent checks #99 around policy management. What constitutes a "policy" vs a "filter" is not immediately clear to me but some separation between a decision to permit and the underlying git request seems like a useful abstraction. It would also allow us to implement an interface that is generic to other policy engines like OPA
• Each phase of the request lifecycle should be able to fire off events in an agnostic way. How we persist those events can be abstracted into a persistence interface and backed by a variety of sources (object storage, SQL, Mongo, etc).
• I included an idea of using git-proxy purely in a localized sense such as a library running on a development machine. This may be preferable to a hosted server so I've separated those out into their own components. At the very least, this allows a developer to directly use git-proxy without the new to run it as a server.

Much of this will likely result in breaking changes so I consider this a version 2.x of git-proxy so we're free to iterate on the new design.

EDIT: Added the dashboard UI, API and user interactions.

[maoo]

Really like this new architecture, and the drawing.

For the latter, I think it would help to draw where users connect to, specifically:

• a Developer (connecting directly to git-proxy-server, or the local block close to it)
• a Compliance manager, which should connect to git-proxy-ui, though I don't see it mentioned ; I also assume that the UI connects to an API layer that is exposed by the core package, which would be helpful to draw.

WDYT?

[JamieSlome]

I'd like to bring in #428 to 2.x as well. Likely sits in the filter chain in the architecture diagram above.

@coopernetes - thoughts?

[coopernetes]

I'll add the API and dashboard (UI) components in here. They should get decoupled and exist as separate packages with various interaction points with a few of the core components.

[JamieSlome]

🔔 Reminder to include for Monday working group agenda @coopernetes @maoo

[coopernetes]

Updated diagram above.

[maoo]

Here's a recap of async discussions with @coopernetes and @JamieSlome around project layout and structure.

• Codebase for 2.x restructuring is in https://github.com/coopernetes/git-proxy/tree/experiment-2.x
• At root level, the packages/ folder contains core application components and the plugins/ folder contains extensions which may be bundled with the app but are standalone functionality
• We plan to leverage Yarn 3+ workspaces to decompose the current monolithic package.json into plugins and packages

Anything to add?

[JamieSlome]

@coopernetes - looking fresh 🚀

One thing I'd like to reflect on is the home for plugins. Whilst it makes sense for us to bundle out-of-the-box plugins with Git Proxy, I wonder if the plugin functionality should itself be a separate repository, housed elsewhere?

It follows on from the discussion and closing comments in #438 as well conversations with GitLab around re-use of our plugins. I'd liken it to the modules ecosystem in Nuxt (massive props to @danielroe and team). I equate plugins for us to highly reusable modules that support a customisable Git Proxy, CI/CD or otherwise experience.

@coopernetes - I'd like to get your thoughts here...

[coopernetes]

@JamieSlome I'm agnostic on the approach. We can keep "in tree" features under the packages/ workspace and leave plugins to be outside the main repo. I think it's worth shipping at least one "example" or "hello world" extension but that can be hosted separately too. That can be provided out as a template repository as well.

[JamieSlome]

@coopernetes - yes, I think we should still support an inline plugin out of the box 📦

This will be extremely useful as not all implementations will be heavy or reusable enough to demand its own "module".

[coopernetes]

Update - once I have some very basic tooling setup & stabilized on my personal fork, I will push up a new 2.x branch here on the main repo so that we can collaborate on a shared branch. #276 alone is proving to be a beast 😅 On the TODO:

• Working TypeScript compilation for a single package (partially complete)
• Workspace setup and intra-package dependencies building successfully via Yarn 4
• ESLint
• ESM compatible test framework (Jest is the best supported one for ESM projects as far as I can tell)

[coopernetes]

@JamieSlome thoughts on starting a 2.0 branch for all these changes? Slowly making progress via my personal fork and I would prefer to collaborate on some of the more involved changes with yourself and others directly here. We could also accept PRs into 2.x work as well while we continue to make incremental enhancements to what's in main for some of the open issues that would be difficult to implement on 1.x.

https://github.com/coopernetes/git-proxy/tree/experiment-2.x

some working notes

[JamieSlome]

@coopernetes - let's do it ❤️

Any issues with branching off of reconcile? We will be merging into main very shortly and will just make our lives a little easier if we anticipate repository wide changes for 2.x. Thoughts?

[coopernetes]

@JamieSlome et. al, a new branch has been created based off of the latest in reconcile. package.json breakages are expected.

https://github.com/finos/git-proxy/tree/version-2x