feat(warehouse)!: artifact discovery and manual Freight creation

[hiddeco]

TL;DR

This pull request addresses the majority of the backend part of #761, by reflecting the last 20 discovered artifacts for each subscription of a Warehouse in its Status, allowing the UI to present the user with options to pick from to "mix-and-match" their selection of artifacts to create Freight for.

Combined with these changes, a "manual" and "automatic" Freight creation mode has been introduced to the Warehouse, in addition to changes to the reconciliation model, and further optimizations to the logic that gathers information about upstream artifacts.

API changes

.status.discoveredArtifacts

For each subscription of a Warehouse, Kargo will discover the 20 latest artifacts based on the selection strategy and reflect them in .status.discoveredArtifacts field of the object. The UI can use this information to provide the user with the option to "mix and match" artifacts for which to create Freight.

An example:

discoveredArtifacts:
  charts:
    - repoURL: oci://ghcr.io/akuity/kargo-charts/kargo
      versions:
        - 0.6.0
        - 0.6.0-rc.2
        - 0.6.0-rc.1
        # ...omitted for brevity
  git:
    - repoURL: https://github.com/akuity/kargo.git
      commits:
        - author: Remington Breeze <remington@breeze.software>
          branch: main
          createdAt: "2024-05-15T18:58:00Z"
          id: c5c74d42069b41461763bc3c21de02b579ada1dc
          subject: 'fix(ui): Promotion confirmation message overflowed freight box (#2015)'
        - author: Remington Breeze <remington@breeze.software>
          branch: main
          createdAt: "2024-05-15T18:55:00Z"
          id: f65a27fcb5bc48861d10768313bb5e4201a8cb86
          subject: 'feat(ui): Ability to abort verification from UI (#2013)'
        # ...omitted for brevity
  images:
    - repoURL: ghcr.io/akuity/kargo
      references:
        - digest: sha256:eae672841a1756683a967b7c25eab31d694ac66ecd09b35191664ec166530f1a
          tag: v0.6.0
        - digest: sha256:79103eb0596ac4c6e12f2eedd8e9066d8557ff86ba58b9a4037d5bdbd74c1ac0
          tag: v0.6.0-rc.2
        - digest: sha256:0ee72b9e8955a7bf4f4b78667ebe9612983e3d0d184eab14f75b1b87c3de31d9
          tag: v0.6.0-rc.1
        # ...omitted for brevity

Note: Not all fields for images and commits are shown above, as the availability of data (e.g. a Git tag, or the creation timestamp of a container image) depends on the chosen selection strategy. Please refer to the in-code changes for further details on all fields.

.spec.freightCreationPolicy

This newly introduced field configures if Freight is automatically created for newly discovered artifacts. It can be set to Automatic (the default) to discover artifacts and automatically create Freight for the latest discovered version(s), or to Manual to discover artifacts without creating Freight.

Behavioral changes

• Warehouse reconciler will first discover the latest artifacts for each subscription, and then (optionally) automatically create Freight for it.
• The discovery of Git artifacts required having access to tree or repository history in many cases. Because of this, and to continue to keep things performant, the cloning process has changed from using a "shallow clone" (--depth=1) to the newer "partial clone" (--filter) option. To read more about this, please refer to this excellent article.
• Git metadata of tags and commits is now gathered using git for-each-ref and git log with specifically designed custom formatters. Instead of performing FS-heavy git checkout, etc. operations.
• Instead of looking at the diff for all commits between the current Freight and HEAD to determine if it contains changes to included and/or excluded paths. Every individual commit is inspected until 20 matching commits are found (or no commits remain).

Other notable changes

• For commits, we do now take note of the author and committer.

Potential follow-ups

• Allow the configuration of the requeue interval, e.g. by adding something like:

  spec:
    discovery:
      interval: 10m0s

  The reason for putting this in a nested struct, is because this would potentially also allow a configurable limit (i.e. .discovery.limit: 50.

• Figure out how to do "on-demand deepening". For example, to allow the user to request metadata about an image outside the subscription's selector. This is a challenging task, as the observed artifacts are continuously overwritten and we would likely need to keep state about a previous request(s) (or requests from two different users).

[netlify]

✅ Deploy Preview for docs-kargo-akuity-io ready!

Name	Link
🔨 Latest commit	e44c41a
🔍 Latest deploy log	https://app.netlify.com/sites/docs-kargo-akuity-io/deploys/664614c7933e8500083fa79f
😎 Deploy Preview	https://deploy-preview-1984.kargo.akuity.io
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

Attention: Patch coverage is 54.62478% with 260 lines in your changes are missing coverage. Please review.

Project coverage is 45.80%. Comparing base (1e3c3e7) to head (e44c41a).
Report is 25 commits behind head on main.

Files	Patch %	Lines
internal/controller/git/git.go	0.00%	62 Missing and 1 partial ⚠️
internal/image/newest_build_selector.go	10.20%	44 Missing ⚠️
internal/controller/warehouses/git.go	77.58%	31 Missing and 8 partials ⚠️
internal/image/lexical_selector.go	12.50%	35 Missing ⚠️
internal/image/semver_selector.go	14.63%	35 Missing ⚠️
internal/helm/helm.go	46.34%	22 Missing ⚠️
internal/controller/warehouses/images.go	58.33%	19 Missing and 1 partial ⚠️
internal/image/digest_selector.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1984      +/-   ##
==========================================
- Coverage   46.21%   45.80%   -0.42%     
==========================================
  Files         235      235              
  Lines       16017    16191     +174     
==========================================
+ Hits         7403     7417      +14     
- Misses       8254     8413     +159     
- Partials      360      361       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hiddeco]

I notice the ! in the PR title, but I cannot spot what this breaks. I'm missing something.

I had to think again about why I thought this was breaking, and initially thought it wasn't, but then recalled the tiny difference.

At present, include and exclude paths for Git subscriptions look at the full diff between $previousCommit and HEAD. If this contains changes to included paths, the current HEAD is greenlighted to create Freight for.

With this pull request, this behavior changes, as we look at every individual commit to see if it contains changes that make it eligible to be used as an artifact for Freight. As the current HEAD may not contain an included path, this means that the newest discovered artifact can change compared to the latest Freight we created (before upgrading), causing the Warehouse to create new Freight with the exact commit that contained these changes (instead of HEAD).

[krancour]

causing the Warehouse to create new Freight with the exact commit that contained these changes

Ah. That makes sense. I doubt many people will notice this very slight behavior change and if anyone does, I think they're likely to find the new behavior make more intuitive sense.

Fine with me if you want to put the ! back in the title so we don't forget about this when reviewing commits to produce v0.7.0 release notes. It may prove to not be worth calling attention to, but at least we won't forget about it.

[krancour]

I am curious about the choice of |*| as a delimiter. I'm not suggesting it should change. Asking for my own benefit.

[hiddeco]

I wanted something that was highly unlikely to show up in some of the fields, I could think of reasons e.g. , or | would show up. But I would be quite baffled about this happening for |*|.

[hiddeco]

(The reason it needs to be a (set of) character(s) is that unlike git log formatting, it git for-each-ref can't insert tabs or other whitespace-like escape sequences.)

[krancour]

Left a few new comments this morning, but they are entirely non-blocking, dealer's choice sort of things.

LGTM as is and it stands if you do choose to do anything about the remaining comments.

This PR is absolute 🔥