[Feature] Package Tags

[herronelou]

For a long time, I've felt that Rez could benefit from some kind of package tagging mechanism.

While I have very specific tags that I'd like implemented for my workflow (experimental, new and deprecated), not every workflow is the same, and keeping these customizable would likely be best.

A tag would specifically not be part of the package.py, but be added as a side-file, in the manner of the .ignore files, as they would represent different "stages" of a package's lifetime. And maybe "tag" isn't the right term to use for this, I'm specifically looking for something that would change over time, hence why it would be a sidecar to preserve the integrity of the package itself.

These tags would be accessible to the package filters and package orderers, and show up in the resolve information similarly to some of the existing qualifiers currently showing up:

Packages would support multiple simultaneous tags.

Motivation
This feature would greatly simplify the management of the lifecycle of packages.

Look at #631:
The user wants to have a way to indicate a preferred version that isn't necessarily the latest version of a package released.

They could create a tag preferred, and in their package orderer indicate that preferred packages should be resolved before non-preferred ones. Whenever they're ready to move on to a newer version, they can tag a different version as preferred without re-releasing anything.

For my own desired workflow:

• experimental packages would be filtered out by default, but users desiring to test the upcoming pipeline changes could opt-in to loading experimental packages.
• new packages would be included by default in the resolve without any specific filters or orderers, but would show on the resolve to indicate to the users that the resolved package is a recent release. Optionally, when a user runs into issues and wants to revert to a "true and tested" environment could add a filter to ignore new packages.
• deprecated packages would be ordered very last, to prevent them from getting picked, unless necessary.

You could almost think of .ignore as the next tag in the lifetime, where after being deprecated for a while, the package is ignored before complete removal, and .build as the very first tag when the package is still building.

[herronelou]

Another use case:

#302, where some pre-release packages resolve higher than the release. A prerelease tag could be created, and the orderers adjusted to take the tag into account.

[herronelou]

#1125 was attempting to use the .ignore files in this manner, and could have benefitted from this feature.

[JeanChristopheMorinPerso]

I think studios usually handle tagging by having multiple repositories and moving packages with rez-mv. Though, there is no priority between repositories.

#302, where some pre-release packages resolve higher than the release. A prerelease tag could be created, and the orderers adjusted to take the tag into account.

This can already be achieved with orderers if I'm not mistaken. I think @maxnbk told me that it's possible.

They could create a tag preferred, and in their package orderer indicate that preferred packages should be resolved before non-preferred ones. Whenever they're ready to move on to a newer version, they can tag a different version as preferred without re-releasing anything.

This is also possible purely with orderers today. Though it's getting quite in the territory of managing/storing your production environment in rez, which we try to avoid in general.

As for the comparison with rez-pkg-ignore (note how I don't say .ignore since that's an implementation detail), that's not a valid comparison IMO. rez-pkg-ignore exists to safely remove a package from a repository, not to filter it out. Most package managers have a similar functionality. You might see terms like "yanking a release".

[herronelou]

I think studios usually handle tagging by having multiple repositories and moving packages with rez-mv. Though, there is no priority between repositories.

That's one way, but it's not particularly flexible or convenient, and moving packages makes un-necessary duplicates.

#302, where some pre-release packages resolve higher than the release. A prerelease tag could be created, and the orderers adjusted to take the tag into account.

This can already be achieved with orderers if I'm not mistaken. I think @maxnbk told me that it's possible.

They could create a tag preferred, and in their package orderer indicate that preferred packages should be resolved before non-preferred ones. Whenever they're ready to move on to a newer version, they can tag a different version as preferred without re-releasing anything.

This is also possible purely with orderers today.

Yes, I also believe it can be done, from what I've seen of custom orderers you can essentially do anything, this would only be for providing an additional mechanism that can be utilized.

Though it's getting quite in the territory of managing/storing your production environment in rez, which we try to avoid in general.

Rez's job is to pick the right versions of packages based on a request, and this could extend these capabilities by allowing more precise control of what it should or shouldn't pick up.

As for the comparison with rez-pkg-ignore (note how I don't say .ignore since that's an implementation detail), that's not a valid comparison IMO. rez-pkg-ignore exists to safely remove a package from a repository, not to filter it out. Most package managers have a similar functionality. You might see terms like "yanking a release".

I agree and disagree at the same time. The rez package is still there, but it's "ignored" because a little tag tells rez to ignore it. From this point of view, I do think it's comparable.
rez-pkg-ignore does have a very specific meaning to rez, and from this point of view, I agree it differs, as my proposed "tags" wouldn't necessarily have an intrinsic meaning for rez, and would leave the implementation details to those using them.

[maxnbk]

I'm not yet agreeing or disagreeing with the idea as a whole, but I would argue that the .ignore / .build files are more of a backend "implementation detail" of other critical components. the .build files are there as a mechanic to prevent against package-access race conditions, and the .ignore files are there as a mechanic to facilitate archival, and as such I view them as more of a "package statefulness" aspect and not a "version management" aspect, though I can understand someone looking at "archival" as "lifecycle state" to a sparse degree.

I think my gut reaction is, "what does this do that a user-written management layer could not handle?" As in, I don't see that this accomplishes anything that manipulating your rez-request to target what you want to target doesn't also accomplish. The .build and .ignore files are distinct in that respect; They accomplish something not possible to do by manipulating your rez-request.

I'm also unclear on whether these tags are meant to be directly meaningful to rez in any way, or just a searchable/filterable attribute, and I'm curious how this concept plays, if at all, with the future idea of cloud repositories, where the filesystem isn't something you can just touch a file inside of.

I will counter myself slightly, in that, it isn't easy to express a series of versions that are preferred to fall back to in a request if one doesn't satisfy it, and package orderers are of course the way to manipulate that. I wonder if you wouldn't be better served by a custom package orderer that references an outside resource to evaluate the "preferredness" of a package version?

[herronelou]

I think my gut reaction is, "what does this do that a user-written management layer could not handle?"

Absolutely nothing, but it's a lot of work to write these management layers, and it feels like Rez isn't really "production" usable without at least some sort of custom implementation, whether it's done via custom filters, directory moving, or something else.
We all end up with vastly different workflows, and while this isn't meant as a single workflow, having some standard option could help new users get up and running faster with less self-discovery needed.

I'm also unclear on whether these tags are meant to be directly meaningful to rez in any way, or just a searchable/filterable attribute, and I'm curious how this concept plays, if at all, with the future idea of cloud repositories, where the filesystem isn't something you can just touch a file inside of.

In my description, I tried to make them not directly meaningful to Rez, so that there isn't an idea of "one single way to do things", rather Rez is aware of their existence, and they can be used in multiple places to influence behaviours, whether it be filters, orderers, maybe packages themselves (haven't had much thoughts about use-cases for this).

The whole implementation as a sidecar file similar to the .ignore files I'm not attached to, the main point being some way to manage the lifecycle of packages without necessarily having to re-release those packages (which I've seen some people do by adding prefixes to their version), nor having to move them to different folders.

I'm open to other suggestions to achieve something similar, it's just a very common question I feel people ask when starting with Rez.

[maxnbk]

To some degree, I wonder if this is falling into the same sort of camp as the "production configuration working-group" sort of idea that @JeanChristopheMorinPerso was suggesting alongside another discussion with someone else in slack the other day.

On the one hand, I'm somewhat opposed to attempting to implement things intended to act like a production configuration system in rez, as there truly is no one "right way" to do it to meet everyone's needs. On the other hand, the idea that we track or manage some level of metadata about rez packages outside of packages themselves might be required in the future for cloud repos. Where my opinion gets fuzzy is on whether that should be a central focus for the purpose of micromanaging resolves.

One other thing leading away from the idea of files-on-disk would simply be, hits on NFS filers are very important to minimize, and checking for hundreds/thousands of these metadata files and checking their contents could get very messy very quickly.

I think if I were implementing this for a hypothetical studio today, I'd probably be investigating if a packageorderer that could query a lightweight DB would be possible / fulfill the need. A simple persistent redis store would probably be the only thing needed. My personal experience with creating separate repositories for stable/staging/dev zones and migrating packages between them is not what I would consider to be ideal and practically requires an entire service-stack to manage intelligently, so I feel your pain in that respect.

I think this might turn out to be a feature request that could be better worked towards as a community discussion towards how rez can best aid configuration systems without actually becoming one, because that is definitely the fear.

[herronelou]

All good, this was meant to get some conversation going. I thought about bringing this up during the TSC yesterday and thought that it was likely to require some discussion and time was running short so I decided to put this in as a ticket instead.

[instinct-vfx]

I will need more time to thoroughly read and give more feedback. I still would like to address some points. Not becoming a configuration system is something i think is very important. Mixing the two has so many hard to predict implications. There is a good thread on the old mailing list with some examples.

I think studios usually handle tagging by having multiple repositories and moving packages with rez-mv. Though, there is no priority between repositories.

That's one way, but it's not particularly flexible or convenient, and moving packages makes un-necessary duplicates.

As a small remark, rez-mv will not duplicate but move the package.

With that said we handle life cycle with sepearate repositories. I think there is priority (the order which they are configured) though, no? Need to double check.

We do have "duplicates" but for reasons. When we go from a dev package to staging to prod the package matures and it is not the same package. Things released to prod may also require additional steps that can sometimes be neglected in earlier states. Also it is not necessarily clear what kind of version bump a release would get in the end. And for dev it is not even clear which features (=branches) may end up in a release.
Most importantly we want to actually restrict usage of these packages to certain people. The main reason is that i can not count anymore how often non-production packages/versions leaked into production.
The same is true for archival. We would not want these to be used. And if someone absolutely has to we would like to know.

[herronelou]

I thought rez-mv did an ignore and a duplicate but maybe I'm wrong, haven't done it for a little while.

Agree with your points Thorsten, and we do things differently in a different repo during the "dev" cycle. Here we differentiate it by using rez-build in the dev cycle and rez-release for the "final" package.

What I'm looking for is a way to further manage the lifecycle of the released packages. Because it's released doesn't mean I necessarily want it to go in prod right away, and it doesn't have to thanks to our custom layer on top to manage all this, but for our few users who write rez-env commands directly they end up with something different than intended if they had used our launcher. Same with some deprecated packages, I may want to keep them so old snapshots of the pipeline keep resolving properly, but do not want it picked up otherwise.

[instinct-vfx]

rez-mv moves a package, rez-cp copies packages between repositories. (Just to be sure: I am not saying our workflow is the "correct" or only way to do this, i just wanted to give more details for why we are doing it). I am leaning towards Stephen's comment that it feels more like a request manipulation case than a resolve manipulation case, but i have not had the time to fully think it through yet.