Cached package is used instead of local version

[mmdanggg2]

When a version of a package has been released and then cached, if I start to develop that package again in local with the same version number, the cached version of the payload is used instead of the one in local despite having package_cache_local = False

Environment

• OS (Rocky Linux 8)
• Rez version (2.113.0)
• Rez python version (3.6)

To Reproduce
1. Setup rez package caching
2. Release a package, foo-1.0
3. rez-env foo to cache the package
4. Modify the package and rez-build -i to local
5. rez-env foo again and see that while the local copy overrode the package definition, REZ_FOO_ROOT is pointing to the cache

Expected behavior
I expect that the local version would always take precedence over a cached version.

Actual behavior
The cached version takes precedence.

[BryceGattis]

@mmdanggg2 Question about your expected behavior statement there, wouldn't the cached package take precedence over the local package depending on their order in your rezconfig.py?

For example, I tested out on Windows two different config files:

packages_path = [
    "D:/packages",
    "~/packages",
]
cache_packages_path = "C:/.rez/cache"
default_cachable = True
local_packages_path = "~/packages"

and then:

packages_path = [
    "~/packages",
    "D:/packages",
]
cache_packages_path = "C:/.rez/cache"
default_cachable = True
local_packages_path = "~/packages"

When using the first config file, the selected package is at D:/packages (whose variant is cached to C:/.rez/cache).

When using the first config file, the selected package is at ~/packages.

I tested with Rez 2.113.0.

[mmdanggg2]

I do indeed have ~/packages first in my packages_path and it was using the local package definition (said as much in the context printout) although I've tried this again now with a really simple package and it's not doing it, behaving as I expect. I'll see if I can figure this out.

[mmdanggg2]

So this would seem to be a much more difficult to catch bug than I originally thought, this package I have the problem with appears to have had the local version cached at some point. There are actually 2 different packages cached under the same version in the cache folder. So I guess then the bug is more that somehow somewhere rez decided to cache a local package.

Just checked and this has seemingly happened to at least 3 packages so it's not completely one off either.

[mmdanggg2]

So the plot thickens, these wrongly cached packages were not even cached by my user. A bit of background, we run a render farm where rez env is being run under different users on our workstations. Additionally, for reasons, the packages folder of my user is readable by other users.
There is a render log on my machine from the exact time these caches were put there but the fun part is that the packages that were cached weren't even part of the resolve of that render.

So my current theory as to what happened it that the render job started up as userB which spun up a rez-pkg-cache daemon as userB. At this same time I must have started a rez-env which will have resolved my local packages and added them to the list of packages to cache. These would usually then rejected by the cache daemon but, since the daemon was not running as my user but userB, those packages did not appear local to that user and so were cached.