uWSGI: Makes stubtest_allowlist_darwin more resilient against CI flakyness

[Daverball]

Don't merge this, just trying out some stuff in relation to #11813

Since this surfaced an old issue with the uWSGI stubs this should be considered for a merge after all as soon as the caching issue has been resolved.

[AlexWaygood]

interesting. now stubtest is no longer crashing for uwsgi (but it is still complaining about not being able to load various modules at runtime), and it seems like this gets rid of the pyaudio error as well. The JACK-Client error looks unchanged, though.

[AlexWaygood]

theoretically #11817 should have invalidated the cache key though (since it edited a METADATA.toml file in the stubs/ directory, so maybe if we did another rerun on the main branch we'd see the same thing

[Daverball]

The uwsgi failures specifically are version specific attributes that are turned on/off depending on what's available on the system, I've had some flakyness with that in the past where the missing attributes weren't consistent between CI runs, I think I've had those very same attributes on the ignorelist for macOS in the past and then they had to be taken off again. Could potentially be mitigated using a RegEx, since IIRC stubtest won't complain if a regex entry doesn't match any failures.

[AlexWaygood]

IIRC stubtest won't complain if a regex entry doesn't match any failures.

You need to use a clever regex that will also match on a zero-length sequence, e.g.

(thing_that_might_exist_or_not)?

[AlexWaygood]

theoretically #11817 should have invalidated the cache key though (since it edited a METADATA.toml file in the stubs/ directory, so maybe if we did another rerun on the main branch we'd see the same thing

I confirmed this by running the daily test manually from my fork (did it from my fork so that the bot wouldn't open a new issue on the run failing again): https://github.com/AlexWaygood/typeshed/actions/runs/8800720075/job/24152501055. Uwsgi now just fails on main rather than crashing on main, and the pyaudio errors are gone. So we shouldn't need to remove the pip cache now that the cache key has been invalidated.

EDIT: but it's still crashing for uwsgi on this PR. Ugh I have no idea.

[Daverball]

@AlexWaygood I think that's because the cache for the regular third party stubtest workflow is separate from the one for the daily run.

[AlexWaygood]

@AlexWaygood I think that's because the cache for the regular third party stubtest workflow is separate from the one for the daily run.

sure but it has the same cache key, right? If the cache key has been invalidated by the recent change to the METADATA.toml file for the daily test, it should also have been invalidated for the regular workflow

typeshed/.github/workflows/stubtest_third_party.yml

Lines 42 to 50 in c13f6e1

	- uses: actions/setup-python@v5
	with:
	# TODO: Use Python 3.12. As of 2024-03-08, several third-party
	# packages fail to install with Python 3.12.
	python-version: "3.11"
	cache: pip
	cache-dependency-path: |
	requirements-tests.txt
	stubs/**/METADATA.toml

[Daverball]

I don't really know enough about how setup-python generates the cache key to answer that question. My assumption would've been that each workflow definition has its own cache given that there's multiple entries for MacOS-Python3.11 on the main branch. Otherwise there would only be one, right? Actually nevermind, it makes sense that there are two entries for 3.11, if the cache-depdendency-path is the only thing informing the cache key other than the platform and python version. For some reason I thought there were more than two on the main branch, but I probably was off by a row when reading the table.

Also I think each fork has its own cache, so I'm not sure it can be tested the way you did it.

[AlexWaygood]

Also I think each fork has its own cache, so I'm not sure it can be tested the way you did it.

Ugh I guess that makes sense

[Daverball]

Looks like having any cache at all breaks things, although it's unclear to me whether my change actually did cause a fresh cache to be created for MacOS specifically, I only see a new cache entry for Windows and Ubuntu and I see the MacOS cache from the main branch was also used recently, despite no recent workflow runs...

So it appears like the cache reuse/invalidation logic is not as simple as it seems. It's possible it's eagerly using the most recent cache that could be used as a base and instead relying on pip causing a reinstall simply based on the version number of the package and only uploading a new version of the cache if any files changed. So cache invalidation would never happen for unrelated packages, only for the package that had its pinned version changed.

stubtest fails, things are weird

[Daverball]

Now Ubuntu started failing for pyaudio as well...

Although that may be because Microsoft borked their aptitude mirror for jammy, also getting errors on an unrelated project.

[Daverball]

@AlexWaygood I'm fairly confident now that the whole cache key thing almost doesn't affect if/which cache it pulls in to restore at the beginning, it seems to either always pull in something regardless of cache key as long as there's a matching cache for the given python version and platform or it completely ignores the state of a branch and calculates the cache key based on the main branch.

It looks to me like the cache key is only used to write out a new version of the cache when it changes, but never to completely invalidate an old cache.

Can we get someone to test deleting all the "setup-python-macOS-python-3.11.9" caches, so we can start from a blank slate?

[Daverball]

Looks like this has resolved itself

[JelleZijlstra]

Do you still think we need this change?

[Daverball]

@JelleZijlstra If we can re-enable these tests on MacOS, then we should, yes.

[JelleZijlstra]

Seems like it passes now, so let's merge.