Use uv instead of pip to manage nox virtualenvs #432

jherland · 2024-05-07T13:59:08Z

Upgrade Nox to (at least) 2024.03.02 (which is the first version with
support for managing virtualenvs with uv.

Add uv as an indirect dependency by depending on "nox[uv]" instead of
"nox" (exception: inside the "lint" dependency group, we only depend on
"nox" in order for Mypy to access Nox' type annotations, uv is not
needed here).

Also we cannot use/depend on uv when using Python 3.7, since uv requires
Python >=v3.8. I'm not actually sure why uv requires >=v3.8, as it is
apparently able to create venvs for Python v3.7 (see e.g.
https://github.com/astral-sh/uv?tab=readme-ov-file#python-discovery),
still astral-sh/uv#1239 prevents uv from being
installed on <=v3.7).

Finally, in noxfile.py, use uv as our default venv_backend instead of
the default (pip), but only when it is in fact available.

A final complication on Nix(OS) happens when we install requirements for
the current session; we do this in two steps, and then we make sure that
whatever we installed was patched appropriately:

    session.install("-r", str(requirments_txt))
    if include_self:
        session.install("-e", ".")

    if not session.virtualenv._reused:  # noqa: SLF001
        patch_binaries_if_needed(session, session.virtualenv.location)

However, with uv in the mix, we have to consider that session.install()
itself runs uv at the same time as the first session.install() may
also install uv itself into the virtualenv. The second
session.install() can then end up running a uv that was installed
by the first session.install(), and this will break on Nix(OS) unless
the uv binary has been patched in the meantime.

We therefore need to insert a call to patch_binaries_if_needed()
between the two session.install() calls. Since the second
session.install() only installs FawltyDeps itself (which does not
introduce any binaries to be patched), we can get away with simply
reordering the second session.install() and the call to
patch_binaries_if_needed():

    session.install("-r", str(requirments_txt))

    if not session.virtualenv._reused:  # noqa: SLF001
        patch_binaries_if_needed(session, session.virtualenv.location)

    if include_self:
        session.install("-e", ".")

Upgrade Nox to (at least) 2024.03.02 (which is the first version with support for managing virtualenvs with uv. Add uv as an indirect dependency by depending on "nox[uv]" instead of "nox" (exception: inside the "lint" dependency group, we only depend on "nox" in order for Mypy to access Nox' type annotations, uv is not needed here). Also we cannot use/depend on uv when using Python 3.7, since uv requires Python >=v3.8. I'm not actually sure _why_ uv requires >=v3.8, as it is apparently able to create venvs for Python v3.7 (see e.g. https://github.com/astral-sh/uv?tab=readme-ov-file#python-discovery), still astral-sh/uv#1239 prevents uv from being installed on <=v3.7). Finally, in noxfile.py, use uv as our default venv_backend instead of the default (pip), but only when it is in fact available. A final complication on Nix(OS) happens when we install requirements for the current session; we do this in two steps, and then we make sure that whatever we installed was patched appropriately: session.install("-r", str(requirments_txt)) if include_self: session.install("-e", ".") if not session.virtualenv._reused: # noqa: SLF001 patch_binaries_if_needed(session, session.virtualenv.location) However, with uv in the mix, we have to consider that session.install() itself _runs_ uv at the same time as the first session.install() may also _install_ uv itself into the virtualenv. The second session.install() can then end up _running_ a uv that was _installed_ by the first session.install(), and this will break on Nix(OS) unless the uv binary has been patched in the meantime. We therefore need to insert a call to patch_binaries_if_needed() _between_ the two session.install() calls. Since the second session.install() only installs FawltyDeps itself (which does not introduce any binaries to be patched), we can get away with simply reordering the second session.install() and the call to patch_binaries_if_needed(): session.install("-r", str(requirments_txt)) if not session.virtualenv._reused: # noqa: SLF001 patch_binaries_if_needed(session, session.virtualenv.location) if include_self: session.install("-e", ".")

Our sample_projects and real_projects tests use CachedExperimentVenv in tests/project_helpers.py to prepare virtualenvs containing the dependencies for each of these tests. Establishing these virtualenvs is costly, which is why we also _cache_ these virtualenvs between test runs (using the pytest cache). Using `uv` instead of `pip` can considerably speed up the creation of these virtualenvs. The speedup is largely due to two factors: 1. `uv` is simply faster than `pip`, even when they essentially perform the same tasks. 2. `uv` also implements its own cache of downloaded packages and will install a package into a virtualenv by _hardlinking_ the package files from its own cache. Here are some measurements before and after this commit. We run `time nox -Rs integration_tests-3.12 -- -k Python:all_reqs_installed` which times the execution of _one_ real_projects test with a fairly large set of dependencies: "The Algorithms - Python:all_reqs_installed". Each scenario is run 3 times: Before this commit (i.e. using `pip`): - Cold pytest cache (after running `rm -rf ~/.cache/pytest/*`): - 1m34.633s - 1m28.204s - 1m37.618s - Warm pytest cache: - 7.138s - 6.732s - 7.406s After this commit (i.e. using `uv` instead of `pip`): - Cold `uv` cache + cold pytest cache (after running `rm -rf ~/.cache/uv ~/.cache/pytest/*`): - 1m28.220s - 1m34.373s - 1m34.682s - Cold pytest cache (after running `rm -rf ~/.cache/pytest/*`): - 9.602s - 9.077s - 9.918s - Warm pytest cache: - 7.575s - 6.600s - 6.780s When both the `uv` cache and our own pytest-based cache are empty, `pip` and `uv` essentially have to perform the same work and the run time is dominated by the time it takes to download and unpack the required packages. In the warm cache case we reuse an existing virtualenv from the pytest cache and `pip`/`uv` is not involved at all. But in the case where we cannot reuse our pytest cache (e.g. because some detail of the experiment has changed), then `uv` will take advantage of its own cache to created the required virtualenvs almost instantaneously. In essence, with `uv` downloaded packages will be cached across test runs whether or not we implement our own caching. In the future - if we can _mandate_ `uv` instead of `pip` - we can consider removing our pytest-based cache with little impact on our test run times.

The previous commit explains why the uv cache has the potential to speed up the execution of our sample_projects and real_projects test cases. However, in order for CI to benefit from the same potential speedup, we need to actually preserve the uv cache across test runs.

jherland force-pushed the jherland/ruff-format-instead-of-black branch from a2c964c to fc666fd Compare May 13, 2024 11:16

Base automatically changed from jherland/ruff-format-instead-of-black to main May 13, 2024 11:59

jherland added 3 commits May 22, 2024 13:45

jherland force-pushed the jherland/uv-replaces-pip branch from e6f968f to ac3e5bd Compare May 22, 2024 15:56

jherland added 2 commits May 22, 2024 19:30

WIP: Dummy change to force pytest cache invalidation in CI

47eb6b8

WIP: Prepare to decouple --install-deps from pip

0ba8f7d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use uv instead of pip to manage nox virtualenvs #432

Use uv instead of pip to manage nox virtualenvs #432

jherland commented May 7, 2024 •

edited

Use uv instead of pip to manage nox virtualenvs #432

Are you sure you want to change the base?

Use uv instead of pip to manage nox virtualenvs #432

Conversation

jherland commented May 7, 2024 • edited

jherland commented May 7, 2024 •

edited