[GitHub Action] python is not installed into the virtualenv

[patrick91]

Current Behavior

Hi there, I'm having an seemingly random issue with nox on GitHub actions, here's an example of a test failure:

https://github.com/strawberry-graphql/strawberry/actions/runs/5790313890/job/15693180713?pr=3015

I'm not sure why this happens, and especially why it only happens on 3.12, maybe I setup the cache wrongly?

The full error is:

Installing the current project: strawberry-graphql (0.200.0)
nox > python -m pip install uvicorn
nox > Error: python is not installed into the virtualenv, it is located at /opt/hostedtoolcache/Python/3.11.4/x64/bin/python. Pass external=True into run() to explicitly allow this.
nox > Session CLI tests-3.12 failed.

Here's the action I'm using:

jobs:
  generate-jobs-tests:
    name: 💻 Generate test matrix
    runs-on: ubuntu-latest
    outputs:
      sessions: ${{ steps.set-matrix.outputs.sessions }}
    steps:
      - uses: actions/checkout@v3
      - uses: wntrblm/nox@main
      - run: pipx install poetry
      - run: pipx inject nox nox-poetry
      - id: set-matrix
        shell: bash
        run: |
          echo sessions=$(
            nox --json -t tests -l |
            jq 'map(
              {
                session,
                name: "\( .name ) on \( .python )\( if .call_spec != {} then " (\(.call_spec | to_entries | map("\(.key)=\(.value)") | join(", ")))" else "" end )"
              }
            )'
          ) | tee --append $GITHUB_OUTPUT

  unit-tests:
    name: 🔬 ${{ matrix.session.name }}
    needs: [generate-jobs-tests]
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        session: ${{ fromJson(needs.generate-jobs-tests.outputs.sessions) }}

    steps:
      - uses: actions/checkout@v3
      - uses: wntrblm/nox@main
        with:
          python-versions: "3.8, 3.9, 3.10, 3.11, 3.12"

      - name: Pip and nox cache
        id: cache
        uses: actions/cache@v3
        with:
          path: |
            ~/.cache
            ~/.nox
            .nox
          key: ${{ runner.os }}-nox-${{ matrix.session.session }}-${{
            hashFiles('**/poetry.lock') }}-${{ hashFiles('**/noxfile.py') }}
          restore-keys: |
            ${{ runner.os }}-nox-${{ matrix.session.session }}-
            ${{ runner.os }}-nox-

      - run: pipx install coverage
      - run: pipx install poetry
      - run: pipx inject nox nox-poetry
      - run: nox -r -t tests -s "${{ matrix.session.session }}"

      - name: coverage xml
        run: coverage xml -i
        if: ${{ always() }}

      - uses: codecov/codecov-action@v3
        if: ${{ always() }}
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          fail_ci_if_error: true
          verbose: true

[patrick91]

On a rerun I got this (different) error:

nox > Session Tests-3.12 raised exception FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/opt/pipx/venvs/nox/lib/python3.11/site-packages/nox/sessions.py", line 792, in execute
    self.func(session)
  File "/opt/pipx/venvs/nox/lib/python3.11/site-packages/nox/_decorators.py", line 83, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pipx/venvs/nox/lib/python3.11/site-packages/nox_poetry/sessions.py", line 43, in wrapper
    function(proxy, *_args, **_kwargs)
  File "/home/runner/work/strawberry/strawberry/noxfile.py", line 48, in tests
    session.run(
  File "/opt/pipx/venvs/nox/lib/python3.11/site-packages/nox/sessions.py", line 386, in run
    return self._run(
           ^^^^^^^^^^
  File "/opt/pipx/venvs/nox/lib/python3.11/site-packages/nox/sessions.py", line 490, in _run
    return nox.command.run(args, env=env, paths=self.bin_paths, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pipx/venvs/nox/lib/python3.11/site-packages/nox/command.py", line [123](https://github.com/strawberry-graphql/strawberry/actions/runs/5790313890/job/15693176817?pr=3015#step:8:124), in run
    return_code, output = popen(
                          ^^^^^^
  File "/opt/pipx/venvs/nox/lib/python3.11/site-packages/nox/popen.py", line 79, in popen
    proc = subprocess.Popen(args, env=env, stdout=stdout, stderr=stderr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/strawberry/strawberry/.nox/tests-3-12/bin/pytest'
nox > Session Tests-3.12 failed.

[ssbarnea]

One extra reason why I do not find nox as being able to challenge tox, not being usable outside the box. I has the same problem.

[Elvynzs]

I have the same issue locally on one of my computer (on the other it works fine). I didn't manage to find any info/documentation on how to fix this.

[rra]

I think I have figured out this problem in GitHub Actions. The short version is that you need to include env.pythonLocation in your cache key.

I believe there may be multiple bugs reported together in this issue, but I think the cause of the initial problem is an interaction between the setup-python action, the cache action, and the way nox virtual environments are cached. (If so, it is not a nox bug.)

As documented in https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#check-latest-version, actions/setup-python by default caches Python. If you install Python by minor version (3.12, for example, not 3.12.1), the patch version that you get therefore depends on the local cache and may flip back and forth depending on what GitHub Actions runner you happen to get.

When Python creates a virtualenv, the python binary in bin is a symlink to the system Python. This symlink is then cached as part of the nox cache when you cache the .nox directory. However, when used with setup-python, the symlink points to a specific patch version of Python. If the cache is then reused on a worker that happens to install a different patch version of Python, the symlink is a broken link, which in turn means python resolves outside of the virtualenv, which causes this error.

The fix is to ensure that you do not use a nox virtualenv cache with a different patch version of Python. The easiest way to do this is to include the value of env.pythonLocation in your cache key as a required match. This is set by actions/setup-python and will look something like /opt/hostedtoolcache/Python/3.12.1/x64; note the patch version. Then, if your runner happens to get a different patch version of Python, it will decline to use that cache and will install all the virtualenvs from scratch with the different version of Python, and they'll have the correct link.

I expect you will see this problem whenever a new patch version of Python is released for one of the minor versions that you test, if the caching isn't by patch version.

Other workarounds would be to always use check-latest with actions/setup-python, or to request specific patch versions instead of just minor versions. But the cache key fix seems the best way to preserve the original semantics.

[rra]

I should also add that while I don't believe this is a nox bug, I think nox could in theory detect that this has happened and recreate the venv for the session, which may fix the other non-GitHub-Actions comments on this issue. It would need to check whether the running version of Python exactly matches the version of Python used to set up the venv, and recreate the venv if this is not the case. I believe tox does this, although I'm not sure.