compile_pip_requirements test fails with proxy

[wingsofovnia]

🐞 bug report

Affected Rule

The issue is caused by the rule: compile_pip_requirements (or rather py_test ($name_test) generated by this macro)

Is this a regression?

Unclear. I am a first-time user of rules_python.

Description

When used behind proxy / VPN, pip-compile creates a lock file successfully (hence recognises proxy configuration), but its test fails. If VPN is turned off, the test passes.

Although I recognise this might be a weird local setup, however, it is strange that the generation itself has connectivity but not test.

Is there a way to disable the test target?

🔬 Minimal Reproduction

compile_pip_requirements(
    name = "requirements_lock",
    extra_args = ["--allow-unsafe"],
    timeout = "moderate",
    src = "requirements.txt",
    requirements_txt = "requirements.txt.lock",
)
gazelle_python_manifest(
    name = "requirements_mapping",
    modules_mapping = ":requirements_metadata",
    pip_repository_name = "pip",
    requirements = "//:requirements.txt.lock",
)
modules_mapping(
    name = "requirements_metadata",
    wheels = all_whl_requirements,
)

# Proxy enabled
> bazel run //:requirements_lock.update && bazel run //:requirements_mapping.update
...
INFO: Running command line: bazel-bin/requirements_lock.update _main/requirements.txt _main/requirements.txt.lock //:requirements_lock.update '--resolver=backtracking' --allow-unsafe --generate-hashes --allow-unsafe
Updating requirements.txt.lock
....
INFO: Running command line: bazel-bin/requirements_mapping.update ./requirements_mapping.generated_manifest ./gazelle_python.yaml
Copying /private/var/tmp/_bazel_d0oldgr/9c1080b7fb0dff5bf53320f61219f86a/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/requirements_mapping.update.runfiles/_main/requirements_mapping.generated_manifest to /Users/d0oldgr/Development/foundation-devicebackend-services/gazelle_python.yaml
....
echo $?                                               
0

🔥 Exception or Error

# Proxy enabled
> bazel test //:requirements_lock_test
...
//:requirements_lock_test                                                FAILED in 8.3s
    /private/var/tmp/_bazel_....../execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/requirements_lock_test/test.log
...
> cat /private/var/tmp/_bazel_....../execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/requirements_lock_test/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:requirements_lock_test
-----------------------------------------------------------------------------
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106b49e10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/apscheduler/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106b49a20>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/apscheduler/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106b49750>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/apscheduler/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106b495a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/apscheduler/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106b49060>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/apscheduler/
  ERROR: Cannot install APScheduler==3.10.1 because these package versions have conflicting dependencies.
Discarding apscheduler==3.10.1 (from -r /private/var/tmp/_bazel_d0oldgr/9c1080b7fb0dff5bf53320f61219f86a/execroot/_main/_tmp/e02825b0d8b6a1f7ef53ba28b513277f/requirements.txt.lock.out (line 18)) to proceed the resolution
  ERROR: Could not find a version that satisfies the requirement APScheduler==3.10.1 (from versions: none)

# Proxy DISabled
> bazel test //:requirements_lock_test
INFO: Analyzed target //:requirements_lock_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //:requirements_lock_test up-to-date:
  bazel-bin/requirements_lock_test
INFO: Elapsed time: 32.787s, Critical Path: 32.44s
INFO: 2 processes: 2 local.
INFO: Build completed successfully, 2 total actions
//:requirements_lock_test                                                PASSED in 32.4s

🌍 Your Environment

Operating System:

  
MacOS 14.3 (23D56)
  

Output of bazel version:

  
bazel version
Bazelisk version: development
Build label: 7.1.1rc2
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Tue Mar 19 20:52:03 2024 (1710881523)
Build timestamp: 1710881523
Build timestamp as int: 1710881523
  

Rules_python version:

  
bazel_dep(name = "rules_python", version = "0.31.0")
bazel_dep(name = "rules_python_gazelle_plugin", version = "0.31.0")
  

[aignas]

To disable the test you can pass tags = ["manual"] and the test would not run unless run explicitly.

[aignas]

If proxy requires authentication you may need keyring addition like discussed in #1578. Is that the case here?

[omar-droubi]

@wingsofovnia : From what I can gather, the failure in the test target is due to sandboxing.
Bazel "run" doesn't run within a sandbox, thus all your enviornemtnatl variables for proxy i.e. http_proxy are available for PIP. Bazel "test" runs within a sandbox which mean you don't get the http_proxy variables. Although I can see the tag ["no-sandbox"] set in the py_test.

I had an issue with using pip.parse() behind a proxy. The solution is to add extra_pip_args = ["--proxy=http://
"] to pip.parse(). I don't think the same can be done with the py_test target.
Another possible solution might be to use --action_env="http_proxy=http://your-proxy" when running the test. I haven't tested that.

[aignas]

Closing this as I think it is not a bug in rules_python. Please comment here/re-open if there is no way to configure in this case.