Support for --user option in pip_install()

[brownag]

On some systems the system python site-packages is not writeable. Often there are limitations on creation/execution of binaries in user-created conda or virtual environments. Sometimes the system python, or python the user can run, is an installation associated with another program with or without a conda or virtual environment associated. The user running install_python() themselves is generally not an option under these conditions--Python can be installed but not executed.

In the past when using reticulate::py_install() or related, it would have pip to fall back to a user installation with message "Defaulting to user installation because normal site-packages is not writeable"

I see that in 1.27 --no-user was made default in the command string, and then an internal argument no_user was added to pip_install() to toggle it for old versions of pip.

reticulate/R/pip.R

Lines 31 to 33 in 5d0a9a5

	# construct command line arguments
	args <- c("-m", "pip", "install", "--upgrade",
	if (no_user) "--no-user")

Would it be possible to optionally add --user flag to get around errors like the one below, exposing it as a user-level argument to e.g. py_install()? Perhaps there is a more elegant way to do what I am trying to do, in terms of bootstrapping existing python installations, and getting the dependencies installed as seamlessly for the user as possible

Z:\>python -m pip install --no-user --upgrade scipy
Requirement already satisfied: scipy in c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages (1.11.2)
Collecting scipy
  Downloading scipy-1.11.4-cp311-cp311-win_amd64.whl.metadata (60 kB)
     ---------------------------------------- 60.4/60.4 kB 401.3 kB/s eta 0:00:00
Requirement already satisfied: numpy<1.28.0,>=1.21.6 in c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages (from scipy) (1.25.2)
Downloading scipy-1.11.4-cp311-cp311-win_amd64.whl (44.1 MB)
   ---------------------------------------- 44.1/44.1 MB 586.2 kB/s eta 0:00:00
Installing collected packages: scipy
  Attempting uninstall: scipy
    Found existing installation: scipy 1.11.2
    Uninstalling scipy-1.11.2:
      Successfully uninstalled scipy-1.11.2
  Rolling back uninstall of scipy
  Moving to c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages\scipy-1.11.2-cp311-cp311-win_amd64.whl
   from C:\Users\Andrew.G.Brown\AppData\Local\Temp\pip-uninstall-v7hlkvu7\scipy-1.11.2-cp311-cp311-win_amd64.whl
  Moving to c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages\scipy-1.11.2.dist-info\
   from C:\Users\Andrew.G.Brown\AppData\Roaming\Python\Python311\site-packages\~cipy-1.11.2.dist-info
  Moving to c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages\scipy.libs\
   from C:\Users\Andrew.G.Brown\AppData\Roaming\Python\Python311\site-packages\~cipy.libs
  Moving to c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages\scipy\
   from C:\Users\Andrew.G.Brown\AppData\Roaming\Python\Python311\site-packages\~cipy
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'C:\\Program Files\\Python311\\Lib\\site-packages\\scipy-1.11.4-cp311-cp311-win_amd64.whl'
Consider using the `--user` option or check the permissions.

In above example, I use a command directly in a windows prompt, but the same message is generated by standard calls to py_install()
Note the error message suggests the use of --user flag: "Consider using the --user option or check the permissions."

I think --no-user is a fine default, but I think there are legitimate use cases that rely on --user. Often a solution to fix the failed reticulate install is to drop to the command line / system call and do the suggested command that failed, with --user flag replacing --no-user.

[t-kalinowski]

I would strongly encourage working in a virtual environment whenever possible. Installing Python packages into a user library with pip install --user is an anti-pattern, a foot-gun, an accident waiting to happen.

virtualenv_create("r-reticulate")
py_install(packages = c("scipy"), 
           envname = "r-reticulate")

That said, if you find yourself working on a machine with draconian restrictions that prevent you from creating or using a Python virtual environment, you can override the default by explicitly passing an argument pip_options = "--user"

py_install("scipy", pip_options = "--user")

[brownag]

Yes, agreed on using a virtual environment wherever possible. I anticipate that some of these restrictions will be relaxed a bit more in the future as the latest versions of python become more prevalent.

I have been working towards getting everything switched over to use a virtualenv preferentially. Other than the current issue with CRAN reticulate and virtualenv_create() failing on some platforms that have the tools installed (fixed in dev version), it is working great to replace old workflows. I am glad for all your work guiding folks towards productive ways of managing environments.

I am aware of the problems with a general user level library and/or breaking system package installs... But in these "draconian" cases, venvs would be likely need to be installed by a special method, or allow-listed by name, due to the way executable permissions get handled.

Thanks for your consideration and the tip about pip_options, that does make sense it can override the default!

[brownag]

Hi, actually with reticulate 1.34.0 in this case I get:

reticulate::py_install("scipy", pip_options="--user")
#> Using Python: C:/Program Files/Python311/python.exe
#> Creating virtual environment "~/.virtualenvs/r-reticulate" ...
#> + "C:/Program Files/Python311/python.exe" -m venv "C:/Users/Andrew.G.Brown/.virtualenvs/r-reticulate"
#> FAILED
#> Error: Error creating virtual environment '~/.virtualenvs/r-reticulate' [error code 1]

Error in system2(python, c("-c", shQuote(command)), stdout = TRUE, stderr = TRUE) : 
  'CreateProcess' failed to run 'C:\Users\ANDREW~1.BRO\VIRTUA~1\R-RETI~1\Scripts\python.exe -c "import sys; import pip; sys.stdout.write(pip.__version__)"'

Whereas this works:

system(paste(shQuote("C:/Program Files/Python311/python.exe"), "-m pip install --upgrade --user scipy"))
#> Requirement already satisfied: scipy in c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages (1.11.4)
#> Requirement already satisfied: numpy<1.28.0,>=1.21.6 in #> c:\users\andrew.g.brown\appdata\roaming\python\python311\site-packages (from scipy) (1.26.2)

Is there some trick to prevent the virtualenv from being created, or otherwise safely skipping over it and falling back to system if/when it fails?

[t-kalinowski]

Unfortunatly, py_install() today only works with virtual environments or conda environments.

This sounds like a somewhat speciality use-case, I would recommend making a custom wrapper like:

py_install_user_base <- function(..., python = Sys.which("python3")) {
  system2(python, c("-m pip install --user", ...))
}

[brownag]

Unfortunatly, py_install() today only works with virtual environments or conda environments.

That is unfortunate.

I understand --user packages or installing into system library can create problems if not done carefully.

In making design decisions that are "opinionated" we can sometimes prevent users from going down the wrong path. There is the idea we should have only one "right" way to do things and it should be "obvious"--I would say this reasoning is why a wrapper function like py_install() is beautiful: it can be used interchangably virtual, conda and (up until recently) system/user installations.

Recent design decisions in reticulate are based on a well-intentioned, but generally incorrect, idea that users are able to create virtual environments, control their contents, and execute the binaries in them. I note that I don't believe there is anything exotic about users without permissions to install their own executables; this is very common situation in industries that are not primarily focused on developing software..

I would suggest an error message in the example above RE: virtualenv creation failing. Also, since there are cases where no amount of installing python, setting up environments etc. will help, I would suggest a method to "opt in" and bypass common sources failure like site packages not being writable. I think you should reconsider that such an option would be out of scope for reticulate--for a package that is the interface to Python in R I think there should be a (non-default) way to use the system python similarly to a venv. There may need to be sidebars or warnings but it should be possible.

venv "whenever possible" is a great goal, but there are cases it is categorically "not possible." I would say that the withdrawal of PEP requiring, rather than suggesting, package managers use venvs speaks a lot to barriers to "forcing" such a pattern for all of Python: https://peps.python.org/pep-0704/. Marking of base environments as "externally managed" (https://peps.python.org/pep-0668/) provides guidance that handles most of the concerns without being so absolute. I note that error messages produced by pip in PEP668 context are highly informative--e.g. telling exactly how to use apt to install Python packages for the system. Instead of being hidden from the user, even the most inadvisable choices (e.g. --user or --break-system-packages) appear as a possible workaround with appropriate warnings.

I have workarounds in a couple of my reticulate based packages to construct system calls... If I must maintain some extra logic in lieu of reticulate to provide a similar environment setup for all users then so be it.

Thanks for your consideration.