Support a way for other tools to assist in environment/interpreter discovery

[brettcannon]

The Launcher would look for binaries on PATH with an appropriate prefix, like _py_launcher_*, and call them. These locators would find relevant environments and/or interpreters to help the Launcher makes it decision of what to use.

The locators should probably be a bit more generic in what they return than what the Launcher might need so that people who have their own search needs can also use the binaries (for instance, the proposal covers everything that the Python extension for VS Code displays to users to help them select an environment). I also expect that a base crate will be created that will be directly embedded in the Launcher to cover its current semantics around things it already supports. That should make it easier to use the code separately and provide structs and such that other crates could use to make these binaries (and would eventually end up on PyPI).

The expectation is that these locators will provide as much cheap information as possible, but otherwise will prefer speed over completeness. The desire to reuse this work for other purposes than the Launcher means we want locators to provide as much info as they can naturally give quickly. But on the other hand we don't want to ask any locators to have to perform any expensive operations like querying the Python binary just to provide extra info that the user may not even care about. In the cases where a tool using these binaries needs more info, the missing info should be available via a Python code snippet. The provided information should also be enough to run Python interpreter/environment to get the rest of the details, e.g. -c "import sys; print(sys.version_info)".

The results from a locator will be returned in JSON format encoded as UTF-8 via stdout for portability. All names are valid Python identifier names.

There is no priority order of executions of binaries. The assumption is most people will at most have one binary on their machine to fit their workflow. This also allows for parallel execution if necessary. Otherwise people can create their own binary if they want to enforce execution order.

Proposed data in the JSON results are:

// Returning an array forces the details to be fully self-contained. This facilitates any library that
// may return a collection of these results by not requiring any post-processing on what a locator
// returns; chaining results from multiple sources is all that is required.
[
    {
        // A unique identifier for the interpreter.
        // The key should be unambiguous but as weakly specified as possible, i.e. to the
        // directory for environments, but to the specific interpreter for something found on `PATH`
        // (i.e. `python3.11`, not just `python3`). That way tools can generally agree on the same key
        // w/o coordination. (🤔 should this be weakly or strongly specified?).
        "path_id": "/home/brett/my-venvs/my-venv",
        // An array specifying what is required to execute the interpreter.
        // The expectation is to append args to the end of the array before
        // execution.
        // E.g. for conda environments:
        // ```
        // ["/path/to/conda", "run", "--path",
        //  "/home/brett/.conda/envs/conda-env", "--no-capture-output"]
        // ```.
        "run": [
            "/home/brett/my-venvs/my-venv/bin/python3.10"
        ],
        "python_version": {
            // `sys.version_info`
            "major": 3, // Optional
            "minor": 10, // Optional
            "micro": 1, // Optional
            "releaselevel": "final", // Optional
            "serial": 0 // Optional
        },
        "implementation": { // Optional
            // `sys.implementation`
            "name": "cpython",
            "version": { // Has the same structure as `python_version` above.
                "major": 3, // Optional
                "minor": 10, // Optional
                "micro": 1, // Optional
                "releaselevel": "final", // Optional
                "serial": 0 // Optional
            }
        },
        "executable": {
            "bits": 64, // Optional; `math.log2(sys.maxsize+1) + 1`
            "architecture": "x86-64"  // Optional; platform.machine()
        },
        "environment": { // Optional
            // What type of environment, e.g. "virtual", "conda", etc.
            "type": "virtual",
            "name": "my-venv"
        },
        // Is the result specific to the workspace?
        "contextual": true,
        // Who created this result.
        "locator_name": "Python Launcher"
    }
]

[brettcannon]

Feedback has been provided via https://discuss.python.org/t/what-information-do-you-gather-for-interpreters-and-environments/13468/ from folks involved in editors/IDEs.

[brettcannon]

Asked for feedback on Twitter via https://twitter.com/brettsky/status/1491168621900402689?s=20&t=v2lxc5_E1U0zLOnKOkjGEw .

[brettcannon]

https://twitter.com/brettsky/status/1491908748129955840 was a poll for potential binary prefix names. Jannis suggested using _ as prefix which does help with the tab completion conundrum, although I don't know what people would think of non-shell stuff having that prefix.

[brettcannon]

I am now leaning towards _py_launcher_*.

[jezdez]

I like this!

[brettcannon]

My current thinking on CLI flags:

1. --version: Any potential hint on what to (not) search for, e.g. ==3.* for -3 for the Python Launcher. Search binaries MAY support this. Could also use *, 3, and 3.10 as values.

A few open questions:

1. Should search binaries define -h/--help?
2. Should search binaries be told to never error out in the face of unknown options?
3. Is specifying the version restrictions (if available) even worth it?

[brettcannon]

The key thing here is how to service the two key use-cases:

1. The Python Launcher that wants the best answer the fastest it can get it.
2. Editors that want to know everything and what the suggested best answer is (which also services the Launcher case of needing to decide what to use).

It might come down to leaving the "best" selection logic to the caller and instead just provide ways to let the plug-ins know what they are searching for in order to maximize performance (e.g. 3 or 3.10).

[brettcannon]

It might also make sense to either support or always pass in a --workspace flag that specifies where the search should be considered to be anchored at. This could also be the current working directory, but explicit is better than implicit. 😁

[brettcannon]

At the moment I'm leaning towards not worrying about any version argument/option, and only really thinking about whether to explicitly pass in the workspace/directory.

[brettcannon]

Instead of a prefix and relying on PATH for locating locators exclusively, an environment variable could be supported to list more directories to prepend/extend PATH with.

[brettcannon]

Request for feedback from pyenv at pyenv/pyenv#2472 .

/cc @Darsstar

[brettcannon]

Twitter poll at https://twitter.com/brettsky/status/1578101990625710080?s=20&t=JCU0NXB3sEwbBIs9qqT6_g about whether to prefer interpreters with more version details or less (e.g. select 3.10 or 3.10.2 when a 3.10 interpreter is requested).

[brettcannon]

And the winner was ... no one! 😅 Too close to call, so I plan to say "unknown" < "0".

[brettcannon]

To record my thought process on something ...

I was considering dropping the environment.type key since it doesn't play into how you run an intepreter thanks to executable.run. But it does play into how to you choose to manually interact with it, e.g. if your editor tells you it's a conda environment and thus you are going to use conda install instead of pip install. So it's more of a UX hint than critical piece of information on how to run code.

[brettcannon]

One thing I'm thinking about is how to display something in a list of interpreters/environments (i.e. py --list)? I do know folks are asking for a which-like experience to just get at the path of the interpreter (see #226). Now as it stands, "path_id" won't serve that purpose because of its "unambiguous but as weakly specified as possible" design. Now if that was changed to "as tightly specified as possible" (e.g. virtual environments would be expected to always specify the pythonN.N interpreter path), then that could play double-duty for both the listing and the UUID for what was found. But the key reason I did the initial "weak" design was in case someone found ~/.virtualenvs/my-venv/bin/python3.11 as just an interpreter as well as a virtual environment. By saying virtual environments would have the ID be ~/.virtualenvs/my-venv, it would work around this ambiguity. But maybe it should be recognized there's always the chance of ambiguity and not worry about it? That way the consumer of the data is left with deciding what should take precedence?

Another solution is to introduce something like a "path_display" that could be ambiguous, but is purely for situations like py --list. But that might be a bit too redundant?

[brettcannon]

https://hynek.me/til/python-project-local-venvs/ covers how direnv users do things (from https://mastodon.social/@hynek/109561175719682337).

[dhellmann]

How dynamic is the data that would be retrieved? How often do you expect the information for a project to change? Would it make sense to keep the information in a file associated with the project, instead of invoking another program to look it up each time? For example, I could envision a plugin for virtualenvwrapper that updated a file (with a format defined by the launcher) with data each time an env is linked to a project directory via setvirtualenvproject or removed with rmvirtualenv. Then the launcher just needs to read that config file, and long running tools like IDEs could cache the results.

[brettcannon]

How dynamic is the data that would be retrieved?

I've thought about potentially caching and such (to the point of letting locators specify glob patterns to file watch for changes; both are one of the hidden comments), but in the end I figured simpler is better and so I don't plan to directly support any caching. If a locator like one for virtualenvwrapper wanted to cache a file, it totally could and is welcome to do so, but I think having the Launcher itself make any assumptions about the freshness/staleness of data is asking for trouble.

I also assume that for most people, they probably going to have 0, 1, maybe 2 locators, and so the overhead of calling them is going to be rather small relatively speaking compared to probably what you're about to do with Python itself.

How often do you expect the information for a project to change?

I have no idea, hence why I don't want to handle the caching. 😉

Would it make sense to keep the information in a file associated with the project

This is why I have thought about whether #165 makes sense to support directly and why I can't make up my mind about it.

instead of invoking another program to look it up each time?

Based on how trying to cache things in VS Code for environments has gone, I would rather avoid that as much as possible and assume folks are not going to use locators that have a drastic cost to run.

[dhellmann]

Based on how trying to cache things in VS Code for environments has gone, I would rather avoid that as much as possible and assume folks are not going to use locators that have a drastic cost to run.

OK, I can buy that running a program each time to answer the question is the best approach.

[brettcannon]

I brought back the executable.architecture field as I realized there wasn't going to be a way to tell when someone was using a WebAssembly build of Python vs a native binary that happened to also be 32-bits. Technically there's still a difference between something built for WASI or generic WebAssembly for the browser using Emscripten, but I don't expect much overlap in use cases in that regard. Plus if it ends up being that critical then people could probably abuse the architecture field somewhat to include that detail.

[brettcannon]

Having path_id be a path to a binary would be useful for PEP 668: EXTERNALLY-MANAGED if sysconfig.get_path("stdlib", sysconfig.get_default_scheme()) can be statically calculated, else executable.run solves the problem of getting that information.

[brettcannon]

I moved run to the top-level as I don't think any of this information is useful if you can't actually execute the interpreter.