tox4: plugin migration use cases

[gaborbernat]

This issue is used to mark the migration plans for various tox plugins. Shows which plugins and use cases we considered when defining the plugin interface for tox 4.

1. tox-travis
2. tox-poetry
3. tox-ansible
4. tox-pipenv
5. tox-current-env
6. tox-ltt
7. tox-py
8. tox-extra
9. tox-asdf
10. tox-bindep
11. tox-pyo3
12. tox-backticks
13. tox-envreport

Need to fork

1. tox-pyenv

POC done

1. tox-conda see https://github.com/tox-dev/tox-conda/tree/main/src/tox_conda

Done

1. tox-pdm see https://github.com/pdm-project/tox-pdm/blob/main/tox_pdm/plugin4.py
2. tox-gh-actions , recommending https://github.com/tox-dev/tox-gh instead (though upstream started adding support for tox 4)
3. tox-docker - thanks to @dcrosta at https://github.com/tox-dev/tox-docker/tree/tox-4

Moved to core

1. tox-battery via Port pip requirements.txt parser #2009
2. tox-external-wheels - via Support for external packages and builders #2235
3. tox-factor
4. tox-envlist

[hroncok]

We have https://pypi.org/project/tox-current-env/ and I'd happily discuss the use case with you. If new tox could be used as a library, we would be able to drop the plugin entirely and just ask tox for the data we need.

[gaborbernat]

Please do describe what functionality you would like to be exposed to, and why would it be better than a plugin? Thanks!

[hroncok]

Will do tomorrow.

EDIT: Or the day after.

[luzfcb]

I had to do some juggling for my tox-lambda-autodiscovery experimental plugin to work. There is no simple way to defer the processing of commands.

At that time, @gaborbernat had some insights into the improvements needed in a future tox 4. What is happening now.
https://mail.python.org/archives/list/tox-dev@python.org/thread/2B2GXTFWCE6FYMVCMBMKAOQAXEOGXEWR/

[gaborbernat]

There is no simple way to defer the processing of commands.

Can you clarify what do you mean by defer here?

[frostming]

Any chance unify the interface of install deps and install pkgs so that we can change the behavior in one hook.

[gaborbernat]

Any chance unify the interface of install deps and install pkgs so that we can change the behavior in one hook.

Definitely not, these two sections have totally different formats and contracts. It just happens pip can handle both using the same interface, but that's not generally true. If you're using an installer that can handle it the same way you can still define two hooks that calls the same function.

[hroncok]

tox-current-env

What do we want

We want a way to run upstream tests when we build RPM packages for Fedora. We want a standardized way to get a list of test dependencies (by we install them ourselves) and a standardized way to get commands to execute (or a standardized way to execute them in a Python environment we choose).

Why do we use tox

Unfortunately, there is no standard. We've been told that tox is the de-facto standard and that any future standard (if any) is likely to evolve from tox. I don't recall where is this information coming from, but it makes sense to us.

What does our plugin do

When certain command line switch is used, it redefines the virtualenv creation (and fakes it with symbolic links to sys.executable) .
When certain command line switch is used, it redefines the "run the tests" part of tox to print some information parsed by tox from the configuration (related to dependencies).

Why would a library be better

If there was a library, we could "ask" tox: What are the dependencies for this toxenv? What are the commands for this toxenv? (Or alternatively: Execute the commands via subprocess.) I think that this approach is better, because it:

• involves no hacks
• does not break when other plugins are installed
• is lighter
• is possibly faster as it does not involve serializing and deserializing data between our tooling and tox to text files or fds

Ideally this library would not even require virtuelnv, but I understand that this would require packaging it separately and that is a lot to ask.

[gaborbernat]

@hroncok for your use case my recommended solution would be to replace tox environment creation with a no op that just serves to tox the host python, translate pypi dependencies to RPM dependencies, and use apt to install them. All this you could do in a plugin. This would be the recommended way, and safer IMHO. Now if you'd insist on rolling your own pipeline to get the raw dependencies and commands you can use tox --showconfig and interpret the output. In tox 4 you'd be able to do tox c -k deps,commands to get just these.

[hroncok]

replace tox environment creation with a no op that just serves to tox the host python

I thought that we need an actual environment, so we create a fake one. I can have look if this can be simplified.

translate pypi dependencies to RPM dependencies, and use apt to install them

Due to technical limitations, we cannot install stuff directly, we need to echo it on stdout and exit. That is what we do, more or less, via the plugin (except that we do the translation via a wrapper).

This would be the recommended way, and safer IMHO.

OK. We'll keep doing it that way then.

Now if you'd insist on rolling your own pipeline to get the raw dependencies and commands you can use tox --showconfig and interpret the output. In tox 4 you'd be able to do tox c -k deps,commands to get just these.

I'll check it out.

[gaborbernat]

In a tox 4 world, you'd just need to implement the abstract parts of https://github.com/tox-dev/tox/blob/rewrite/src/tox/tox_env/python/api.py#L73, by default we use a virtualenv+pip based https://github.com/tox-dev/tox/blob/rewrite/src/tox/tox_env/python/virtual_env/api.py#L28, but you're free to replace those parts with host python + dnf. Note the API is not stable yet 👍🏻 so don't jump on this bandwagon just yet, but this is the midterm plan.

[obestwalter]

Hi, haven't looked at tox-current-env in detail yet, but it sounds a lot like the plugin, I wrote to facilitate running tests and tasks via tox inside a container that already contains a prepared venv or needs packages installed in that prepared venv: https://pypi.org/project/tox-direct/ circumvents the venv creation and uses the env that tox was started from.

I am just getting acquainted with the rewrite after a long time off from the whole project, but if I understand correctly one reason for the rewrite is to be able to accommodate different run time environments directly, including docker containers, so it would be interesting to find out what and how much of plugins like tox-direct and tox-current-env is actually still needed in tox4.

[gaborbernat]

They're still needed. You want to be able to plug in the system python as the tox python env. The subject is not how to skip X, but more like how do we integrate X to work with tox. The expectation here is that users will implement abstract base classes, such as https://github.com/tox-dev/tox/blob/rewrite/src/tox/tox_env/python/runner.py#L22; and use that as the runner class.

[gaborbernat]

A POC of tox-conda has been now done under https://github.com/tox-dev/tox-conda/tree/main

[gaborbernat]

https://tox.readthedocs.io/en/rewrite/plugins.html

[frostming]

Hi, I've updated tox-pdm to support both tox 3 and 4. It turns out not easier than v3(232 lines v.s. 157 lines). I don't know if I am missing any best practices.

[gaborbernat]

Thanks. I don't think it would be easier (well definitely not code line wise), but in tox 4 you can do more with fewer hacks. Would have been nice if you opened a PR on the new plugin so I can add inline comments to the relevant parts.

[frostming]

Yes, the v3 plugin is really HACKY. There are two main points that make me feel uncomfortable:

1. The Execute -> ExecuteInstance chain is a bit long for overriding.
2. The package env and running env have to perform multiple subclassing to share some code, this makes the inheritance chain really long and hard to trace, like I don't know what abstract methods are there remaining unimplemented. I did this by importing them in the REPL and print the __abstractmethods__ attribute.

If you are interested, here is the PR that you can leave comments on: https://github.com/pdm-project/tox-pdm/pull/11/files

[gaborbernat]

1. The Execute -> ExecuteInstance chain is a bit long for overriding.

Can't be helped sadly 🤔

2. The package env and running env have to perform multiple subclassing to share some code, this makes the inheritance chain really long and hard to trace, like I don't know what abstract methods are there remaining unimplemented. I did this by importing them in the REPL and print the __abstractmethods__ attribute.

Yeah, this is a trade-off for allowing easy override of things, while making explicit what needs to be implemented (via abstractmethods). IDEs are generally really good at saying what's missing, they likely use the same attribute though.

[ymyzk]

tox-gh-actions moved to https://github.com/tox-dev/tox-gh

Clarification for users, tox-gh-actions supports tox v4: https://github.com/ymyzk/tox-gh-actions/tree/tox4. @gaborbernat made a new implementation at https://github.com/tox-dev/tox-gh.

[dcrosta]

Happy to report, tox-docker now works with 4.0.0a9. I'm happy to cut a release once you feel the APIs are stable and unlikely to change any more @gaborbernat, just let me know.

[frenzymadness]

I'm trying to rewrite tox-current-env for the new tox and I have to admit that I'm kinda lost. The new structure of environments, runners, installers, and executors is too complex for me right now. But, I have one general question related to my implementation. Is it possible to read tox options before I register my new tox environment? I'm asking because I see two possible ways forward for me now:

1. Either I can register my new environment every time. In that case, I'd probably inherit from the default VirtualEnv, override the important methods, and run our code (which skips the default behavior) only if the plugin is activated by one of the provided CLI options - otherwise, fall back to the original implementation in the VirtualEnv methods.
2. Or I can implement very minimal implementation of a tox environment and register it only if our plugin is activated by one of the provided CLI options. This means that I don't have to manually check it in all the methods and manually call the original implementations in the superclass - this sounds better for future maintenance if the default implementation changes.

I think that the second option is better and something like that might be useful also for other plugins working in the way "do nothing unless explicitly activated by an CLI option". What do you think @gaborbernat ?

One more question: Do I understand it correctly that the tox_before_run_commands is executed after all the installation and preparation steps and before the actual tests are executed?

[gaborbernat]

I'm not sure I understand all your questions, but few major points:

• You definitely should not inherit from VirtualEnv as that would imply your host environment is a Python environment which is not the case in general. You should inherit from https://github.com/tox-dev/tox/blob/rewrite/src/tox/tox_env/python/runner.py#L18 as all host environments will be a Python environment.
• You likely want to define a host runner type, and set that as a default runner from https://tox.wiki/en/rewrite/plugins.html#tox.plugin.spec.tox_register_tox_env

One more question: Do I understand it correctly that the tox_before_run_commands is executed after all the installation and preparation steps and before the actual tests are executed?

Yes, but I don't think you want to do changes there in any shape or form. Can discuss more interactively over https://discord.gg/tox if you want.

[frenzymadness]

I'm not sure I understand all your questions, but few major points:

Thanks for that but the main question remains open. It is possible to do something like this:

@impl
def tox_register_tox_env(register):
    if option.current_env or option.print_deps_to or option.print_extras_to:
        register.add_run_env(CurrentEnvRunner)

Are options available in any form during the registration of environments?

[gaborbernat]

Not at the moment, there was no use case yet needing the CLI arguments for env registration 🤔 Why would you want to keep the registration optional?

[frenzymadness]

Because our plugin can be enabled via CLI arguments only (no configuration options), which means that if you run just tox without any arguments, our plugin does nothing and should not affect the testing process. It sounds reasonable to me to not register our plugin and its environments if it should have no effect at all.

[gaborbernat]

Registering the environment runner does not imply activating it. It should be safe to register it and only active it based on CLI flags within https://tox.wiki/en/rewrite/plugins.html#tox.plugin.spec.tox_add_core_config. If registering the environment has side-effect you might need to rethink your design of the plugin. Can you share the code you have, I'm happy to chime in via a review on it, to guide the best path ahead.

[frenzymadness]

Thanks to your help, @gaborbernat , I was able to create this: fedora-python/tox-current-env#42

Could you please take a look and tell me if I'm doing something completely wrong in the implementation? It seems to work even some parts look kinda hacky but the whole plugin seems to be a corner case.

[Cadair]

Hello 👋

I have been trying to update the pypi-filter plugin for tox 4. The plugin runs a PyPI proxy server as a subprocess and then used to set the indexserver option so that requests to PyPI went through the proxy. (The proxy can filter out versions of any packages being installed, including transitive deps etc).

I have mostly managed to convert it to tox 4 but the issue I have is there doesn't seem to be a hook at the right point in the cycle for me to use. Using tox_add_env_config works, but causes me to run a proxy server for all defined envs (logically enough), but using tox_before_run_commands is too late because I need to spin up my proxy server before any of the deps or installation of the package happens.

I looked around at overriding the virtualenv builder to achieve what I wanted but my head started spinning pretty quick, so I thought I would ask for input on how best to approach the goals of this plugin in the tox 4 land. Thanks!

[gaborbernat]

Ideally, when do you think this plugin should start/stop?

[Cadair]

I would like it to start before the install_deps phase so that it is running for install_deps, install_package_deps and install_package, then it could stop at before_run_commands or maybe after_run_commands.

99.9% of the time it wouldn't matter if it was running for the package build step, it's goal isn't really to fix issues in building the package.

[gaborbernat]

I would like it to start before the install_deps phase so that it is running for install_deps, install_package_deps and install_package, then it could stop at before_run_commands or maybe after_run_commands.

Those phases are env phases, so do you want to run one instance per env then?

[Cadair]

Yes, that's what we were doing in tox3 land. I don't think personally, I have ever configured it with different filters per-env but it's possible. That's wrong, we disable it for some envs (i.e. tox-conda ones).

[gaborbernat]

So my recommendation is to:

• set your plugin configuration in https://tox.wiki/en/latest/plugins.html#tox.plugin.spec.tox_add_core_config
• use https://tox.wiki/en/latest/plugins.html#tox.plugin.spec.tox_add_env_config to load the set_env and then use update to set PIP_INDEX_URL https://github.com/tox-dev/tox/blob/main/src/tox/config/set_env.py#L95
• use Add tox_on_install and tox_env_teardown plugin hooks #2687 to start the index server, note this can be invoked in parallel by multiple threads, so use some global thread lock while starting the server
• https://tox.wiki/en/latest/plugins.html#tox.plugin.spec.tox_after_run_commands is not a good place to teardown as this will not be called with --no-test. We likely want some new teardown endpoint for that, PR welcome.

[gaborbernat]

@Cadair https://tox.wiki/en/latest/changelog.html#v4-0-9-2022-12-13 has now all you need

[Anvil]

Good morning everyone

I was unhappy with the fact that tox-backticks has not been ported to tox4 yet, and also unhappy by some bugs (curly braces escaping does not work) and/or the behaviour of this plugin (trailing newlines not being stripped...), so I decided to start my own plugin.

I've been learning the tox internals for the past few days, and produced a very early-stage tox-backtocks plugin. This is a work in progress but it currently works with some basic cases, and, if you can believe me, I intend to do something robust, useful and user-friendly.

I'm gonna start testing it and using it on the 100+ projects we have at work that currently use tox-backticks, in order to finally being able to upgrade to tox>=4.

If it's not too much to ask, I would appreciate some feedbacks on the code (I'm using some protected attributes of some tox objects... and did not find any other way to achieve my goal) and also if any tox-backticks user is reading me, feel free to report any failure and/or misbehaviour.

Thank you.

[gaborbernat]

Would be easier if you'd have a PR we can open. You are accessing here a private attribute that's probably not a good idea https://github.com/Anvil/tox-backtocks/blob/main/tox_backtocks/__init__.py#L54. I'll close this issue, feel free to open a new discussion (rather than issue for feedback).