pytest-plugin-list features

[mcsitter]

Features

As described in pytest-dev/pytest#5105 this issue discusses the features and functionality of the pytest plugin list.

Previously a list of pytest plugins was retrieved from the package index and integration tests were performed. The table showed success and failure of the integration tests of the plugin for different python versions. This did not prove to be very useful so a simpler approach is desired.

Plugin definition

• A pytest plugin is defined as any package that matches the pattern pytest-*.

Excluded from this definition are the following:

• Plugins which do not follow this standard are not supported until they can be automatically identified properly.
• The trove classifier for the pytest framework is also not used due to a huge amount of false positives, since many projects test with pytest.

Columns

The plugin list should contain the following columns:

• name of the plugin
• summary of the plugin functionality

These are usually provided by the package metadata.
Other metadata (source) which may be desirable:

• compatible python versions (shield could be embedded)
• minimum required pytest version (metadata)

About development status and maintenance:
Indicating the maintenance status of a package could prove very useful! Here are a couple ideas:

• show last release date
• show development status as given by the classifier
• create flag for packages that do not match maintenance criteria (last release, classifier etc.)

I am in favor of excluding plugins which have the Development Status :: 7 - Inactive classifier from the list. I am not so sure about the criteria for exclusion, since the classifier is actively set by the maintainer, while some plugins very rarely get a new release, but seem to be relatively maintained and functional according to download statistics and the repository homepage.

It would be great if there is no need for a blacklist imo.

It may be worth thinking about exclusion of projects, which have a development status like Planning or another unstable development status.

Docs Integration

Different solutions for the integration with the pytest docs have been proposed. All would be automatically triggered by a cronjob in a ci system. These are the most common

• job which automatically creates a PR to pytest-dev/pytest if there are changes
• plugin list served through github pages

Open for feedback!

[mcsitter]

@nicoddemus , I have made some progress. The basic functionality (project name and summary) of the script is implemented and i created a github workflow, which automatically creates a pull request to update the plugin list. I believe this is the best docs integration. The cron pattern can also be discussed. If there are some features like last release date or version numbers, the cron should not run to often to avoid the maintenance overhead of merging PRs. Once every week could be enough if it is basically just a list of names. Otherwise one could also set up automatic pull request merges.

I would love some feedback to the features described above regarding exclusion/ inclusion criteria and the included columns/ data. I have written the code for the features above already, so no worries there.

One more minor thing: Naming of the script, commit message, pull request title and body are open to feedback.

I mentioned you, because i did not get any feedback so far.

Since you initially raised issue, @ssbarnea , you may also want to give feedback.

This will probably end up being a pull request to the main repo, but i would like to get things working as desired here and locally first

[nicoddemus]

Hi @mcsitter!

I have made some progress. The basic functionality (project name and summary) of the script is implemented and i created a github workflow, which automatically creates a pull request to update the plugin list.

Sounds great!

I believe this is the best docs integration. The cron pattern can also be discussed. If there are some features like last release date or version numbers, the cron should not run to often to avoid the maintenance overhead of merging PRs. Once every week could be enough if it is basically just a list of names. Otherwise one could also set up automatic pull request merges.

I think PRs from the cron job could be merged automatically. With that, it can be executed once a day, as the cron job should be fairly quick (last jobs took ~2m only).

I would love some feedback to the features described above regarding exclusion/ inclusion criteria and the included columns/ data. I have written the code for the features above already, so no worries there.

About the features:

compatible python versions (shield could be embedded)

👍 for the shield: should be just really simple to embed.

minimum required pytest version (metadata)

Desirable, can we get that directly from PyPI, or do we need to process a wheel?

One more minor thing: Naming of the script, commit message, pull request title and body are open to feedback.

Those are all your personal choices to be honest, but I like to add some information that a commit/PR was done by the bot, such as a [bot] or [automated] prefix.

About development status and maintenance:
Indicating the maintenance status of a package could prove very useful! Here are a couple ideas:

• show last release date
• show development status as given by the classifier
• create flag for packages that do not match maintenance criteria (last release, classifier etc.)

All sounds great to me, and would be useful to users. Packages marked as Inactive could be explded

I am in favor of excluding plugins which have the Development Status :: 7 - Inactive classifier from the list. I am not so sure about the criteria for exclusion, since the classifier is actively set by the maintainer, while some plugins very rarely get a new release, but seem to be relatively maintained and functional according to download statistics and the repository homepage.

Excluding Inactive packages sounds like a great start. We can refine from there once we have things running.

I mentioned you, because i did not get any feedback so far.

Yeah sorry about that, I made a mental note to check here but forgot. Thanks for the ping, it helps because then I get a notification! 👍

All in all this is shaping up great, I'm really happy to the thought and the work you have put on it so far, thanks a lot!

[mcsitter]

Update! What i did so far:

• Add [automated] prefix to commit message & PR
• Run Update Plugin List daily
• Add pyversions shield
• Add development status info
• Add pytest requirement

Parsing the requirements can be done without installing the wheel. It is not very fancy, but it works.

If the coding style can be more pythonic let me now. Iterating, checking and returning values while requiring defaults are rather complex syntactically and i find it hard to decide between for loops, comprehension and where to use else statements.

I could not figure out yet, how pypi determines the last release date. The API gives a list of releases, but there can be a release without any files uploaded and only the uploaded fiels have a release date. With some fiddling locally, i got it to work in a way that satisfies me. I may have to look more into the source code for warehouse to do it the way they do it.

A maintenance flag also has not been added so far. I suggest using ("✅" and "❌"). Development status 5 (stable) and 6 (mature) would be considered maintained if it has a release within 12 (?) months. I am not sure about including status 4 (beta).

ToDos could be:

• Add last release date
• Add maintenance flag based on last release date over threshhold and possibly development status over threshhold (excluding inactive)

I will add the release date next! If the threshhold for maintenance is determined i will also add that one. Then i think i should open a PR to the pytest codebase.

[nicoddemus]

Update! What i did so far:

Cool! Any time frame in your mind to have this up in GH sites? Note it doesn't have to be perfect to be up and running, it helps seeing what works or not on the live page.

If the coding style can be more pythonic let me now. Iterating, checking and returning values while requiring defaults are rather complex syntactically and i find it hard to decide between for loops, comprehension and where to use else statements.

TBH I'm not looking at the code at all, but I can take a look some time if you like.

I could not figure out yet, how pypi determines the last release date. The API gives a list of releases, but there can be a release without any files uploaded and only the uploaded fiels have a release date. With some fiddling locally, i got it to work in a way that satisfies me. I may have to look more into the source code for warehouse to do it the way they do it.

I wouldn't worry about releases without files, only taking the latest released file is fine I think.

A maintenance flag also has not been added so far. I suggest using ("✅" and "❌"). Development status 5 (stable) and 6 (mature) would be considered maintained if it has a release within 12 (?) months. I am not sure about including status 4 (beta).

I also wouldn't put too much weight on the "development status" classifier... I suspect many plugins might have a "alpha" status and have been working for years. Before we think too much about it, might be more useful to just display that information to users and let them decide.

I will add the release date next! If the threshhold for maintenance is determined i will also add that one. Then i think i should open a PR to the pytest codebase.

I got from our previous discussions this would live in its own repository under the pytest-dev org eventually, just like plugincompat does now, not in pytest-dev/pytest. Or is that what you mean?

[mcsitter]

Cool! Any time frame in your mind to have this up in GH sites? Note it doesn't have to be perfect to be up and running, it helps seeing what works or not on the live page.

I will be busy towards the end of the week, but will probably be able to do it tomorrow or on thursday. I never set anything up with it, but should not be too hard.

• job which automatically creates a PR to pytest-dev/pytest if there are changes

This will probably end up being a pull request to the main repo, but i would like to get things working as desired here and locally first

I thought this could be incorporated into the main pytest repository since this will be 1 workflow file, 1 script and the automatically generated rst. This is why i adopted the same directory layout as is present in pytest-dev/pytest . With automatically merged PRs it would'nt even produce to much clutter. Also there will not necessarily be a daily PR if nothing changes. Read more on the create-pull-requests GitHub Action. Also tests will automatically be run on this part of the docs as well.

I am of course open to transfering ownership to pytest-dev and also to leaving it as a GH page.

just display that information to users and let them decide.

It's probably a good idea. I will not add a maintenance flag, since it may be misleading and does not add any information.

I wouldn't worry about releases without files, only taking the latest released file is fine I think.

Alright, i will figure out how to do precisely that.

[nicoddemus]

Thanks again for all the hard working on this!

I am of course open to transfering ownership to pytest-dev and also to leaving it as a GH page.

I think it would be better, otherwise pytest-dev.github.io/pytest would be a list of plugins, which is a bit weird. 👍

[ssbarnea]

I wanted to propose something like: have the plugin list generator/updater as a standalone project under pytest-dev. I would have personally tried to use a JSON file as "database" and have other scripts that build various kind of reports based on it. JSON file can also be queried from outside (official pytest doc page).

[mcsitter]

I added the creation of a html page to the script. This is not done by rendering the rst, but it allows to get a feel for how it may look. You can find it here.

The proposition of @ssbarnea at #6 should not be to hard to implement, since i can easily store the dicts with the data in a json. How it should be stored depends on how it will be used. Should the pyversions shield be included as a url, a html tag, rst?

What i do not understand yet is why the script should not be part of the main repository?
It is just 1 script, 1 workflow and 1 generated document. Once implemented it should not require much maintenance and also will give version control to the docs, which is not the case with the current implementation or the other proposals.
For example with the release of pytest 6 some plugins drop support for previous versions with new releases. If one is locked to older pytest versions for whatever reason, there is no way to see the plugin list from the past.

[nicoddemus]

What i do not understand yet is why the script should not be part of the main repository?

As I mentioned in #1 (comment):

I think it would be better, otherwise pytest-dev.github.io/pytest would be a list of plugins, which is a bit weird.

For example with the release of pytest 6 some plugins drop support for previous versions with new releases.

It is not so much about the maintenance per-se, but to have it published under a separate name. Or do you mean to integrate it into the main documentation?

If one is locked to older pytest versions for whatever reason, there is no way to see the plugin list from the past.

In its own repository I suppose it would be just as easy to investigate previous versions of the documentation.

[mcsitter]

I mean integration into the main documentation. Thats why i placed the plugin list in the doc directory here.

[nicoddemus]

I mean integration into the main documentation. Thats why i placed the plugin list in the doc directory here.

Ahh I see. Yeah that would just be another docs page, which is periodically updated by a cron job via PRs, I like the idea!

I thought the intent was to have a separate website, that's why I didn't think it was a good idea initially.

[mcsitter]

If the implemented features are satisfactory i would open a PR with the workflow and script to the main repo. I think i would pick the previous version of the script, which does not include the html table in index.html.

[nicoddemus]

Sounds good.

[mcsitter]

Implemented in pytest-dev/pytest repo