Plugins split plan

[ana]

After reading previous PR/issues, I have made a first approach for splitting the plugins. Starting only with the HTML plugin, I split out the plugin in this (test) repository: https://github.com/avocado-framework/avocado-result-html
(Please, take a quick look because we need to agree on the name of the tags and branches)

I made a first PR/commit with the github actions. You can see the result at https://github.com/avocado-framework/avocado-result-html/actions/runs/987162409

There are only GitHub Actions, do we need to add more? It won't be so easy to combine the two repositories with Travis or Cirrus.

Finally, what is still missing is the spec file for packaging the plugin independently. More about this below.

With respect the avocado repository, I made a draft PR at #4755
In this draft PR, I removed the plugin, update GHA, cirrus and travis. I tried updating the packaging part but that proved tricky. After removing all the packaging bits of the HTML plugin, there are still the checks that are hard to avoid. I added --disable-plugin-checks=html but it doesn't work as it does with the robot plugin.

I don't know what's the best option now:

1. Update checks.py to be able to opt-out all the plugins tests so we can split the plugins and still keep the tests in the avocado repo. After all, testing the plugins must be done within avocado.
2. Move the tests with the plugin, then split them out. I don't fully see this one because it's nice to test all the changes of avocado with the plugins as well and have only one test suite. I don't even know if we have tests that involve two plugins together at the same time.
3. Is multi-source packaging possible with packit? This way packit not only fetchs the current git repository, it would also fetch all the git repositories with the plugins. This would be an intermediary step while we split the packaging properly.

Comments welcome!

References: #4389 #4608

[beraldoleal]

After reading previous PR/issues, I have made a first approach for splitting the plugins. Starting only with the HTML plugin, I split out the plugin in this (test) repository: https://github.com/avocado-framework/avocado-result-html
(Please, take a quick look because we need to agree on the name of the tags and branches)

I understand that all current plugins don't have a prefix right now, but I have the impression that would be nice to have a prefix/suffix on the repositories names so we can easily distinguish what are plugins and what are misc repositories (i.e: plugin-result-html).

I don't know what's the best option now:

1. Update checks.py to be able to opt-out all the plugins tests so we can split the plugins and still keep the tests in the avocado repo. After all, testing the plugins must be done within avocado.

Besides a better organization, another advantage of this split is to keep Avocado main repository as clean as possible, including dependencies. So if some would like to try a basic Avocado without additional plugins, the installation process would be easier. Dependencies for the plugin would be on the plugin repository (as you did with jinja2). Because of that, IMO we should try to follow this rule not only for users, during the installation process, but also for development. If I'm hacking Avocado, Would be nice if wouldn't need to install dependencies from all plugins. However, it is still important to know if I'm breaking the compatibility with those plugins, during CI stages. I remember discussing with @clebergnu something like this:

https://blog.marcnuri.com/triggering-github-actions-across-different-repositories/

What do you think about this approach?

[ana]

After reading previous PR/issues, I have made a first approach for splitting the plugins. Starting only with the HTML plugin, I split out the plugin in this (test) repository: https://github.com/avocado-framework/avocado-result-html
(Please, take a quick look because we need to agree on the name of the tags and branches)

I understand that all current plugins don't have a prefix right now, but I have the impression that would be nice to have a prefix/suffix on the repositories names so we can easily distinguish what are plugins and what are misc repositories (i.e: plugin-result-html).

Yeah, the naming of the repositories is also important. So we can go with something like:

• name of the repository: plugin-$NAME
• migrated tags from avocado/avocado-framework: avocado-framework-$VERSION.
  (This leads to a question: how are we planning to release the plugins. Every time there is a new avocado, we also tag the plugins?)
• Branches for LTS: do we keep them?

I don't know what's the best option now:

1. Update checks.py to be able to opt-out all the plugins tests so we can split the plugins and still keep the tests in the avocado repo. After all, testing the plugins must be done within avocado.

Besides a better organization, another advantage of this split is to keep Avocado main repository as clean as possible, including dependencies. So if some would like to try a basic Avocado without additional plugins, the installation process would be easier. Dependencies for the plugin would be on the plugin repository (as you did with jinja2). Because of that, IMO we should try to follow this rule not only for users, during the installation process, but also for development. If I'm hacking Avocado, Would be nice if wouldn't need to install dependencies from all plugins. However, it is still important to know if I'm breaking the compatibility with those plugins, during CI stages. I remember discussing with @clebergnu something like this:

https://blog.marcnuri.com/triggering-github-actions-across-different-repositories/

What do you think about this approach?

I'm fine when triggering workflow executions across different GitHub repositories as mentioned in that article, but I'm not yet there. First, I would like to discuss where/how we'll keep the tests of the plugins. A first thought, it seems logic to move the tests of every plugin to the plugin repository, but thinking further, when testing the plugins, you want to test it all together because some plugins might change behavior if they're used together.
I just realized now there is even an option 4, that's keeping all the selfests in a different repository that's used by all the avocado and plugins.

[ana]

In the meantime, I have started working in the first option and improving checks.py to exclude plugins, see #4761

[ana]

A summary of what I remember from our live discussion today. I'm ignoring all the part related the internal CI pipelines - because GitHub Actions are very flexible and will allow a lot of options.

Discussed possibilities

Split every plugin in their own repository (avocado-plugin-foo), with their own separate RPM source package.

Pros:

• More granularity, it's easy to deprecate a plugin and its package.
• It gives the impression to external contributors that's easy to contribute a plugin to avocado.
• A more pythonic approach.
• Makes it more clear that plugins are optional.

Cons:

• More complex installation for external users when installing from source. It changes nothing for people using the packages.
• Version numbering can be misleading and complicated to get right.
• We need to monitor closely which version of each plugin works with a given version of avocado.
• More complex and time consuming release process.
• Unsure what we gain from splitting.

Split every plugin in their own repository (avocado-plugin-foo), all plugins are packaged together in the same source package.

Similar to the previous option but packaging is relatively easier: all the plugins are released together with the same version number.

Split all the plugins in the same repository (avocado-optional-plugins) and package all the plugins from the same source package.

Pros:

• It's pretty much the same we have now but with the plugins in a different repository.
• Makes it more clear that plugins are optional.
• Minimal extra work compared to the current situation.

Cons:

• Users installing from source need to git or pip optional plugins.
• Some extra work for integration pipelines between repositories.

Not splitting them.

Pros:

• Nothing to do!
• Considerable amount of time saved that can be directed to features having a bigger impact.

Cons:

• Gives the impression that the plugins are required for avocado while they're optional.
• Need to continue working in making easier enable/disable optional plugins and installing their dependencies.

(Biased) Big picture summary

Splitting the plugins won't have any advantage for users of the packages. Rather, they'll sometimes find themselves having to check if avocado-plugin-foo version 0.90 is compatible with avocado version 0.95.

For users of git/pip, we can invest the time in making the current process easier and handle dependencies better. This still would be less of the time we'd need to invest to split the plugins out and further time invested in release work.

[beraldoleal]

ack, +1.

[willianrampazzo]

I understand the reasons for splitting the utilities. I'm not advocating against it, but reading the previous comments made me think about what message the Avocado project is passing to the users.

One of the biggest advantages of using Avocado is the set of tools (utilities) it has. Looking at some tests written by users of Avocado, it is clear the hard dependency of utilities when writing Avocado tests. What does it mean for Avocado users if we split utilities? I can imagine 2 things: 1 - They will always need to install Avocado+utilities. It makes me think about one of the objectives for the project this year of making Avocado easier for users; 2 - They can use another test framework with the utilities package.

As I mentioned, I'm not advocating against the split, but we need to make sure we pass the right message to the users.

[clebergnu]

I understand the reasons for splitting the utilities. I'm not advocating against it, but reading the previous comments made me think about what message the Avocado project is passing to the users.

One of the biggest advantages of using Avocado is the set of tools (utilities) it has. Looking at some tests written by users of Avocado, it is clear the hard dependency of utilities when writing Avocado tests. What does it mean for Avocado users if we split utilities? I can imagine 2 things: 1 - They will always need to install Avocado+utilities. It makes me think about one of the objectives for the project this year of making Avocado easier for users; 2 - They can use another test framework with the utilities package.

Yes, 1 is true, because we can expect even Avocado (the application, test runner, etc) to continue to depend on avocado.utils, just the way it currently does. But this will happen automatically, that is pip install avocado-framework will get "avocado.utils" installed too, simply because it's a dep. So, AFAICT, for users, nothing changes.

2 is also true, but the only way that it can impact Avocado is that it may attract more users + contributors -- even if using other test runners / frameworks with it.

As I mentioned, I'm not advocating against the split, but we need to make sure we pass the right message to the users.

Sure, it's good to double check we're not causing unintended damage. So, can you see any flaws in what I described above?

Thanks!

[clebergnu]

BTW, I meant that 1 is true in the sense that users will end up needing to have the "avocado.utils" package installed alongside with the "avocado-framework" one, but it should happen automatically anyway.

[willianrampazzo]

I understand the reasons for splitting the utilities. I'm not advocating against it, but reading the previous comments made me think about what message the Avocado project is passing to the users.
One of the biggest advantages of using Avocado is the set of tools (utilities) it has. Looking at some tests written by users of Avocado, it is clear the hard dependency of utilities when writing Avocado tests. What does it mean for Avocado users if we split utilities? I can imagine 2 things: 1 - They will always need to install Avocado+utilities. It makes me think about one of the objectives for the project this year of making Avocado easier for users; 2 - They can use another test framework with the utilities package.

Yes, 1 is true, because we can expect even Avocado (the application, test runner, etc) to continue to depend on avocado.utils, just the way it currently does. But this will happen automatically, that is pip install avocado-framework will get "avocado.utils" installed too, simply because it's a dep. So, AFAICT, for users, nothing changes.

Okay, I forgot we can make avocado-framework have the utilities as a dependency. That makes the user's life easier.

2 is also true, but the only way that it can impact Avocado is that it may attract more users + contributors -- even if using other test runners / frameworks with it.

Maybe, just time will tell us.

As I mentioned, I'm not advocating against the split, but we need to make sure we pass the right message to the users.

Sure, it's good to double check we're not causing unintended damage. So, can you see any flaws in what I described above?

No, that is fine, thanks for taking the time to think about it :D

Thanks!

[ana]

I'll look into the Avocado utils split then. Will open another discussion when I have some plan to share!

[ana]

A new idea appeared while discussing with @beraldoleal. This would be removing entirely the concept of optional plugins and making plugins either a core plugin or an extra plugin.

In practice, this would mean:

• some plugins will be moved from optional_plugins directly to avocado/plugins and installed (e.g. HTML)
• remaining plugins will stay in optional_plugins now renamed to extra_plugins

If any of the extra plugins ends up having very big requirements (such as a database) then yes, it's the time to split it.

Please, beraldo, leave a comment if I remember this wrong.
If people like the idea, I can make a first draft of what to do with the 8 plugins we have now.

[beraldoleal]

@ana I think you captured well the discussion.

We just need to double-check before promoting a plugin to a core plugin if it has some requirements. For instance: html plugin has jinja2 as requirement and we are trying to keep core without extra requirements.

I would vote for this approach, as I mentioned, the "optional_plugins" seems to be redundant, so "extra" seems a better name.

[ana]

As reference, the list of plugins and their requirements:

Plugin	Requirements (except avocado-framework)
golang
html	jinja2
resultsdb	resultsdb-api
result_upload
robot	robotframework<=3.1.2'
varianter_cit
varianter_pict
varianter_yaml_to_mux	'PyYAML>=4.2b2

[pevogam]

In practice, this would mean:

* some plugins will be moved from optional_plugins directly to avocado/plugins and installed (e.g. HTML)

* remaining plugins will stay in optional_plugins now renamed to extra_plugins

What is the main difference between optional plugins and extra plugins? Could we use some naming that is telling of the purpose and role of the separate types of plugins? At least to me it is not clear from the current names chosen. This being said, I also seem to be for some simpler deployment and only let plugins have separate repos once this is really justified. Working across many different repos, often in correlated fashion and sets of changes, could add significant additional costs with time and we have already been through some of it with some splitting we have done before.

[ana]

This PR #4941 contains important context about the tests of the plugins and what steps would need to be done in case of a split.