Plugins, & getting Lektor out of the package management business

[nixjdm]

For a long time now, it has become clearer that having Lektor manage the installation of a project's Python plugins is a significant pain point for maintainers and users. There are a lot of issues that have popped up over the years, from bugs, to missing features. I and some other maintainers are leaning toward minimizing Lektor's role in installing and managing a project's plugin Python packages. @mitsuhiko do we have your go-ahead? Thoughts or concerns?

Regardless of the exact goal chosen, there are the normal concerns about breakages and deprecations in order to prepare users for the changes and not make things unnecessarily hard.

We need to decided what exactly this looks like. For example, this could mean:

Rough option 1) - minimize plugin management:

• Removing the lektor plugins command, and it's sub-commands
• Removing the use of the [packages] section from project files
• Remove the automated use of packages/ subdirectory
• Give specific and simple guidelines for installing packages in the docs, such as how to use a very basic Poetry setup, or requirements.txt.
• Possibly leave in the ability run a similar command to lektor dev new-plugin, though this would likely be modified, at least to match the changed docs instructions. I don't see this, isolated, to be very burdensome, and is potentially very helpful.

In my opinion this option is the best, because

• At presents everyone with a clear demarcation of responsibility, and lets a dedicated package manager do what it was made for
• It reduces maintenance burden to the minimum on this issue.

This plan has the con of potentially removing some convenience features for users, such as only having to understand and use a single program (Lektor) instead of a minimum of two (Lektor and pip/poetry/etc). The burden of choice at least can be minimized by clear instructions in our docs.

Rough option 2) - do some level of wrapping of a preferred packaging tool

Should we try to keep some pieces of lektor plugins, such as add, remove, and list, and adapt them to match or wrap a chosen tool like Poetry? Should we leave in [packages] and somehow translate that to invoke a tool like Poetry? Should we auto-install things from packages/? These goals are somewhat independent, and they may offload some maintenance burden, but they clearly add some too. Choosing this path could mitigate some of the drawbacks of the option 1). In my opinion, that is not a good enough tradeoff.

N.B. there are also several discussions about using a different package management strategy internally to maintain this repo. This wouldn't matter much if we choose option 1), above, but the less we approach option 1), the more it might reduce mental overhead to use the same tool for integrations for projects and to manage this repo itself.

[mitsuhiko]

I have no strong opinions here. Maybe it's worth checking if pip can be instructed these days to install into a local path for lektor.

[frostming]

While I agree to remove the plugins command, the automatic loading of the local packages/ folder would be still useful for project-specific hooks that are not likely to share. I prefer to keep it.

[dairiki]

Here's a somewhat meandering brain dump...

Background on "The Troubles"

The headaches with the current system revolve around the issue that there seems to be no simple, stable, clean way to manage dependencies in what is, essentially, a stacked, second site-packages directory. (Or if there is a way, we haven't found it.)

We start with Lektor already running, then install some more packages into a separate local package cache, then add that local package library to sys.path.

But what about the dependencies of those local packages? Many of them may already be installed in Lektor's environment. Ideally, we don't want to install them again — if we do one of the copies is going to be ignored. What if there are version conflicts among the dependencies?

None of the package management libraries or tools (e.g. pip, poetry, whatever) I've seen really support this case. They are all geared to manage a single site-packages directory. As a result, currently, Lektor's plugin installation system more-or-less punts on this: any already-installed dependencies are installed a second time (though PR #1065 improves on this, I think), and there is no check for version conflicts.

Other Issues

For quite a while, pip install --target <private-cache> didn't work for installing local packages in editable mode. (Thanks to the efforts of our @dwt, that may now be fixed?) This caused Lektor to jump through hoops to install local packages.

Lektor's package management system is (appropriately) simple, with the result that it is not very flexible.
E.g. What if a plugin distribution specifies dependency extras? I don't think Lektor provides a way to specify that they should be installed. What if one wants to pin a dependency of a plugin?

How We Got Here

I suspect a lot of the motivation for getting Lektor into the package management business in the first place was the desire to be able to ship Lektor as a self-contained, packaged app. Initially, Lektor did ship a GUI app for macOS (I think?), but we've since given up on that, at least for now.

---

Is There a Reason to Stay in the Package Management Business?

Use Cases: Per-Project vs Shared Lektor Installation

Per-Project Lektor Installation

My workflow for Lektor projects (i.e. for each website I author using Lektor) involves a per-project virtualenv into which (a specific version) of Lektor is installed (along with perhaps other dependencies). In that case, I really see no reason to use Lektor's dependency management. Plugins can just as well be installed using whatever package management system I'm using to set up that per-project virtualenv. (This works, today. Just don't configure any plugins in the .lektorproject file, and put your local plugin source somewhere other than the packages subdirectory. PR lektor/lektor-website#328 by @yagebu does this for the Lektor website.)

The advantages of this method are likely apparent to any Python developer. One gets all the features of whatever package-management tool one chooses to use. (E.g. poetry, with its lock files for reproducible builds and command-line sugar for adjusting dependencies.)

Shared Lektor Installation

There is still an alternative workflow that involves using a shared installation Lektor, wherein a single installation of Lektor is used to build multiple Lektor projects. That Lektor might be installed globally as part of an OS distribution, or it might be installed on a per-user basis using pipx (as our docs currently recommend).
Each Lektor project, in general, may specify a different set of plugins, so to continue to support his use case, I think Lektor would have to remain in the package-management business.

Questions

So, the question is, who are our users? Do we have users for whom the added requirement to use an external dependency manager (e.g. Poetry) would be particularly onerous?

I don't know the answer to that, but my guess is probably not. Most Lektor projects in the wild, I would guess, have someone involved who is capable of writing the HTML templates, CSS, and JS for the site, as well as writing any project-local custom Lektor plugins, etc.

In any case, I would argue that for all but the simplest of Lektor projects, the build-reproducibility that comes from using a per-project virtualenv outweighs any additional effort that might be required.

---

My Preference

As mentioned above, it's already possible to use your dependency-management tool of choice to manage a per-project virtualenv into which Lektor as well as all the project's required plugins are installed. No changes are required to permit this.

• There is one change in Lektor that would make this easier. Currently, Lektor requires a plugin named "foo-bar" to come from a distribution named lektor_foo_bar. This is an (arguably) arbitrary requirement. A side effect of this is that any project-local plugins must be packaged into their own distribution. I.e. each plugin must be in its own subdirectory, which includes a setup.py or some other configuration for all of its own distribution metadata. This seems like an unnecessary hoop for plugins that are only intended for use in a single Lektor project. (For more, see Is there a reason Lektor is so strict about plugin distribution names? #875, in particular, my comments near the bottom of that issue.)

• Our documentation should be updated to describe the per-project virtualenv workflow and provide a concrete example (using a specific package management tool).

• Once those are done, we can deprecate all of Lektor's package management.

[dairiki]

• Possibly leave in the ability run a similar command to lektor plugin add, though this would likely be modified, at least to match the changed docs instructions. I don't see this, isolated, to be very burdensome, and is potentially very helpful.

If I understand what you are suggesting, wouldn't that lock the user into a particular dependency manager?

Poetry has poetry add lektor-markdown-highter, PDM has a similar pdm add, pipenv has pipenv install. I'm not sure there's much benefit to wrapping those.

[dairiki]

While I agree to remove the plugins command, the automatic loading of the local packages/ folder would be still useful for project-specific hooks that are not likely to share. I prefer to keep it.

If we retain the ability for Lektor to install distributions from packages, I think we might as well retain the ability to install from PyPI as well. (I think both of those require maintaining a separate, project-local package cache as is done now.)

Note that if we relaxed Lektor's requirement on what distribution name a plugin has to come from, installing local plugins into a project-local virtualenv becomes pretty simple using just about any package management tool. One needs just declare the necessary lektor.plugins entry points.

[frostming]

If we retain the ability for Lektor to install distributions from packages, I think we might as well retain the ability to install from PyPI as well. (I think both of those require maintaining a separate, project-local package cache as is done now.)

IMO it is not necessary to require the local plugin to be a distribution(with a setup.py or other metadata description). It should only expose an entry file, like conftest.py in pytest and lektor reads that file to find hooks to load. Other dependencies of the plugin, if any, should be added to the project by the package manager.

Yeah, that is very different from the current local packages folder. We can pick another name for the directory, like plugins to reduce the impact.

[nixjdm]

If I understand what you are suggesting, wouldn't that lock the user into a particular dependency manager?

Your confusion here is understandable! My mind slipped - I meant lektor dev new-plugin might be independently valuable and easy enough to keep, regardless of the other choices. And it wouldn't lock in anything but the suggestion to use a specific tool, which should match the docs. I've edited my original post.

[nixjdm]

I think I like the idea of being able to use local plugins that aren't distributions. It makes sense to me that you should just be able to supply just a single script if you aren't planning on publishing the plugin. If you are publishing it, then you are probably fine using a package manager and can let Lektor find the plugin as installed by the package manager, whether local or not.

If local plugins aren't distributions, how are they detected? I am in favor of using plugins/ for this and deprecating packages/ with everything else. Allowing local non-distribution plugins could introduce a non-arbitrary naming scheme, like pytest has, for modules and plugin classes, all for the sake of discoverability. Like pytest, we could have plugins.py and lektor_*.py, and search those modules for Lektor* classes. Or just only load plugins.py which either has plugins inside it or imports them from anywhere, and we could use Plugin.__subclasses__() after loading the modules. So there's a few options.

Plugins that are distributed, not local, I don't think have to require a naming scheme though, as long as the entry points are discoverable.

[dwt]

While I like the idea of getting rid of lektor managing its own plugins and just using a normal package manager instead, I am not sure I that this warrants the complexity building it's own plugin discovery and loading mechanism to allow plugins that are not distributable.

For two reasons: 1) Building an installable plugin is really simple (and can be even simpler for other package managers) and 2) the progression from an installable plugin to one that can be distributed through pypi is very simple / clear and something I think worth something to us as a community.

For example, to have a simple plugin 'date' that just adds some date functions to the ninja environment looks like this:

❯ pwd
/path/to/lektor-project/packages/date

❯ find .
./setup.py
./lektor_date.py

❯ cat setup.py 
from setuptools import setup

setup(
    name='lektor-date',
    version='0.1',
    author=u'Martin Häcker',
    author_email='my@email',
    license='MIT',
    py_modules=['lektor_date'],
    url='https://github.com/dwt/lektor-date',
    entry_points={
        'lektor.plugins': [
            'date = lektor_date:DatePlugin',
        ]
    }
)

❯ cat lektor_date.py 
from datetime import datetime, date
from lektor.pluginsystem import Plugin

class DatePlugin(Plugin):
    name = u'Lektor Date plugin'
    description = 'Add date objects to your jinja templates.'
    
    def on_setup_env(self, **extra):
        self.env.jinja_env.globals.update(
            date=date, datetime=datetime,
        )

And for sure, that setup.py file could still be shortened considerably. But I opted to already prepare its eventual publication.

Some documentation might go a long way - and I would be happy to provide some of it.

Parting thought: I think it is easier to document how to build installable plugins very simply, than building a custom plugin loader. That would still allow us to get rid of the custom pip install stuff and would still promote the ease of publishing plugins that we already have.

[nixjdm]

2. the progression from an installable plugin to one that can be distributed through pypi is very simple / clear and something I think worth something to us as a community

I think that's a pretty good point.

I agree that leaving it out would minimize maintenance cost on us. My comment pitching a few ways of doing it I think illustrates this point. It's all unneeded if we just make people install a plugin like any other local package.

Leaving out plugin loading makes things a bit more inconvenient for the user, but not a lot. Especially with good docs, a user could pretty easily modify their primary pyproject.toml (for example) to add a line to add a local plugin in editable mode. Then their env setup is basically unchanged from then on. To give a clear example, a user could run lektor dev new-plugin followed by poetry add --editable ./plugins/my-package/, then poetry install as normal. It's a bit more inconvenient, but not by a lot.

[dairiki]

3. the progression from an installable plugin to one that can be distributed through pypi is very simple / clear and something I think worth something to us as a community

I think that's a pretty good point.

I disagree, I think. We've been through this before (see #875), and I suspect I'm still in the minority, but...

While I agree that our documentation should include some pointers and encouragement on how to publish a plugin the PyPI and I think the plugin showcase in the docs is a great way to encourage and publicize plugins, I think it's possible to push too hard (and make it too "easy") to publish plugins to PyPI. It does a disservice to both the Python and Lektor communities to have crufty, stale, orphaned, non-maintained distributions to PyPI. It's not a good look for Lektor to have a number of lektor-* plugins on PyPI that look vaguely interesting but don't actually work with recent versions of Lektor.

It takes more than a functioning setup.py to make a healthy public distribution: e.g. hopefully, there is a GitHub repo (or similar) for the plugin to foster community feedback, some CI testing, etc. I don't think it's Lektor's place to provide a lot of guidance and leadership on these basic open-source, and python-wide best practices.

Obviously, everyone who is going to be writing a Lektor plugin and then publishing it to PyPI (and then maintaining it to any degree) is going to have some level of competence in Python development. It's just not that hard to create a PyPI distribution and there's plenty of general documentation out there on how to do this. I don't think it's in Lektor's best interest to do too much hand-holding in this regard.

---

On the other side, nearly every Lektor project I work on includes a local Lektor plugin which includes "throw-away", local-to-that-particular-project, extensions. Mostly, these include things like custom jinja globals and filters, but often it includes custom link formatting in Markdown text, etc. Including distribution metadata for these plugins (other than, perhaps, entry point declarations) is just superfluous. It's some trouble (not a lot, admittedly), but, more importantly, it's pointless — e.g. the version number in that setup.py means nothing.

I suspect it would be more "beginner-friendly" to make basic local plugin authoring as simple as possible, rather than getting hung up and ensuring that all plugins are "PyPI-ready".

[dairiki]

I'm neutral on the idea of adding a new custom plugin loader.

It would be simple enough (i.e. minimal additional code) to do something like: check a particularly directory for .py files (or just check for a single particular .py file); load all matching files; look through each loaded module for subclasses of lektor.pluginsystem.Plugin and treat each of those as a plugin.

At the same time, once one switches to using a package manager (e.g. Poetry) to manage the Lektor plugins for a particularly Lektor project, it's really not that complicated to manually configure the appropriate entry point(s) in the top-level metadata file for a project.

E.g. here's a Poetry pyproject.toml that configures a local plugin (as well as installing lektor-atom from PyPI):

[tool.poetry]
# Lektor currently requires that the dist name be "lektor-<plugin-name>".  This should, IMO, be relaxed.
# If it were, we could name this project "test-demo-lektor-website" or something else
# more descriptive.  (We could also configure more than one local plugin.)
name = "lektor-demo"
version = "0.1.0"
description = "Test"
authors = ["Jeff Dairiki <dairiki@dairiki.org>"]
readme = "README.md"
#packages = [{include = "demo_plugin.py"}]

[tool.poetry.plugins."lektor.plugins"]
# plugin code is in ./lektor_demo/plugin.py
demo = "lektor_demo.plugin:DemoPlugin"

[tool.poetry.dependencies]
python = "^3.7"
Lektor = "^3.3.7"
lektor-atom = "^0.4.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Note that this works just fine with current versions of Lektor.

[frostming]

At the same time, once one switches to using a package manager (e.g. Poetry) to manage the Lektor plugins for a particularly Lektor project, it's really not that complicated to manually configure the appropriate entry point(s) in the top-level metadata file for a project.

Oh, good point, I didn't consider it before. Yes, we can define the local plugins directly in the current project's metadata and after it is installed, the plugins are activated already. Except that it comes with a minor inconvenience -- the lektor project itself must be a PyPI-ready package, which is not usually the case for a website. It seems Poetry ensures that when initializing a project but others like Pipenv may not force that.