"How to create plugins" needs additional development context

[pkt-nspktr]

• I have searched the issues (including closed ones) and believe that this is not a duplicate.

Issue

Environment: FreeBSD 13.1-RELEASE-p1; Python 3.9.13 (installed from FreeBSD pkg(8)); Pelican 4.8.0 (installed in virtualenv via pip install pelican)

I'm attempting to develop a new plugin ("csv-reader") for Pelican. I've followed the documentation for how to create plugins and even taking the steps to do a fresh install of Pelican to eliminate other factors. Nevertheless, I do not understand how to develop or test the plugin based on the current documentation.

The namespace plugin structure section recommends using the cookiecutter template to set up the proper structure. Running this created the recommended plugin directory structure under ~/virtualenvs/pelican/csv-reader/.

The how to use plugins section says, "If you leave the PLUGINS setting as default (None), Pelican will automatically discover namespace plugins and register them." Running pelican --ignore-cache --print-settings | grep -i plugin returns

'PLUGIN_PATHS': [],
'PLUGINS': [],

My expectation based on the documentation would be Pelican auto-discovers the plugin created by the cookiecutter template, but when executing pelican --debug, the plugin messages only says, "DEBUG Finding namespace plugins"

I'm clearly missing some additional steps to test a plugin under development. I don't expect the Pelican documentation to be the definitive guide to plugin development, but it would be nice to have it link to a more comprehensive description of how to configure Pelican to load the cookiecutter-created directory structure for testing and then how to package it up and make it available on the pelican-plugins repository.

If this is the wrong Issue tracker, please point me in the correct direction. I hope that I can learn what I need to do to test my plugin and improve the documentation for other who wish to do the same.

[justinmayer]

Hi @pkt-nspktr. The short answer is that you must install the plugin, and it sounds like it has not yet been installed. In a development context, it's best to install plugins in "editable" mode so you can make changes and observe the results in real time.

You can refer to the plugin development documentation for more detail. There are multiple ways to install Python packages in editable mode, and the method utilized in that documentation is to run the invoke setup task, which calls the setup function in the tasks.py file to poetry install the current plugin.

[pkt-nspktr]

@justinmayer , thank you for the response - I am certainly ignorant of this process. Under the link you provided there's only one sentence about creating a new plugin, with a link to the cookiecutter template. That documentation does not provide a description of the plugin installation process. The following section describing contributions to existing projects uses the invoke setup command, but it was not clear how to modify the setup script or if the cookiecutter template provided everything for new projects.

Your description about installing in editable mode is new to me, I'll have to look into this more as well as the poetry install process. These are the kinds of pointers I think would be good to have as part of the plugin development documentation. Obviously there are many talented developers who figured this out before me without needing it spelled out, so I appreciate any help to get up to speed.

[justinmayer]

Happy to help. As noted above, the canonical approach for now is to use invoke setup, which in turn runs poetry install. Outside of a Poetry-managed process, installing a Python project in the current directory in editable mode is normally done via: pip install -e .

What other questions do you have?

[pkt-nspktr]

It's been a few days and I've been working through the process of installing my plugin so I can start fixing it. But first I had to get the plugin to actually install. I've kept an ongoing write up of my experience of this process which I'll publish on my blog at a future date, but I'll leave my list of recommendations for Pelican's documentation. When the Pelican developer's feel this issue is addressed, they can close it. If I encounter development issues specifically with my plugin, I'll open up a new issue. Thank you again for your assistance, @justinmayer.

Recommendations for Pelican documentation

As I encounter issues that I feel should be addressed in the Pelican documentation, I'll update this section. As of the writing of this post, Pelican 4.8.0 is the most current version and I'm using Python 3.9. I hope future releases of Pelican will incorporate these changes and make this post obsolete.

1. Specify that creation of new plugins requires installing the Pelican development environment tools/dependencies
2. Add the following programs and packages to the dependency list for development of plugins as well as Pelican proper:
   • git
   • invoke
   • poetry, listed as optional under Setting up the development environment, but apparently is required?
   • bash
   • pre-commit
3. Provide a description of the configuration questions and files generated by the cookiecutter command optionally within the Pelican documentation, but certainly within the cookiecutter-pelican-plugin documentation.
4. Update the Namespace plugin structure documentation to reflect the actual output of the cookiecutter directory structure and files. As of the 4.8.0 documentation, cookiecutter does not create a "setup.py" file and the documentation does not provide any examples or references to what needs to be included in setup.py.
5. Specify that to install the plugin locally for troubleshooting, the directory must be turned into a git repository first, according to the cookiecutter templates installed.
6. pelican-plugins can be used during development and testing of plugins for plugins installed via invoke setup as well as via pip.

[justinmayer]

Thanks for writing down your notes. Some comments:

1. Specify that creation of new plugins requires installing the Pelican development environment tools/dependencies

I imagine that a link from the Plugins section to the Plugin Development section should suffice. I don't believe such a link currently exists.

2. Add the following programs and packages to the dependency list for development of plugins as well as Pelican proper:

First, the list of development dependencies is currently maintained inside pyproject.toml. This is true for both Pelican and its plugins. Links to those files could be added to the documentation, but I don't want to maintain separate lists of dependencies that will eventually become out-of-sync and need constant updating.

Second, I don't believe any of those items are actually necessary to develop a Pelican plugin. They are useful, and that's why they are mentioned in the development docs, but it should be possible to do plugin development without any of those tools installed. Git should only be required if one wants to make commits, Invoke merely automates a bunch of commands that could otherwise be executed manually, someone (such as I) might use Fish instead of Bash, etc. While Poetry would be needed for folks who want to locally build Python packages, even Poetry is not strictly required since the CI system handles building/publishing, and installation can alternatively be performed via: pip install -e ./

In short, the development documentation describes, in detail, one possible workflow for development that can be used, but that is not the only possible workflow.

3. Provide a description of the configuration questions and files generated by the cookiecutter command optionally within the Pelican documentation, but certainly within the cookiecutter-pelican-plugin documentation.

I can see how the latter would be useful, and a pull request to the cookiecutter-pelican-plugin repo to that effect would be welcome.

4. Update the Namespace plugin structure documentation to reflect the actual output of the cookiecutter directory structure and files. As of the 4.8.0 documentation, cookiecutter does not create a "setup.py" file and the documentation does not provide any examples or references to what needs to be included in setup.py.

This is intentional — setup.py is not required. As per PEP 517, pyproject.toml files can now take the place of setup.py, rendering the latter unnecessary when using non-setuptools build systems. In this case, we assume the use of Poetry as that build system.

5. Specify that to install the plugin locally for troubleshooting, the directory must be turned into a git repository first, according to the cookiecutter templates installed.

I don't understand this. It's not clear why initializing the plugin root into a Git repository is required. Could you elaborate?

6. pelican-plugins can be used during development and testing of plugins for plugins installed via invoke setup as well as via pip.

I'm not sure how the installation method (e.g., pip install, poetry install, etc.) relates to this. The pelican-plugins command looks for installed plugins, regardless of installation method.

[pkt-nspktr]

Thank you for the comprehensive responses -- clearly there is a wealth of Python knowledge assumed of potential developers within the documentation, which in my case is lacking. I'll look at the references you've provided to expand my own knowledge scope. I'll see if I can get my full writeup published this weekend where those interested can see the steps and errors I encountered trying to get my plugin to just install which lead to this list of recommendations.

Regarding the pull request to the cookiecutter-pelican-plugin repo; I've not done one before, but I'll attempt to submit one in the next few days.

Regarding setup.py vs. pyproject.toml, for a new developer learning as I go, the differences between these approaches are unknown to me and as I read the Pelican documentation, I expected to read about the recommended (or canonical way) of creating a plugin to extend the functionality of Pelican. After spending several evenings working through the recommended way of creating a new plugin -- via cookiecutter templates --, the mismatch between the documentation and reality was frustrating. I can see where a more experienced developer could choose to bypass the cookiecutter template files, but that is beyond my own capabilities at this point.

After I got through installing all the dependencies, running invoke setup resulted in the following error message:

Installing the current project: pelican-csv-reader (0.0.0)
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at /usr/home/pi/.cache/pre-commit/pre-commit.log

I did have git installed, but the plugin directory was not a git repository. Once I performed a git init; git add; git commit and re-ran invoke setup, the setup/install process completed successfully:

% invoke setup
Requirement already satisfied: pip in /usr/home/pi/virtualenvs/pelican/lib/python3.9/site-packages (22.2.2)
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: pelican-csv-reader (0.0.0)
pre-commit installed at .git/hooks/pre-commit

Development environment should now be set up and ready!
%

This was the background of the comment that the plugin needed to be a git repository. Again, I used the default responses to the cookiecutter template prompts, so this may have been a result of the the template providing a git repository, but I don't know for sure. I certainly don't know enough at this point which configuration options to change to avoid having to create a git repository to get invoke setup to complete successfully.

Finally, the first paragraph under the How to use plugins says (my emphasis):

Starting with version 4.5, Pelican moved to a new plugin structure utilizing namespace packages that can be easily installed via Pip. Plugins supporting this structure will install under the namespace package pelican.plugins and can be automatically discovered by Pelican. To see a list of Pip-installed namespace plugins that are active in your environment, run:

pelican-plugins

The documentation calls out "Pip-installed namespace plugins" specifically, which to my inexperienced eyes, appears that there is a difference between "pip-installed" (production-ready and publicly published) plugins and invoke setup (in development and not published) plugins. By the time I got to this point, I realized the documentation was not as comprehensive as I had assumed so tried pelican-plugins and found that it did actually see the invoke setup installed plugin. If pip uses invoke setup internally, that would be news to me as I have not reached the step of attempting to publish the plugin.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your participation and understanding.