Separate extensions from core

[waylan]

From time-to-time we have mentioned possibly separating the built-in extensions from the core. This would involve breaking each of the 18 existing extensions out into a separate repo and Python package. This has a few advantages.

1. Development of core would not hold up release of an update to an extension or visa versa.
2. Issue tracking would be more focused as each repo would have its own issue tracker.
3. It would provide a clearer distinction between extensions and core behavior, which would (hopefully) reduce requests for additional extensions to be added to the core.
4. Some existing extensions could potentially be clearly marked as being in "maintenance mode" without any effect on the core or other extensions. For example, an extension could be clearly marked as only receiving updates to continue working with future versions, but to not receive any new feature additions. Or an extension could be marked end-of-life, which means no updates at all going forward. But it could continue to be used so long as it supports the dependencies the user has installed.

I propose that each extension and its related tests would be moved to a new repo (under the Python-Markdown org) and be distributed as a separate Python package. Presumably, each extension would be packaged as a namespace package, Therefore they would still be importable from the same path (markdown.extensions.{extension_name}).

There is one major issue that still needs addressing, which is what has stopped us from moving forward with this thus far. Specifically, most existing users expect to get the existing extensions when installing markdown. Therefore, we still need to ensure each separate package gets installed with it. There are a few different ways we could approach this.

Option 1 (the namespace method)

Keep the core functionality in the existing Python-Markdown/markdown repo and list all 18 of the extensions as dependencies which get installed as namespace packages. There would be no way to install the core without all 18 extensions.

This option would be simple to implement, and would not even be noticed by most users. However, it might be confusing for a user who was inclined to report an issue or make a feature request. They would have to track down the repo for the relevant package that they had not explicitly installed .

On the other hand, releases would be easy here. Any updates to core or an extension could just be released. Individual extensions would simply indicate the earliest version of markdown that they support, which could differ from extension to extension over time. It might be worth exploring if a future namespace package would override/replace a submodule of the existing (non-separated) version of markdown. We might need to limit to specific versions in an extension's dependencies to avoid weirdness here.

Option 2 (the core method)

Move the core functionality and related tests to a new repo Python-Markdown/core. The markdown-core distribution package would install a markdown package and each extension would install a namespace package. The existing Python-Markdown/markdown repo would no longer contain any code, but just be a distribution package (markdown) which lists the core and all 18 extensions as dependencies.

This would provide the additional option to users of installing the core only (via markdown-core) or the core and only the specific extensions they need/use.

Conceivably, the markdown wrapper package would not need to be updated to get updates to core, which could result in the versions becoming out-of-sync. We could lock the versions between markdown and markdown-core, but then every time we do a release for markdown-core we would also need to release a new version of markdown. Presumably each extensions should list a minimum supported version of markdown-core.

Option 3 (the optional-dependency method)

We could implement either option 1 or 2 above with one change. Specifically we could use optional dependencies for the 18 extensions. Then a user could install markdown[extension_name] to get a specific extension. The issue with this is that all standard dependencies always get installed with or without any optional dependencies. Therefore, none of the extensions could be listed as standard dependencies. And that means that installing the markdown package (without any option dependencies indicated) would not get any extensions. Users would need to use markdown[all] to get the same behavior they have now with markdown. Even though this is an attractive option (and what I would likely prefer if we were starting out new), it would be a breaking change and so I don't recommend it. I only included it for completeness.