Support latest realise of Markdown library

[Samreay]

I believe there has been some update to the Markdown library and how it internally records its version that is breaking things.

With a brand new environment and a fresh install of mkdocs, a mkdocs build --strict --verbose fails my project with this error:

DEBUG    -  Loading configuration file: /Users/sh/Projects/dataportalapiclient/mkdocs.yml
ERROR    -  Config value: 'markdown_extensions'. Error: module 'markdown' has no attribute 'version_info'

At this point, mkdocs has a dependency on Markdown==3.4.1, which was released three days ago.

After running a pip install Markdown==3.3.7 to downgrade the version, rerunning the build is successful:

DEBUG    -  Loading configuration file: /Users/sh/Projects/dataportalapiclient/mkdocs.yml
...
DEBUG    -  mkdocstrings: Tearing handlers down
INFO     -  Documentation built in 3.45 seconds

I notice in this commit from May 27th on the Markdown repository, the deprecated version_info info object was removed, and replaced with the __version_info__ object, as per this table:

Deprecated Object	Replacement Object
markdown.version	markdown.__version__
markdown.version_info	markdown.__version_info__
markdown.util.etree	xml.etree.ElementTree
markdown.util.string_type	str
markdown.util.text_type	str
markdown.util.int2str	chr
markdown.util.iterrange	range
markdown.util.isBlockLevel	markdown.Markdown.is_block_level
markdown.util.Processor().markdown	markdown.util.Processor().md
markdown.util.Registry().__setitem__	markdown.util.Registry().register
markdown.util.Registry().__delitem__	markdown.util.Registry().deregister
markdown.util.Registry().add	markdown.util.Registry().register

Hopefully the fix is a simple change to this dunder object! Whether this repository is the right place for the packaged markdown extension or not, I'm unsure, I couldn't quite see where that config gets run either here or in the Python Markdown library.

If this isn't the place, I'd appreciate if you can please point me towards the right repo.

[lyz-code]

I'm also facing errors with this update, when I try to run mkdocs build I get the next error:

ERROR    -  Config value: 'markdown_extensions'. Error: extendMarkdown() missing 1 required positional
            argument: 'md_globals'

Rolling back works as expected

[ultrabug]

@oprypin maybe we could first limit the Markdown version in a "hotfix" release?

See #2893

Else, we can also work on directly updating mkdocs to support newer Markdown itself of course. Thoughts?

[oprypin]

Hi. I could still be missing something, but my first point is, I am detecting absolutely no issue with MkDocs. Tests pass, random websites that I had laying around build without problems. That's one basic check that I'd hope to see. And another one is that if there's no mention of markdown.version_info and no mention of md_globals (also as a parameter) in the whole repository, then there's no reasonable way to make a blanket bug report on this repository claiming that everything is broken and that something should done about it (about what exactly?).

That is, this would require at least the list of plugins that you're using (mkdocs.yml) to be able to conclude anything. Or better, a full minimal website source that causes this error.

I am only now able to attend to this, but all I can do is look through every Python-Markdown plugin (possibly all old versions, too, because who knows..?) to guess which ones it might have been that caused these errors. (So far I found one but the error looks differently still.) But let's just step back and go to the previous paragraph.

Actually if anyone is running into such an issue, for all such cases I'd like to figure out which plugin is causing that for them. For all we know, it could be just 1 or 2 plugins in common use that should just be fixed.

Speaking of which, that is actually my main point - I do believe that it's some Markdown plugins that are breaking with the version update, not MkDocs breaking, at all. Sure, the error report comes from MkDocs and in a way that conceals which plugin is at fault. But in the end, I'll still want to see the list of those plugins to make a decision.

[oprypin]

And yes indeed, it is the Python-Markdown release 3.4 that causes all these breakages of extensions. I do believe that the best course of action would be for them to backtrack on the breakages (the PR itself doesn't even sound certain that the breakages were OK to do), but I have no influence there.

As a workaround (which I admit would immediately help people), to punt this issue some time into the future (where we will presumably be back at square one), we could indeed pin the version within MkDocs, even if that is technically clearly wrong, as MkDocs has no such limitation. And pinning deps like that is in itself a ticking time bomb. So I am not so sure about this.

What we will have to do in any case, it seems, is to track down the Python-Markdown extensions that break with this change in Python-Markdown. Concealing this issue now (as in the previous paragraph) might hinder the progress of that.

[oprypin]

And I did just mention that MkDocs does poorly here in how it reports these breakages. Absolutely all information about where this breakage comes from is concealed. Actually a big part of the "concealment" already happens in Python-Markdown itself but then MkDocs doesn't even keep the name of the extension that broke (I still haven't dug into where this happens).
This is absolutely a high priority to improve as well. I might provide a patch soon so people can at least see which extension breaks.

[ojacques]

That is, this would require at least the list of plugins that you're using (mkdocs.yml) to be able to conclude anything. Or better, a full minimal website source that causes this error.

My mkdocs sites, with extensions, broke also since the introduction of markdown-3.4.
I am able to reproduce with mdx-truly-sane-lists or mkdocs-featherlight.

This example mkdocs.yml is enough to reproduce (starting from mkdocs new .):

site_name: My Docs
markdown_extensions:
  - mdx_truly_sane_lists

The error I get for both plugin:

$ mkdocs build
ERROR    -  Config value: 'markdown_extensions'. Error: Failed loading extension "mdx-truly-sane-lists".
Aborted with 1 Configuration Errors!

Not the same as the version info from @Samreay, but I can link it to markown extension.

I also used the <3.4.0 version constraint in the meantime as a workaround.

[ultrabug]

Hi. I could still be missing something, but my first point is, I am detecting absolutely no issue with MkDocs.

I agree with you on the strict sense.
But this can become arguable in a more general sense where people and projects rely on the mkdocs requirement and thus on its listed dependencies, which is now clearly broken by Python-Markdown.

So yes, Python-Markdown broke us but I have the feeling that we ought to preserve our community from a cascade of potential failures now that we're aware of the situation.

Now, unless I'm mistaken I also would like to point out that MkDocs itself is affected by this problem when building our own mkdocs.yml website:

(fresh_virtualenv) ➜  mkdocs git:(master) mkdocs build                
ERROR    -  Config value: 'markdown_extensions'. Error: Failed loading extension "mdx_gh_links".

[ultrabug]

And I did just mention that MkDocs does poorly here in how it reports these breakages.
This is absolutely a high priority to improve as well.

I agree with you and if you feel like it, ping me on IRC so we can maybe join forces to tackle this at the same time.

[oprypin]

@ultrabug Ah at that point I was already finishing it up. That would be #2894

[oprypin]

For anyone affected by extension breakages, please install this patch (with the below command) and provide the output of MkDocs with it.

pip install git+https://github.com/mkdocs/mkdocs.git@refs/pull/2894/head

This output will also basically be ready as a bug report for whichever extension is affected, that you certainly should report in the respective repositories in any case.

[oprypin]

@ojacques The message indeed hasn't changed, that shouldn't have happened if this actually installed properly
For the venv you'll probably need

venv/bin/pip install -U git+https://github.com/mkdocs/mkdocs.git@refs/pull/2894/head

[ojacques]

venv/bin/pip install -U git+https://github.com/mkdocs/mkdocs.git@refs/pull/2894/head

Indeed:

$ mkdocs build
ERROR    -  Config value: 'markdown_extensions'. Error: Failed to load extension 'mdx-truly-sane-lists'.
           ModuleNotFoundError: No module named 'mdx-truly-sane-lists'

As you are tracking a list of extensions, you can add featherlight to the list of affected extensions.

[oprypin]

Indeed I started a list of known broken extensions. There, you can see whether your issue is a known issue.

• Tracking list: external extensions broken by release 3.4 Python-Markdown/markdown#1277

Please refrain from replying there, you can continue to reply here instead. See #2892 (comment)

[oprypin]

Even with those^ numerous breakages, I think:

• We will definitely need to merge this
  When a Markdown extension fails to load, print its name and traceback #2894
  and make a release soon - that's my main plan.
  Then people will see what's broken and will be able to report a bug.

• I still prefer to not merge
  requirements: fix Markdown version to >=3.2.1 and <3.4 #2893
  but I could still be convinced and we could still make a followup release.

@ultrabug what do you think?

[Samreay]

#2894 is super useful, I now know what project to hit up:

ERROR    -  Config value: 'markdown_extensions'. Error: Failed to load extension 'mdx_include'.
              File "/Users/sh/miniconda3/envs/dataportal/lib/python3.9/site-packages/mdx_include/__init__.py", line 4, in <module>
                from .mdx_include import makeExtension
              File "/Users/sh/miniconda3/envs/dataportal/lib/python3.9/site-packages/mdx_include/mdx_include.py", line 42, in <module>
                MARKDOWN_MAJOR = markdown.version_info[0]

[chancsy]

Another broken external extension: mkdcomments
mkdocs' build error message:

ERROR - Config value: 'markdown_extensions'. Error: CommentsExtension.extendMarkdown() missing 1 required positional argument: 'md_globals' Aborted with 1 Configuration Errors!

Issue raised #7.

[ultrabug]

@oprypin my general feeling is that I put our community contract before technical righteousness. MkDocs' community contract -to me at least- is that it is simple to setup and easy to use.

Those known breakages will affect our community, having a tracking list will help for those who have the skills/habit of searching for their errors, which I don't believe represents the MkDocs community as a whole. I'd bet most of our users just use MkDocs because it just works.

I'm not here to start a flame war or to judge who's right about how they approach a technology, I'm just trying to say that at some point in time we got aware of something that could affect a lot of people and possibly wouldn't release a fix to keep things simple.

On a more selfish side, I'd bet we'll get a constant flow of issues about this from now on and I'd bet that even a better traceback error message won't help us getting those reports (and yes most of our potential new users won't even report the problem, it will fail and they will move somewhere else).

• As such I argued that we could add a better message for people instead of a purely technical traceback
  When a Markdown extension fails to load, print its name and traceback #2894

• requirements: fix Markdown version to >=3.2.1 and <3.4 #2893
  If we do not go for this one I propose to do another PR that looks for the Python-Markdown version and report even better the problem when loading extensions if 3.4+.
  Something like:
  Python-Markdown version 3.4.xx is known to break some extensions, please check [1] to see if your extension is listed or report the problem described below to the appropriate extension project.

My whole point is that MkDocs should remain robust, helpful and easy for everyone.

[oprypin]

@ultrabug Again, the other big issue in my view is that pinning the version is just punting the problem into the future. Then, in most cases I don't see any reason why extensions would even know to release an updated version. That means that some time in the future I'll have to arbitrarily decide "well, it's time to break stuff again, can't keep it pinned anymore". How do you see that happening?

[ultrabug]

pinning the version is just punting the problem into the future

I totally agree, but...

some time in the future

From users' perspective, maybe at least that's not right now

arbitrarily decide "well, it's time to break stuff again, can't keep it pinned anymore"

So we'll have time to warn people beforehand by updating the docs maybe, discuss this further with the community

I don't see any reason why extensions would even know to release an updated version

That's why I proposed to add a warning message with hints when things go wrong with extensions, how do you feel about that @oprypin ?

[oprypin]

Yes you convinced me.
Indeed, otherwise to non-technical users this will always just look like MkDocs is broken.
Let's pin the version. I'll release soon (in ~minutes).

[oprypin]

And regarding further improving the warning messages, it's really not that simple. I replied on #2894 (comment)

[lyz-code]

Thank you @oprypin and @ultrabug for maintaining the package, keeping and open mind and taking care of us :)

[oprypin]

As things currently stand, I plan to keep the "<3.4" version pinned for up to a year. That version has minor features that seem useful only for usages from code and also improves extensions that people almost never use. So it's just a worse version pretty much.

A big part of the problem is that warnings that were supposed to be there, were never actually shown. So I want to make them shown.

• Show DeprecationWarnings as INFO messages #2907

Making users aware of breakages in advance will help by making it not a surprise when things break.
That will be the main difference when we eventually un-pin the version. (addressing my prior comment:)

in the future I'll have to arbitrarily decide "well, it's time to break stuff again, can't keep it pinned anymore".

I think that maybe that's enough.

[ultrabug]

I agree and will review #2907.

[Egorand]

As things currently stand, I plan to keep the "<3.4" version pinned for up to a year.

Any plans to review the version pin? We often encounter transitive dependency conflicts, e.g. mkdocs-material just bumped markdown to "~=3.4", breaking compatibility: squidfunk/mkdocs-material#5970.

[pawamoy]

The pin was removed: 562d5e1. The change is part of 1.5.0+

[Egorand]

@pawamoy true, sorry, looks like we also pinned the version in our own project, which was causing issues.