Extracting unformatted fences just as SuperFences would

[StevenMaude]

Hi, thanks for the extension; it's used in a couple of projects I help maintain 👋

I'm using SuperFences and have a slightly related question to #1291. Is there a suggested way to extract the source form of fences via the same parsing as SuperFences?

My use case is reading Markdown source files, extracting what would be permitted SuperFences code blocks from that source, and generating tests from those blocks for pytest. ¹

That is, given a simplified Markdown document such as follows:

This is some text.

```lang1
codeblock1
```

This is some more text.

> ```{.lang2 .extra-class linenums="1"}
  codeblock 2
  ``` 

I would like to be able to access the content of the code in the Markdown source — codeblock1 and codeblock2 — with their languages — lang1 and lang2 respectively.

It's the first time I've really had to look at the pymdown-extensions or Python-Markdown source. I've spent a couple of hours trying things with the SuperFences source, and looking at the Python Markdown API documentation. But I'm having trouble getting anywhere, and not sure if I'm trying to do something that isn't easily possible in a hack-free, maintainable way.

Do I have to resort to writing my own code entirely, reimplementing the parsing of SuperFences to extract the code blocks? Or is there a simple approach I'm missing due to my naivety here?

Footnotes

1. I've looked at several existing packages. Those I've seen are either MkDocs plugins, which is not what I need. Or they are pytest plugins, which would probably work. Except the existing pytest plugins aren't configurable enough, and, the reason for my question, only match against simple fences with backticks at the start of the line: mktestdocs, pytest-codeblocks, pytest-examples. ↩

[facelessuser]

You can access the content with custom fences which we document: https://facelessuser.github.io/pymdown-extensions/extensions/superfences/?h=custom+fe#custom-fences.

You can do whatever you want with them. We use them to do some things like code blocks that we execute the content of: https://facelessuser.github.io/coloraide/

We also have an example where we take the content, and parse it in a Markdown sandbox: https://github.com/facelessuser/pymdown-extensions/blob/main/tools/pymdownx_md_render.py

[StevenMaude]

@facelessuser: thanks for the quick reply!

I'd seen custom fences and wasn't sure if that was the route or not in this case, so it saves me looking.

One quick follow-up question: I'm guessing the originating line numbers — the start and end of the block in the source file — aren't easily available to custom fences? This is from a quick look at superfences.py again. (I want to annotate my generated code block tests with the originating source line numbers.)

[facelessuser]

Unfortunately, that is not available. It should be noted that we try to handle fenced code blocks early, but it is very possible that some other preprocessor could alter the content before us as there are a number of preprocessor extensions out there, even some that we include (Snippets as an example). In short, line numbers could quite quickly become inaccurate. You'd probably have an easier time if you were tagging the blocks with unique identifiers. Granted, it sounds like you are trying to do some pretty "advanced" things.

[StevenMaude]

Thanks for the full explanation 👍

Thanks also for your compliment, but not sure my code's particularly "advanced" 😉

I'm searching for code blocks in documentation source and then using them to generate tests with pytest. It's using the naive "look at start of line for backticks" approach which some of the plugins I mentioned do. That was a starting point and I only later realised that SuperFences has its own rules for matching blocks. (And that Python Markdown allows tildes for fences.)

Anyway, appreciate your fast responses and the work on your code 😎