Allow custom tags to be parsed as raw content like the behavior of fence

[66Leo66]

The builtin node "fence" has no children and its content is stored in attributes.content without and process.

Such behaviour would be useful for creating custom nodes, especially for rendering other syntax, e.g. LaTeX math, Mermaid diagram ...

Currently, tag children is always processed, making the content unsuitable for above cases;

Workaround is to customize transform of fence node as in this discussion and example below, but it would add complexity handling normal codeblock and special renderers.

if (lang.startsWith("math-")) {
  const markup = convertLatexToMarkup(attributes.content as string, {
    mathstyle: lang.split("-")[1] === "inline" ? "textstyle" : "displaystyle",
  });
  return unescapeHTML(markup) as any;
}

Workaround:
```math-inline
x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}
```

Wanted (currently, if there are parts matching markdoc syntax it will be parsed by ast):
{% math %}
x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}
{% /math %}

[rpaul-stripe]

Our perspective on this is that the fence is the standard CommonMark way to indicate unprocessed content in a Markdown document, so we're reluctant to support literal text in any arbitrary tag as it would create a lot of interoperability challenges with other parts of the Markdown ecosystem.

We also really prefer to avoid having any schema dependencies inside of the parser, we want to be able to do the parse step without having any awareness of what tags the user has defined, and it would be hard to do that if we made it so that certain tags accepted literal bodies.

Workaround is to customize transform of fence node as in #411 and example below, but it would add complexity handling normal codeblock and special renderers.

Yes, I agree that the current steps that you have to take to implement a custom fence processor are cumbersome, add unnecessary complexity, and have very poor composability. I think it'd make sense to have a better extensibility model around this — a dedicated way to register individual fenced block types without having to put a bunch of conditional statements in a single fence transform function.

We could perhaps come up with a hybrid approach where we make fences behave more like tags while remaining aligned with the CommonMark syntax. Maybe instead of just allowing annotations in the fence info line, we allow it to be a named tag, in which case that tag gets the literal content. e.g.:

```latex  {% math %}
x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}
```

And then you could register a math tag instead of having to handle this as a special case in a fence transform.

We'd need to figure out what the behavior is around processing markdoc tags within the fence. Right now, the literal text still gets parsed for Markdoc tags unless the fence is annotated with process=false, but I'm not totally sure if that's the behavior we'd want for tagged fence block.

[66Leo66]

I like your idea and demo syntax! Btw, what's the current behaviour if fence content get parsed? Does that tag appear as a child of fence? Parsing within fences appears confusing to me.

[rpaul-stripe]

In a fenced code block AST node, the full literal string content goes in attributes.content and the children array is populated with the parsed body. The parsing (you can see how the tokenization is handled here in the code) scans forward in the text looking for a tag, and when it finds one it parses it as a Markdoc tag node and then turns the preceding plain text into a text node (an AST node where type is text and the actual text is stored in attributes.content).

So, for the following example:

```
this is a {% foo %}tag{% /foo %} in a fence
```

You get roughly the following AST:

• document
  • fence
    • text: "this is a "
    • tag: foo
      • text: "tag"
    • text: " in a fence"