Penguin-flavoured markdown

[pngwn]

In #259 I alluded to other changes in v1, besides the configuration API. The biggest one is something that has been one of my biggest frustration in some regards: markdown itself. Now, lets get this out of the way: markdown is excellent and easily the best document markup format we have ever had. Kudos to Mr Gruber and all who have pushed markdown forward. Nothing in this document should be considered a derogation of markdown, I have nothing but respect and admiration for everything that has brought us to this point. This is a criticism, in many ways, but hopefully a constructive one. It is also just the opinion of a single penguin, and a penguin of questionable character, at that.

---

Penguin-flavoured markdown

Link to easier to navigate gist.

• Context
• Principles
• The differences
  • Indentation
  • Link references
  • Headings
  • Lists
  • Emphasis
  • Line breaks
  • Superscript
  • Subscript
  • Strikethrough
  • Tables
  • Generic directives

Context

This document is an early suggestion of what markdown in mdsvex might look like (it will also exist as a standalone markdown flavour—yes, another one), but it takes a slightly different approach, removing features, as well as adding them. I still feel as though it can be considered 'markdown' in spirit (a 'suberset' that shares most of the syntax and semantics), although technically that probably isn't true.

Over the past few months I have been doing a lot of reading and research around markdown, the various flavours, and the history. I have read the CommonMark spec more times that I wish to think about, I have thousands of words of notes. I have also looked very deeply into various markdown flavours (GitHub Flavoured Markdown, Markdown Extra, R Markdown, MultiMarkdown) (and others), as well as other document markup formats (reStructured text, AsciiDoc(tor), textile, text2tags, org mode, setext). Some of these formats are almost 3 decades old, and at this stage I have learned more than I ever wished to know about human-readable document markup formats and their history, but honestly learning about the history and evolution of this ecosystem has been very rewarding and I bow down to all of those who have played a part; it is thanks to them that we have such a rich set of tools and such a vibrant ecosystem.

This document uses CommonMark as the baseline (not Gruber's original 'spec'), all markdown variants are considered just that. Many of the older formats would be too much of a break from what folk are used to today but learning about them was interesting and provided useful context.

This document will detail a list of differences from markdown. The aim is to remove ambiguity (by removing, restricting or modifying syntax) and add some essential features out of box (via a few syntax additions).

On reducing ambiguity, I have to give all of the credit to John MacFarlane, he is smarter than I am and I have stolen his ideas, mainly from here

Principles

Some over-arching principle. Things that I believe to be true. Some will disagree but these are some of the key guiding principles that guide what I'm about to suggest. There are always exceptions but i generally find these things to hold true.

1. Markdown is a learned language. It is not 'intuitive' or 'natural'; this is a myth. Tweaking it will not compromise its integrity.
2. Ambiguity is confusing for users and complex for machines.
3. Invisible syntax is confusing (I don't consider line breaks invisible).
4. Having multiple ways to achieve the same thing make for confusing design.
5. Markdown should solve the 80% case without extension.
6. Markdown is not a 'compile to HTML' language. It is a way of adding simple documents semantics to a text file, the compilation target varies. However, this is the most common usecase, and must be considered.
7. Where possible things should render well enough when fed into a standard parser. An odd constraint for such a proposal but a practical one. I do not care about tooling much but I do care about GitHub/ GitLab rendering a bit.

Indentation

Indentation ins markdown creates code blocks or literal/ verbatim text. This is naturally problematic for mdsvex (when mixing HTML and markdown you may want to indent for clarity) but it also introduces confusing rules around when certain markdown syntax will render a code block and when it will do nothing. Indented code blocks arguably violate the invisible syntax principle but definitely violate the 'more than one way to do things' principle because fenced code blocks exist.

Fenced code blocks exist and are generally more useful. They allow you to include a variety of useful metadata.

PFM will remove the significance from indentation.

Link references

In markdown you can do this, it is quite explicit and very useful:

[link text][link_ref]

[link_ref]: google.com

But you can also do this:

[link_text_and_ref]

[link_text_and_ref]: google.come

The problem is, we don't know if the second one is a shorthand link until we discover the link reference definition. This is complex for parsers but I also think it is confusing for humans, there is a degree of context switching that is required by jumping around a document to search for the possible link reference definition. If no definition is found then this is actually just a normal paragraph: <p>[link_text_and_ref]</p>.

PFM will require shorthand link references to be explicit, the shape of the syntax alone will dictate its nature:

[link_text_and_ref][]

[link_text_and_ref]: google.come

Should this be an error if no reference is found? This would be a dramatic break for markdown, markdown does not have parser errors.

Headings

These are all valid headings:

hello one
===

hello two
---

### hello three ###

#### hello four

The first two are setext style headings and are confusing especially when they are near to horizontal rules, the others are atx style. These styles are named after the language they were borrowed from. All but one of these will be removed.

Leading octothorpes will distinguish headings, nothing else will be supported, trailing octothorpes will be considered part of the heading text:

# one

## two

Lists

Lists with strange numbers will be supported because it seems sensible, the auto-heading functionality of markdown is very confusing:

11. eleven
27. twenty-seven

Markdown has a concept of loose and tight lists as well as strange rules around nested lists, I haven't quite got my head around these yet. They are useful features but i'm still confused. I'll update when I have thought it through properly.

Emphasis

There is no single feature in markdown that is as confusing as emphasis. The markdown spec as 17 rules for emphasis, each more confusing than the last. That is seventeen. This ambiguity is mainly due to the fact that there are many wasy to emphasise and strongly emphasise text. You can read more here

What is this:

**hello* world**

or this:

hello***a*friendsss

(I'm not sure GitHub is rendering these in accordance with the commonmark spec).

As you can see, it gets confusing fast. Intraword emphasis is even more confusing.

PFM will have distinct syntax for emphasis and strong emphasis. This is a hard break from the spec but a necessary one. Intraword emphasis is very rare but needs to be supported. Distinct syntax will be used for that case.

_emphasis_

*strong emphasis*

*_emphasis inside strong emphasis_*

_*strong emphasis inside emphasis*_

intraword emphasis is fan~_tas_~tic

Line breaks

Line breaks come in a few forms, hard and soft.

Lets do a small experiment, should these two paragraphs render the same?

hello
friends

hello  
friends

They don't:

<p>hello
friends</p>

<p>hello<br>
friends</p>

The second paragraph actually had two space characters after the work "hello" this made it a hard line break. The first is a 'soft' line break (it basically renders in HTML as if there is no line break), the second is a 'hard' line break and will render a br for times when line breaks add semantic meaning (poems, addresses).

I personally find invisible syntax quite hard to see. Markdown has an explicit version too:

hello
friends

hello\
friends

PFM will support both hard and soft breaks, hard breaks will only be supported via the explicit backslash syntax.

Superscript

Superscript is not part of the markdown spec, PFM will support it via a pair of ^ characters (rules TBD):

Coming Soon ^TM^

Will render:

<p>Coming Soon <sup>TM</sup></p>

Coming Soon ^TM

Subscript

Superscript is not part of the markdown spec, PFM will support it via a pair of ^ characters (rules TBD):

x~1~ ... x~n~

Will render:

<p>x <sub>1</sub> ... z<sub>n</sub></p>

x ₁ ... z_n

Strikethrough

Strikethrough is not supported by markdown but I'd like to support them in PFM. They will match common syntax for strikethrough ~~word~~. Undecided on intraword strikethrough but it is simpler than emphasis.

Some options:

~~strikethrough~~

This is very much TBD, would be nice but very much a nice to have.

Tables

Here we go. I think tables are a critical feature and i haven't really worked this one out yet. It is very complex and there are a million edge cases. The basis should be GFM pipe tables but I'd like the ability to have headings on left, in addition to or instead of across the top, as well as the ability to merge cells (at least horizontally but probably both). I was thinking something like this:

| Header  | Header | Header           |
| --------|--------|------------------|
| Header || Cell   | Cell             |       
| Header || Cell spanning two columns |

This looks weird and will almost definitely not work but it is a start. It may be better to have distinct syntax for simple pip tables (GFM style) and more complex tables with more explicit syntax, this does violate a principle though. Needs more thought.

Generic directives

This one is a bit left of field, but I think it is very important. Markdown needs a way for users to 'plugin' with a generic syntax that covers most usecases. This essentially brings wordpress/ hugo style 'shortcodes' into the spec, except it is more flexible and more powerful. This feature has been co-opted from here.

It is powerful and flexible due the different forms.

Inline:

:name[content]

This is a very great article on :wikipedia[penguins]

Would be accompanied by a function. API TBD, realistically this would be a custom AST compile handler (becaue this would have its own AST node type with the necessary info in it) rather than the below but this is illustrative:

const config = {
  directives: {
    inline: {
      penguin(content){
        const { url, title } = look_up_wikipedia(content);
        return `<a href="${url}">${title}</a>`;
      }
    }
  }
}

Leaf block (block, cannot be inside inline content; leaf, cannot have children):

::name[content]

::youtube[cool video]

Similar function as above to render a youtube video.

Container block (block, cannot be inside inline content content; container, can have children):

:::name[content]
<!-- childf content -->
:::


:::warning[bad shit be happening]
here is some text that will go inside this block
:::

The linked proposal also suggest {key=val} for 'attributes' but { and } are sacred in mdsvex and not viable for markdown syntax. I think the capability to pass props would be nice (::youtube[title]{start=180} for example) but I'm not sure what that syntax should look like.

What is really great about the generic directive proposal is that it can actually replace a lot of other github extensions (it could actually be used for addition, delete, etc. from above), as well as allowing easy customisation. It is very, very powerful, in my opinion. PFM will support it by default.

---

This is where i am up to. Please feed me back!

---

Edit: Replaced 'Additions and deletions' with 'Strikethrough'. Update syntax to use ~~word~~.

[rchrdnsh]

Liking what you got so far…will go through in more detail soon :-)

[swyxio]

nice list! more here https://dev.to/swyx/6-things-markdown-got-wrong-124 eg i think headers should always always slugify and autogenerate their own #anchors.

i still feel like intermixing html in markdown in html is too much effort and unpredictable, i wonder if theres a way to add some rules that would make it easier.

i would love an

vs a

when i use >, perhaps we could introduce a >> variant

[pngwn]

Regarding HTML/Markdown—mdsvex is basically exactly that and will have its own spec on top of this pure markdown one. Both the markdown and svelte (which is basically html+) will all be parsed into an AST, it won't just be a HTML raw tag, the parser will understand HTML semantics as well as markdown. I still need to think a little bit about how the two will interplay but stuff like:

<h1>*hello*</h1>

Will become:

<h1><em>hello</em></h1>

and

<h1>

*hello*

</h1>

Will become:

<h1><p><em>hello</em></p></h1>

This leans on the significance of double new lines in markdown but this:

<div>

<div></div>

</div>

Should not mean the inner div is wrapped in a paragraph for obvious reasons.

[pngwn]

anchors and slugs could easily be generated form that AST. Whether that should be switched on by default, I'm not sure. I was thinking a blog preset could switch things like that on but nothing is decided there. Would all be powered by AST transforms internally regardless of the API and any defaults.

I think formatting swallowed some of your text but the </> issue should be resolvable. Bit of an implementation detail but > is always supportable because we know when we are inside an HTML tag or not, the parser can disambiguate a close element character from a '>` text node value.

< is more difficult because it marks the start of an HTML element. We can disambiguate this by requiring < to be followed by a [a-zA-Z] or / character, so whitespace would not enter a 'start html tag' state and just be a plain text node.

> as the first character after a double newline or at the start of the document is probably a bit more challenging (because it marks a blockquote) but we could probably just escape it per normal markdown parsing rules: \>

[rmunn]

For handling generic directives, I feel that they will need to be implemented as functions taking an AST node and returning an AST node-or-nodes that would be substituted for the original. #52 (comment), for example, could be implemented with something like this:

const config = {
  directives: {
    container_block: {
      sample(content) {
        const codeBlock = new_markdown_block('```', content);
        const rendered = content;
        return [ ...codeBlock, ...rendered ];
      }
    }
  }
}

An empty array would, of course, end up removing the directive from the output (could be used to conditionally include sections based on various factors). And perhaps for convenience, a non-array object would be considered a single AST node.

The drawback to writing them as AST-to-AST transformation functions is that then you have to make the AST API available for library users to import in their config, somehow. Perhaps for convenience, returning a string would count that string as Markdown to be parsed? E.g.:

const config = {
  directives: {
    container_block: {
      sample(content) {
        let mdContent = content.asMarkdownString();
        if (! mdContent.endsWith('\n')) mdContent = `${mdContent}\n`;
        const codeBlock = '```\n' + mdContent + '```\n';
        const rendered = mdContent;
        return `${codeBlock}${rendered}`;
      }
    }
  }
}

Not sure if either of those ideas is a good one; I'm mostly just spitballing here. I am sure, though, that the output of a generic directive needs to be Markdown (whether as an AST or as a string), rather than HTML. Because as you say, Markdown is not exclusively a compile-to-HTML language.

[pngwn]

Another super late reply but it is what it is.

I think this is tricky because what happens if the thing you want to convert your directive into cannot be easily expressed as markdown? In those cases people are likely to jump to HTML (which is technically valid markdown) and that kind of defeats the object.

I think some kind of AST or ast-like structure makes sense because what if someone wants to output a component, they need a way to inject the imports to that component as well. This is very difficult with a string of content.

I don't think the AST needing to be a stable API is too much of an issue though because the plugin system will require that as well.

One possible option would be for directives config to be a two stage step:

const config = {
  directives: {
    directive_name: {
      // takes in the arguments to the directive, and a possible tree of child nodes
      // returns a new AST node
      generate(args, child_content) {
        walk(child_content, (tree, node) => {
          // do stuff
        })
        return node(attributes, children)
      },

      // takes in a node and returns and html string (or possibly an html ast)
      compile_html(node) {
        return <h1>hi</h1> 
      }
    }
  }
}

This way, plugins that wish to run against the AST will still have a full AST to do so without being poisoned by raw strings (which are the words nodes).

[archiewood]

We at Evidence.dev make heavy use of html-style components in markdown that is preprocessed by MDSveX e.g.

# Sales Dashboard

- This data is pulled from Salesforce
- For more details see the chart below

<BarChart 
    data={my_query} 
    title="Sales over time"
    x=month
    y=sales
/>

The full data can be seen here

<DataTable data={my_query}>
   <Column id=month fmt="mmm yyyy"/>
   <Column id=sales fmt=usd/>
</DataTable>

I've personally always thought the html tag syntax is one of the more confusing things we do, so if we could find something more markdown-y that could output to a valid component, I think that would help.

For this to work, we'd need to be able to support:

• passing attributes to components
  • which could be strings, numbers or objects
• passing in components as slots to other components

On attributes, an alternative idea which is floated in the same commonmark thread listed above

# Sales Dashboard

- This data is pulled from Salesforce
- For more details see the chart below

::BarChart[my_query] (   <!--how do we identify this is an object without {}?-->
    title="Sales over time", 
    x=month, 
    y=sales
)

The full data can be seen here

:::DataTable [my_query]
    ::Column (id=month, fmt="mmm yyyy")
    ::Column (id=sales, fmt="mmm yyyy")
:::

• You let the directive accept multiple, comma separated inputs inside () rather than {}
• both [] and () are optional

[ryanatkn]

I like this a lot. Lovely details and directives are exciting. I consider myself a frustrated Markdown user and this addresses most of my complaints. (happy aquatic bird flapping noises)

Any thoughts on contentless directives? (the linked proposal says it "may" be followed with content but there's no example without) Thinking of an emoji replacement usecase, so this:

how much time? :emoji[turtle] :emoji[rabbit]

could hypothetically be shortened to:

how much time? :turtle :rabbit

If not, then this would also work:

how much time? :turtle[] :rabbit[]

For directive attributes, double square brackets seem unsurprising but idk:

:reminder[check the thing][when=tomorrow]

Directives are really intriguing. I'm not sure this is the best usecase, but here's a poll example that comes to mind:

:::poll

What is your preferred stimulant?

- caffeine
- social media
- healthy living
- an exquisite flavour of markdown

:::

Wondering if that would transform into something functionally like this:

<Poll
	prompt="What is your preferred stimulant?"
	options={['caffeine', 'social media', 'healthy living', 'an exquisite flavour of markdown']}
	id="maybe_a_generated_id_or_something"
/>

Makes me wonder about the tradeoffs between directives and components.

[pngwn]

So super late reply but I don't think this direct would compile to a component:

:::poll

What is your preferred stimulant?

- caffeine
- social media
- healthy living
- an exquisite flavour of markdown

:::

I think the main distinction should be static vs dynamic content.

Directives are static build time 'functions'. They are abstractions that can take arguments but those functions resolve at build time and don't depend on dynamic data and can be compiled into a markdown string.

I think an important note here is that it may be valuable at some point to determine whether or not an mdsvex file is entirely static or contains dynamic content, if a component is used then it is automatically 'dynamic' because of the nature of components (there could anything inside a component), whereas 'static' markdown could be rendered to HTML and served much more simply when compared to a svelte component (javascript module).

So in this way i think directive == static, component == dynamic is a good way to think about it because that offers a better optimisation path in the future but there is certainly no real issue with this kind of compilation i guess. Any analysis would be done after plugins and configuration had been applied.

In terms of ergonomics, i think this example of a poll is a good one, it is just much nicer and more 'markdownish' with the generic directive syntax (imo). Even if it just a proxy for a component.

[pngwn]

Regarding the emoji example, I guess there is a possibility of adding a 'default' generic directive (possibly for each type, leaf, block leaf, block) but in the specific case of emojis, I think we could just support them directly.

In general though 'default' directives might be interesting to explore.

[ryanatkn]

The static/dynamic distinction makes sense, and the optimization for static sounds very useful.

I'm guessing directives can return other directives, which recursively get resolved to enable composition?

[pngwn]

I'm guessing directives can return other directives, which recursively get resolved to enable composition?

I think this makes sense conceptually but it could have an impact on performance and the order in which things are applied would need to be taken into account. So I'd need to think about the implementation, but in principle, yes.

[kelvindecosta]

Hey @pngwn , I hope you're doing well!
I am curious if you have any updates on this, thank you!

[pngwn]

hi!

I took a bit of a break from some of my open source work but I'm back again and working on this. I should have some more concrete updates over the coming months!

[kelvinsjk]

Thanks for the well thought out points: many times when I'm struggling with edge behavior working with a particular dialect of markdown I often come back to this post and there is a salient note on my problem here.

And now that John MacFarlane himself has tried to address the issues around markdown with djot, will be interesting to hear if you have any thoughts on potentially using that

[pngwn]

I think host is really interesting and some of John MacFarlane's reflections on common mark and markdown in general have had a significant influence on this proposal.

The goals of this proposal and jdot are pretty well aligned, the only problem is I can't think of a way to resolve curly brace collisions in mdsvex itself. In PFM it wouldn't be an issue but curly braces are reserved in svelte and I don't think there is a great way to disambiguate them as they can appear pretty much anywhere in a markdown document. Jdot uses curly braces pretty extensively, so some of those features are a bit of a non starter.

I'm also not sure I agree with adding arbitrary attributes to any markdown element. I think it compromises readability as it could potentially introduce a lot of noise. I'd much rather people reach for something a little cleaner like a generic directive.

Definitely a step in the right direction and definitely an improvement over common mark. If I were writing a markdown implementation that only needed to work for markdown I would consider it but sadly there are too many incompatibilities.