Add position information to tokens

[segevfiner]

It could be cool to add position information as specified by https://github.com/syntax-tree/unist to the resulting HAST.

[wooorm]

Heya!

What do you plan on using that for?

To do that, we’d have to only add it to the text nodes, because the text is preset in the original file, but we wouldn’t add them to the elements, as those aren‘t written in the original source. In unist terms, the elements are generated

[smith-kyle]

This would be helpful for me too! I'd be able to apply diff highlights to the tree if I knew the source positions

[wooorm]

Can you explain?

[smith-kyle]

Sure thing, not sure how much context you need so I'll lean towards oversharing:

I'm building a notebook diff review tool and use lowlight to do syntax highlighting in the cells. Prior to syntax highlighting the diff is calculated using the raw source code. The diffs are a list of range mappings indicating the start line, end line, start column, and end column in the raw source code. In order to find out where these diff highlights belong in the HAST I need to know what the source positions of each node is.

Specifically for markdown rendering, I use this chain:

  const processor = unified()
    .use(remarkParse, { position: true })
    .use(remarkRehype, { allowDangerousHtml: true })
    .use(raw, { position: true })

I'd like to add syntax highlighting for the code cells within the markdown using lowlight, but when I do so, the resulting nodes have no position information. Without position information I don't know what source code the nodes map to so I can't apply the diff highlights.

I have a workaround where I split the code HAST into lines then count the characters, but it'd be a big help if position was an option

[wooorm]

The notebook diff tool, with just lowlight, that sounds like an okay reason to have this.

But the second part, using lowlight inside unified for markdown -> html, and transforming it, in that case the positional info should not be there.
Positions refer to places in the source file. But the <span>s being generated are not in the source file. Across unified, when tools generate nodes, they don’t have positional info.
From earlier, the “In unist terms, the elements are generated”