remark@next (13)

[wooorm]

Hi!

remark is switching to a new parser (and compiler) internally (micromark, remarkjs/remark#536), which will break several of these plugins.
Several of the remark plugins, such as frontmatter, footnotes, math, a new remark-gfm, and a couple others, will update soon too.
Those might server well as inspiration on how to tokenize with micromark in remark. There are several micromark extensions already as well (which deal with tokenizing markdown), and corresponding mdast extensions, which deal with mapping those tokens to mdast.

Anyway, feel free to ask me questions!

[LikaKavkasidze]

Thanks a lot for having taken time to inform us on this issue.

I'll experiment with micromark in the upcoming weeks, and ask you if I have any questions.

[vladar]

Any thoughts on remark 13 support? I am interested in upgrading remark-custom-blocks plugin, so can prepare a PR if you are looking for help. But not sure if we can bump only one plugin or do we have to update all at once?

[LikaKavkasidze]

We are planning to update them all at once, but the migration should start fairly soon. In fact, I already started working on it for some plugins (but not remark-custom-blocks). If you are interested in working on remark-custom-blocks, I'd would be happy to accept the pull request, and can arrange to publish a version without problem. Please note however that this will require significant structural changes; here are my current thoughts on how we are going to do it:

• publish a new package micromark-extension-custom-blocks, that will in the end contain both a micromark tokenizer (lib/index.js) and an HTML serializer;
• update the current remark-custom-blocks to use the tokenizer mentioned above, and render an AST similar to the one we generate today.

In fact, I'm planning to initialize a specific branch for working on micromark plugins. The best you can do is maybe to wait for it to get created as it will provide an plugin example (micromark-extension-kbd). I can arrange to do it next week, so stay tuned, I'll post here when it's done. You can also of course start working on it by now, and we will integrate it after. There is a great margin on what we can do as the migration is just getting started.

[vladar]

Thanks for the update! 👍 I was hoping the migration will be easy 😅 But happy to try anyway when your micromark extensions are ready as you suggest.

[LikaKavkasidze]

Quick update: a plugin was migrated today, remark-kbd. It took longer than expected and still have some problems, but I plan to solve them later.

If you feel confident enough, then technically, you can do the same as I did with this first plugin:

• write a micromark extension, the most complicated part, especially considering that remark-custom-block is a block element, making it more difficult to handle than an inline one such as remark-kbd;
• rewrite the existing remark plugin to use the micromark extension, and add support for the AST.

If you feel like waiting more isn't a big deal, then I'm writing a guide to porting our extensions (and more generally, to writing extensions) for micromark, that may help you get started. If you've never written any plugin for the new parser, then maybe waiting for this guide will help, but if you're experienced enough, our codebase is technically completely ready for micromark on the branch next, and I would be happy to accept your PR! 🙂

[ghost]

Any updates on this @StaloneLab ? @vladar Said he was willing to help

[LikaKavkasidze]

Still in progress, it takes a long time, and I'm almost the only one working on it. Still no due date planned, and there likely won't ever be any, we'll release plugin-by-plugin when they will be ready.

By now, I'm still writing a guide on how to update the plugins, so that maybe more people will be able to participate. This guide will lead to two more plugins being ported (namely, remark-ping and remark-iframes), and I cannot guarantee anything after.

People that already know how to write micromark plugins are encouraged to join in whenever they want, to work on any other plugin that the three already "in progress", taking for example remark-kbd. I'm willing to help anyone who wants to get involved, just drop in a PR with your proposal.

EDIT: just saw that micromark added a guide to writing extension to it's own documentation, this might help speed up the process! I'll have a look at it.

[daleperforce]

@StaloneLab I have been investigating your remark-ping plugin and been trying to use it with react-markdown and noticed it not working with the latest remark-parser. Your comment on 16 Jul suggests you had a fix and was publishing this but it still not working.

I am wondering if you had a plan to migrate that plugin over to using the new remark-parser format in the near future? Happy to open a new Issue for just this plugin if you don't want to pollute this ticket.

[LikaKavkasidze]

Actually, I think it is a great thing you used this issue to ask your question. In July, I said I was writing a guide on porting plugins, and it would lead to some plugins being ported. This is still what is planned, but I'm running out of time by now, hence very little progress has been made.

The plugins will be made to work on remark 13 at some point, but we do not have any due date, as I'm almost the only one working on it. If you are interested in taking part (for instance porting remark-ping), you can refer to some work that has been done in the next branch, and follow the guide on the micromark repository to rewrite the plugin, there are no more blocking things on our side.

To make it short, any PR is welcome (and will be reviewed as fast as possible), I am ready to provide help to anyone willing to participate, and any plugin ported will be published whenever ready.

[janpe]

@StaloneLab I'd want to use the remark-disable-tokenizers plugin, but it is not ported to support this yet. Do you see it being updated at some point? I'd love to make a contribution and make the changes, but I'm not familiar with the technologies here. Would someone be able to point me in the right direction and tell me what kind of logic the new version would need to follow to achieve the same functionality?

[wooorm]

See https://github.com/micromark/micromark#case-turn-off-constructs.
Example: https://github.com/micromark/micromark-extension-mdx-md/blob/main/index.js. Wrapped in https://github.com/micromark/micromark-extension-mdxjs. In turn in https://github.com/mdx-js/mdx/blob/main/packages/remark-mdx/index.js.

[janpe]

See https://github.com/micromark/micromark#case-turn-off-constructs. Example: https://github.com/micromark/micromark-extension-mdx-md/blob/main/index.js. Wrapped in https://github.com/micromark/micromark-extension-mdxjs. In turn in https://github.com/mdx-js/mdx/blob/main/packages/remark-mdx/index.js.

Thanks a lot for these, much appreciated! Tried playing around with this but I'm still unsure how to incorporate the disable into a plugin.

[wooorm]

function myPluginToTurnOffThings(options = {}) {
  const data = this.data()
  const list = data.micromarkExtensions || (data.micromarkExtensions = [])
  list.push({disable: {null: options.disable || []}})
}

[janpe]

function myPluginToTurnOffThings(options = {}) {
  const data = this.data()
  const list = data.micromarkExtensions || (data.micromarkExtensions = [])
  list.push({disable: {null: options.disable || []}})
}

Thanks a lot! Using remark-gfm is there a way to disable parts of GFM?

[wooorm]

Can you open a question somewhere and thoroughly explain what you want? That would prevent us going in circles and off topic

[imhul]

please ping back if any updates regarding micromark-extension-custom-blocks. Gatsby updated minimum remark version, and now custom blocks plugin blocks upgrade of Gatsby :(

[nosovk]

yep, micromark is bit more complex comparing to previous parser.
I can't understand how nested md parsing should be done, in my opinion best starting point is
https://github.com/micromark/micromark-extension-gfm-table
but I don't have clue how they describe that cell is a valid MD to parse.

[imhul]

I hope somebody can update it, Because I have no idea how to do that :(

[BlindDespair]

Any plans to update remark-align?

[LikaKavkasidze]

Full migration is still an objective for us, but it is not a priority.
Any PR will be accepted, and I am willing to help anyone working on the migration.

Apart from that, no release date is planned. Some plugins, namely remark-kbd and remark-iframes, are actually ready, but updating them would require us to freeze the old plugins before, so it is uncertain whether it will be done.

[eerison]

Hello guys

I was needing that this https://github.com/zestedesavoir/zmarkdown/tree/master/packages/remark-custom-blocks supports remark 13 (But I saw that it is in version 14 😄 ), does make sense to extract this package and release a new major version to support remark 14?

But in theory it leads to split all packages in small ones! and I don't know if it's a good idea, because the packages's code are quite small!

[nosovk]

It would be super if you create some solution for latest Gatsby.

[eerison]

It would be super if you create some solution for latest Gatsby.

yep, the goal is bump it for Gatsby 😄 !

But I would like to know what is the best approach for this ;) !

[eerison]

But as I can see, have many packages into one repository is a common approach for others repositories then I guess we need to keep it in this mono repo 😢

@StaloneLab what do you think we create a 12.x branch to make the this repository support remark 14?

Edit: or should we use next branch

[artragis]

The "next" branch is indeed the "work in progress" branch for support of remark 13+ and therefore micromark.

I try to dig into it many times and, the new way is really hard to understand, I'm quite stuck.

[eerison]

Hey @artragis thanks to confirm it!

is there anywhere a todo list with packages that are missing to support the new major version?

for example:

• mdast-util-split-by-heading
• rebber-plugins
• ......

[artragis]

Here is the full affected plugin list :

• remark-kbd
• remark-sub-super
• remark-iframes
• remark-abbr
• remark-grid-tables
• remark-special-blocks
• remark-emoticons
• remark-ping
• remark-fix-guillemets
• remark-comments (this one may have an alternative, we need to check)
• remark-align

mdast-util, rebber, rebber-plugins do not need to be upgraded because they work on the AST, not on the markdown itself.
Same for heading-shift.

[LikaKavkasidze]

Hello everyone, @artragis has answered most of the questions asked, but seeing people commenting on this issue again and again makes me think that I might not have been clear before about our goals regarding the migration. This message aims to give some additional information to @eerison and anyone having questions about why migrating to remark@13 takes us so long.

What is ZMarkdown?

Although it might be obvious for some people, I think it is interesting to give a bit of context on what is ZMarkdown and why it exists. To start from the beginning, ZMarkdown (this repository) is a Markdown engine powering “Zeste de Savoir”, a French website, hosted and maintained by volunteers on their spare time.

Being a Markdown engine based on remark, this repository hosts some remark plugins, which were initially developed to serve our own purpose. These plugins have been published for anyone to use, and we are always very happy to accept both external work and suggestions. Nonetheless, the development made by ourselves are mainly oriented towards the needs of the website mentioned above.

What is the migration?

That being said, let’s talk a bit about remark@13. Currently, the plugins hosted on this repository are not compatible with this version of remark. Why is it? Because the plugins need to be almost entirely rewritten in order to be compatible with the new parser.

So when this version of remark came out (and even before to be honest), we started to wonder whether we would benefit from migrating to the new system, and came to the following conclusion: we likely do not need to migrate our plugins, but it would be a nice investment, both for us (to get fixes from remark) and for people who need the packages.

How is it organized?

Because we do not actively need to switch to the new parser, and considering that at most one person in our team (that’s me) knows how the new system works, I decided that I was going to work on migrating the plugins according to the following rules:

• the old plugins will be supported (because our website need them) until all of our plugins have been migrated to the new remark version;
• I will occasionally work on the migration, depending on the time I have available;
• if other people need some plugins for their own use, then we will welcome any pull request, and I am ready to help new contributors.

This policy is what guides all the answers to this issue: whenever some asks if a plugin will soon be ported, I answer that it likely will, but that no due date is planned, and that any pull request will be considered. Pull requests regarding this issue, are to be opened towards the next branch, on which you can find two examples of plugins already ported: remark-kbd and remark-iframes.

How do I get my plugin ported?

To summarize, there are two ways to get your plugin to work with the new remark version: the first one is to simply wait (potentially for a long time) for us to do it, in that case there is not guarantee of any kind, and asking here when a given plugin will be ready is not necessary: we know there is work to be done.

A better (and likely faster) way is to contribute to the project in order to migrate one, or even many, plugins you need. Apart from making your own schedule, there are at least two advantages for you in contributing to this project: first, you will help the whole community of people using the stack; second, I am willing to help anyone who wants to work on the project (including professionally, contact me by email in this regard), so you gain insight on the new system, with which I have worked.

Can you release my plugin?

This question was not asked before, so I am putting extra attention to answer it: while we do not normally release plugins before all the others are migrated, if you help us migrating a plugin, and need the new version to be published, I am ready to do it. This is essentially quite simple to do, and will cause little trouble, even if it is not optimal for us (hence why I am not doing it automatically).

What that means is that we do not need to split our monorepo into “smaller packages” as mentioned. The way is currently works already allows plugins to be independently published, in fact we have already done it in the past.

TL;DR

Feel free to contribute! We will not do the migration for you, but I am ready to help if necessary.

I hope this message will clarify everything, but I am ready to answer any specific question not answered here. I shall also thank @artragis again for the list of plugins, and for his previous answers.

[elbakerino]

I've used micromark/micromark-extension-gfm-strikethrough/blob/1.0.4/dev/lib/syntax.js as base to get an sub-super plugin running again.

Feel free to check my implementation and copy when needed (MIT from unified and MIT from me), all plugins i've "modifed" (no tests...) can be found in this repository.

came along this issue, but not sure when i find time to make a PR here

[LikaKavkasidze]

Duely noted, thanks for your work, I'll see if it can be ported here!