API to Compile to fragment instead of component

[jquense]

Subject of the feature

This is either a question or feature request. I'd like to be able to take a "partial" of mdx and render to just a jsx fragment, instead of a full component with a layout, for rendering into an existing mdx page. In other words it'd be nice to get just the return value of the MDXContent component.

Problem

Not a problem, but the use case is extracting md/mdx comments from code, via tools like react-docgen or TS typedoc and render them into other mdx pages. E.g. enhancing hand written MDX doc files with auto generated docs. Being able to output fragments allows "stitching" a few compiled fragments into a single file.

Expected behavior

You could imagine something like the following for rendering out importable metadata in, e.g. Docusaurus.

let imports = []
let exports = []

propsMetadata.forEach((prop) => {
  const result = mdx.compileTofragment(prop.description)
  prop.mdx = result.fragment
  imports.push(...result.imports)
  exports.push(...result.exports)
})

writeFile(`
  ${imports.join(';\n')}

  ${exports.join(';\n')}

  export default ${stringifyToJs(propsMetadata, null, 2)
`)

The idea here is that the jsx fragments could be rendered where ever (in the context of another MDX page). I actually find the need for imports/exports less important since you probably aren't likely to be using them in the context of a "mdx" fragment, so it's a bit weird semantically. I don't think it's necessary to support but figured I could show a possible API to be comprehensive.

Alternatives

The current API can be used to write each block out to its own file and import it manually, but it's a bit clunky and gives unwanted behavior like wrapping every block in a layout as if it was a standalone page. Perhaps this is already possible and i'm just missing an API!

[wooorm]

Hmm, I’m not sure I totally understand this feature request: because you can already use one MDX file inside another one (assuming you have set up your bundler to do that):

import Section from 'some-place.mdx'
import Other from 'some-other-place.mdx'

# Heading

<Section />

<Other />

---

In other words it'd be nice to get just the return value of the MDXContent component.

MDXContent is a function: you can call it to get the result? 🤔

[jquense]

Yeah so the request would be to avoid needing to write each fragment to a file, when it's overkill to do so for tiny bits, or when you want to manually construct a single component from a few mdx fragments.

[wooorm]

Those sound like two things to me:

The first seems to be: I want several MDX “files” in one file.
The second seems to be: I want to use several MDX files in one file.

I think the second is I believe answered by my pseudocode above.
The first might be #454. But this might also be an XY problem: perhaps you’re asking about what you think the solution is.

[jquense]

The second seems to be: I want to use several MDX files in one file

Yeah sorry for the lack of clarity here, i'm not asking about this. It seems already true that you can compose multiple files just fine via normal component rendering at runtime. However this is not very ergonomic at compile time with the compiler API (without creating a lot of small files). What I would like to do is avoid creating multiple files at all. I am looking for something similar to #454, but even that is maybe too high level.

perhaps you’re asking about what you think the solution is.

Totally could be the case. I'd be happy to describe the problem more directly if that would be helpful. Basically I have mdx/md in JSDoc code comments I'm rendering in a React app, with .mdx file support using https://v2.docusaurus.io/

As part of the build plugin I am extracting JSDOC comments from a lot of files and writing out the metadata as its own file that can be imported for use in a react component or mdx page e.g. import editorMetadata from './metadata/Editor.js'. Aside from the mdx, the build script is basically

fs.writeFile('Editor.js', 'module.exports = ' + JSON.stringify(metadata))

I would like instead of writing out the uncompiled mdx in the metadata, compile it to a simple jsx fragment that a user can render when and how they like. That can done with the current API like:

fs.writeFile('Editor_desc.js', mdx(metdata.description))
metadata.description = 'require("./Editor_desc.js")'

fs.writeFile('Editor.js', 'module.exports = ' + stringify(metadata))

but you end with a lot more IO and modules than necessary, especially for projects with lots of metadata, and lots of jsdoc comments.

That is a lot for what amounts to looking for a way to optimize my tooling output I realize! Thanks for taking the time to read through it 🙇

[wooorm]

Thanks! Hmm, some loose things:

I think I can now better see where you’re coming from: @mdx-js/mdx right now compiles to a “whole” file: it has top-level imports/exports, which don’t work for you: you want something more light weight, correct?

And, all those “files” use the same identifiers: MDXWrapper, MDXContent, which would conflict when multiple “files” are strung together..

Still, MDX compiles its format to a string of JavaScript, which somehow needs to be evaluated. Is the runtime maybe better for you? https://github.com/mdx-js/mdx/tree/main/packages/runtime.

Or, perhaps, you could use Babel to postprocess your “fragment”s and generate one file from them? You can use skipExport: true and babel-plugin-remove-export-keywords to encapsulate MDX into a function. In fact, all of our tests do something like that, and the runtime too, and the demo on the website, so it might be something to add to core: compiling to an “encapsulated” function

[ChristopherBiscardi]

@jquense If you copy/paste a bit of code and take line 19 out:

mdx/packages/mdx/index.js

Line 19 in c126925

.use(mdxAstToMdxHast)

you can parse to the remark ast, which you can then merge at will with other remark asts.

I've done this sort of thing before and while it feels a bit off to be copy/pasting code like that, that particular file doesn't change often, so should be low-effort if you want to pursue this route.

@johno we've chatted about ast-level transclusion before, do you think exposing more of the compiler-level APIs makes sense? An MDX editor I'm working on, for example, also needs to go from mdx string to remark-mdx ast and back. unified tends to make it kinda of hard to work with intermediate objects outside of it's plugin system.

here's a file I'm using for the purpose

import footnotes from "remark-footnotes";
import remarkMdx from "remark-mdx";
import remarkMdxJs from "remark-mdxjs";
import squeeze from "remark-squeeze-paragraphs";
import toMDAST from "remark-parse";
import unified from "unified";
import remarkStringify from "remark-stringify";
import json5 from "json5";
import visit from "unist-util-visit";

export function pluckMeta(value) {
  const re = new RegExp(`^export const meta = `);
  let meta = {};
  if (value.startsWith(`export const meta = `)) {
    const obj = value.replace(re, "").replace(/;\s*$/, "");
    meta = json5.parse(obj);
  }
  return meta;
}
// a remark plugin that plucks MDX exports and parses then with json5
export function remarkPluckMeta({ exportNames }) {
  return (tree, file) => {
    file.data.exports = {};
    exportNames.forEach((exportName) => {
      const re = new RegExp(`^export const ${exportName} = `);
      visit(tree, "export", (ast) => {
        if (ast.value.startsWith(`export const ${exportName} = `)) {
          const obj = ast.value.replace(re, "").replace(/;\s*$/, "");
          file.data.exports[exportName] = json5.parse(obj);
        }
      });
    });
    return tree;
  };
}

/// Stringify mdxast from nodes
export const processor = unified()
  .use(remarkStringify, {
    bullet: "*",
    fence: "`",
    fences: true,
    incrementListMarker: false,
  })

  .use(remarkMdx)
  .use(remarkMdxJs);

// Parse mdxast to nodes
const DEFAULT_OPTIONS = {
  remarkPlugins: [],
  rehypePlugins: [],
};

function createMdxAstCompiler(options) {
  const plugins = options.remarkPlugins;

  const fn = unified()
    .use(toMDAST, options)
    .use(remarkMdx, options)
    .use(remarkMdxJs, options)
    .use(footnotes, options)
    .use(squeeze, options);

  plugins.forEach((plugin) => {
    // Handle [plugin, pluginOptions] syntax
    if (Array.isArray(plugin) && plugin.length > 1) {
      fn.use(plugin[0], plugin[1]);
    } else {
      fn.use(plugin);
    }
  });

  return fn;
}

function createCompiler(options = {}) {
  const opts = Object.assign({}, DEFAULT_OPTIONS, options);
  const compiler = createMdxAstCompiler(opts);
  return compiler;
}

export const parse = createCompiler().parse;
export const stringify = processor.stringify;

[wooorm]

This is now possible with the latest RC for MDX 2. See https://v2.mdxjs.com.
The heavy lifting can be done by setting options.outputFormat to 'function-body' to compile to, well, a function body, rather than a whole program.
You can wrap those function bodies in IIFEs:

import {compile} from '@mdx-js/mdx'

main(['# some mdx', 'Some more {1 + 1}', '> Ta da!', 'export const a = "b"'])

async function main(descriptions) {
  const file =
    'export default function createFragments(jsxRuntime) { return [' +
    (
      await Promise.all(
        descriptions.map(async (d) => {
          return (
            '(function () {' +
            (await compile(d, {outputFormat: 'function-body'})) +
            '})(jsxRuntime)'
          )
        })
      )
    ).join(',') +
    ']}'

  console.log(file)
}

The above prints something along these lines (formatted and abbreviated):

export default function createFragments(jsxRuntime) {
  return [
    (function () {
      const {Fragment: _Fragment, jsx: _jsx} = arguments[0]
      function MDXContent(props = {}) {
        const _components = Object.assign({h1: 'h1'}, props.components)
        const {wrapper: MDXLayout} = _components
        const _content = _jsx(_Fragment, {children: _jsx(_components.h1, {children: 'some mdx'})})
        return MDXLayout ? _jsx(MDXLayout, Object.assign({}, props, {children: _content})) : _content}
      }
      return {default: MDXContent}
    })(jsxRuntime),
    /* … */
  ]
}

Of course, you might want slightly different wrapper code and maybe use an object mapping names to fragments.
Note that exports are supported and imports could be supported with options.useDynamicImport (and async IIFEs).
Now, assuming you wrote that to the file system as fragments.js, and imported it somewhere where React/Preact/Emotion/etc is available, it could be used like so:

import * as jsxRuntime from 'react/jsx-runtime'
import createFragments from './fragments.js'

console.log(createFragments(jsxRuntime))

Which prints:

[
  { default: [Function: MDXContent] },
  { default: [Function: MDXContent] },
  { default: [Function: MDXContent] },
  { a: 'b', default: [Function: MDXContent] }
]