Docs: Conversion from custom HTML generated documentation to regular Markdown generated documentation

[AxiomeCG]

Description

Today the documentation is generated from custom tags in HTML, and makes it difficult to contribute to it (lot of friction). The documentation is already a topic that is a pain for most of the devs. It is often the neglected part, I think it would benefit to reduce the friction by returning to industry standards and common tooling.

Usage of markdown would benefit by being able to generate documentation with whatever we want, (like docusaurus, ...)

We would access to many new automatic features like table of contents, "Edit this page" type of buttons to be able to encourage the contribution.
We could go further by supporting MDX in the future, to help craft wonderful documentation. It would be opening doors to new tools from there.

The discussion is not clear in my head if we should go on a different repository for the documentation, or keep it with the main repository.

Three.js is very popular among the 3D developers versus the rest of the frameworks like Babylon and stuffs. But the documentation does not live up to the popularity of our beloved lib, in my opinion.
If we take react-three-fiber, it takes advantages of the three.js documentation, by referencing it. So there is a whole ecosystem that would really benefit from a clearer, cleaner, modern documentation.

Also, I would take advantage of this modernisation to describe the documentation using Typescript convention of typing. This would enable syntax highlighting based on that for free, even if the underlying implementation is javascript. We would just use the ts code fences available with markdown.

Solution

I began to work on a python script to convert our custom HTML to markdown files and took docusaurus as a test bed for this documentation.

I used a separate repository to do so, just as a proof of concept.
I added the typescript code fences from markdown to explicit the typing on the method and properties of each pages.

You can see a demo here : https://threejs-docs-markdown.vercel.app/docs/en/animation/AnimationAction
Link to the codebase : https://github.com/AxiomeCG/threejs-docs-reloaded

Basically the script is parsing the HTML and producing a markdown file from it. It converts most of the custom elements like [property: ...], [method: ...] and <code> tags.

The structure we know for the folders would be the same.

The script is working by giving a folder in input and it produces an equivalent output by preserving the folder hierarchy and outputting the markdown equivalent files. I need to test on internalization files from other folders different from the en one. But as soon as it respects the same linking and convention on the custom elements (which I assumed true due to the automatic generation of the actual docs), I think it would not be troublesome.

Alternatives

We should examine the best alternative to a markdown system. Do we want to keep on going with this adhoc system or get back on an industry standard ?

Additional context

There is some of the things that are known broken like linking in the current script. I just submit the first beta result as a proof of concept to see if it is validated to be continued, and not wasting time on a tool that would not be well received ! :)

[AxiomeCG]

Relates to #25381

[Mugen87]

@donmccurdy has written an interesting proposal regarding the documentation in context of #24984.

https://gist.github.com/donmccurdy/db75b86a28a775135e33317ce2694a32

I support this proposal since I think it fits best to our current development process.

[AxiomeCG]

@Mugen87 Would the JSDoc be a full replacement of the current documentation ?

There is a set of documentation elements that fit more as articles/tutorials and would not be able to survive the JSDoc conversion.
Like introduction articles (for instance: https://threejs.org/docs/index.html#manual/en/introduction/Creating-a-scene)

I'm in favor of the JSDoc mentioned in the proposal, though it will only solve a part of documentation.

We need to differenciate what is API documentation, from the rest of the documentation like the Getting Started, the Examples and stuff, that my actual proposition still gravitates around in my opinion.

EDIT: if we look at Babylonjs for example, there is the API part that seems obviously generated out of JSDoc or equivalent, and there is the core documentation, that I would assume correspond to a Markdown based documentation with a set of beginner friendly documentation, explanation of core concepts that the API doc would not embrace.

[Mugen87]

AFAIK, @mrdoob wanted to separate the API documentation from a manual with guides and tutorials.

The API documentation would be: https://threejs.org/docs/
The manual would be: https://threejs.org/manual/

Don's proposal is about the API documentation (@donmccurdy Please correct me if I'm wrong^^).

Right now, the manual contains the articles that were originally written by @greggman and published as "three.js fundamentals". In the long term, guides like "Creating a scene", "Installation" or "How to update things" should be part of the manual and not the API documentation.

Of course the API documentation may refer to articles published in the manual.

Whether the current approach of the manual is ideal or not is open for discussion. I agree that something like Markdown could be interesting in this context.

[donmccurdy]

I agree that thinking about the API documentation separately from the long-form written content ("manual") is best. My proposal affects only the API documentation, but would be compatible with static generators like Docusaurus or Astro, if we decide to go that direction for the manual.

Babylon.js uses Typedoc for their API documentation. Frankly, my experience with Typedoc has not been great, and I would prefer to avoid it. On my project https://gltf-transform.dev — the long-form pages are written in Markdown, and the API doc pages are generated automatically from TSDoc comments using a custom script.

Using Markdown or MDX for the manual would be OK with me. We can still embed basic HTML like iframes when needed. I enjoy writing in both Markdown and MDX. But rewriting pages is a large task that I wouldn't have much bandwidth to help with at the moment.

[AxiomeCG]

@donmccurdy
I think I will continue to work towards the markdown conversion script until we figure out how to merge everything, and then begin to work on the JSDoc migration of the actual API documentation. This way we will have both tools to get the best output for a first move into a modern tooling.