Components for ReadTheDocs stuff

[pradyunsg]

A bit of context: this theme is basically an attempt to create a new base, for Sphinx themes to reduce the amount of duplicated effort in the ecosystem.

In #11, @choldgraf and I (for now, hopefully more folks will pitch in soon) are brainstorming what "reusable chunks of HTML" we should provide downstream theme authors, to make their lives easier. We're calling them components.

For reasons that I hope are obvious, it'd likely be a worthwhile investment to have this theme's components work well on RTD. This is basically an issue for having a place to discuss about how to make that happen! :)

/cc @humitos @stsewd @ericholscher

[pradyunsg]

Looking at https://github.com/readthedocs/readthedocs.org/tree/master/readthedocs/core/static-src/core/js/doc-embed, I can see what RTD's embed JS does. Here's a few notes + comments on potential changes that'll make it easier to get things working here:

• version-compare.js: Injects a warning admonition, on outdated docs.
  • This currently assumes that the page has a div.body and div.document while will almost certainly not be True for themes deriving from this one (or Furo FWIW).
  • Suggestion: Change this to look for a tag with role="main", since that's what Sphinx's search uses as well.
• sponsorship.js: Injects the ad.
  • This just works with the new and clean data-ea-publisher: https://github.com/pradyunsg/furo/blob/main/src/furo/theme/furo/sidebar/ethical-ads.html
• sphinx.js: Handles the version selector tracking logic.
  • I can't figure out how it populates this. Likely with footer.js? :)
• footer.js: Can't figure out what this does.

[pradyunsg]

So, I guess... here's my questions for the RTD folks:

• Version selectors:
  • What does this JS need from an element in the page's DOM, to trigger embedding the RTD-version-selector's HTML in that element?
  • What does the embedded HTML look like? Is there a good way to test for this locally?
• Old version warning:
  • Could you start embedding the admonition to the [role=main] element, rather than in div.body div.document?

[ericholscher]

@pradyunsg I'd recommend you take a peek at readthedocs/readthedocs.org#8052 -- where we're doing a design doc on the future of the integration here to make this kind of thing easier.

[choldgraf]

@ericholscher ah very cool - given that this "base theme" will likely not become totally integrated with docs for a little while (I suspect a few weeks at the very earliest, @pradyunsg lemme know if I sound off there!), does it make sense to design for the new implementation of the ads integration and such?

[pradyunsg]

@ericholscher Thanks for that link! Any idea how long the v3 API would possibly take?

TBH, I'm actually willing to put in the time to figure out how to stylize /api/v2/footer_html's output (it shouldn't be that difficult) assuming someone either points me to how to get that OR I figure out how to "look like" RTD-Sphinx-Theme to the JS. :)

FWIW, I'm definitely on board for straight up upgrading this theme for v3, once that's available -- I think it' personally don't think we need all that though, just information about how stuff works right now w.r.t. the version selector/popup would be sufficient.

(I suspect a few weeks at the very earliest, @pradyunsg lemme know if I sound off there!)

If "few" means >= 2 weeks, yea. I'm hoping to cut a release over this weekend, and maybe switch Furo over to this over the next weekend. That's of course assuming that this (and a couple of pip things I've lined up for myself) won't take more than 2 weekends + 1 week's free-time-that-goes-towards-OSS. I'm also hoping to put together something that's like GitBook (with Furo's style language) that's built off of this as well. That's probably another 2 weeks. :)

[choldgraf]

Sounds good - I was also imagining similar (or longer) timelines.

I'm also hoping to put together something that's like GitBook (with Furo's style language) that's built off of this as well

Just a quick thought - why make another 3 column layout-style theme? Why not either turn that into the Sphinx Book Theme, or the PyData Sphinx Theme? (assuming both will wish to use this base theme component, and have very similar design goals to gitbook)

[pradyunsg]

Just a quick thought - why make another 3 column layout-style theme?

Hah. This is a great question! This sent me down a fun mental spiral and I needed to step away, pace up and down in my house to figure out why I wanted to do this.

[snip story I rambled on about, with too much information for a publicly visible message]

To put it simply: I've found the process of writing a theme fun, so I wanna write one with no strings attached. :)

---

Basically, I don't want to be doing the "difficult" bits right now; only the ones that are fun in my book and changing existing Sphinx themes just isn't one of them. Yes, the folks are great, but I'm probably not gonna find the actual work fun. And I'm obviously optimising for fun here. :)

Changing an existing theme has all of the things I do NOT want to be doing right now:

• Thinking about backwards compatibility concerns (because they have existing user bases).
• Dealing with change-management overhead of too-many-user-facing-knobs and users-depend-on-things-we-never-promised.
• Needing to discuss ideas and communicate around them, instead of just... doing them and navigating by feel.
• Expectation management with others and concensus building.

[stsewd]

Hi, answering some questions

What does the embedded HTML look like? Is there a good way to test for this locally?

footer.js calls this API https://docs.readthedocs.io/_/api/v2/footer_html/?project=docs&version=stable&page=index&theme=sphinx_rtd_theme&format=jsonp&docroot=%2Fdocs%2F&source_suffix=.rst which returns the content to be injected.

Could you start embedding the admonition to the [role=main] element, rather than in div.body div.document?

Yeah, but we would still need to fallback to the other selector and maybe also look for the main tag.

[pradyunsg]

Notes for future self:

• https://github.com/readthedocs/readthedocs.org/blob/c70c6e8edc97d1f8a0e8420c5456c3696bd72fb3/readthedocs/api/v2/templates/restapi/footer.html is the template that's rendered for HTML.
• https://github.com/readthedocs/readthedocs.org/blob/c70c6e8edc97d1f8a0e8420c5456c3696bd72fb3/readthedocs/api/v2/views/footer_views.py#L145 is the context for that render.
• https://sphinx-basic-ng.readthedocs.io/_/api/v2/footer_html/?callback=callback&project=sphinx-basic-ng&version=latest&page=index&theme=furo&format=jsonp&docroot=%2Fdocs%2F&source_suffix=.md is the call made by the RTD script, for the home page of these docs.

Yeah, but we would still need to fallback to the other selector and maybe also look for the main tag.

Not really? Every page is basically required to have [role=main], because Sphinx search depends on it. :)

[stsewd]

oh, sorry, I thought you meant other thing, this can be updated yeah https://github.com/readthedocs/readthedocs.org/blob/4b4ca8e3e2231b4219dc8ab8ac199eceff7e05cf/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L28-L31

[pradyunsg]

Heyo folks! Is there any way to enable the version selector (flyout) menu in a local build -- like the one that sphinx-rtd-theme has?

I don't mind if it needs to reach out to somewhere on ReadTheDocs to populate it. I'm basically looking for any way to get a "sample" version selector flyout, that's available while developing a theme locally, so that theme authors can try stylising it.

[choldgraf]

+1 this would be very helpful (even better would be a little guide to how authors can add support for this)

[stsewd]

Heyo folks! Is there any way to enable the version selector (flyout) menu in a local build -- like the one that sphinx-rtd-theme has?

I don't mind if it needs to reach out to somewhere on ReadTheDocs to populate it. I'm basically looking for any way to get a "sample" version selector flyout, that's available while developing a theme locally, so that theme authors can try stylising it.

sorry for the late reply, had this open in a tab and today was the day for this one.

You can do this by adding this in your conf.py file

html_css_files = ['https://assets.readthedocs.org/static/css/readthedocs-doc-embed.css', 'https://assets.readthedocs.org/static/css/badge_only.css']
html_js_files = ['rtd_dummy_data.js', 'https://assets.readthedocs.org/static/javascript/readthedocs-doc-embed.js']

Where rtd_dummy_data.js is a file on _/static with this content

var READTHEDOCS_DATA = {
  'project': 'your-project-slug',
  'version': 'latest',
  'language': 'en',
  'proxied_api_host': 'https://readthedocs.org',
};

note that this is just a hack, it isn't guaranteed that this will always work.

[pradyunsg]

Thanks @stsewd! I finally tried this today, and am happy to confirm that it works seemlessly. 🎉 🌮

Which brings me to a very straightforward next question: Is there any ReadTheDocs-based documentation site, that has got translations + multiple "active" versions? It would be nice to have the slug for that site, so that I can figure out how to get both aspects working properly.

[stsewd]

This project makes good use of translations https://docs.wpilib.org/en/stable/.

[pradyunsg]

@ericholscher @stsewd One of the things that has come out of a bunch of discussions is wanting to have separate version and language selectors (like docs.python.org does today).

Would you be willing to make it possible to have versions and languages in separate dropdowns/flyouts, embedded on page based on the location of something like a #readthedocs-embed-versions / #readthedocs-embed-translations? With #readthedocs-embed-flyout being similar to the current blurb that gets injected?

[ericholscher]

@pradyunsg That is definitely something we can do. I'm happy to inject the data into a user-defined place, and that seems like an obvious extension of the work we're already doing. We also want to have a lot more customization here, but I think giving users more control over how to inject the data from the API is a great first step towards that.

Would love your feedback on readthedocs/readthedocs.org#9357, which is the first step down this road, and will likely be the pattern we build on top of for any other logic here. It wouldn't be hard to break out the API data and pass it somewhere else in the HTML, the question then is mostly the HTML structure, but it seems like folks have been able to work around our HTML as long as it's reasonably semantic.

[pradyunsg]

That looks great to me. Furo and multiple other themes already work with the markup that RTD spews, so keeping it as-is seems reasonable to me. :)

[humitos]

@pradyunsg

@ericholscher Thanks for that link! Any idea how long the v3 API would possibly take?

Unfortunately, we haven't had the time to continue with this work yet. However, we are recovering all these conversations now and deciding how to move forward.

TBH, I'm actually willing to put in the time to figure out how to stylize /api/v2/footer_html's output (it shouldn't be that difficult) assuming someone either points me to how to get that OR I figure out how to "look like" RTD-Sphinx-Theme to the JS. :)

I think that you were able to do this for now. The Read the Docs flyout and the ads are already integrated in Furo and they look great! Besides, the HTML structure for the flyout is now documented at https://docs.readthedocs.io/en/stable/flyout-menu.html which should be useful when styling it.

So, I'm not sure if all the work required for this issue is done or there are more work required from Read the Docs team to do here -- but keep reading my next comment 😄

[humitos]

While reviewing lot of PRs and issues on the Read the Docs side, I arrived at this issue here and I wanted to let you all know that I started working on a new implementation of the readthedocs-doc-embed.js.

The immediate main goal is to support projects building with build.commands (currently in beta) to have all the same features that projects building without any customization of the build process. Together with this, it will bring all the Read the Docs features to other doctools (e.g. Pelican, etc) that are currently not fully supported. In other words, this means, for example, that a project building a site with build.commands and Pelican, will show the Read the Docs' flyout, the version warning and the pull request warning by default.

The initial proposal and main conversation is happening at readthedocs/readthedocs.org#10127. Feel free to give us any early feedback you may consider useful at this point. I'd personally really appreciate it and consider at this design stage we currently are. Some of the important places to give feedback are:

• what is the data that Read the Docs should provide via environment variables at build time to the doctool? (related to Build: expose VCS-related environment variables readthedocs/readthedocs.org#9423)
• what is the data the doctool (e.g. Sphinx, Pelican, MkDocs, Docsify, etc) needs to communicate at build time to Read the Docs for correct integration?
• what is the data that Read the Docs should provide at serve time to the frontend?

Thanks for all the work you do for the community 😄

[humitos]

Hi 👋🏼 . I wanted to give you an update here since we have done a lot of work related to this issue since my last comment.

While reviewing lot of PRs and issues on the Read the Docs side, I arrived at this issue here and I wanted to let you all know that I started working on a new implementation of the readthedocs-doc-embed.js.

This work is done and we called "Addons". This is a completely new re-designed and self-contained JavaScript that's inject at serve time in the HTML and does not interfere with the build process at all (there isn't --won't be-- hacky readthedocs-sphinx-ext anymore). We published a blog post about this at https://blog.readthedocs.com/addons-flyout-menu-beta/

The code lives on its own repository at https://github.com/readthedocs/addons and it's implemented following new JavaScript standards and using WebComponents done with Lit (https://lit.dev/).

This new approach, splits each "feature" on its own addon (Flyout, Notifications, Search, Analytics, DocDiff, EthicalAds, etc) and allows theme authors to integrate and customize each addon separately. I did a proof of concept as a "theme maintainer" using the JavaScript API exposed by the addons to integrate the flyout in our theme at readthedocs/sphinx_rtd_theme#1526 that you can take a look as example. I'd really appreciate feedback about this implementation since we want to build something simple and useful for theme maintainers.

The main thing to highlight on that PR is that theme authors have access to all the Read the Docs data in a JSON structure that they can use as they prefer. The Read the Docs flyout is using that exact data to generate the flyout on the client side (instead of on server side as it was before).

The immediate main goal is to support projects building with build.commands (currently in beta) to have all the same features that projects building without any customization of the build process. Together with this, it will bring all the Read the Docs features to other doctools (e.g. Pelican, etc) that are currently not fully supported

Some live examples of other doctools using the new beta addons can be seen at:

• Docusaurus: https://test-builds.readthedocs.io/en/docusaurus/
• Pelican: https://test-builds.readthedocs.io/en/pelican/
• Material for MkDocs: https://test-builds.readthedocs.io/en/mkdocs-material/
• Docsify: https://test-builds.readthedocs.io/en/docsify/
• AsciiDoctor: https://test-builds.readthedocs.io/en/asciidoctor/
• Furo: https://test-builds.readthedocs.io/en/furo/