Confirm API reference plan for Python

[mircubed]

We need to confirm the API reference plan for the Python launch. In #3658, you can see an example from @vikram-dagger that is hosting our own API docs.

Issues with hosting ourselves:

• the format isn't great automatically, so we will still need to it
• Linter doesn't work, so we'd need to update linter roles for it
• hosting ourselves

Let's use this issue to come to an agreement on what the Python API reference plan should be. We need a final decision by Monday, November 7th if we want the launch to stay on track.

[mircubed]

@helderco @shykes need your thoughts

[shykes]

What are the options?

[vikram-dagger]

• Option 1: Build the API reference ourselves using a markdown generator like pdoc. I tried this and it works, although with some issues

  • Docusaurus metadata is not present in the generated markdown. It needs to be added as a separate step.
  • Some of the formatting of the generated markdown is weird. I'm not sure if this is an issue with the code comments or the generator tool.
  • We need to manually regenerate the API reference every time we release a new version of the SDK.

  Sample of this at https://deploy-preview-3658--devel-docs-dagger-io.netlify.app/ -> Python SDk -> Reference

• Option 2: Host the API reference externally eg. on readthedocs.org, similar to the approach for the Go SDK. readthedocs automatically rebuilds the docs every time we push a new version. However:

  • We will need to add some configuration to the Python project in our repo so that the docs are generated properly. I'll need @helderco 's help with this.

My recommendation is option 2, as it already includes the required automation, should require less time to set up, and also fits the pattern we established with the Go SDK.

[mircubed]

@shykes we will need a final decision on this by Monday EOD at the latest, so we can stay on track. Let us know if you need additional information to make a decision. Thanks!

[mircubed]

@shykes @aluzzardi need a decision on this one to keep us on track. Thanks!

[shykes]

Option 2 makes sense to me. Unless @helderco disagreed, let's do that.

[aluzzardi]

Option 2 sounds good to me too -- I believe readthedocs is the standard way for Python reference documentation

[helderco]

I'm going to experiment with it today. Been many years since I've used it and not in a monorepo. If it doesn't work out well I'll explore other solutions. Will close this when one is found.

By the way, we can host it ourselves but it doesn't have to be integrated with docusaurus. The core tool here is Sphinx. That's what readthedocs does, just publishes the html output from Sphinx. You can push that html wherever you can publish a static website. For example Netlify can also be automated to run a command (i.e. build docs) when a new release is pushed to git. Or we can do it in GHA when deploying a new release.

Back to RTD, I haven't confirmed if they still do it, but we used to be able use our own domain with that service if you guys are interested.

[jpadams]

Finding that RTD is not "the" standard at this point. Some packages in PyPI don't have any docs linked and many popular packages are hosting their API docs on Vercel or Netlify or wherever.

some packages on PyPI don't have a docs link: https://pypi.org/project/aws-cdk.aws-lambda-python-alpha/, while others host their own docs: https://pypi.org/project/poetry/ -> https://python-poetry.org/docs (on Vercel)

So a mixture. No obvious benefit to hosting on RTD since we have to generate API docs (in Sphinx for them) anyway.

on RTD: https://anyio.readthedocs.io/en/latest/

on something else (Netlify/Vercel?): https://fastapi.tiangolo.com/

[jpadams]

I'm going to experiment with it today. Been many years since I've used it and not in a monorepo. If it doesn't work out well I'll explore other solutions. Will close this when one is found.

I'll try to experiment and will report back

[jpadams]

Got API docs working! @helderco @vikram-dagger 😄

pip install sphinx sphinx_rtd_theme
pip install --pre dagger-io
git clone https://github.com/dagger/dagger
cd dagger/sdk/python
sphinx-apidoc -F -o docs/ src/dagger/
# edit docs/conf.py (use file below)
make -C docs html
cd docs/_build/html
python3 -m http.server
# open browser to localhost:8000

final conf.py

# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = 'Dagger Python SDK API'
copyright = '2022, Dagger'
author = 'Dagger'
release = 'v0.1.0'

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.viewcode',
    'sphinx.ext.todo',
]

templates_path = ['_templates']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']

language = 'en'

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = 'sphinx_rtd_theme'
html_static_path = ['_static']

# -- Options for todo extension ----------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/extensions/todo.html#configuration

todo_include_todos = True

[helderco]

Just a status update. My problem right now is for readthedocs to pick up the right version since tags aren't fully semver due to the sdk prefixes.

[helderco]

Closing this and tracking remaining work in new issues:

• Trigger correct versions for ReadTheDocs.org #3767
• Trigger ReadTheDocs.org build only on successful publish #3768