Creating Vespa documentation

All Vespa features must have user documentation - this document explains how to write documentation. See introduction to documentation for styles and examples.

Practical information

Vespa documentation is served using GitHub Project pages with Jekyll. To edit documentation, check out and work off the master branch in this repository.

Documentation is written in HTML or Markdown. Use a single Jekyll template _layouts/default.html to add header, footer and layout.

Install bundler, then

$ bundle install
$ bundle exec jekyll serve --incremental --drafts --trace

to set up a local server at localhost:4000 to see the pages as they will look when served. If you get strange errors on bundle install try

$ export PATH=“/usr/local/opt/ruby@2.6/bin:$PATH”
$ export LDFLAGS=“-L/usr/local/opt/ruby@2.6/lib”
$ export CPPFLAGS=“-I/usr/local/opt/ruby@2.6/include”
$ export PKG_CONFIG_PATH=“/usr/local/opt/ruby@2.6/lib/pkgconfig”

The output will highlight rendering/other problems when starting serving.

Alternatively, use the docker image jekyll/jekyll to run the local server on Mac

$ docker run -ti --rm --name doc \
  --publish 4000:4000 -e JEKYLL_UID=$UID -v $(pwd):/srv/jekyll \
  jekyll/jekyll jekyll serve

or RHEL 8

$ podman run -it --rm --name doc -p 4000:4000 -e JEKYLL_ROOTLESS=true \
  -v "$PWD":/srv/jekyll:Z docker.io/jekyll/jekyll jekyll serve

The layout is written in denali.design, see _layouts/default.html for usage. Please do not add custom style sheets, as it is harder to maintain.

Writing good documentation

Learn how to contribute to documentation, then read the following guide before writing some.

Guides and references

A document cannot be both comprehensive and comprehensible. Because of this, documentation is split into guides and reference documents.

Guides should be easy to understand by only explaining the most important concepts under discussion. Reference documents on the other hand must be complete but should skip verbiage meant to aid understanding.

Reference documents are those that are placed in reference/ subdirectories.

Maintainability

Prioritize maintainability higher than usability:

• Don't include unnecessary details, especially ephemeral ones such as that a feature is "recently added" or how things was before, etc. The guide/reference distinction helps here: Guides are harder to maintain as they contain more verbiage, and they should not unnecessarily repeat information found in a reference doc. Write such that the document will still be correct in a half decade.

• Don't repeat information found in other documents. It is tempting to make life easier for users by writing use-case oriented documentation on how to accomplish specific tasks, but this backfires as it leads to a lot of repetition which we fail to maintain. In the long run it is better to explain the concepts clearly and succinctly and leave it to the users to piece together the information. Use the same principles for documentation as for code: DRY, refactor for coherency etc.

• Be wary of adding code in the documentation. The code will becomes incorrect over time and should in most cases be placed in git as continuously built code and referenced from the doc.

Text quality

Documentation is not high prose, and not a podcast. Users want to consume the information as soon as possible with as little effort as possible and get on with their lives.

Make the text as short, clear, and easy to read as possible:

• Describe things plainly "as they are". You usually shouldn't worry about explaining why, what you can do with it etc.
• Use short sentences with simple structure.
• Avoid superfluous words such as "very".
• Avoid filler sentences intended to improve the flow of the text - documents are usually browsed, not read anyway.
• Use consistent terminology even when it leads to repetition which would be bad in other kinds of writing.
• Use active form "index the documents", not passive "indexing the documents".
• Avoid making it personal - do not use "we", "you", "our".
• Do not use " , — and the likes - makes the document harder to edit, and no need to use it.
• Less is more - <em> and <strong> is sufficient formatting in most cases.

Links

Add an id attribute to each heading such that link can refer to it: Use the exact same text as the heading as id, lowercased and with spaces replaced by dashes such that references can be made without checking the source. Don't change headings/ids unless completely necessary as that breaks links.

Example: <h2 id="my-nice-heading">My nice Heading</h2> If this algorithmic transformation is followed it is possible to link to this section using <a href="doc.html#my-nice-heading"> without having to consult the html source of the page to find the right id.

Link to Javadoc

• Link to javadoc for an artifact: https://javadoc.io/doc/com.yahoo.vespa/container-search
• Link to javadoc for a package: https://javadoc.io/doc/com.yahoo.vespa/container-search/latest/com/yahoo/search/federation/vespa/package-summary.html
• Link to javadoc for a class: https://javadoc.io/doc/com.yahoo.vespa/vespa-feed-client-api/latest/ai/vespa/feed/client/JsonFeeder.html

By Jon Bratseth, June 2016

Appendix: Vespa Documentation Search

See the Vespa Documentation Search sample application for architecture.

Below is a description of the job for indexing this repository's documentation. File locations below refer to this repo's root.

1. Build a Vespa feed from the source in this repo:

   1. Use Jekyll to generate HTML from the content (some files are in Markdown)
   2. Use Nokogiri to extract text from HTML
   3. Implement HTML-to-text in a Vespa feed file by using a Jekyll Generator, see _plugins-vespafeed/vespa_index_generator.rb
   4. The generated open_index.json can then be fed to Vespa

2. Feed changes to https://console.vespa.oath.cloud/tenant/vespa-team/application/vespacloud-docsearch using feed_to_vespa.py:

   1. Visit all content on the Vespa instance to list all IDs
   2. Determine whether or not to remove documents
   3. Feed all content

3. Automate these steps using GitHub Actions

   1. Store the keys required to feed data as secrets in Github
   2. Find workflow at .github/workflows/feed.yml

Local development:

$ bundle exec jekyll build
$ ./feed_to_vespa.py   # put data-plane-private/public-key.pem in this dir in advance