Scaladoc: Refactor static site loading and directory structure #14378

pikinier20 · 2022-01-28T14:43:45Z

Static site in Scaladoc is probably one of its core features. Since it didn't have a lot of attention and also due to changing requirements, static site used to be not-so-clear piece of code. Having in mind upcoming changes to the Scaladoc, I decided to do a bigger refactor mainly in static site loading phase and also in directory structure.

I think that before merging we should also update documentation.

Static site loading

Previously, we loaded all files from filesystem and the sidebar.yml file was only used to generate LHS navigation. This solution was a bit problematic. Firstly, we could have pages that were only navigable by other pages or by sharing a link. It doesn't seem as a good thing because we could easily loose control over what's in the navigation and what is rendered. Also, it caused a problem with index pages: every section in navigation needs to have a index page otherwise we would get 404 after entering a section. Since the sidebar.yml definition had nothing to do with location of files in filesystem, there was a problem with finding a location where the index should be rendered. (Navigation could e.g. aggregate files among more than one directory). This solution is also not flexible as we can't change the place where the file is rendered (Rendered directory structure is always the same as the source directory structure).

I've changed that so we have two ways of loading static site:

We don't create sidebar.yml file and Scaladoc will load all documents that are present in /_docs/ directory and automatically generate navigation.
We create sidebar.yml file (probably we also need to rename it?) which declares sections and pages that should be rendered and navigable. (We can also add a option: hidden to render without entry in navigation). The main advantage though is that the YAML pages structure defines the rendered directory structure. By default, for each section there will be directory created by converting its name to kebab case (e.g. Other New Features => other-new-features/). There's a setting directory that overrides this convention. We can also define a section with title and index but without defining children which will cause loading documents from the directory of index file. e.g.

- title: Some section
  index: some-section/index.md

will load all files from some-section/ directory.

Static site directory structure

After my changes, static site directory structure looks like this:

.
└── <site-root>/
    ├── _docs/
    │   └── ...
    ├── _blog/
    │   ├── index.md
    │   └── _posts/
    │       └── ...
    └── _assets/
        ├── js/
        │   └── ...
        ├── img/
        │   └── ...
        └── ...

All documents should be in _docs directory. File-system loading crawls this directory and also every entry in YAML configuration is relative to this directory.

Whole blog content should be in _blog directory. We check if the directory exists and based on that, we generate Blog section or not.

All assets should be in _assets directory. The whole content of this directory is copied to the documentation and there's a check if asset exists by checking if the given path resolved in _assets exists. e.g. for images/logo.png we check if _assets/images/logo.png exists.

This change makes a static site structure clearer and also simplifies a lot of logic in codebase.

The `-Yapi-subdirectory` setting

The setting was introduced to resolve possible conflicts between static site and API. However, we also decided to use it in order to define what is a top-level documentation. With the -Yapi-subdirectory off (default), API documentation is in top-level and whole static site is inside _docs subdirectory. Otherwise, the static side goes to top-level and API is in api subdirectory.

BarkingBad · 2022-01-28T15:42:10Z

docs/sidebar.yml

-  - title: Blog
+rootIndex: index.md
+pages:
+  # - title: Blog


Let's remove this

BarkingBad · 2022-01-28T16:09:02Z

scaladoc/src/dotty/tools/scaladoc/renderers/Locations.scala

@@ -34,9 +34,9 @@ trait Locations(using ctx: DocContext):
    cache.get(dri) match
      case null =>
        val path = dri match
-          case `docsRootDRI` => List("docs", "index")
+          // case `docsRootDRI` => List("docs", "index")


Also leftover

BarkingBad · 2022-01-28T16:11:13Z

scaladoc/src/dotty/tools/scaladoc/renderers/Renderer.scala

@@ -82,34 +74,17 @@ abstract class Renderer(rootPackage: Member, val members: Map[DRI, Member], prot
            )
          updatedTemplates.result()

-        val newTemplates = updateSettings(templates, newSettings.to(ListBuffer))
+        val newTemplates = updateSettings(Seq(siteContext.staticSiteRoot.rootTemplate), newSettings.to(ListBuffer))


I guess it was supposed to be Seq(rootTemplate) or wh do we need property in line 43?

BarkingBad · 2022-01-28T16:13:37Z

scaladoc/src/dotty/tools/scaladoc/site/StaticSiteContext.scala

-    if files.size > 1 then
-      val msg = s"ERROR: Multiple root index pages found: ${files.map(_.getAbsolutePath)}"
-      report.error(msg)
+  val relativizeFrom = if args.apiSubdirectory then docsPath else root.toPath


Nice trick :D

BarkingBad · 2022-01-28T16:26:43Z

scaladoc/src/dotty/tools/scaladoc/site/StaticSiteLoader.scala

+        LoadedTemplate(withUpdatedTitle, List.empty, pathFromRoot.resolve(file.getName).toFile)
+      case Sidebar.Root(_, _) =>
+        // Cannot happen
+        ???


Maybe we could log it instead of throwing NotImplementedException even though it will never hit this branch

Or, should we define the type Sidebar in a different way so that we wouldn’t even have to handle the Root case here?

BarkingBad · 2022-01-28T16:27:50Z

scaladoc/src/dotty/tools/scaladoc/site/templates.scala

+          case "diff" =>
+            s"<pre><code class=\"language-diff hljs\" data-lang=\"diff\">${super.asString(nodes(1).render(context), context)}</code></pre>\n\n"
+          case _ =>
+            report.warn("Unsupported highlight value. Currenlty supported values are: `diff`", file)(using ssctx.outerCtx)


Currently typo

julienrf

Nice work @pikinier20!

There are several parts that I don’t understand in the diff, I have left some comments, do you mind clarifying?

Other than that, here are a few comments about the PR description:

I think that before merging we should also update documentation.

Sure! Thank you!

We don't create sidebar.yml file and Scaladoc will load all documents that are present in /_docs/ directory and automatically generate navigation.

So, the navigation bar will mirror the hierarchy of directories and files in _docs? What would be the order of the files? Alphabetical, I guess?

We create sidebar.yml file (probably we also need to rename it?)

Why do you want to rename it?

(We can also add a option: hidden to render without entry in navigation)

What do you mean? I don’t understand this idea.

There's a setting directory that overrides this convention.

Could you please show an example?

With the -Yapi-subdirectory off (default), API documentation is in top-level and whole static site is inside _docs subdirectory. Otherwise, the static side goes to top-level and API is in api subdirectory.

Did you make any changes to -Yapi-subdirectory in this PR? I can’t see them.

scaladoc-testcases/src/_docs/tests/Adoc.scala

scaladoc/src/dotty/tools/scaladoc/renderers/HtmlRenderer.scala

scaladoc/src/dotty/tools/scaladoc/renderers/Renderer.scala

scaladoc/src/dotty/tools/scaladoc/site/SidebarParser.scala

scaladoc/src/dotty/tools/scaladoc/site/StaticSiteLoader.scala

scaladoc/src/dotty/tools/scaladoc/site/templates.scala

scaladoc/test-documentations/basic/index.md

julienrf · 2022-02-01T14:59:45Z

docs/index.html

@@ -1,109 +0,0 @@
---


How does the landing page look like now?

This is a thing that we need to discuss. As I said in comment above - I don't think that custom landing pages are a responsibility of Scaladoc. For me, the entry point to the generated documentation should be standard doc page with the navigation etc. If we want to keep the "red" landing page on dotty.epfl.ch maybe we should just deploy it in the root directory and the entire generated nightly documentation would land into separate directory e.g. "reference" or "documentation"?

Ah, so the landing page would be the one we currently see at https://dotty.epfl.ch/docs/index.html?

I realize now that the part that generates the sidebar can not be controlled by the users. There is a global layout that is applied to all the .md files that don’t set a layout in their front matter, is that right? However, that layout is not applied to .html files such as this index.html that you removed?

I don’t (yet) have a strong opinion about the idea of not supporting custom landing pages in Scaladoc.

This may have an impact on the way we could integrate the Scaladoc-generated website into the Jekyll-generated website.

I don't really see any big advantage of generating custom landing page by Scaladoc. One can always do it manually. Usually, the links on landing page are not changed very often. Also, if we want to integrate Scaladoc and Jekyll, we can generate this page with Jekyll.

pikinier20 · 2022-02-02T12:39:50Z

Nice work @pikinier20!

There are several parts that I don’t understand in the diff, I have left some comments, do you mind clarifying?

Other than that, here are a few comments about the PR description:

I think that before merging we should also update documentation.

Sure! Thank you!

We don't create sidebar.yml file and Scaladoc will load all documents that are present in /_docs/ directory and automatically generate navigation.

So, the navigation bar will mirror the hierarchy of directories and files in _docs? What would be the order of the files? Alphabetical, I guess?

Alphabetical sounds good for me.

We create sidebar.yml file (probably we also need to rename it?)

Why do you want to rename it?

I just want to support multiple filenames because sidebar associates with just a navigation and in this case it decides what lands in static site or not.

(We can also add a option: hidden to render without entry in navigation)

What do you mean? I don’t understand this idea.

The hidden option in loading using YAML would be meant for pages that we want to have in static site but we don't want to have an entry in navigation for them. (If the page is not declared in YAML, it won't be rendered)

There's a setting directory that overrides this convention.

Could you please show an example?

Loading using YAML by default converts section title to the directory name that will contain the section. So for example:

- title: Some section
  subsection:
    ...

would be rendered inside some-section/ directory. Sometimes we want to override the directory name. To do so, we can define section as:

- title: Some section
  directory: custom-directory
  subsection:
    ...

which results in rendering this section in custom-directory/ directory.

With the -Yapi-subdirectory off (default), API documentation is in top-level and whole static site is inside _docs subdirectory. Otherwise, the static side goes to top-level and API is in api subdirectory.

Did you make any changes to -Yapi-subdirectory in this PR? I can’t see them.

There's an added line in StaticSiteContext which decides if the docs index lands into root directory or in the _docs subdirectory. Same logic is added in Locations which does the same for API.

julienrf · 2022-02-02T13:49:50Z

I just want to support multiple filenames because sidebar associates with just navigation and in this case it decides what lands in static site or not.

Do you mean that when there is a sidebar there is no way to produce a web page that is not linked from the sidebar? Or, it would be rendered only if we put hidden: true in its front matter? I like the idea of making it explicit to create those pages with a required property. Does the process fail if a page is present in the file system but not in the sidebar? Or is this file silently ignored? (BTW, I would call the property orphan instead of hidden, but YMMV)

pikinier20 · 2022-02-02T15:14:44Z

Does the process fail if a page is present in the file system but not in the sidebar? Or is this file silently ignored? (BTW, I would call the property orphan instead of hidden, but YMMV)

Currently, it silently ignores the file - actually, it just never checks any file that is not declared in YAML. We can change that to scan the directory and warn about files that exist but are not present in config

scaladoc/src/dotty/tools/scaladoc/site/SidebarParser.scala

scaladoc/src/dotty/tools/scaladoc/site/common.scala

julienrf · 2022-02-08T12:33:44Z

Reading again the specification:

We can also define a section with title and index but without defining children which will cause loading documents from the directory of index file

Does that overlap with the hidden property? It seems that it renders all the pages that are under the given index path, but without showing them in the side bar?

julienrf · 2022-02-08T12:42:33Z

I think this is going in a good direction, could you add some user documentation? Most probably the best place to do so for now would be here: https://github.com/scala/docs.scala-lang/tree/main/_overviews/scala3-scaladoc

pikinier20 · 2022-02-08T12:46:13Z

Reading again the specification:

We can also define a section with title and index but without defining children which will cause loading documents from the directory of index file

Does that overlap with the hidden property? It seems that it renders all the pages that are under the given index path, but without showing them in the side bar?

If we specify a section without children, we get all pages that are in the directory and we add them all to navbar. - It's a shortcut to people that just want to have all pages without writing long YAML files.

pikinier20 · 2022-02-08T14:27:33Z

I think this is going in a good direction, could you add some user documentation? Most probably the best place to do so for now would be here: https://github.com/scala/docs.scala-lang/tree/main/_overviews/scala3-scaladoc

scala/docs.scala-lang#2323

scaladoc/src/dotty/tools/scaladoc/site/SidebarParser.scala

pikinier20 · 2022-02-10T12:16:11Z

@julienrf I think I addressed all issues that you pointed out. We need there @BarkingBad to re-review the PR. In the meantime we can take a look at the entire work and apply some polishing if there's a need.

EDIT: Also, in terms of documentation on docs.scala-lang, I think we can merge these two PRs asynchronously. Maybe it will be even simpler to first merge the PR here and then finish documentation because we'll know that nothing will be changed.

julienrf

Nice!
Just to double-check, so how does the landing page look like now?

pikinier20 · 2022-02-11T08:52:32Z

Nice! Just to double-check, so how does the landing page look like now?

Right now, it's the main page of docs.

…ctory to the new implementation

… destination in sbt project

…h sidebar.yaml

Nicolapps · 2022-02-13T19:00:10Z

It looks to me that many URLs are broken by this change. For instance, https://dotty.epfl.ch/docs/usage/scaladoc/ is now located at https://dotty.epfl.ch/usage/scaladoc/.

Would it be possible to put redirections in place?

Also, it might be necessary to update links in the official Scala repositories (I can count 80 links in docs.scala-lang and 214 links in Dotty itself).

julienrf · 2022-02-13T20:08:25Z

My 2 cents on this is that the docs on dotty.epfl.ch are experimental (and updated every night). People should not expect much stability here.

On the other hand, the stable docs can be found on docs.scala-lang.org (although they are a bit outdated now). We are currently working on a process to publish the docs on docs.scala-lang.org from the content of the dotty.epfl.ch repository, but this is not ready yet.

Nicolapps · 2022-02-13T21:52:44Z

@julienrf Thank you for your reply. So, what should be done in the meantime for all the broken links in the official Scala repositories? Would a pull request fixing them be welcome?

julienrf · 2022-02-14T10:08:45Z

Hey Nicolas, that’s really nice of you to propose your help!
I think that should not be necessary because there is an ongoing PR to restore the old URLs: #14475

pikinier20 requested review from romanowski, julienrf and BarkingBad January 28, 2022 14:43

BarkingBad suggested changes Jan 28, 2022

View reviewed changes

anatoliykmetyuk assigned pikinier20 Jan 28, 2022

julienrf self-assigned this Jan 31, 2022

julienrf reviewed Feb 1, 2022

View reviewed changes

julienrf removed their assignment Feb 1, 2022

pikinier20 force-pushed the scaladoc/static-site-refactor2 branch from 5af46b0 to 1e0ea84 Compare February 8, 2022 11:23

pikinier20 requested review from BarkingBad and julienrf February 8, 2022 11:24

pikinier20 force-pushed the scaladoc/static-site-refactor2 branch 2 times, most recently from d5d0433 to e7c6003 Compare February 8, 2022 12:11

julienrf reviewed Feb 8, 2022

View reviewed changes

scaladoc/src/dotty/tools/scaladoc/site/SidebarParser.scala Show resolved Hide resolved

scaladoc/src/dotty/tools/scaladoc/site/SidebarParser.scala Outdated Show resolved Hide resolved

scaladoc/src/dotty/tools/scaladoc/site/common.scala Outdated Show resolved Hide resolved

pikinier20 force-pushed the scaladoc/static-site-refactor2 branch from e7c6003 to 513088c Compare February 8, 2022 13:04

julienrf reviewed Feb 8, 2022

View reviewed changes

scaladoc/src/dotty/tools/scaladoc/site/SidebarParser.scala Outdated Show resolved Hide resolved

pikinier20 force-pushed the scaladoc/static-site-refactor2 branch 2 times, most recently from 429f0d0 to 752f751 Compare February 10, 2022 12:10

julienrf approved these changes Feb 10, 2022

View reviewed changes

pikinier20 added 2 commits February 11, 2022 09:52

Refactor static-site loading

3642c11

Change static site directory structure. Adjust Dotty static site dire…

c85e4b0

…ctory to the new implementation

pikinier20 added 4 commits February 11, 2022 09:52

Refactor blog handling

a7f847e

Adjust testcases static site to the new structure. Change assets copy…

5a873e0

… destination in sbt project

CRs from review. Add hidden property to hide page configured throug…

2ef7753

…h sidebar.yaml

Change requests: Uniformize property names in YAML config

c5575b8

pikinier20 force-pushed the scaladoc/static-site-refactor2 branch from 752f751 to c5575b8 Compare February 11, 2022 08:53

BarkingBad approved these changes Feb 11, 2022

View reviewed changes

pikinier20 merged commit b472e26 into scala:main Feb 11, 2022

pikinier20 deleted the scaladoc/static-site-refactor2 branch February 11, 2022 10:29

griggt mentioned this pull request Feb 13, 2022

Dotty docs not found #14470

Closed

griggt mentioned this pull request Feb 15, 2022

CI failing with tons of dead dotty.epfl.ch links scala/scala-lang#1342

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scaladoc: Refactor static site loading and directory structure #14378

Scaladoc: Refactor static site loading and directory structure #14378

pikinier20 commented Jan 28, 2022 •

edited

BarkingBad Jan 28, 2022

BarkingBad Jan 28, 2022

BarkingBad Jan 28, 2022

BarkingBad Jan 28, 2022

BarkingBad Jan 28, 2022

julienrf Feb 1, 2022

BarkingBad Jan 28, 2022

julienrf left a comment

julienrf Feb 1, 2022

pikinier20 Feb 2, 2022

julienrf Feb 2, 2022

pikinier20 Feb 8, 2022

pikinier20 commented Feb 2, 2022

julienrf commented Feb 2, 2022

pikinier20 commented Feb 2, 2022

julienrf commented Feb 8, 2022

julienrf commented Feb 8, 2022

pikinier20 commented Feb 8, 2022

pikinier20 commented Feb 8, 2022

pikinier20 commented Feb 10, 2022 •

edited

julienrf left a comment

pikinier20 commented Feb 11, 2022

Nicolapps commented Feb 13, 2022

julienrf commented Feb 13, 2022

Nicolapps commented Feb 13, 2022

julienrf commented Feb 14, 2022

Scaladoc: Refactor static site loading and directory structure #14378

Scaladoc: Refactor static site loading and directory structure #14378

Conversation

pikinier20 commented Jan 28, 2022 • edited

Static site loading

Static site directory structure

The -Yapi-subdirectory setting

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienrf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pikinier20 commented Feb 2, 2022

julienrf commented Feb 2, 2022

pikinier20 commented Feb 2, 2022

julienrf commented Feb 8, 2022

julienrf commented Feb 8, 2022

pikinier20 commented Feb 8, 2022

pikinier20 commented Feb 8, 2022

pikinier20 commented Feb 10, 2022 • edited

julienrf left a comment

Choose a reason for hiding this comment

pikinier20 commented Feb 11, 2022

Nicolapps commented Feb 13, 2022

julienrf commented Feb 13, 2022

Nicolapps commented Feb 13, 2022

julienrf commented Feb 14, 2022

pikinier20 commented Jan 28, 2022 •

edited

The `-Yapi-subdirectory` setting

pikinier20 commented Feb 10, 2022 •

edited