Slow documentation generation

[soywiz]

Describe the bug
I have noticed that the generation is really slow. Then I have investigated thru VisualVM, and noticed that it is expending most of the time with HTTP requests. Why is that? What is it downloading?

Expected behaviour
To be much faster. Do not download anything, since I'm generating a local documentation from local source code.

Screenshots

To Reproduce

git clone https://github.com/korlibs/korge.git
./gradlew dokkaHtmlMultiModule

Dokka configuration

Default configuration.

Installation

• Operating system: macOS
• Dokka version: 1.7.10

[IgnatBeresnev]

Hi! Thanks for your investigation!

Performance issues have indeed been overlooked for a long time, but we do plan to spend some time profiling and optimizating before Dokka goes into stable, which should be relatively soon

As for your particular case, I might be wrong, but I would assume it's downloading package lists for external documentation links. By default it includes links to Kotlin's stdlib, Java's JDK, Android sdk and androidx references as can be seen here.

You can try setting noJdkLink, noStdlibLink and noAndroidSdkLink to false to verify it, but then links to types inherited from stdlib or jdk will not work (i.e they won't be links anymore).

I haven't really dug into it deep, but given how these links end up being java.net.URL, maybe these downloaded package lists could be cached locally (or through some shared index like in IDEA) and then passed as file:///whatever to externalDocumentationLinks configuration property

[soywiz]

In my case offline mode was a solution: https://github.com/korlibs/korge/blob/d296abdc2657cd3680efdaa573a3ae38eedb8ea3/build.gradle.kts#L165-L167

But indeed I have the feeling that it is downloading too many things, because it expends several minutes downloading, and now sure why.

---

BTW also a performance thing (I can open another ticket):

The other thing I noticed is that in Chrome after a few navigations after some time, the performance starts degrading while navigating thru the documentation: https://dokka.korge.org/
It seems that the navigation is setting the html content of the toc everytime the page loads. It seems that Chrome is degenerating not sure why, because firefox and safari seems to handle it reasonably well.

Maybe at some point dokka could use the History API: https://developer.mozilla.org/en-US/docs/Web/API/History_API , when clicking internal links do not fully load the new page but only the content, so the Table of Contents is kept and only updated. In my case the TOC is really huge, so avoiding setting it everytime would be much better in terms of battery.

Alternatively the TOC could be constructed dynamically (it can load the toc html as it is, cache them in the local storage based on the contents hash, and put nodes on demand when opening sections). By profiling I noticed that the problem was when setting the html of the content, and then the browser re-rendering everything (tried to disabled setting the html of the table of contents and fixed the performance issue). Also chrome pseudo-hangs while closing the tab with the korlibs dokka documentation.

Screen.Recording.2022-10-10.at.11.47.29.mov

[IgnatBeresnev]

Just... wow. I've just tried it out in chrome and it loads considerably faster, but I can still see the lag - probably because I'm on a beefy desktop machine

Anyhow, TOC is indeed a problem, but more for the fact that it's annoyingly resetting position every time. There was an attempt at re-writing navigation to react, which should hopefully solve both problems, but it was left unfinished. Hopefully we'll get back to it some day in the future

In the meantime, maybe we can indeed come up with something to mitigate the slow loading problem.

Could you please create a separate issue for that so that it doesn't get lost? Documentation you generated for korlibs and korge (which are amazing btw) might very well be the largest we've seen yet in terms of the amount of classes in ToC

[soywiz]

Thanks! The libraries have indeed a lot of code; before I had separate dokka documentations, but wanted to have them all together for convenience.
I have already created a separate issue for the navigation performance thing

[IgnatBeresnev]

The initial problem with slow network calls seems to have been resolved. I outlined this problem and possible solutions in new documentation, so it should hopefully clear things up.

If you set offlineMode to true, but still want Dokka to generate external links to types from the standard library or JDK (or some other dependency), you can try caching package-lists locally and provide them as file URLs:

import org.jetbrains.dokka.gradle.DokkaTask

tasks.withType<DokkaTask>().configureEach {
    dokkaSourceSets.configureEach {
        externalDocumentationLink {
            packageListUrl.set(
                rootProject.projectDir.resolve("serialization.package.list").toURL()
            )
        }
    }
}

---

As for the rest of the slowness, I think it's best to submit individual issues for specific problems. We already have #2699 and #2741 to begin with. So I'll close this issue for the lack of Definition of Done, but do let us know if you encounter anything else!