Dokka K2 analysis

[vmishenev]

Compiler: Kotlin 1.9.0

The PR stems from #2995, but it's using a recent refactoring #3034.

---

The PR description is WIP.

Dokka has its own documentable model to represent analyzed code, The analysis is performed by a compiler frontend.

In K1 the compiler frontend has descriptors that use the underlying Binding Context (global shared stateful structure). Dokka just maps descriptors to Documentable by DefaultDescriptorToDocumentableTranslator.

K2 compiler has FIR tree, which means “Frontend Intermediate Representation”, instead of Binding Context. But we do not use FIR in Dokka directly, since it is too low-level for analysis. The Kotlin compiler provides high-level Analysis API for this case. The API is used by KSP too. Analysis API represent elements of FIR (declarations, parameters and so on) as Symbols. For more details see KtSymbolByFirBuilder, KtSymbol.
For Dokka symbol is the replacement of descriptors in K2.

The general data pipeline is:

flowchart LR

a(Source file)  --> b(Parse tree)  --> c(Raw FIR) --> d(Resolved FIR) -->  e(Symbols)  --> f(Documentable Model)

In code:

flowchart LR
a1(.kt files)  --> b2(KtElement)   -->  c2(FirElement)  -->   d2(FirElement)   --> |KtSymbolByFirBuilder|e2(KtSymbol)  --> |DefaultSymbolToDocumentableTranslator|f2(Documentable)

Also, to set up the environment of project analysis in K1 we use idea dependencies (or copy-past from there). In K2 for these aims there is Standalone Analysis API from Analysis API.

Overview

Main principles

• Do not use FIR directly, only via Analysis API.
• Keep compatibility with the Documentable model from K1. It does not break our and user plugins,
• Do not do double-checking. Dokka supposes that the output from the analysis API is correct. For example, we do not check the correctness of the annotation target.
• Do not keep symbols or KtAnalysisSession. Symbols are bounded by KtAnalysisSession (although KtAlwaysAccessibleLifetimeToken does not check their lifetime; a session caches them, and a session is cached by KtAnalysisSessionProvider). The documentable model keeps almost full information from symbols. Also, there are analysis principles, but Dokka does not have ReadAction unlike IDE.

Entry point

The main entry point is DefaultSymbolToDocumentableTranslator (this is an extension for the extension point ), that used by the Dokka core to build Documentable model by source set.

Across running Dokka we keep StandaloneAnalysisAPISession and KtSourceModule instances into AnalysisContex per source set. (See createAnalysisContext).
AnalysisContex is used in 'DefaultSymbolToDocumentableTranslator' and other services, that need additional analysis. (see classes in services package)

Differences from K1 in documentable model

• We do not put empty extra properties anymore (e.g. AdditionalModifiers, Annotations).
• KotlinModifier.Empty is used only for constructors
• If while translating an exception is thrown, Dokka rethrows it with TranslatorError that contains a file path where it has happened. In this case, It fails to generate documentation. (Or should Dokka skip a declaration with exception and only emit a warning in the logger?)

Current state

• Analysis API does not support MPP.
• Standalone API is a prototype. For now, it does not support libraries and JDK. See KT-60884 Currently, it supports only modular JDK.

Testing

Currently, K2 translator is tested on unit tests of base plugin. To run it, you can use symbolsTest task.
Some tests (currently, 37 from 843) in K2 analysis are muted by a special annotation (@OnlyDescriptors or @OnlyDescriptorsMPP):

• In K1 Documentable Java field does not have modality modifiers. In K2 they have open.
• Order of constructors and inheritors is different in K1 and K2.
• ... Todo

Plans

• Fix muted unit tests (fix in K1/fix in K2/have 2 sets of tests)
• Migrate to new Standalone
  • Update the compiler to 1.9.20
  • Use expect/actual from Symbols
• Add integration tests for only JVM project
• Support kotlin-as-java and javadoc plugins in K2
• Use K2 analysis on only JVM projects by default

[darthorimar]

I've only reviewed the Analysis API usage; I did not look at logic or Dokka-specific code

[darthorimar]

KtAnalysisSession is supposed to be used as a context receiver, this way you can always use your extension receiver :

context(KtAnalysisSession)
private fun KtSymbol.getFileLevelAnnotationsFrom() = ...

[IgnatBeresnev]

I believe removing this line will break 3rd party plugins, even for K1, as their tests will have no analysis implementation. It's worth checking or reverting it to play it safe.

[IgnatBeresnev]

Isn't this exception throw from the markdown module? Why is the depth different with K2? Maybe we need to re-wrap it somewhere to remove excessive layers?

[vmishenev]

Isn't this exception throw from the markdown module?

Yes

In K2 it is wrapped TranslatorError

[IgnatBeresnev]

Two questions:

1. Is this the only copy-pasted code from IDE?
2. Is this the only copy-pasted code in general, from anywhere? Maybe something was copy-pasted from the compiler or something?

[vmishenev]

1. Yes
2. No, there is some code in KotlinAnalysis, but it is related to the Standalone prototype.