Kotlin Analysis API - information and requests

[IgnatBeresnev]

Dokka 1.9.0 introduces some breaking changes to the plugin API - namely, the analysis part. It is expected that only a small fraction of Dokka plugins will get affected, so if your Dokka plugin compiles without any problems -- you may ignore this information.

This issue contains an explanation for the changes, and aims to collect feedback from affected plugins. This will allow us to provide a stable replacement for the removed API.

(if something got broken, but you are short on time and don't want to read the wall of text below, skip to the Feedback section)

What exactly is not available anymore?

• All of the K1 compiler types (such as descriptors), as well as IntelliJ platform classes -- basically, anything from the org.jetbrains.kotlin and com.intellij.psi packages. These classes should've never leaked into public API or as transitive dependencies.
• Anything from the org.jetbrains.dokka:dokka-analysis artifact, including classes like AnalysisEnvironment and DescriptorDocumentableSource / PsiDocumentableSource. This module should never have been used outside of Dokka itself, and all the declarations inside it should've been marked as internal.
• Some extension points in DokkaBase, like kotlinAnalysis, externalDocumentablesProvider and externalClasslikesTranslator - it is not possible to support them going forward due to the rest of the changes.
• Some extensions like DefaultDescriptorToDocumentableTranslator, DefaultSamplesTransformer and JavadocParser - it is also not possible to support them due to the rest of the changes.

Why isn't this API available anymore?

Because we refactored all of the analysis-related code in #3034, breaking things on purpose, but at the same time laying a solid foundation for the future to make sure it doesn't happen again.

Why was it necessary?

There are three major reasons:

First: The missing API was exposing Dokka's internal dependencies (kotlin-compiler, intellij-core, etc) that we have no control over. So if the maintainers of these dependencies chose to refactor everything overnight - not only would Dokka have big problems, but all of the downstream Dokka plugins that use these libraries transitively would also have problems.

Second: With the new K2 compiler in sight, the old K1 compiler will be deprecated and no longer supported at some point. This means that, sooner or later, everything would break anyway, and we'd rather do it in a controllable manner.

Third: Dokka itself needs to migrate the internal analysis to the K2 compiler. We could just replace all K1 classes with the corresponding K2 alternatives, but that would leave us with a long-lived branch that we would have to support until we verify that there's no regression (spoiler: there's quite a bit already). At the same time, it wouldn't make it any easier for downstream Dokka plugin now or in the future, should the K2's API also change in a major way (we know for a fact it will).

What's the solution?

To resolve/mitigate the issues outlined above and to make sure it doesn't happen again in the future (should K3 ever happen), we will no longer expose the underlying analysis dependencies/types to Dokka plugins, instead providing implementation-independent stable API. This way, you as an API consumer don't have to worry about what changes under the hood - the migration between K1 and K2, and different internal compiler refactorings will be our pain, not yours.

We will now provide a single artifact for all analysis-related API - analysis-kotlin-api. It will contain plain interfaces that you will be able to use to get various information about the source code, project and similar things. We will make sure this API is stable (i.e changes are backward-compatible, or an alternative is provided).

The actual implementations of these analysis API interfaces will be injected during runtime automatically, and they will adhere to the interface's specification/contract. Under the hood, Dokka will have two implementation - analysis-kotlin-descriptors (K1) and analysis-kotlin-symbols (K2), giving the users the ability to switch between K1 and K2 by toggling a flag. This also allows us to test for regressions and to migrate users gradually. Should K3 ever appear, it will be but a third implementation of the stable analysis API, and the migration to it should be practically unnoticeable for the downstream Dokka plugins.

Feedback

Long story short, at the moment there are no alternatives / replacements for the removed API, because the scope is too large and we don't actually know what API was used by which plugins and for what purposes.

If you are affected by these changes, and something doesn't compile or doesn't work anymore, please leave a comment with an explanation of what exactly you are doing (preferably with source code links), why you need it, and what a good alternative would look like in your case.

We will use this information to design thought-out interfaces for everyone to use, or at the very least try to help you with finding an alternative solution.

[ftomassetti]

We are using this API to extract comments from Kotlin code and generate documentation for a system of hours (combining different sources: Kotlin comments are just one of them). Given we have lost this API we cannot migrate to newer versions of Kotlin. This is a snippet of some code using the API removed:

    private fun buildAST(file: File): AST {
        val sourceSet =
            DokkaSourceSetImpl(
                displayName = "ast",
                sourceSetID = DokkaSourceSetID("ast", "ast"),
                sourceRoots = setOf(file.canonicalFile),
            )
        val dokkaContext =
            DokkaContext.create(
                DokkaConfigurationImpl(sourceSets = listOf(sourceSet)),
                DokkaConsoleLogger(),
                listOf(),
            )
        val translator = DefaultDescriptorToDocumentableTranslator(dokkaContext)
        val module = translator.invoke(sourceSet, dokkaContext)
        return AST(module, sourceSet)
    }

Successively we were getting the comments in this way:

    private fun recordDocumentation(ast: AST) {
        ast.module.packages.forEach { pkg ->
            val documentationNode = pkg.documentation[ast.sourceSet]
            documentationNode?.let { commentsByFqn[pkg.packageName] = it }
            pkg.classlikes.forEach { cls ->
                val classDoc = cls.documentation[ast.sourceSet]
                val name = cls.name
                classDoc?.let {
                    name?.let { commentsByFqn[pkg.packageName + "." + name] = classDoc }
                }
                name?.let {
                    cls.properties.forEach { property ->
                        val propertyDoc = property.documentation[ast.sourceSet]
                        propertyDoc?.let {
                            commentsByFqn[keyForProperty(pkg.packageName, name, property.name)] = propertyDoc
                        }
                    }
                }
            }
        }
    }

Any suggestion on how to replace this?

[vmishenev]

DefaultDescriptorToDocumentableTranslator is directly unavailable after the refactoring. Although you can use it via an interface of extension point. In this case it is CoreExtensions.sourceToDocumentableTranslator
Also, see the developer guide: sourceToDocumentableTranslator.
In your example the code should be like:

val translators = dokkaContext[CoreExtensions.sourceToDocumentableTranslator]

By default, in the configuration provided by Maven or Gradle plugins CoreExtensions.sourceToDocumentableTranslator contains two translators: Java and Kotlin.

Also, pay the attention that your DokkaConfigurationImpl .pluginsClasspath provides no plugins.
You need to provide paths at least to the artifacts analysis-kotlin-api and analysis-kotlin-descriptors (analysis-kotlin-symbols )
Note: they also can be loaded via ServiceLoader without configuring if a runtime classpath contains them.