build config: Use buildSrc convention plugins to configure the Dokka subprojects #2704

aSemy · 2022-10-11T17:58:21Z

This is a large part of the implementation for #2703

Status: ready for review

set up buildSrc convention plugins
move shared configuration into the convention plugins
tidy up the subproject build files to remove duplicated configuration
set up centralized repo definitions

Important to note:

This should have no impact on the actual source code or published artifacts, this is only updating the build config
All Gradle plugins must be defined in ./buildSrc/build.gradle.kts, so they are aligned across all subprojects

This PR duplicates the change proposed in Move plugin version management to buildSrc/build.gradle.kts #1321

aSemy · 2022-10-11T17:59:26Z

I made some good progress but then I got stuck when I reached the integration test config, because it's a bit of a beast.

https://github.com/Kotlin/dokka/blob/83174361becb2af227159834cdf6e14db9300c53/integration-tests/build.gradle.kts

I think most of the config in here should be converted to a specific 'integration test' buildSrc plugin.

aSemy · 2022-10-12T20:44:27Z

@IgnatBeresnev I see you've added the 'Gradle Plugin' label, and while this PR would help with that, it's more about the improving the general Gradle build config of the entire project :)

IgnatBeresnev · 2022-10-12T20:57:00Z

Yeah, true. I don't want to create too many labels, but I definitely want to mark all Gradle-related PRs in case we ask for anyone's help with review and whatnot

If it makes anyone uneasy, we can certainly create a separate one or rename the existing. We'll discuss it with the team this week, just in case

aSemy · 2022-10-13T11:36:14Z

With regards to labels, what about a 'build config' or 'ci/cd' label that would cover the build scripts, and also GitHub Actions, Qodana, TeamCity etc?

…ntion_plugins # Conflicts: # build.gradle.kts

aSemy · 2022-10-13T20:47:54Z

I've made some good progress! All shared build-config now comes from buildSrc convention plugins, not from subprojects{} or allprojects{} blocks. This makes the build much more modular, because it's not restricted by inheritance, and easier to configure because a build script is more self contained - it doesn't have config applied from some arbitrary parent.

…subprojects

plugins/android-documentation/build.gradle.kts

…ption, which is specific to the convention plugin, not for implementations)

(kotlin-dsl plugin will auto-add dependencies)

…ntion_plugins # Conflicts: # runners/gradle-plugin/build.gradle.kts

IgnatBeresnev · 2023-03-07T15:21:44Z

Hi! Not sure if you noticed that the integration tests for coroutines are unable to be run because of the toolchain being set to 1.8 globally. This refactoring has uncovered quite a number of problems already, which is good, I guess :)

A problem occurred evaluating root project 'kotlinx.coroutines'.
> Failed to apply plugin 'jdk-convention'.
> Project required JDK 11+, but found 1.8

Can you please fix it when you have the time, so that the checks pass? I believe that's the last thing holding up the merge

Not sure what the best solution for the fix is, but I'd say any that builds the same bytecode as before and makes the tests pass. We can revisit our requirements for the used java/kotlin versions later and separately

aSemy · 2023-03-07T15:42:53Z

thanks for checking @IgnatBeresnev, I didn't realise that was the problem.

I set the toolchain for Dokka to use Java 8 because that matches the existing jvmTarget

dokka/build.gradle.kts

Line 33 in ded804e

jvmTarget = "1.8"

If I set the toolchain to 11, then the :runners:maven-plugin build fails:

> Task :runners:maven-plugin:helpMojo
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building dokka-maven-plugin 1.8.20-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-plugin-plugin:3.5.2:helpmojo (default-cli) @ dokka-maven-plugin ---
> Task :plugins:base:frontend:nodeSetup
> Task :plugins:base:frontend:npmSetup SKIPPED
> Task :runners:maven-plugin:helpMojo
[WARNING] Using platform encoding (UTF-8 actually) to read mojo source files, i.e. build is platform dependent!
[INFO] java-javadoc mojo extractor found 0 mojo descriptor.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.425 s
[INFO] Finished at: 2023-03-07T16:44:34+01:00
[INFO] Final Memory: 12M/68M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo (default-cli) on project dokka-maven-plugin: Execution default-cli of goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo failed: Unsupported class file major version 55 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo (default-cli) on project dokka-maven-plugin: Execution default-cli of goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo failed: Unsupported class file major version 55

And there's a warning for :runners:gradle-plugin

> Task :runners:gradle-plugin:compileTestKotlin
'compileTestJava' task (current target is 11) and 'compileTestKotlin' task (current target is 1.8) jvm target compatibility should be set to the same Java version.
By default will become an error since Gradle 8.0+! Read more: https://kotl.in/gradle/jvm/target-validation
Consider using JVM toolchain: https://kotl.in/gradle/jvm/toolchain

It looks like kotlinx.coroutines has a custom check for the JDK version...

https://github.com/Kotlin/kotlinx.coroutines/blob/4116d4a178f32b1241db8247a38cd2823fb8b03e/buildSrc/src/main/kotlin/jdk-convention.gradle.kts#L7-L14

I'll look into it a little more. Perhaps the kotlinx.coroutines int. test requires a little more config.

# Conflicts: # build.gradle.kts Merge pull request Kotlin#3582 from isaialvarado/patch-1 Fix example code Converted Serialization formats list to table PR Kotlin#2010 Updating docs after 7.0.0 release

IgnatBeresnev · 2023-03-07T16:07:32Z

I set the toolchain for Dokka to use Java 8 because that matches the existing jvmTarget

Yeah, that's the most important part - we cannot break compatibility with Java 8 for now.

We can set jvmTarget to 8 in the kotlin convention + the gradle plugin, but run the whole project with java 11 - I don't mind that. Adding the toolchain is definitely a good idea, even if jvmTarget stays at 8

But whatever works is fine :) We can revisit the used versions later

Tell me once it's ready to be looked at, I'll trigger the teamcity build

aSemy · 2023-03-07T16:12:25Z

@IgnatBeresnev Do you know why the jvm-version is set to 12 in the example workflows?

dokka/.github/workflows/gh-actions-artifacts-snapshots.yml

Line 21 in 2e78b7b

java-version: 12

aSemy · 2023-03-07T16:29:42Z

We can set jvmTarget to 8 in the kotlin convention + the gradle plugin, but run the whole project with java 11 - I don't mind that. Adding the toolchain is definitely a good idea, even if jvmTarget stays at 8

Toolchains don't let you separately define a language version; the JVM target is set to that of the Java version of the Toolchain. So if the project needs to target Java 8, the Toolchain needs to be Java 8. Don't ask me why jvmTarget can't be set separately...

Fortunately Toolchains are very flexible, so I set the integrationTest task launcher to be Java 11.

1bbbd9b

@IgnatBeresnev Tests have passed, it's ready for the TeamCity build!

IgnatBeresnev · 2023-03-07T17:02:51Z

@IgnatBeresnev Do you know why the jvm-version is set to 12 in the example workflows?

It was copied from s3-snapshots.yml, but why it's set there to 12 - zero clue :( I think it's fine to set it to 11 like you did

@IgnatBeresnev Tests have passed, it's ready for the TeamCity build!

Thank you once again! Triggered the builds, will merge once they pass :)

aSemy · 2023-03-08T10:30:50Z

@IgnatBeresnev By the way, good job on the GitHub labels! They look really smart, and they're helpful.

IgnatBeresnev · 2023-03-08T13:47:30Z

@aSemy yeah, going through all of the issues :) Part of the reason is actually for the dokkatoo DSL. In particular, the configuration and the Gradle plugin tags could be of interest to go along with the re-write since we're breaking compatibility anyway

Goooler · 2023-03-13T08:18:52Z

buildSrc/src/main/kotlin/org/jetbrains/conventions/base-java.gradle.kts

+java {
+    toolchain {
+        languageVersion.set(JavaLanguageVersion.of(8))
+    }
+}


If we apply toolchain for all modules, they'll run tests on Java 8 instead of the JDK matrix on CI, we need to apply more toolchains in Test which will slow down the build, I believe we should not use toolchain for now. Check

Configure languageVersion via toolchain detekt/detekt#5735

dokka/.github/workflows/gradle-test.pr.yml

Lines 12 to 13 in f5581c5

matrix:

version: [ 8, 11, 17 ]

Good catch!

Toolchains can still help with this though, because it's possible to set them per-task. I did something similar for MockK - I'll put together a PR.

Yeah, I used it, but it's slow, without parallelism, check

Configure languageVersion via toolchain detekt/detekt#5735

Run tests parallelly on CI pinterest/ktlint#1787

Ah right, yes that would be slower. I agree, using GitHub parallel workflows is best, not multiple Gradle tasks.

ilya-g · 2023-04-06T19:27:56Z

Hi @aSemy.
I've noticed that after this change the artifact dokka-analysis started publishing an additional shadowRuntimeElements variant with the following attributes:

    {
      "name": "shadowRuntimeElements",
      "attributes": {
        "org.gradle.category": "library",
        "org.gradle.dependency.bundling": "shadowed",
        "org.gradle.libraryelements": "jar",
        "org.gradle.usage": "java-runtime"
      },
      "files": [
        {
          "name": "dokka-analysis-1.8.20-local-1-all.jar",
          "url": "dokka-analysis-1.8.20-local-1-all.jar",
          "size": 100899229,
          ...
        }
      ]
    }

This is unfortunate, because it causes variant resolution ambiguity with the runtimeElements variant in ...Plugin configurations of manually created DokkaTaskPartial tasks in the stdlib docs build.

Is this an expected consequence of this change?

if no, could you investigate why it has happened and how to stop publishing this variant?
if yes, could you provide a guidance how to overcome this variant ambiguity in a project?

aSemy · 2023-04-06T20:00:25Z

I will take a look! I'm not very familiar with the Shadow plugin, but it looks like there are a few issues relating to shadowRuntimeElements https://github.com/johnrengelman/shadow/issues?q=shadowRuntimeElements

What might be a quick fix is disabling publication of this specific configuration, with a similar way to how Test Fixtures publication can be disabled.

val javaComponent = components["java"] as AdhocComponentWithVariants
javaComponent.withVariantsFromConfiguration(configurations["testFixturesApiElements"]) { skip() }
javaComponent.withVariantsFromConfiguration(configurations["testFixturesRuntimeElements"]) { skip() }

try setting up buildSrc convention plugins

3f77497

aSemy marked this pull request as draft October 11, 2022 17:58

IgnatBeresnev added the runner: Gradle plugin An issue/PR related to Dokka's Gradle plugin label Oct 12, 2022

aSemy added 8 commits October 13, 2022 21:40

finish creating buildSrc plugins

b591e34

Merge remote-tracking branch 'origin/master' into feat/buildsrc_conve…

16657a2

…ntion_plugins # Conflicts: # build.gradle.kts

create MavenProjectLocal helper

a8abe34

dokka-gradle-plugin: warn on duplicate resources

2dd31cf

dokka-gradle-plugin: fix kotlin-dsl compatibility

58bc1c8

disable empty subproject :plugins:base:search-component

89a504f

formatting

7a1a430

re-add dependencies that were in :dokka:plugins subproject {} block

968a123

re-add project(":integration-tests") dependency to integration-tests …

63a7da4

…subprojects

aSemy commented Oct 13, 2022

View reviewed changes

plugins/android-documentation/build.gradle.kts Show resolved Hide resolved

aSemy added 4 commits October 13, 2022 23:16

fix dokka-integration-test integrationTest configurations

766af55

replace println with logger.lifecycle

770a27b

update description for base plugin (else it provides a default descri…

ae0f4e1

…ption, which is specific to the convention plugin, not for implementations)

rm commented out code

bba37c7

aSemy marked this pull request as ready for review October 18, 2022 10:08

aSemy added 4 commits October 18, 2022 12:31

update test code after applying kotlin-dsl plugin

549c11c

tidy up gradle-plugin build.gradle.kts

9bfedff

(kotlin-dsl plugin will auto-add dependencies)

add repositoriesMode

54e68e3

move repositories to settings.gradle.kts

a1cfab1

aSemy marked this pull request as draft October 18, 2022 10:42

Declare the Node.js & Yarn download repositories

d2e05d0

aSemy marked this pull request as ready for review October 18, 2022 11:15

aSemy added 6 commits February 28, 2023 22:46

tidy up compiler args

7c49e09

restore suppress-version-warnings

27c639c

Merge remote-tracking branch 'origin/master' into feat/buildsrc_conve…

bccbdd7

…ntion_plugins # Conflicts: # runners/gradle-plugin/build.gradle.kts

remove unused import

7212695

update API dump

f1dcf92

remove unnecessary MavenPublishPlugin check

8fb0319

aSemy added 5 commits March 7, 2023 16:46

centralise repos in buildSrc

4024f36

set jvmToolchain in buildSrc

ff45720

Merge branch 'master' into feat/buildsrc_convention_plugins

2e78b7b

# Conflicts: # build.gradle.kts Merge pull request Kotlin#3582 from isaialvarado/patch-1 Fix example code Converted Serialization formats list to table PR Kotlin#2010 Updating docs after 7.0.0 release

set test Java Launcher toolchain to 11

1bbbd9b

rm unnecessary task (clean already deletes buildDir contents)

d06036f

change java-version from 12 to 11 in snapshot workflows

ac58931

IgnatBeresnev merged commit 62c9807 into Kotlin:master Mar 7, 2023

aSemy mentioned this pull request Mar 8, 2023

Gradle build config: Replace hard dependency on JVM 11 with Java Toolchain Kotlin/kotlinx.coroutines#3665

Closed

aSemy mentioned this pull request Mar 8, 2023

Update build config to be more modular, using buildSrc convention plugins #2703

Open

Goooler reviewed Mar 13, 2023

View reviewed changes

aSemy mentioned this pull request Mar 13, 2023

Regression: All versions of Java are not tested in CI/CD #2917

Closed

aSemy deleted the feat/buildsrc_convention_plugins branch March 13, 2023 10:31

aSemy mentioned this pull request Mar 16, 2023

Fix test runtime Java versions #2918

Merged

4 tasks

aSemy mentioned this pull request Apr 16, 2023

disable publication of shadowRuntimeElements variant #2961

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build config: Use buildSrc convention plugins to configure the Dokka subprojects #2704

build config: Use buildSrc convention plugins to configure the Dokka subprojects #2704

aSemy commented Oct 11, 2022 •

edited

aSemy commented Oct 11, 2022 •

edited

aSemy commented Oct 12, 2022 •

edited

IgnatBeresnev commented Oct 12, 2022 •

edited

aSemy commented Oct 13, 2022

aSemy commented Oct 13, 2022

IgnatBeresnev commented Mar 7, 2023 •

edited

aSemy commented Mar 7, 2023 •

edited

IgnatBeresnev commented Mar 7, 2023 •

edited

aSemy commented Mar 7, 2023

aSemy commented Mar 7, 2023

IgnatBeresnev commented Mar 7, 2023 •

edited

aSemy commented Mar 8, 2023

IgnatBeresnev commented Mar 8, 2023

Goooler Mar 13, 2023

aSemy Mar 13, 2023

Goooler Mar 13, 2023 •

edited

aSemy Mar 13, 2023

ilya-g commented Apr 6, 2023

aSemy commented Apr 6, 2023 •

edited

build config: Use buildSrc convention plugins to configure the Dokka subprojects #2704

build config: Use buildSrc convention plugins to configure the Dokka subprojects #2704

Conversation

aSemy commented Oct 11, 2022 • edited

aSemy commented Oct 11, 2022 • edited

aSemy commented Oct 12, 2022 • edited

IgnatBeresnev commented Oct 12, 2022 • edited

aSemy commented Oct 13, 2022

aSemy commented Oct 13, 2022

IgnatBeresnev commented Mar 7, 2023 • edited

aSemy commented Mar 7, 2023 • edited

IgnatBeresnev commented Mar 7, 2023 • edited

aSemy commented Mar 7, 2023

aSemy commented Mar 7, 2023

IgnatBeresnev commented Mar 7, 2023 • edited

aSemy commented Mar 8, 2023

IgnatBeresnev commented Mar 8, 2023

Goooler Mar 13, 2023

Choose a reason for hiding this comment

aSemy Mar 13, 2023

Choose a reason for hiding this comment

Goooler Mar 13, 2023 • edited

Choose a reason for hiding this comment

aSemy Mar 13, 2023

Choose a reason for hiding this comment

ilya-g commented Apr 6, 2023

aSemy commented Apr 6, 2023 • edited

aSemy commented Oct 11, 2022 •

edited

aSemy commented Oct 11, 2022 •

edited

aSemy commented Oct 12, 2022 •

edited

IgnatBeresnev commented Oct 12, 2022 •

edited

IgnatBeresnev commented Mar 7, 2023 •

edited

aSemy commented Mar 7, 2023 •

edited

IgnatBeresnev commented Mar 7, 2023 •

edited

IgnatBeresnev commented Mar 7, 2023 •

edited

Goooler Mar 13, 2023 •

edited

aSemy commented Apr 6, 2023 •

edited