The Kotlin and Groovy DSLs should have similar performance characteristics

[eskatos]

Expected Behavior

The Kotlin and Groovy DSLs have as much as possible similar performance characteristics.
IOW there's no outlier use case for which the performance characteristics make using one or the other DSL a penalty.

Current Behavior

What follows is the current state of affairs as Gradle 6.8 for four use cases:

• first use (e.g. ephemeral CI agent or clean workstation without build cache)
• buildSrc abi change
• buildSrc non-abi change
• nothing changes

The data below comes from performance tests run on the Gradle CI. Those tests use a build with a large number of projects and build scripts, the build scripts being rather simple, and they run on both the Groovy and Kotlin DSL for comparison.

Use case	Groovy	Kotlin	Differences
First use	🟢 33.5s	🔴 76.2s	Groovy DSL is 2.2x faster
buildSrc abi change	🟢 13.2s	🔴 42.3s	Groovy DSL is 3.2x faster
buildSrc non-abi change	🔴 13s	🟢 5.2s	Kotlin DSL is 2.5x faster
Nothing changes	🔵 1.7s	🔵 1.8s	Similar performance

Context

The numbers above are, at the time of writing, for current release branch of Gradle that is the upcoming 6.8.1. See:

• https://builds.gradle.org/buildConfiguration/Gradle_Check_PerformanceTestSlowLinux_Trigger/40748694?buildTab=report_project941_Performance
• https://builds.gradle.org/buildConfiguration/Gradle_Check_PerformanceTestTestLinux_Trigger/40757415?buildTab=report_project944_Performance

Use Login as guest and note that these links will eventually get outdated.

This issue is a follow up to gradle/kotlin-dsl-samples#902 which resides in an archived repository.

One of the reasons the Kotlin DSL is slower when it is is the time it takes to the Kotlin compiler to compile the scripts.
See https://youtrack.jetbrains.com/issue/KT-24668

[kaushalyap]

thanks for the benchmark @eskatos

Any issues filed on the youtrack issue tracker against Kotlin compiler related to kotlin dsl performance? if so please link the associated issues.

[LifeIsStrange]

random thoughts from a lambda Kotlin dev:

It could be a useful addition to add a variant for kotlin and groovy on this benchmark:
That could be called e.g Kotlin-optimized and that would enable non-default gradle.properties such as

org.gradle.vfs.watch=true
kapt.use.worker.api=true
kapt.include.compile.classpath=false
org.gradle.parallel=true
org.gradle.caching=true

(btw its unclear whether kapt.include.compile.classpath=false is still useful in 6.8 given compiler avoidance)

Also worth exploring would be to benchmark with the new compiler backend and the new compiler frontend

finally setting a continuous benchmark (in CI) of this would be useful for showing progress and for catching regressions!

note1: I cannot see the linked benchmark results, Login as guest doesn't seems to work (I don't have an account)
note2: reading the guide on how to improve gradle performance:

The Gradle Java plugin allows you to run the compiler as a separate process by using the following configuration for any JavaCompile task:
This process is reused for the duration of a build, so the forking overhead is minimal. The benefit of forking is that the memory-intensive compilation happens in a different process, leading to much less garbage collection in the main Gradle daemon. Less garbage collection in the daemon means that Gradle’s infrastructure can run faster, especially if you are also using --parallel.
It’s unlikely to be useful for small projects, but you should definitely consider it if a single task is compiling close to a thousand or more source files together.

https://docs.gradle.org/nightly/userguide/performance.html#compiler_daemon
This optimization doesn't seems to be provided for Kotlin currently

note3: minor: I wonder whether using the Kotlin BOM incurs a slowdown vs not using it

[gildor]

maybe worth adding one more case, change build.gradle file (not src), I think it even more common comparing to buildSrc

[eskatos]

Guest access to our CI is currently temporarily disabled, it'll be enabled back soon.

@kaushalyap there's already a link to https://youtrack.jetbrains.com/issue/KT-24668. I'm not aware of other Youtrack issues related to this.

@LifeIsStrange this issue is about the Gradle Kotlin DSL performance, not the Kotlin Gradle Plugin performance in general. For instance KAPT is out of the picture. Also, the embedded Kotlin compiler used internally by the Gradle Kotlin DSL isn't ready to be moved to the new compiler infrastructure. The numbers above represent the default Gradle user experience which is a good thing.

@gildor yes this use case is missing from the set above and I agree it's very common. We started with the buildSrc use case because it potentially invalidates a lot of things and shows the worst situation.

[kaushalyap]

@kaushalyap there's already a link to https://youtrack.jetbrains.com/issue/KT-24668. I'm not aware of other Youtrack issues related to this.

@eskatos I know about that issue. Hope Kotlin team close that issue in the first half of this year with the movement to new compiler as mentioned in the roadmap, so android devs can move to kotlin dsl for android projects without increased build time.

[gildor]

@kaushalyap This is the new backend of the compiler, it has nothing to do with compilation speed. You probably talk about the new frontend of the compiler, which promises up to 2 times faster compilation, but it will not be ready in 6 months, in the best case it would be some experimental version available this year

[cristan]

I wonder if this is still an issue. Gradle 7 has been out for a while and it boosts massive performance improvements for Kotlin DSL script compilation. An updated benchmark would be very appreciated :).

[eskatos]

Those benchmarks are automated and constantly run against Gradle as it evolves. You can browse the results on our CI, e.g. here https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger/46861573?buildTab=report_project55641_Performance

Here I captured the current state:

Use case	Groovy	Kotlin	Differences
First use	🟢 34s	🔴 57.5s	Groovy DSL is 1.7x faster
buildSrc abi change	🟢 20.5s	🔴 32s	Groovy DSL is 1.6x faster
buildSrc non-abi change	🔴 21s	🟢 4.3s	Kotlin DSL is 4.8x faster

Things indeed got better since Gradle 7, especially the non-abi change scenarios. The performance isn't similar though.
The Groovy DSL got slower in general since Groovy 3.
The Kotlin DSL is still almost twice slower in first use and abi-change scenarios.
The Groovy DSL is now much slower than the Kotlin DSL in a non-abi change scenario thanks to compilation avoidance introduced in the Kotlin DSL. This can be added to the Groovy DSL too to make it faster in such scenarios, see #14829.

[kaushalyap]

@eskatos Thanks for the update, I wonder what would be the build times for composite builds for the same project?

There is an issue reporting converting buildSrc to implicit included.

[rowi1de]

Could you explain what ABI / Nop-ABI means 🙈

[eskatos]

@rowi1de sure thing!

ABI stands for Application Binary Interface.

An ABI change is a change that modifies the ABI, iow the "public api" that downstream consumers compile against. All consumers need to be recompiled.

A non-ABI change is a change that does not modify the ABI, iow only touches the "private implementation". Consumers compilation can be skipped.

[blundell]

@eskatos clicked your like this one and after signing in a guest I get:

Build with id "46861573" does not exist

( was hoping to check the latest benchmarks :) )

[eskatos]

@blundell that individual run got garbage collected, here you can see the list of runs https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger?branch=master&buildTypeTab=overview&mode=builds

[blundell]

I think I've done this right :D

Gradle 7.4/7.5 numbers
(buildSrc = 7.4, First use = 7.5)

Use Case	Groovy	Kotlin	Difference
First use	🟢 38.855s	🔴 63.54s	Groovy DSL is 1.6x faster
buildSrc abi change	🟢 25.307	🔴 35.014s	Groovy DSL is 1.4x faster
buildSrc non-abi change	🔴 24.526s	🟢 4.732s	Kotlin DSL is 5x faster

Info from 11th Feb 2022 CI run

For example, from the performance tab, I looked up the buildSrc abi change and took the first row, here.
Kotlin:

Groovy:

and the first use numbers from here.
Groovy:

Kotlin:

[eugene-krivobokov]

@blundell , could you please explain the difference calculations?

In the previous results: 21s VS 4.3s - 4.8x faster. I get it.
In the last results: 24.526s VS 4.732s - 135% faster. How have these percents been calculated?

[blundell]

@eugene-krivobokov Sorry about that, wrong equation. Updated to calculate as a percentage.

[mochadwi]

@blundell that individual run got garbage collected, here you can see the list of runs https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger?branch=master&buildTypeTab=overview&mode=builds

does it private? seems unable to open with my team city account

[liutikas]

Is there any sort of break down on what goes in "first use" and why it is so much slower?

[stigkj]

@liutikas I guess "first use" is slower with Kotlin DSL as that needs to be compiled.

[eskatos]

Here are numbers updated for Gradle 8.0

Use case	Groovy	Kotlin	Differences
First use	🟢 30s	🔴 75s	Groovy DSL is 2.5x faster
buildSrc abi change	🟢 9s	🔴 35s	Groovy DSL is 3.8x faster
buildSrc non-abi change	🔴 9s	🟢 5.5s	Kotlin DSL is 1.7x faster
configuration	🟢 1.9s	🔴 2.2s	Groovy DSL is 0.15x faster

This is not looking great. The upgrade to Kotlin 1.8 in Gradle 8.0 caused a regression in Kotlin DSL script compilation. Also those scenarios are not the best at representing reality as they don't even use the plugins {} block for which Gradle 8.0 can provide a 20% performance boost. This would not make the overall build times faster than with the Groovy DSL though.

The gap is not just the Kotlin compiler performance change. Groovy got faster. There are also some changes due to other areas of Gradle at play. But there definitely was a performance regression with 1.8 in our isolated tests. We are discussing this with the compiler team and will try to mitigate.

[Jacks0N23]

@eskatos could you please check gradle 8.1 performance

[little-fish]

8.2 is out with Kotlin DSL is Now the Default for New Gradle Builds. I really would like to know whether there are any performance improvements related to previous versions. Is there any chance to do some testing against the latest Gradle release? Thanks.

[Brankale]

I haven’t run any benchmark on the latest release but looking at the Gradle’s public roadmap, I think we need to wait Gradle 8.4 at least for first-use performance improvements

[orubel]

Since Groovy is OBVIOUSLY faster and a more mature lang, any chance there will be a reversion back?

NOTE: I notice the thumbs down so am confused as people want speed but also want 'Kotlin'??? You can't complain about speeds and then reject the one thing that gave you that speed. This is a 'cake and eat it' situation folks

[rakshitsoni02]

thanks for sharing the insights, Since Google is the default kotlin for builds, curious to know about performance with AGP 8.1 onwards before I make a move & update the whole project which has at least 50 modules & Build performance is a higher priority for big team, i tried in older version which was very slow
cc @eskatos

[illarionov]

It would be nice to see updated numbers for Gradle 8.7 given the changes in #27271.

[eskatos]

Here are numbers updated for Gradle 8.7

Use case	Groovy	Kotlin	Differences
First use	🟢 32s	🔴 76s	Groovy DSL is 2.4x faster
buildSrc abi change	🟢 10s	🔴 33s	Groovy DSL is 3.3x faster
buildSrc non-abi change	🔴 10s	🟢 5.3s	Kotlin DSL is 1.9x faster
configuration	🟢 2s	🔴 2.2s	Groovy DSL is 0.11x faster

Things have not changed much overall since 8.0. The gap between the two DSLs closed a tiny bit.
Again, note that these numbers are from synthetic benchmarks.

[orubel]

Lol. soooo... now would be the time a sensible person would say 'let's revert back to #groovy. This isn't working out'.

[ckoca]

I wonder if the new (not yet released) Kotlin K2 compiler is going to make things better here.

[orubel]

I wonder if the new (not yet released) Kotlin K2 compiler is going to make things better here.

Thats a fun game to play; bet on future compiler against future compiler; this is actually how we got here. Instead, why not just use PRESENT tools UNTIL other tools meet requirements. GASP RADICAL IDEA!

[dsvoronin]

I wonder if the new (not yet released) Kotlin K2 compiler is going to make things better here.

Thats a fun game to play; bet on future compiler against future compiler. Instead, why not just use PRESENT tools UNTIL other tools meet requirements. GASP RADICAL IDEA! GASP

I missed some news?
Something happened with groovy support in gradle?

[orubel]

I missed some news? Something happened with groovy support in gradle?

Oh didn't you hear? https://android-developers.googleblog.com/2023/04/kotlin-dsl-is-now-default-for-new-gradle-builds.html#:~:text=gradle.,Compose%2C%20and%20now%20build%20scripts!

[dsvoronin]

I missed some news? Something happened with groovy support in gradle?

Oh didn't you hear? https://android-developers.googleblog.com/2023/04/kotlin-dsl-is-now-default-for-new-gradle-builds.html#:~:text=gradle.,Compose%2C%20and%20now%20build%20scripts!

This one I've heard.
It's the default, not the requirement. Nothing stops you from using groovy, it only affects new project creation and probably future support of some Android Studio UI-side Gradle tooling, which I never touched, so can't justify how crucial it is to others.

My pretty big(couple of mln lines of code, and thousands of Gradle modules) project also suffers from some Gradle performance issues, and it's actually still on groovy scripts exactly because of the data from this thread and personal benchmarks.

However until groovy is on par with kotlin-dsl how is this even a problem? Please explain, I probably missing something

[orubel]

This one I've heard. It's the default, not the requirement.

Uh huh... BUT you are using 'the Microsoft Monopoly argument' : 'IE is the default browser, not the required browser'

You converted to a language that is less scalable to please a BILLION DOLLAR COMPANY.

The end user has to KNOW to go into THIS ISSUE to see the benchmarks and understand the differences/deficiencies

The will not know to do this. So you are making all decisions for the end user.

Thus most will use the default... much like most used the DEFAULT BROWSER in Microsoft (and why they got sued for a monopoly)

Here is where you blow off my response as meaningless.

[dsvoronin]

This one I've heard. It's the default, not the requirement.

Uh huh... BUT you are using 'the Microsoft Monopoly argument' : 'IE is the default browser, not the required browser'

You converted to a language that is less scalable to please a BILLION DOLLAR COMPANY.

The end user has KNOW to go into THIS ISSUE to see the benchmarks and understand the differences.

The will not know to do this. So you are making all decisions for the end user for them.

Thus most will use the default... much like most used the DEFAULT BROWSER.

Here is where you blow off my response as meaningless.

I agree that making it the default is a questionable decision.
I don't know the motives and has no intention to defend any company here.

Probably it's working "good enough" for most of the project around here, but it definitely not the best choice for 1k+ modules monsters right now.

I also have this thing to add:
Making kotlin-dsl the default would possibly attract more attention to current problems and will facilitate it's faster resolution, which will benefit me as a potential user, as I definitely see kotlin-dsl superior in terms of editing/discovery.
It's an egoistic position as it is basically beta-testing on less experienced devs, but it is what it is

[orubel]

I also have this thing to add: Making kotlin-dsl the default would possibly attract more attention to current problems and will facilitate it's faster resolution, that will benefit me as a potential user, as I definitely see kotlin-dsl superior in terms of editing/discovery. It's an egoistic position as it is basically beta-testing on less experienced devs, but it is what it is

But the ORIGINAL reasoning for the push was 'KOTLIN would be FASTER'.

it wasn't. It isn't. It's been proven.

Now we sit here and try to compromise on the reasons for the decision after the fact like a bad management decision rather than engineers accepting that this is (and was) a bad engineering decision.

So the question is: is this a management decision or an engineering decision?

Because thats what ends this discussion :)

A management discussion is just so they can please their corporate benefactors.

An engineering discussion is to have a better product.

So that is the real question. Is this going to be an engineering conversation or a management one?

[gabrielittner]

But the ORIGINAL reasoning for the push was 'KOTLIN would be FASTER'.

What's the source of that? The story was always around improving the editing experience by using a type safe language that can offer proper auto complete and other IDE features.

https://blog.gradle.org/kotlin-meets-gradle
https://blog.gradle.org/kotlin-dsl-1.0

[orubel]

What's the source of that? The story was always around improving the editing experience by using a type safe language that can offer proper auto complete and other IDE features.

You are showing articles for adding Kotlin to Gradle... not for making it the DEFAULT; thats what we are discussing.

But you are 100% right. No one EVER said speed was a reason to adopt Kotlin https://blog.jetbrains.com/kotlin/2023/04/kotlin-dsl-is-the-default-for-new-gradle-builds/

It was :

• IDE support in Jetbrains (both have this)
• Strongly typed (both have this; Groovy can be BOTH - stop the FUD)
• Simplified, declarative plugin syntax (oh no... I have to declare 'id' for the plugin; this is SOOOOO hard to change)
• A growing community of Kotlin developers(ie pressure from Google)

So it comes down to what I stated before : one of DISTINCTLY FASTER and one just has pressure from Google (ie engineering decision or management decision)

[jamesward]

Wow this is sure spiraling - please keep it civil so the thread doesn't get locked. Since I wrote the referenced Android blog let me add some thoughts... When we look at the experience of editing a build and using a build there are many factors, performance being just one. For build editing, many developers I've talked to have ultimately faster iterations with KTS than with Groovy due to better help in the IDE (red squiggles if something is invalid, etc). The benchmarks here don't show the end-to-end developer feedback loop time. When using a build, KTS has shown to be sufficiently on-par with Groovy. KTS also gets some benefits from better build cacheability (ie remote build caches which I don't think are possible in the same way with Groovy).

As been said before, users can choose Groovy or KTS. Android went KTS as a default for a number of reasons, including, Android is Kotlin-First from build to UI. There is a significant downside to developers needing to know multiple languages. And we've seen from many Android users that satisfaction is higher with KTS. However, if Android users want to use Groovy for their build, that is totally fine.

[orubel]

As been said before, users can choose Groovy or KTS. Android went KTS as a default for a number of reasons, including, Android is Kotlin-First from build to UI.

Right... but Android isn't Gradle. Android is Google/Alphabet.

I don't speak for all users only myself but I pretty sure not all users are Google/Alphabet; and I think Gradle is used for other things (not sure).

Its the 'Microsoft Explorer by default for all' decision because Google 'decided'

This is why I am asking if this was an engineering decision or a management decision? Because after seeing these benchmarks again and again, this CAN'T be an engineering decision. I mean I don't see any attempt at an engineering discussion between the two.

[jamesward]

Back in 2022, Gradle, JetBrains, and Android engineers all got together and decided together that we would all switch our defaults. Some reasons were different but ultimately we all decided together that this was the best default.

[jamesward]

The performance numbers represented here may be the only thing some users care about, but they aren't the only thing that every user cares about.

[orubel]

Back in 2022, Gradle, JetBrains, and Android engineers all got together and decided together that we would all switch our defaults. Some reasons were different but ultimately we all decided together that this was the best default.

I saw some Google engineers posts some of those reasons in their Google Gradle forum; some of it is FUD. Its concerning that misinformation can lead to decision that effect everyone

And the performance numbers were enough to bother Google that they put warnings on their site. Thats seem like it bothered mre than a 'few'.

[orubel]

An example, its one of the hottest topics to talk about in Kotlin... https://dev.to/cdsap/performance-impact-analysis-of-gradle-87-in-android-projects-5288

[joshfriend]

KTS also gets some benefits from better build cacheability (ie remote build caches which I don't think are possible in the same way with Groovy).

Gradle 8.7 enables remote build caching for Groovy build scripts: https://docs.gradle.org/8.7/release-notes.html#avoiding-groovy-build-script-compilation-thanks-to-build-cache-support