Build and test pipeline takes too long

[Zelldon]

Issue Description

The current compilation time with maven mvn clean install -DskipTests -T1C is for me locally ~1.5 min, but this highly depends on the machine. The execution of all the tests is way too long and I never do it locally (which is bad). If I do changes I normally just run the test in the affected module.

The CI pipeline takes ~33 min, to build, test and deploy snapshots. See this 33m 28s.

If we do a calculation of how many engineers we are working on this project ~9 and say everyone is pushing at least once to a branch we run 33 * 9 min = 297 min per day. If we say we have around 200 working days in a year, we have 59400 min CI time, only for our work.

This doesn't include daily builds, dependabot (which are a lot), stable branches, etc.

If we are reducing the test and compile time, this will have a major effect on costs (since we using self-managed runners) but also gives us a boost in effectivity, since we getting faster feedback.

Potential Idea:

I think there are several ideas to tackle this issue, improving the tests, reducing integration tests and increasing unit test coverage, running partial builds, etc.

But there is also another thing that might be interesting for us to investigate. We have several modules we haven't touched for years. There are also modules that we touch only from time to time, maybe every quarter once.

I would propose we release one version of each of these modules and pin them in our dependency. This would allow us to skip the compiling and running tests of these modules on every build. For example, it is not necessary to compile and run tests every time for Zeebe Msgpack Core, Zeebe Msgpack Value, or others.

I think it is important that if we change one of these modules we directly release a new version of it. IMHO that is ok since these modules are and should only be used internally. As long we keep our public release intact and release dist and clients with the correct release number this should be fine. This would decrease the time of the general CI pipeline by a lot.

BTW When we started with Zeebe (or TNGP) we had each module its own REPO, but we realized that since our APIs were so fragile we had to change a lot on multi modules so it didn't make much sense to have them in several repos and release them every time (or use snapshot). But I think this has changed since a lot of modules are more mature now.

First iteration scope

Optimize overhead of test jobs => target max time 10m

Tasks

Beta Give feedback

1. ci(integration): split module and integration test jobs #12406 - will optimize the order of tests module tests always offset qa integration tests
2. ci(test): skip test artifact upload on success #12497 - will optimize duration on test success (most impact on unit test job)
3. Optimise long running tests #12417 - single long running test classes limit the yield of forks
4. ci: always cache maven dependencies #12424 - a considerable amount of time is spend downloading deps, we use cahces in some but not all jobs with this PR caching will be used by default

[korthout]

I would propose we release one version of each of these modules and pin them in our dependency. This would allow us to skip the compiling and running tests of these modules on every build. For example, it is not necessary to compile and run tests every time for Zeebe Msgpack Core, Zeebe Msgpack Value, or others

This makes sense to me. Especially as you say:

But I think this has changed since a lot of modules are more mature now.

Before choosing one solution, I want to propose an alternative:

• only run ci on modules with changes (or changes in dependencies) (can be easily checked using git, but there are also maven extensions for this)

[Zelldon]

Yep this is what I meant with running partial builds but the last time I checked was the maven support not that optimal (at least the plugin was no longer maintained etc.)

[korthout]

ZPA triage summary:

• Let's discuss with infra to see the actual CI costs, so we can help quantify these improvements
• @abbasadel Please discuss with @megglos to coordinate efforts

[megglos]

ZDP planning:

• we will look into a reasonable scope for the next iteration

[megglos]

Given with recent improvements we brought the CI stage down to about 10m I would conclude the first iteration scope complete. See e.g. https://github.com/camunda/zeebe/actions/runs/5068424932?pr=12850

I would still consider #12417 worth doing anytime soon to ensure the forking of IT jobs is efficient.

Let's check in on how to proceed.

[megglos]

ZDP-Triage:

• right now flaky tests seem to be a bigger annoyance
• would be worth revisiting while
• moving back to backlog pausing for for now