Android kokoro build is very flaky

[dapengzhang0]

Android kokoro build became very flaky after #6103

[dapengzhang0]

https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod%3Agrpc%2Fjava%2Fmaster%2Fpresubmit%2Fandroid

[voidzcy]

The Gradle build seems to hit memory limit easily after we adding ./gradlew clean. I was try to increase the Metaspace limit in #6148, but it doesn't seem to work.

[ejona86]

* What went wrong:
Metaspace

https://source.cloud.google.com/results/invocations/c66a5a4a-ff2c-4698-b313-a416a8155e0b/targets/grpc%2Fjava%2Fmaster%2Fpresubmit%2Fandroid/log

[dapengzhang0]

Removing the highly flaky label as it is not seen again this week. It may also be possible that flakiness occurred after #6103 is by coincidence.

[ejona86]

#6103 actually seems a reasonable trigger, because it means the gradle task rebuilds more code. It looks like the failures resolved after #6131 was merged (it is the only PR between the last failure and first success), which is not wholly unreasonable. It isn't completely clear to me why that PR fixed it, but the change impacted gradle enough it does make some sense.

I'm going to mark this as resolved. It could easily rear its head again, but it seems resolved for the while.

[dapengzhang0]

I believe #6131 resolves this issue. Actually #6131 itself fails the android build because it was rebuilding head^ with the old build.gradle, but once it's merged no more PR fails.

[ejona86]

Uh, oh. This may be resurfacing. I just saw https://source.cloud.google.com/results/invocations/8dcbb963-10c3-43e5-9345-9e0a3a7ed6b6

[ejona86]

Actually. That's because it is on the v1.24.x branch. I guess we need a fix for the branch.

[dapengzhang0]

An easy way to fix 1.24.x branch is to revert #6103

Edit: The "easy fix" above actually does not fix.

[ejona86]

@dapengzhang0, ah, yeah, that would probably work well. Would you mind sending out a PR?

[creamsoup]

I encountered same issue of my backport to 24. this fix is also backported, so reopening the issue.

https://source.cloud.google.com/results/invocations/f47ddafa-0b41-4c79-97cc-01e050129fa8/targets/grpc%2Fjava%2Fv1.24.x%2Fpresubmit%2Fandroid/log

[dapengzhang0]

@creamsoup This happens in some old branches (where #6131 is not in), even #6103 is reverted.

[creamsoup]

okay. this is only happening in older branches, closing it. please reopen if this happens in 1.25.x or newer.