Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop Gradle Deamons on-demand #342

Closed
wants to merge 3 commits into from
Closed

Stop Gradle Deamons on-demand #342

wants to merge 3 commits into from

Conversation

milis92
Copy link

@milis92 milis92 commented Jun 21, 2022

As stated in #341, post action step is canceling all Gradle tasks being executed on the host.
This PR gives consumers the ability to disable cancellation and keep the daemons running after-action run is completed

@bigdaz
Copy link
Member

bigdaz commented Jun 21, 2022

Thanks for the contribution. I'm curious about your use case:

  • How are you running GitHub Actions with non-ephemeral runners?
  • How would you expect (or want) the Gradle User Home save/restore to function on these runners?

I'm concerned that there a bunch of places in the code where it is assumed that a Job starts with a fresh state.

@milis92
Copy link
Author

milis92 commented Jun 21, 2022

So basically we have a single physical host that has 6 runners configured under a single user.
This is causing all 6 runners to share the same Gradle User Home, which is permanent on the host, so caches are not saved/restored using the action.

@bigdaz
Copy link
Member

bigdaz commented Jun 22, 2022

@milis92 Thanks for the info. I have more questions that will inform any additional work we need to for your use case:

  • Are you running with GitHub Enterprise Server?
  • Do the 6 runners each have an isolated workspace? Specifically, gradle-build-action uses the GITHUB_WORKSPACE, RUNNER_TEMP directories: are these shared between runners? Are they cleaned between Jobs?
  • If an environment variable is set by writing to $GITHUB_ENV in one runner, is it possible that this would leak to other runners, or to later invocations with the same runner?

@bigdaz
Copy link
Member

bigdaz commented Jun 22, 2022

Regarding the functionality added by your PR, I'd prefer to avoid having a configuration option and instead to only stop the daemon processes when it's required to save the Gradle User Home directory. It would be pretty simple to change stopAllDaemons() so it's only called when we're about to save the cache state, and to do nothing in the case where the Gradle User Home is not going to be cached.
That should make things just work in your environment without requiring an additional config param. WDYT?

@milis92
Copy link
Author

milis92 commented Jun 22, 2022

Are you running with GitHub Enterprise Server?

No, its a out-of-the-box MacMini with MacOs (Monterey)

Do the 6 runners each have an isolated workspace? Specifically, gradle-build-action uses the GITHUB_WORKSPACE, RUNNER_TEMP directories: are these shared between runners? Are they cleaned between Jobs?

Yes, they all have their isolated workspaces and temp directories, managed completely by the runners (we are not cleaning up anything before/after the run)

This is how it looks

flowchart LR
    subgraph h[HOST]
        direction TB
        
        gh(("~/.gradle"))     
        
        subgraph r1["action_runner_1"]
        direction TB    
            w1[_work]
            t1[_temp]
        end r1

        subgraph r2 ["action_runner_2"]
        direction TB
            w2[_work]
            t2[_temp]
        end r2;
  
        subgraph r3 ["action_runner_3"]
        direction TB
            w3[_work]
            t3[_temp]
        end r3

        r1 --> gh
        r2 --> gh
        r3 --> gh
    end h

If an environment variable is set by writing to $GITHUB_ENV in one runner, is it possible that this would leak to other runners, or to later invocations with the same runner?

Nope, this shouldn't happen. Each run is supposed to start with a clean environment. The only thing thats preconfigured is JAVA_HOME and ANDROID_SKD_ROOT environment variables. These values are hardcoded in the .env file in each of the runners.
One thing worth mentioning is that we use the default GRADLE_HOME that causes everything in ~/.gradle to be shared between runners.

@milis92
Copy link
Author

milis92 commented Jun 22, 2022

Regarding the functionality added by your PR, I'd prefer to avoid having a configuration option and instead to only stop the daemon processes when it's required to save the Gradle User Home directory. It would be pretty simple to change stopAllDaemons() so it's only called when we're about to save the cache state, and to do nothing in the case where the Gradle User Home is not going to be cached.

Yeah, this seems like a better approach. Having an extra config param was more of a quick fix ;)
Is there any other scenario where we can expect GRADLE_HOME not to be cached? For example, if we set cache-read-only: false

@bigdaz
Copy link
Member

bigdaz commented Jun 22, 2022

Thanks so much for the details @milis92. I'm completely unfamiliar with self-hosted runners (until now).

One potential issue I can see is if RUNNER_TEMP isn't cleared between runs, then the generated Build Results table will display results from a previous Job run. What's happening is that gradle-build-action is injecting an init-script into Gradle User Home, and then each Gradle invocation will write a build-results file to a ${RUNNER_TEMP}/.build-results directory. The action then lists all build-results on job completion and renders the summary report.

  • Do you see a Job Summary like this for your GitHub Actions runs? (Not sure if this functionality is supported for self-hosted runners yet).
  • If so, do you observe pollution of the summary from previous executions?

For now, I'm going to assume things are working as expected for self-hosted runners, as I don't currently have the capacity to set this up for testing. Please report any issues that you find. Thanks.

@bigdaz
Copy link
Member

bigdaz commented Jun 22, 2022

Is there any other scenario where we can expect GRADLE_HOME not to be cached? For example, if we set cache-read-only: false

Yes there are, and I think it makes sense not to stop Gradle daemons in these cases.

Signed-off-by: Ivan Milisavljevic <cartman.dev@gmail.com>
Signed-off-by: Ivan Milisavljevic <cartman.dev@gmail.com>
…GRADLE_HOME is going to be cached

Signed-off-by: Ivan Milisavljevic <cartman.dev@gmail.com>
@milis92
Copy link
Author

milis92 commented Jun 22, 2022

From the docs on RUNNER_TEMP

The path to a temporary directory on the runner. This directory is emptied at the beginning and end of each job. Note that files will not be removed if the runner's user account does not have permission to delete them. For example, D:\a_temp

Job summary works well on our side, il take a deeper look and open a separate issue if I find any problems.

I proposed new changes with following updates:

  • Remove stop_daemons configuration flag
  • Stop daemon only if shouldSaveCaches() and !isCacheReadOnly()
  • Updated README to explain new changes

@bigdaz
Copy link
Member

bigdaz commented Jun 22, 2022

Yep, those changes are just what I'm going for in #344

@bigdaz
Copy link
Member

bigdaz commented Jun 22, 2022

@milis92 Apologies, looks like I jumped the gun by implementing this in a separate PR. I really appreciate you taking the time to submit this PR, and hope that you'll find time to contribute other fixes/improvements in the future.
Cheers!

@bigdaz
Copy link
Member

bigdaz commented Jun 22, 2022

I stole the README text directly from your PR (with credit): 3292389

@milis92
Copy link
Author

milis92 commented Jun 22, 2022

No worries, I'm really glad I could help. 👍
Feel free to close this PR in favor of #344

@bigdaz bigdaz closed this Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants