Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

actions/checkout@v2 breaks on a (self-hosted runner, repository) pair with submodules; no way to revert #385

Open
yeputons opened this issue Nov 11, 2020 · 9 comments

Comments

@yeputons
Copy link

How to reproduce:

  1. Spin off a self-hosted GitHub Actions runner. This is important because for reasons I do not understand or accept (to put it mildly) self-hosted runners are fully stateful and do not clean up in any ways between runs for a specific repository.
  2. Ensure that an arbitrary workflow containing actions/checkout@v2 succeeds. Let's say that you have it in a repository main-repo.
  3. Accidentally clone another random repository other-repo inside main-repo, and then git add . && git commit -m"" && git push, effectively adding other-repo as a submodule to main-repo.
  4. Ensure that the workflow runs on push the first time only.
  5. Re-run. Ensure that it fails like this (here labs-yeputons is main-repo, and labs is the other-repo that was accidentally add as a submodule):
Run actions/checkout@v2
  with:
    repository: hse-spb-2020-cpp/labs-yeputons
    token: ***
    ssh-strict: true
    persist-credentials: true
    clean: true
    fetch-depth: 1
    lfs: false
    submodules: false
Syncing repository: hse-spb-2020-cpp/labs-yeputons
Getting Git version info
  Working directory is '/home/platypus/git/actions-runner/_work/labs-yeputons/labs-yeputons'
  /usr/bin/git version
  git version 2.25.1
/usr/bin/git config --local --get remote.origin.url
https://github.com/hse-spb-2020-cpp/labs-yeputons
Removing previously created refs, to avoid conflicts
  /usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
  refs/heads/lab03-list-heap
  /usr/bin/git checkout --detach
  HEAD is now at d5cced7 break everything by adding submodule
  /usr/bin/git rev-parse --symbolic-full-name --branches
  refs/heads/lab03-list-heap
  /usr/bin/git branch --delete --force lab03-list-heap
  Deleted branch lab03-list-heap (was d5cced7).
  /usr/bin/git rev-parse --symbolic-full-name --remotes=origin
  refs/remotes/origin/lab03-list-heap
Cleaning the repository
  /usr/bin/git clean -ffdx
  /usr/bin/git reset --hard HEAD
  HEAD is now at d5cced7 break everything by adding submodule
Disabling automatic garbage collection
  /usr/bin/git config --local gc.auto 0
Setting up auth
  /usr/bin/git config --local --name-only --get-regexp core\.sshCommand
  /usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
  Error: fatal: No url found for submodule path 'labs' in .gitmodules
  Error: The process '/usr/bin/git' failed with exit code 128

So, after the first workflow run the working directory is forever contaminated and all subsequent builds break even if we reset branch to the previous state. The only way to fix this is to either manually clean up working directory on each affected worker, or to add an extra step to nuke repository before checking it out from scratch.

@marcelopontipedra
Copy link

marcelopontipedra commented Feb 26, 2021

Hey Guys. Is there a way that my python app should not terminate. A job for publish shouldn't be terminated because it should publish an app. It's been a week, but still couldn't find solution to disable this auto terminate process.
image

@yeputons
Copy link
Author

@marcelopontipedra I do not see how your comment is relevant to the original issue.

Please avoid asking for help in irrelevant issues: a) you've already created your own; b) there is GitHub Support Community.

@marcelopontipedra
Copy link

@yeputons Sorry about that, I freak out about this issue. My bad. I've also created a ticket for the issue the same time I commented here in the support community.

AndrewFasano added a commit to panda-re/panda that referenced this issue Jul 1, 2021
Github Actions checkout@v2 fails to properly cleanup state in self-hosted runners
which leads to errors from prior CI runs.

actions/checkout#385

Adding the created submodule directory to gitignore should hopefully
prevent this from happening: actions/checkout#424 (comment).
jamcleod pushed a commit to panda-re/panda that referenced this issue Jul 22, 2021
Github Actions checkout@v2 fails to properly cleanup state in self-hosted runners
which leads to errors from prior CI runs.

actions/checkout#385

Adding the created submodule directory to gitignore should hopefully
prevent this from happening: actions/checkout#424 (comment).
@yeputons
Copy link
Author

May be a duplicate of #354

@andychongyz
Copy link

Adding the step below before checkout works for me.

- name: Clear repository
        run: sudo rm -fr $GITHUB_WORKSPACE && mkdir $GITHUB_WORKSPACE

@jsoref
Copy link
Contributor

jsoref commented Aug 22, 2022

There's a workaround for this in #590 (comment)

@jokreliable
Copy link
Contributor

#964 might be a fix in the actual code, but i'm having issues getting any eyeballs on it.
That should bring it back in line and prevent the failures most of us have seen from that type of workflow.

@smarter
Copy link

smarter commented Feb 22, 2023

It hadn't happened in a long time, but I'm using the latest actions/checkout and just saw a runner bricking itself after a submodule got into a bad state: https://github.com/lampepfl/dotty/actions/runs/4244282953/jobs/7378178653 (in this case I didn't even figure out how it got stuck in this state, I think it might have been because a job got cancelled while the submodule was being updated?)

I don't really get why the action insists on running commands on submodules even though submodules: false is the default according to the documentation.

@kaidokert
Copy link

Ftr, i have a rebased PR here that takes care of this: #1321

The repro case described at the top of this issue helped reliably reproduce it, e.g. just add a random git clone & commit somewhere in your actions code to mess up the repo state ( youtube/cobalt_sandbox@367623e )

Note again: Only happens if you are using non-ephemeral runners.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants