CI jobs failing after v0.7.10 images were published #848

atc0005 · 2023-02-10T13:06:35Z

Overview

After the v0.7.10 images were uploaded I began working on a new release for a different project. As part of a routine CHANGELOG PR the CI jobs began falling over:

linting
building

Example of a CI failure involving the ghcr.io/atc0005/go-ci:go-ci-stable-mirror-build image:

Run make all

fatal: detected dubious ownership in repository at '/__w/check-cert/check-cert'
To add an exception for this directory, call:

	git config --global --add safe.directory /__w/check-cert/check-cert
fatal: detected dubious ownership in repository at '/__w/check-cert/check-cert'
To add an exception for this directory, call:

	git config --global --add safe.directory /__w/check-cert/check-cert
Removing object files and cached files ...
Removing any existing release assets
Removing any existing quick build release assets
Removing any empty asset build paths
Building release assets for windows x86 ...
  running go generate for check_cert 386 binary ...
2023/02/10 12:51:54 failed resolving git tag: exit status 128
main.go:8: running "go-winres": exit status 1
  running go generate for lscert 386 binary ...
2023/02/10 12:51:54 failed resolving git tag: exit status 128
main.go:8: running "go-winres": exit status 1
  running go generate for certsum 386 binary ...
2023/02/10 12:51:54 failed resolving git tag: exit status 128
main.go:8: running "go-winres": exit status 1
make: *** [Makefile:253: windows-x86-build] Error 1
Error: Process completed with exit code 2.

Example of a CI failure involving the ghcr.io/atc0005/go-ci:go-ci-stable image:

Run go build -v -mod=vendor ./...
# cd /__w/check-cert/check-cert; git status --porcelain
fatal: detected dubious ownership in repository at '/__w/check-cert/check-cert'
To add an exception for this directory, call:

	git config --global --add safe.directory /__w/check-cert/check-cert
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
Error: Process completed with exit code 1.

References

The text was updated successfully, but these errors were encountered:

atc0005 · 2023-02-10T13:19:22Z

This is a slightly older report, but may be relevant to what I'm seeing here:

fatal: detected dubious ownership in repository at actions/runner-images#6775

The reporter used these commands to generate diffs of the images:

container-diff diff --type=apt IMAGE1 IMAGE2
container-diff diff --type=node IMAGE1 IMAGE2

I'll lookup how to obtain that tool and run the first command.

atc0005 · 2023-02-10T13:28:02Z

Digging around, the consensus is that it's related to recent Git updates.

The workarounds appear to be providing a "safe directory" for the checked out Git repos.

Oddly enough, the GHAWs are already doing this?

For example, from the Check out code step:

Run actions/checkout@v3.3.0
/usr/bin/docker exec  caef991dd03714847a4e14f5628cea562fb367ef7881fe81c57b8bf201abf0e9 sh -c "cat /etc/*release | grep ^ID"
Syncing repository: atc0005/check-cert
Getting Git version info
Temporarily overriding HOME='/__w/_temp/e566fbaa-e8f7-4044-a373-0e2f6aaa6832' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /__w/check-cert/check-cert
Deleting the contents of '/__w/check-cert/check-cert'
Initializing the repository
Disabling automatic garbage collection
Setting up auth
Fetching the repository
Determining the checkout info
Checking out the ref
/usr/bin/git log -1 --format='%H'
'54108fed46834543713242db8cac0ce5b70d7e4e'

and then the next step:

0s
Run go build -v -mod=vendor ./...
# cd /__w/check-cert/check-cert; git status --porcelain
fatal: detected dubious ownership in repository at '/__w/check-cert/check-cert'
To add an exception for this directory, call:

	git config --global --add safe.directory /__w/check-cert/check-cert
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
Error: Process completed with exit code 1.

This command is being run:

git config --global --add safe.directory /__w/check-cert/check-cert

and that is the path referenced by the earlier code checkout?

/__w/check-cert/check-cert

atc0005 · 2023-02-10T13:37:06Z

Explicitly downgraded the shared CI workflows from using the latest atc0005/go-ci images to the specific v0.7.9 release version to see whether that helps.

atc0005 · 2023-02-10T13:39:22Z

Explicitly downgraded the shared CI workflows from using the latest atc0005/go-ci images to the specific v0.7.9 release version to see whether that helps.

It did, but only after I asked for all jobs to be re-ran. Re-running just the failed jobs seemed to continue referencing the latest images.

atc0005 · 2023-02-10T13:55:04Z

$ docker container run -it --rm -v $PWD:$PWD -w $PWD ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.9-0-gdf9564d3 git --version
git version 2.30.2
$ docker container run -it --rm -v $PWD:$PWD -w $PWD ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.10-0-gcc0a2418 git --version
git version 2.30.2

atc0005 · 2023-02-10T14:03:09Z

The reporter used these commands to generate diffs of the images:

container-diff diff --type=apt IMAGE1 IMAGE2

container-diff diff --type=node IMAGE1 IMAGE2

I'll lookup how to obtain that tool and run the first command.

Installed the tool per directions on https://github.com/GoogleContainerTools/container-diff

$ container-diff diff --type=apt ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.9-0-gdf9564d3 ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.10-0-gcc0a2418

-----Apt-----

Packages found only in ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.9-0-gdf9564d3: None

Packages found only in ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.10-0-gcc0a2418: None

Version differences:
PACKAGE                 IMAGE1 (ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.9-0-gdf9564d3)        IMAGE2 (ghcr.io/atc0005/go-ci:go-ci-stable-v0.7.10-0-gcc0a2418)
-curl                   7.74.0-1.3 deb11u3, 429K                                              7.74.0-1.3 deb11u5, 430K
-git                    1:2.30.2-1, 34.2M                                                     1:2.30.2-1 deb11u1, 34.3M
-git-man                1:2.30.2-1, 1.8M                                                      1:2.30.2-1 deb11u1, 1.8M
-libcurl3-gnutls        7.74.0-1.3 deb11u3, 735K                                              7.74.0-1.3 deb11u5, 736K
-libcurl4               7.74.0-1.3 deb11u3, 739K                                              7.74.0-1.3 deb11u5, 740K
-linux-libc-dev         5.10.158-2, 5.7M                                                      5.10.162-1, 5.7M

git package version diff:

-1:2.30.2-1
+1:2.30.2-1 deb11u1

atc0005 · 2023-02-10T14:12:57Z

root@7c4235dc9081:/home/ubuntu/Desktop# apt list -a git
Listing... Done
git/stable-security,now 1:2.30.2-1+deb11u1 amd64 [installed]
git/stable 1:2.30.2-1 amd64

root@7c4235dc9081:/home/ubuntu/Desktop# dpkg -l | grep git
ii  dash                       0.5.11+git20200708+dd9ef66-5   amd64        POSIX-compliant shell
ii  git                        1:2.30.2-1+deb11u1             amd64        fast, scalable, distributed revision control system
ii  git-man                    1:2.30.2-1+deb11u1             all          fast, scalable, distributed revision control system (manual pages)
ii  librtmp1:amd64             2.4+20151223.gitfa8646d.1-2+b2 amd64        toolkit for RTMP streams (shared library)

root@7c4235dc9081:/home/ubuntu/Desktop# apt-get install git=1:2.30.2-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn
Recommended packages:
  patch less
The following packages will be DOWNGRADED:
  git
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 5527 kB of archives.
After this operation, 22.5 kB disk space will be freed.

Downgrading is an option, at least for now.

Work around potential "dubious ownership" complaints from Git regarding the workspace used by GitHub task runners by explicitly marking the current working directory as trusted. Since we're not sharing that space with other user accounts this should be a safe assumption to make. refs GH-848

Swap from hard-coded version (used as a workaround) of images from the atc0005/go-ci project to using a GitHub Actions Workflow variable set at the top of each workflow. As of this commit we've switched back to the `latest` tag for each image, but we can now easily switch all atc0005/go-ci images to a specific version for testing purposes in the future. refs GH-56 refs atc0005/go-ci#848

Work around potential "dubious ownership" complaints from Git regarding the workspace used by GitHub task runners by explicitly marking the current working directory as trusted after checking out content from the repo. Since we're not sharing that space with other user accounts this should be a safe assumption to make. refs atc0005/go-ci#848

This container image does not use Git for any of its operations so we skip this step, but leave the commented out task as an explicit indicator that we're not performing that step (for a reason). refs atc0005/go-ci#848

atc0005 · 2023-02-11T18:57:42Z

Workarounds applied at the workflow level:

atc0005/shared-project-resources
atc0005/mysql2sqlite
- about to apply this change shortly

atc0005 · 2023-02-11T18:58:21Z

Workarounds applied at the workflow level

This was done to avoid making a change directly at the image level that could impact other users.

Work around potential "dubious ownership" complaints from Git regarding the workspace used by GitHub task runners by explicitly marking the current working directory as trusted. Since we're not sharing that space with other user accounts this should be a safe assumption to make. refs atc0005/go-ci#848

atc0005 added bug Something isn't working question Further information is requested oldstable unstable stable labels Feb 10, 2023

atc0005 added this to the Next Release milestone Feb 10, 2023

atc0005 self-assigned this Feb 10, 2023

atc0005 mentioned this issue Feb 11, 2023

Explicitly mark CWD as trusted by Git #850

Merged

atc0005 mentioned this issue Feb 11, 2023

Explicitly use specific release version atc0005/shared-project-resources#57

Merged

atc0005 mentioned this issue Feb 11, 2023

Explicitly trust CWD by Git after checkout atc0005/shared-project-resources#58

Merged

atc0005 closed this as completed Feb 11, 2023

atc0005 mentioned this issue Feb 11, 2023

Explicitly mark CWD as trusted by Git atc0005/mysql2sqlite#218

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI jobs failing after v0.7.10 images were published #848

CI jobs failing after v0.7.10 images were published #848

atc0005 commented Feb 10, 2023 •

edited

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 11, 2023

atc0005 commented Feb 11, 2023

CI jobs failing after v0.7.10 images were published #848

CI jobs failing after v0.7.10 images were published #848

Comments

atc0005 commented Feb 10, 2023 • edited

Overview

References

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 10, 2023

atc0005 commented Feb 11, 2023

atc0005 commented Feb 11, 2023

atc0005 commented Feb 10, 2023 •

edited