Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Failing Test] build-master-fast failure with gcr.io/k8s-staging-releng/k8s-ci-builder:v20240124-v0.16.4-107-gd5a2bc91-default #122939

Closed
MaryamTvk opened this issue Jan 24, 2024 · 13 comments · Fixed by kubernetes/release#3431
Closed

[Failing Test] build-master-fast failure with gcr.io/k8s-staging-releng/k8s-ci-builder:v20240124-v0.16.4-107-gd5a2bc91-default #122939

MaryamTvk opened this issue Jan 24, 2024 · 13 comments · Fixed by kubernetes/release#3431
Labels
area/release-eng Issues or PRs related to the Release Engineering subproject kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/release Categorizes an issue or PR as relevant to SIG Release.
Milestone
v1.30

Comments

@MaryamTvk
Copy link

Which jobs are failing?

https://testgrid.k8s.io/sig-release-master-blocking#build-master-fast

Which tests are failing?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-fast/1750059289608720384

Since when has it been failing?

Before: wrapper.sh] [INFO] Running in: gcr.io/k8s-staging-releng/k8s-ci-builder:v20240118-v0.16.4-99-g0716b946-default
Failing: wrapper.sh] [INFO] Running in: gcr.io/k8s-staging-releng/k8s-ci-builder:v20240124-v0.16.4-107-gd5a2bc91-default

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#build-master-fast

Reason for failure (if possible)

New image Failing: wrapper.sh] [INFO] Running in: gcr.io/k8s-staging-releng/k8s-ci-builder:v20240124-v0.16.4-107-gd5a2bc91-default

Anything else we need to know?

No response

Relevant SIG(s)

/sig
@MaryamTvk MaryamTvk added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Jan 24, 2024
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 24, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@MaryamTvk
Copy link
Author

/milestone v1.30

@k8s-ci-robot
Copy link
Contributor

@MaryamTvk: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.30

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@pacoxu
Copy link
Member

pacoxu commented Jan 24, 2024

/milestone v1.30

@k8s-ci-robot k8s-ci-robot added this to the v1.30 milestone Jan 24, 2024
@pacoxu
Copy link
Member

pacoxu commented Jan 24, 2024

/sig release
/area release-eng
/cc @saschagrunert @xmudrii

@k8s-ci-robot k8s-ci-robot added sig/release Categorizes an issue or PR as relevant to SIG Release. area/release-eng Issues or PRs related to the Release Engineering subproject and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 24, 2024
@pacoxu
Copy link
Member

pacoxu commented Jan 24, 2024

/priority critical-urgent

@k8s-ci-robot k8s-ci-robot added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Jan 24, 2024
@pacoxu
Copy link
Member

pacoxu commented Jan 24, 2024 •
edited

In /etc/init.d/docker

		ulimit -Hn 524288

failed

root@2a05586ed8a2:/#            ulimit -Hn 524288
bash: ulimit: open files: cannot modify limit: Invalid argument

https://github.com/kubernetes/release/blob/d5a2bc91d8336b3962768676d03d57ca7f74078a/images/releng/k8s-ci-builder/Dockerfile
I install docker in debian:bullseye and run the ulimit to reproduce it.

https://github.com/kubernetes/release/blob/d5a2bc91d8336b3962768676d03d57ca7f74078a/images/releng/k8s-ci-builder/wrapper.sh#L90-L92

@pacoxu
Copy link
Member

pacoxu commented Jan 24, 2024

I'm in favor of LimitNOFILE with a TODO to remove after systemd <= 240 (EL 7) falls away; I think it's past time to get this in to a release, and I'll try to make sure we get this reviewed this week.

moby/moby@c893010#diff-329a7d3594b1ed91f76deded4734c7d9390c4dce74aeaa5a772fbf52b0a58c2eR62
https://github.com/moby/moby/blame/314ea05f8d76b8cdfc72a41f49a1035cf26b1669/contrib/init/sysvinit-debian/docker#L62

This is updated that systemd <= 240 will fail. See moby/moby#45534 (comment).

Either we should update the base image with a newer systemd, or we have to set the docker version to an older one than v25.0.0.

@pacoxu pacoxu mentioned this issue Jan 24, 2024
Closed
@rjsadow
Copy link
Contributor

rjsadow commented Jan 24, 2024

Seems related docker/cli#4807

@pacoxu
Copy link
Member

pacoxu commented Jan 24, 2024 •
edited

Seems related docker/cli#4807

@rjsadow Good catch.

I update according to docker maintainer's suggestions in kubernetes/release#3430 (comment).

@xmudrii
Copy link
Member

xmudrii commented Jan 24, 2024

@pacoxu @rjsadow How feasible is it to pin Docker to a lower version instead?

@pacoxu
Copy link
Member

pacoxu commented Jan 24, 2024

@pacoxu @rjsadow How feasible is it to pin Docker to a lower version instead?

But that will keep the old systemd version. BTW, when I open the PR, I find the 'next' in the k8s-ci-builder is already using bookworm.

@xmudrii
Copy link
Member

xmudrii commented Jan 24, 2024

There are concerns with updating k8s-ci-builder/k8s-cloud-builder images to bookworm, I left a comment here kubernetes/release#3430 (comment)

@pacoxu pacoxu mentioned this issue Jan 25, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/release-eng Issues or PRs related to the Release Engineering subproject kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/release Categorizes an issue or PR as relevant to SIG Release.
Projects
CI Signal (SIG Release / Release Team)
Status: RESOLVED
Milestone
v1.30
5 participants
@pacoxu @rjsadow @xmudrii @k8s-ci-robot @MaryamTvk