figure out DinD + podutils

[BenTheElder]

This is a specific issue to track the need for Docker in Docker with the podutils. I am also filing a related but more general issue for the podutils and init / forking that may wind up solving this one. xref: #8779

In short, with the old bootstrap + scenario scripts and their images we have support for using docker at runtime in prowjobs, with the podutils we do not. We need to sort this out. A complication here is that the podutils now control the entrypoint, we can run a similar script under the entrypoint but before the actual test command, but that leaves other questions, like how we will handle zombie processes...

xref: #8763 (comment), #7844 (comment)

[BenTheElder]

/assign

[stevekuznetsov]

The current implementation of the utils is super generic and there is merit to that, but if we wanted to have a dind job type that we could handle in the infra by allowing access to a socket in the test container, that seems OK with me. Would allow us to run the container runtime in a separate container and share the socket while still ensuring things about the lifecycle and logging since the infra would know about the extra container.

[BenTheElder]

@stevekuznetsov I don't think that will work properly due to volume mounts. See #8779 (comment)

Seperately from DinD, most things we run fork, avoiding zombies is something to handle more generally. What needs to happen here specifically is a different snippet to boot docker (instead of having it in the images/bootstrap entrypoint.sh)

[BenTheElder]

We should be able to refactor out just the DinD parts from images/bootstrap/runner and let the pdoutils job be entrypoint test_cmd where test_cmd is something like dind_runner actual-test-command.

This just reminded me that zombies are also coming to eat our nodes. For that something like /bin/sh entrypoint test_cmd would probably be sufficient.

I'll handle the refactor bit now and circle back on zombies. Kubernetes may also soon have better ability to manage this problem as well... (it does have some alpha gated enhancements now IIRC) .. we're not the only ones with "the PID1 problem".

[stevekuznetsov]

I'll handle the refactor bit now and circle back on zombies. Kubernetes may also soon have better ability to manage this problem as well

I'd check where the pause container work is, that was supposed to handle the issue IIRC

[BenTheElder]

It looks like we are waiting on kubernetes/kubernetes#1615 before the pause container can reap zombies. It has some logic already, but currently it doesn't work as the pod doesn't share a PID namespace yet.
The last update is that 1.10 had this as an alpha feature kubernetes/kubernetes#1615 (comment), until just two hours ago A PR opened to push it to beta kubernetes/kubernetes#66507 🤞

It will be a while before Prow is running on this though, as even 1.11 has it as alpha, 1.12 may have it as beta.

[stevekuznetsov]

It will be a while before Prow is running on this though

You are stock GKE, right? I think we may be running with it right now on ours, but we fiddle with cluster flags a lot.

[stevekuznetsov]

let the pdoutils job be entrypoint test_cmd where test_cmd is something like dind_runner actual-test-command.

Are there benefits from integrating more tightly for classes of jobs (DinD, jobs that want VMs, stuff like that) or do we just want to black-box everything and keep it as run by entrypoint?

[BenTheElder]

Stock GKE does not enable alpha features to my knowledge. We could almost certainly sidestep this but it would be hacky.

Are there benefits from integrating more tightly for classes of jobs (DinD, jobs that want VMs, stuff like that) or do we just want to black-box everything and keep it as run by entrypoint?

We should keep it flexible. I don't think we want to support VMs etc as part of PRow. I do think we need to support forking, as tons of useful utilities fork. This issue is really for tracking that our images also need an update because they assume they can do $magic in the entrypoint before running bootstrap.

[verb]

FYI, GKE supports alpha clusters, but they're only suitable for testing.

@stevekuznetsov Are you running with PodShareProcessNamespace=true? I haven't received much feedback on this feature. I'd like to gather a few examples of people using it successfully.

[BenTheElder]

Yeah, alpha clusters are not a suitable option for Prow. Some end to end testing uses them though.

[BenTheElder]

In theory after #8874 a job should be able to use sysv DinD + podutils relatively fine. I'll experiment with this or help someone port a job to it soonish. perhaps @MHBauer

[spiffxp]

/milestone v1.13
I actually have no clue here, I'm going based off of the priority label, otherwise I would have punted from the milestone. Any updates?

[BenTheElder]

This is definitely v1.13 I expect to start poking this more next week :-)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[idealhack]

/remove-lifecycle stale

[spiffxp]

/milestone 2019-goals

@BenTheElder I punted this out given that I saw no activity here in the v1.13 cycle. Please add back to v1.14 or close out or do nothing as appropriate.

[BenTheElder]

pod-utils + dind works with our sysv dind, it's not quite ideal though, but we should probably track options for not hijacking ENTRYPOINT in another issue.