Enhance exec operation to survive CRI-O restarts

[hasan4791]

What happened?

Currently cri-o holds the streaming server which manages the exec connection to/from the running containers. This is causing the exec to break when cri-o is restarted. If possible, we can offload exec operation to intermediate application like conmon or something similar, it would ideally survive the cri-o restarts. Open to discussion on this topic.

What did you expect to happen?

NA

How can we reproduce it (as minimally and precisely as possible)?

NA

Anything else we need to know?

NA

CRI-O and Kubernetes version

NA

OS version

NA

Additional environment details (AWS, VirtualBox, physical, etc.)

NA

[saschagrunert]

This is a feature we can elaborate on, maybe it makes sense to keep the connection open in conmon-rs.

[kwilczynski]

/retitle Enhance exec operation to survive CRI-O restarts

[kwilczynski]

@hasan4791, do you have some things that require to be run for longer?

Why would a momentary interruption be a problem? While CRI-O restarts, which should be momentary, hopefully. What would be the use case you have in mind?

I am asking as there is also a push to enable and ensure a timeout for any streaming operations from the container back to the user.

[haircommander]

I think this is an all or nothing thing. If CRI-O restarts, it looses the connection to the kubelet which will tear down the endpoint assuming the container stopped . I think delegating to conmon-rs is a good idea

[hasan4791]

@kwilczynski Not in my case, but i was fiddling with registry config in the cluster nodes and had to restart the runtime which disconnected the session. I thought it would be a good to have thing to keep the session open unless user disconnects or times out on ideal.

[hasan4791]

@haircommander Is it possible to extend the same for log streaming as well?

[haircommander]

Yeah the mechanism could be similar

[saschagrunert]

So, exec and attach are pretty much the same, portforward is slightly different but should work as well. Logs is tricky, because the log reader in the kubelet (used by crictl as well) is doing a container status to ensure that the container is running.

See: https://github.com/kubernetes/kubernetes/blob/9c50b25/pkg/kubelet/kuberuntime/logs/logs.go#L418-L444

If the runtime is not available, then it assumes that the container is not running as well (which is wrong in case of CRI-O) and therefore aborts the log reading.

[saschagrunert]

The solution for the log streaming is available in kubernetes/kubernetes#124025

[kwilczynski]

/assign saschagrunert
/assign haircommander
/assign kwilczynski