pod exec exception: failed to attache process,err: failed to upgrade to a WebSocket connection: failed to switch protocol: 404 Not Found

[vCassius]

environment:
container: amd64/debian:12.1-slim
k8s version: v1.24.6

[dependencies]
k8s-openapi = { version = "0.19.0", default-features = true }
kube-runtime = "0.85.0"
kube = { version = "0.85.0", features = ["runtime", "derive"]}
openssl = { version = "0.10.45", features = ["vendored"] }
tokio-stream = { version = "0.1", features = ["net"] }
crossterm = {version = "0.27.0" }
[features]
default = ["k8s-openapi/v1_24","kube/derive","kube/client", "kube/openssl-tls","kube/runtime", "kube/unstable-runtime","kube/oauth", "kube/oidc","kube/ws"]

pod exec exception: failed to attache process,err: failed to upgrade to a WebSocket connection: failed to switch protocol: 404 Not Found

but it's strange,in my computer,it works. but run in container. exception like this.

tracing::warn!("pod exec name: {} namespace: {} exec: {} container: {}",pod.clone(),namespace.clone(),exec.clone(),container.clone());
    let pods: Api<Pod> = Api::namespaced(cli, namespace.clone().as_str());
    let mut attached: AttachedProcess = match pods
        .exec(
            pod.clone().as_str(),
            vec![exec],
            &AttachParams::default()
                .stdin(true)
                .tty(true)
                .stderr(false)
                .container(container.clone().as_str()),
        )
        .await
    {
        Ok(attached) => attached,
        Err(e) => {
            tracing::error!("failed to attache process,err: {e}");
            return;
        }
    };

and terminal enable raw mode exception like this

  2023-10-30 17:06:34.795 ERROR v_kubernetes_action::pkg::salvo::router::pod: failed to enable raw mode,err: No such device or address (os error 6)
    at src/pkg/salvo/router/pod.rs:456

  2023-10-30 17:06:34.795  WARN v_kubernetes_action::pkg::salvo::router::pod: pod exec name: v-kubernetes-action-8b95b8884-vjfh5 namespace: bckid-devops-prod exec: bash container: v-kubernetes-action
    at src/pkg/salvo/router/pod.rs:459

End of terminal size stream
thread 'tokio-runtime-worker' panicked at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/tokio-1.32.0/src/runtime/task/core.rs:381:22:
JoinHandle polled after completion
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

    match crossterm::terminal::enable_raw_mode() {
        Ok(_) => {}
        Err(e) => {
            tracing::error!("failed to enable raw mode,err: {e}")
        }
    };

[clux]

you could try to raise RUST_LOG=debug to see if there are other things.

a common problem in-cluster; rbac. do you have pods/exec perms?

- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

[vCassius]

it's full access.i will try RUST_LOG=debug
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin

[vCassius]

  2023-10-31 13:16:16.303 DEBUG hyper::proto::h1::io: parsed 15 headers
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-1.0.0-rc.4/src/proto/h1/io.rs:203

  2023-10-31 13:16:16.303 DEBUG hyper::proto::h1::conn: incoming body is empty
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-1.0.0-rc.4/src/proto/h1/conn.rs:231

  2023-10-31 13:16:16.404 DEBUG hyper::proto::h1::io: flushed 166 bytes
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-1.0.0-rc.4/src/proto/h1/io.rs:330

  2023-10-31 13:16:16.405 DEBUG salvo_extra::websocket: websocket upgrade complete
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/salvo_extra-0.58.2/src/websocket.rs:180

  2023-10-31 13:16:16.405 ERROR v_kubernetes_action::pkg::salvo::router::pod: failed to enable raw mode,err: No such device or address (os error 6)
    at src/pkg/salvo/router/pod.rs:456

  2023-10-31 13:16:16.405  WARN v_kubernetes_action::pkg::salvo::router::pod: pod exec name: v-kubernetes-action-78fdb54ffb-pqrwv namespace: bckid-devops-prod exec: bash container: v-kubernetes-action
    at src/pkg/salvo/router/pod.rs:459

  2023-10-31 13:16:16.405 DEBUG tower::buffer::worker: service.ready: true, processing request
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/tower-0.4.13/src/buffer/worker.rs:197

  2023-10-31 13:16:16.405 DEBUG kube_client::client::builder: requesting
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/kube-client-0.85.0/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://192.168.0.1:443/api/v1/namespaces/bckid-devops-prod/pods/v-kubernetes-action-78fdb54ffb-pqrwv/exec?&stdin=true&stdout=true&tty=true&container=v-kubernetes-action&command=bash, otel.name: "exec", otel.kind: "client"

  2023-10-31 13:16:16.405 DEBUG hyper::client::connect::http: connecting to 192.168.0.1:443
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-0.14.27/src/client/connect/http.rs:537
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://192.168.0.1:443/api/v1/namespaces/bckid-devops-prod/pods/v-kubernetes-action-78fdb54ffb-pqrwv/exec?&stdin=true&stdout=true&tty=true&container=v-kubernetes-action&command=bash, otel.name: "exec", otel.kind: "client"

  2023-10-31 13:16:16.407 DEBUG hyper::client::connect::http: connected to 192.168.0.1:443
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-0.14.27/src/client/connect/http.rs:540
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://192.168.0.1:443/api/v1/namespaces/bckid-devops-prod/pods/v-kubernetes-action-78fdb54ffb-pqrwv/exec?&stdin=true&stdout=true&tty=true&container=v-kubernetes-action&command=bash, otel.name: "exec", otel.kind: "client"

  2023-10-31 13:16:16.413 DEBUG hyper::proto::h1::io: flushed 1297 bytes
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-0.14.27/src/proto/h1/io.rs:342

  2023-10-31 13:16:16.448 DEBUG hyper::proto::h1::io: parsed 4 headers
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-0.14.27/src/proto/h1/io.rs:207

  2023-10-31 13:16:16.448 DEBUG hyper::proto::h1::conn: incoming body is empty
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-0.14.27/src/proto/h1/conn.rs:222

End of terminal size stream
thread 'tokio-runtime-worker' panicked at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/tokio-1.32.0/src/runtime/task/core.rs:381:22:
JoinHandle polled after completion
  2023-10-31 13:16:19.178 DEBUG hyper::proto::h1::decode: incoming chunked header: 0x89 (137 bytes)
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-0.14.27/src/proto/h1/decode.rs:298

  2023-10-31 13:16:19.525 DEBUG hyper::proto::h1::decode: incoming chunked header: 0x30C (780 bytes)
    at /Users/vincent/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/hyper-0.14.27/src/proto/h1/decode.rs:298

[vCassius]

this problem occur only when running inside kubernetes cluster with a container, tryed on amd64/debian:12.1-slim & amd64/centos:centos7,same result,and tryed use token or serviceAccount to access apiserver.same result.but if run in my macbook,or linux server,it's running good