Kubectl 1.10.0 is unable to create deployments on clusters running 1.8.7

[ihoegen]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

/kind bug

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-27T00:13:02Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS: macOS 10.13.3
• Kernel: Darwin 17.4.0
• Install tools: brew

What happened:

When creating a sample deployment using the command kubectl create -f deployment.yaml, the following output appears:

Error from server (NotAcceptable): unknown

What you expected to happen:

When running the same command again, kubectl create -f deployment.yaml, with kubectl version 1.9.3, the following output appears:

deployment "nginx-deployment" created

A better error message describing incompatibility might be useful.

How to reproduce it (as minimally and precisely as possible):

1. Use the following config, and save it as deployment.yaml:

apiVersion: apps/v1beta2 
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template: 
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

2. Run kubectl create -f deployment.yaml using kubectl 1.10.0

[ihoegen]

/sig cli

[jhorwit2]

@ihoegen are you by chance a cluster admin? I can reproduce this when I impersonate a non-admin user but not as an admin. It seems to be RBAC related since /openapi/v2 isn't part of system:discovery. If you run the kubectl command with --v=6 I'm assuming it fails on that request.

[jhorwit2]

Output

I0330 13:35:54.068481   67702 loader.go:357] Config loaded from file /Users/<user>/.kube/config
I0330 13:35:54.192461   67702 round_trippers.go:405] GET https://<url>l/openapi/v2 406 Not Acceptable in 121 milliseconds
I0330 13:35:54.193710   67702 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "unknown",
  "reason": "NotAcceptable",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "unknown"
      }
    ]
  },
  "code": 406
}]
F0330 13:35:54.193740   67702 helpers.go:119] Error from server (NotAcceptable): unknown

[jhorwit2]

/sig auth

@liggitt This response isn't correct for an RBAC deny, right? I see the DENY in the apiserver logs.

[ihoegen]

You're correct, I switched to the admin role, and I was able to create it. I was able to delete it from the developer role though.

The developer rules are:

rules:
- apiGroups: ["*"] 
  resources: ["*"]
  verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]

[ihoegen]

I was able to create it with cluster-admin, and my namespace admin roles as well. Namespace admin rules are:

rules:
- apiGroups: ["*"] 
  resources: ["*"]
  verbs: ["*"]

with a clusterrole of

rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "watch", "list"]

The non admins also use this same clusterrole.

[liggitt]

looks like a duplicate of #61805

[jhorwit2]

@ihoegen this is a non-resource url so it's not covered by those rules.

[ihoegen]

Why would it work with kubectl 1.9.3 then?

…

On Fri, Mar 30, 2018, 11:03 AM Josh Horwitz ***@***.***> wrote: @ihoegen <https://github.com/ihoegen> this is a non-resource url so it's not covered by those rules. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61943 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKflaoNCdsqt03F702bpVglVLxVRZ4tLks5tjnOCgaJpZM4TB42n> .

[liggitt]

Why would it work with kubectl 1.9.3 then?

Because kubectl 1.9.x doesn't attempt to fetch that schema as a prereq to creating/updating all objects, in order to maintain compatibility with 1.8.x servers (which are within the supported version skew of +/- one version)

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Issue Labels Incomplete

@ihoegen @liggitt

Action required: This issue requires label changes. If the required changes are not made within 1 day, the issue will be moved out of the v1.10 milestone.

priority: Must specify exactly one of priority/critical-urgent, priority/important-longterm or priority/important-soon.

Help

• Additional instructions
• Commands for setting labels

[liggitt]

fixed in master in #61949
fixed in release-1.10 branch in #62024