support annotations for admission webhook #58679

CaoShuFeng · 2018-01-23T10:45:07Z

Depends on: #58143
Release note:

Support annotations for remote admission webhooks.

CaoShuFeng · 2018-01-28T11:42:36Z

@liggitt @tallclair @sttts This is also ready for review.
I had deployed a little remote admission controller to test it.

sttts · 2018-01-30T15:13:01Z

staging/src/k8s.io/api/admission/v1beta1/types.go

@@ -94,6 +94,10 @@ type AdmissionResponse struct {
 	// The type of Patch. Currently we only allow "JSONPatch".
 	// +optional
 	PatchType *PatchType `json:"patchType,omitempty" protobuf:"bytes,5,opt,name=patchType"`
+
+	// Annotations set by remote admission controller.


there should be a specified format. What are the keys?

This Annotation is a little bit different from annotation in admission attribute.
It doesn't contain the fully qualified plugin name.

I use this plugin name to call attributesRecord.SetAnnotations().

sttts · 2018-01-30T15:13:46Z

staging/src/k8s.io/apiserver/pkg/admission/attributes.go

+	if record.annotations == nil {
+		return false
+	}
+	key = plugin_name + "/" + key


plugin_name + ".admission.k8s.io/" + key ?

We also have to check that the key is a DNS segment.

We also have to check that the key is a DNS segment.

Done.

plugin_name + ".admission.k8s.io/" + key ?

plugin_name is already fully qualified.
Webhook plugin do not use .admission.k8s.io in their plugin name, I think.

sttts · 2018-01-30T15:15:02Z

staging/src/k8s.io/apiserver/pkg/admission/interfaces.go

@@ -49,6 +49,12 @@ type Attributes interface {
 	GetKind() schema.GroupVersionKind
 	// GetUserInfo is information about the requesting user
 	GetUserInfo() user.Info
+	// GetAnnotations returns annotation map for the admission chain.
+	GetAnnotations() map[string]string


can this be queried by plugins in the chain? I don't like to open up this channel without good reasons.

sttts · 2018-01-30T15:16:19Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/admission.go

@@ -299,7 +300,7 @@ func (a *ValidatingAdmissionWebhook) shouldCallHook(h *v1beta1.Webhook, attr adm
 	return a.namespaceMatcher.MatchNamespaceSelector(h, attr)
 }

-func (a *ValidatingAdmissionWebhook) callHook(ctx context.Context, h *v1beta1.Webhook, attr admission.Attributes) error {
+func (a *ValidatingAdmissionWebhook) callHook(ctx context.Context, h *v1beta1.Webhook, attr admission.Attributes, annotationLock sync.Mutex) error {


this is ugly

What about we move this mutex into SetAnnotations()?

yes. And get rid of the single k/v SetAnnotations variant, but only for a whole map.

sttts · 2018-01-31T11:07:03Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/admission.go

@@ -297,6 +297,10 @@ func (a *MutatingWebhook) callAttrMutatingHook(ctx context.Context, h *v1beta1.W
 		return &webhookerrors.ErrCallingWebhook{WebhookName: h.Name, Reason: err}
 	}

+	for k, v := range response.Response.Annotations {
+		attr.SetAnnotations(h.Name, k, v)


these names can theoretically overlap with the built in names. I guess we need h.Name + ".mutatingwebhook".

This h.Name is already full qualified, like imagepolicy.kubernetes.io.
This name is set by cluster administrator.
What about we trust cluster administrator that he will not set a name overlap with the built in names?

kubernetes/staging/src/k8s.io/api/admissionregistration/v1beta1/types.go

Line 126 in f3942e7

// Name should be fully qualified, e.g., imagepolicy.kubernetes.io, where

Not sure it's obvious that this name shows up in auditing. But imagepolicy.kubernetes.io.mutatingwebhook.admission.k8s.io is not nice either.

What about we ignore h.Name?
And allow remote admission to set pluginName themselves?
If so, remote admsssion will be consistent with the build-in admission.

we could also enforce that webhooks use *.mutatingwebhook.admission.k8s.io as key.

liggitt · 2018-07-27T15:16:12Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go

@@ -97,6 +97,13 @@ func (a *mutatingDispatcher) callAttrMutatingHook(ctx context.Context, h *v1beta
 		return &webhookerrors.ErrCallingWebhook{WebhookName: h.Name, Reason: fmt.Errorf("Webhook response was absent")}
 	}

+	for k, v := range response.Response.Annotations {
+		if err := attr.AddAnnotation(h.Name+"/"+k, v); err != nil {


~~if the key already is already namespaced to the webhook, noop?~~

edit: that would require the webhook to know the name by which it was registered, which isn't great, so forget that

erroring on an existing key means that a mutating and validating webhook with the same name cannot set the same key, since they share a namespace

erroring on an existing key means that a mutating and validating webhook with the same name cannot set the same key, since they share a namespace

Local mutating and validating admission may have same problem.
What about we trust the cluster admin here?

liggitt · 2018-07-27T15:20:22Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go

+	for k, v := range response.Response.Annotations {
+		if err := attr.AddAnnotation(h.Name+"/"+k, v); err != nil {
+			return &webhookerrors.ErrCallingWebhook{WebhookName: h.Name,
+				Reason: fmt.Errorf("Failed to set annotations for mutating webhook %s: %v.", h.Name, err)}


message should say "validating webhook"

liggitt · 2018-07-27T15:21:48Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go

@@ -97,6 +97,13 @@ func (a *mutatingDispatcher) callAttrMutatingHook(ctx context.Context, h *v1beta
 		return &webhookerrors.ErrCallingWebhook{WebhookName: h.Name, Reason: fmt.Errorf("Webhook response was absent")}
 	}

+	for k, v := range response.Response.Annotations {
+		if err := attr.AddAnnotation(h.Name+"/"+k, v); err != nil {
+			return &webhookerrors.ErrCallingWebhook{WebhookName: h.Name,


I'm not sure this should be a fatal error... the in-tree annotation error handling just logs a warning:

kubernetes/plugin/pkg/admission/security/podsecuritypolicy/admission.go

Lines 168 to 170 in 4d5d266

if err := a.AddAnnotation(key, pspName); err != nil {

glog.Warningf("failed to set admission audit annotation %s to %s: %v", key, pspName, err)

}

Updated to glog.Warningf and make them consistent with each other.

CaoShuFeng · 2018-07-31T06:51:27Z

/test pull-kubernetes-e2e-kops-aws

liggitt · 2018-08-10T20:23:32Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go

@@ -102,6 +102,12 @@ func (a *mutatingDispatcher) callAttrMutatingHook(ctx context.Context, h *v1beta
 		return &webhookerrors.ErrCallingWebhook{WebhookName: h.Name, Reason: fmt.Errorf("Webhook response was absent")}
 	}

+	for k, v := range response.Response.AuditAnnotations {
+		if err := attr.AddAnnotation(h.Name+"/"+k, v); err != nil {
+			glog.Warningf("Failed to set annotations for mutating webhook %s: %v.", h.Name, err)


should probably indicate the reason why in the warning message like the other places we do this (e.g. glog.Warningf("Failed to set annotations[%q] to %q for audit:%q, it has already been set to %q", key, value, ae.AuditID, ae.Annotations[key]))

jimangel · 2018-08-17T02:32:29Z

@CaoShuFeng do you have a PR for the 1.12 docs branch for this? Thanks!

CaoShuFeng · 2018-08-17T02:50:30Z

@CaoShuFeng do you have a PR for the 1.12 docs branch for this? Thanks!

Will work on it.

k8s-ci-robot · 2018-08-17T05:51:37Z

@CaoShuFeng: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-unit	ad223e87a26cc4fd0f1017a93d55a5e10b879215	link	`/test pull-kubernetes-unit`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@liggitt

Automatic merge from submit-queue (batch tested with PRs 55600, 67386). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. update Annotations description about audit.Event ref: #58679 (comment) **What this PR does / why we need it**: **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes # **Special notes for your reviewer**: /assign @liggitt **Release note**: ```release-note NONE ```

@liggitt

Automatic merge from submit-queue (batch tested with PRs 55600, 67386). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. update Annotations description about audit.Event ref: kubernetes/kubernetes#58679 (comment) **What this PR does / why we need it**: **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes # **Special notes for your reviewer**: /assign @liggitt **Release note**: ```release-note NONE ``` Kubernetes-commit: 4c5e6156525b96b72961b86ff5bd82c44ea0cd96

liggitt · 2018-08-22T20:11:02Z

/lgtm

k8s-ci-robot · 2018-08-22T20:11:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CaoShuFeng, liggitt, tallclair

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/apis/OWNERS~~ [liggitt]
~~staging/src/k8s.io/api/OWNERS~~ [liggitt]
~~staging/src/k8s.io/apiserver/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-08-22T21:15:57Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-08-22T22:31:33Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@liggitt

Automatic merge from submit-queue (batch tested with PRs 55600, 67386). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. update Annotations description about audit.Event ref: kubernetes/kubernetes#58679 (comment) **What this PR does / why we need it**: **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes # **Special notes for your reviewer**: /assign @liggitt **Release note**: ```release-note NONE ``` Kubernetes-commit: 4c5e6156525b96b72961b86ff5bd82c44ea0cd96

@liggitt

Automatic merge from submit-queue (batch tested with PRs 55600, 67386). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. update Annotations description about audit.Event ref: kubernetes/kubernetes#58679 (comment) **What this PR does / why we need it**: **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes # **Special notes for your reviewer**: /assign @liggitt **Release note**: ```release-note NONE ``` Kubernetes-commit: 4c5e6156525b96b72961b86ff5bd82c44ea0cd96

k8s-ci-robot requested review from deads2k, derekwaynecarr, lavalamp, liggitt, mikedanese, php-coder, soltysh, tallclair and vishh January 23, 2018 10:45

k8s-github-robot added the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label Jan 23, 2018

CaoShuFeng mentioned this pull request Jan 23, 2018

Add PodSecurityPolicy information to audit logs #58143

Merged

CaoShuFeng force-pushed the admission_webhook branch 2 times, most recently from 1d0b1cd to 0c96dec Compare January 24, 2018 06:30

CaoShuFeng mentioned this pull request Jan 25, 2018

Add PodSecurityPolicy admission information to audit logs #56209

Closed

CaoShuFeng force-pushed the admission_webhook branch from 0c96dec to 9def9cf Compare January 27, 2018 08:46

CaoShuFeng changed the title ~~[WIP] support annotations for admission webhook~~ support annotations for admission webhook Jan 28, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 28, 2018

CaoShuFeng force-pushed the admission_webhook branch from 9def9cf to 7db77c5 Compare January 28, 2018 11:45

sttts reviewed Jan 30, 2018

View reviewed changes

CaoShuFeng force-pushed the admission_webhook branch from 7db77c5 to bdb4e11 Compare January 31, 2018 09:53

sttts reviewed Jan 31, 2018

View reviewed changes

CaoShuFeng force-pushed the admission_webhook branch from bdb4e11 to 43d7def Compare February 1, 2018 10:42

liggitt reviewed Jul 27, 2018

View reviewed changes

CaoShuFeng force-pushed the admission_webhook branch from 56785ed to 0c4fa2b Compare July 31, 2018 05:24

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 31, 2018

CaoShuFeng force-pushed the admission_webhook branch from 0c4fa2b to df62d48 Compare August 8, 2018 01:40

liggitt reviewed Aug 10, 2018

View reviewed changes

CaoShuFeng force-pushed the admission_webhook branch from df62d48 to ade4f52 Compare August 14, 2018 06:14

CaoShuFeng mentioned this pull request Aug 14, 2018

update Annotations description about audit.Event #67386

Merged

CaoShuFeng force-pushed the admission_webhook branch from ade4f52 to c903285 Compare August 14, 2018 12:50

CaoShuFeng force-pushed the admission_webhook branch from c903285 to cd2307c Compare August 17, 2018 05:19

add annotations to admission webhook

b4a71bb

CaoShuFeng force-pushed the admission_webhook branch from cd2307c to 0ebfc3e Compare August 20, 2018 01:15

CaoShuFeng added 2 commits August 20, 2018 09:15

run make update

edc3e40

support annotations for admission webhook

0ebfc3e

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 22, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 22, 2018

k8s-github-robot merged commit 4e76bb4 into kubernetes:master Aug 22, 2018

	if err := a.AddAnnotation(key, pspName); err != nil {
	glog.Warningf("failed to set admission audit annotation %s to %s: %v", key, pspName, err)
	}

support annotations for admission webhook #58679

support annotations for admission webhook #58679

Conversation

CaoShuFeng commented Jan 23, 2018 • edited by sttts

CaoShuFeng commented Jan 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CaoShuFeng Jan 31, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Jul 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Jul 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CaoShuFeng commented Jul 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimangel commented Aug 17, 2018

CaoShuFeng commented Aug 17, 2018

k8s-ci-robot commented Aug 17, 2018 • edited

liggitt commented Aug 22, 2018

k8s-ci-robot commented Aug 22, 2018

k8s-github-robot commented Aug 22, 2018

k8s-github-robot commented Aug 22, 2018

CaoShuFeng commented Jan 23, 2018 •

edited by sttts

CaoShuFeng Jan 31, 2018 •

edited

liggitt Jul 27, 2018 •

edited

liggitt Jul 27, 2018 •

edited

k8s-ci-robot commented Aug 17, 2018 •

edited