Add TLS support to exec authenticator plugin #61803

awly · 2018-03-28T01:19:33Z

What this PR does / why we need it:
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/kubectl-exec-plugins.md#tls-client-certificate-support

Allows exec plugin to return raw TLS key/cert data. This data populates
transport.Config.TLS field.
This requires a change to AuthProvider interface to expose TLS configs,
not only RoundTripper.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #61421

Special notes for your reviewer:

Release note:

Exec authenticator plugin supports TLS client certificates.

ericchiang · 2018-03-28T14:59:31Z

/ok-to-test

awly · 2018-03-28T17:43:11Z

/retest

make bazel-test passes locally
make bazel-test-integration fails at //test/integration/master:go_default_test on this PR and parent commit

ericchiang · 2018-03-28T17:48:49Z

staging/src/k8s.io/client-go/rest/plugin.go

 )

 type AuthProvider interface {
-	// WrapTransport allows the plugin to create a modified RoundTripper that
-	// attaches authorization headers (or other info) to requests.
-	WrapTransport(http.RoundTripper) http.RoundTripper


I think you can make this change without changing the interface since the exec provider doesn't use it. If you want to keep the diff smaller.

Good point, reverted interface changes

ericchiang · 2018-03-28T17:50:23Z

staging/src/k8s.io/client-go/transport/transport.go

 	if err != nil {
 		return err
 	}

-	c.TLS.CertData, err = dataFromSliceOrFile(c.TLS.CertData, c.TLS.CertFile)
+	c.TLS.CertData, err = dataFromSliceFuncOrFile(c.TLS.GetCert, c.TLS.CertData, c.TLS.CertFile)


I expected using crypto/tls.Config.GetClientCertificate instead. That way you could actually rotate these on a live transport.

This is only called on client create. I think we also need to close old connections when a rotation occurs. See:

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/certificate/transport.go#L149

Implemented GetClientCertificate.
Will work on getting old connections closed on cert rotation next.

ericchiang · 2018-03-28T17:51:21Z

This looks like a real failure

--- FAIL: TestTLSCredentials (0.05s)
    --- FAIL: TestTLSCredentials/valid_TLS_cert (0.00s)
    	exec_test.go:508: TLSConfigFor: decode stdout: couldn't get version/kind; json parse error: invalid character '\n' in string literal
    --- FAIL: TestTLSCredentials/untrusted_TLS_cert (0.00s)
    	exec_test.go:508: TLSConfigFor: decode stdout: couldn't get version/kind; json parse error: invalid character '\n' in string literal
    --- FAIL: TestTLSCredentials/valid_TLS_cert_again (0.00s)
    	exec_test.go:508: TLSConfigFor: decode stdout: couldn't get version/kind; json parse error: invalid character '\n' in string literal
FAIL

awly · 2018-03-28T23:17:50Z

Added connection rotation on cert change, PTAL

awly · 2018-03-29T23:04:58Z

Tests fixed (thanks @mikedanese for help).

mikedanese · 2018-04-02T23:02:17Z

staging/src/k8s.io/client-go/util/conntrack/conntrack.go

@@ -0,0 +1,102 @@
+/*
+Copyright 2017 The Kubernetes Authors.


awly · 2018-04-03T00:26:19Z

/retest

mikedanese

looks pretty good, can you squash fixup commits?

mikedanese · 2018-04-02T23:08:08Z

staging/src/k8s.io/client-go/util/conntrack/conntrack.go

+
+// Package conntrack implements a connection dialer that tracks and can close
+// all created connections.
+package conntrack


Just a note, conntrack is already a pretty overloaded name.

It'd be great to make this internal, but that won't be easy between k8s.io/kubernetes and k8s.io/client-go.

Please pick a different name.

mikedanese · 2018-04-02T23:18:45Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/exec/exec.go

+		token: cred.Status.Token,
+	}
+
+	// Avoid setting non-nil key/cert when plugin didn't return them.


When do you imagine this would happen? Why is cert data treated differently than token data?

This would happen if exec plugin provides tokens.
Since a.cachedCreds.key/cert are []byte fields, i want to keep them nil instead of []byte{} when plugin didn't return key/cert. This just makes checks cleaner everywhere (foo == nil vs len(foo) == 0)

Clarified that it's about []byte{} in the comment

mikedanese · 2018-04-02T23:22:58Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/exec/exec.go

@@ -191,40 +229,66 @@ func (r *roundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
 	return res, nil
 }

-func (a *Authenticator) tokenExpired() bool {
+func (a *Authenticator) credsExpired() bool {
 	if a.exp.IsZero() {
 		return false
 	}
 	return a.now().After(a.exp)
 }

 func (a *Authenticator) token() (string, error) {


Make these names consistent. Rename to getToken() or rename getCert() to cert() and getKey() to key(). I think I prefer cert() and key().

mikedanese · 2018-04-03T00:57:03Z

staging/src/k8s.io/client-go/transport/transport_test.go

-				continue
+				t.Fatalf("got %#v, expected no TLSClientConfig", transport)
+			}
+			if testCase.TLS {


Here and for TLSCert, do like you do above:

if !testCase.TLS{ return }

awly · 2018-04-03T17:20:50Z

Squashed all commits

ericchiang · 2018-04-03T18:11:37Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/exec/exec.go

+	return creds.token, nil
+}
+
+func (a *Authenticator) cert() ([]byte, error) {


This has the potential to race with key()

cert := a.cert() // key could rotate here key := a.key()

Nice catch. Merged GetCert/GetKey and cert/key here.

ericchiang · 2018-04-03T18:13:13Z

staging/src/k8s.io/client-go/transport/config.go

@@ -98,4 +103,7 @@ type TLSConfig struct {
 	CAData   []byte // Bytes of the PEM-encoded server trusted root certificates. Supercedes CAFile.
 	CertData []byte // Bytes of the PEM-encoded client certificate. Supercedes CertFile.
 	KeyData  []byte // Bytes of the PEM-encoded client key. Supercedes KeyFile.
+
+	GetCert func() ([]byte, error) // Callback that returns PEM-encoded client certificate. Supercedes CertData and CertFile.


To avoid GetCert and GetKey racing, maybe just:

GetCert func() (*tls.Certificate, error)

ericchiang · 2018-05-30T19:50:26Z

/test pull-kubernetes-e2e-kops-aws

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/kubectl-exec-plugins.md#tls-client-certificate-support Allows exec plugin to return raw TLS key/cert data. This data populates transport.Config.TLS fields. transport.Config.TLS propagates custom credentials using tls.Config.GetClientCertificate callback. On key/cert rotation, all connections using old credentials are closed

liggitt · 2018-05-30T21:28:55Z

/lgtm
/status approved-for-milestone

liggitt · 2018-05-30T21:36:34Z

@awly can you open a doc PR against the release-1.11 branch of https://github.com/kubernetes/website updating the doc for https://github.com/kubernetes/website/blob/release-1.11/content/en/docs/reference/access-authn-authz/authentication.md#input-and-output-formats with info about the TLS response

liggitt · 2018-05-30T21:38:04Z

/assign @cblecker @sttts
for approval of added godep package dependency

k8s-github-robot · 2018-05-30T21:38:20Z

[MILESTONENOTIFIER] Milestone Pull Request: Up-to-date for process

@awly @cblecker @ericchiang @liggitt @mikedanese @sttts

Pull Request Labels

sig/api-machinery sig/auth: Pull Request will be escalated to these SIGs if needed.
priority/important-soon: Escalate to the pull request owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
kind/feature: New functionality.

Help

cblecker · 2018-05-30T22:01:15Z

All internal packages
/approve

k8s-ci-robot · 2018-05-30T22:01:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: awly, cblecker, ericchiang, liggitt, mikedanese

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiextensions-apiserver/Godeps/OWNERS~~ [cblecker]
~~staging/src/k8s.io/apiserver/Godeps/OWNERS~~ [cblecker,liggitt]
~~staging/src/k8s.io/client-go/OWNERS~~ [liggitt]
~~staging/src/k8s.io/kube-aggregator/Godeps/OWNERS~~ [cblecker]
~~staging/src/k8s.io/metrics/Godeps/OWNERS~~ [cblecker]
~~staging/src/k8s.io/sample-apiserver/Godeps/OWNERS~~ [cblecker,liggitt]
~~staging/src/k8s.io/sample-controller/Godeps/OWNERS~~ [cblecker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

liggitt · 2018-05-30T22:14:00Z

/retest

awly · 2018-05-30T22:17:39Z

@liggitt sent docs PR kubernetes/website#8826

awly · 2018-05-30T23:06:00Z

/retest

k8s-github-robot · 2018-05-31T00:34:11Z

Automatic merge from submit-queue (batch tested with PRs 61803, 64305, 64170, 64361, 64339). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot · 2018-05-31T00:36:18Z

@awly: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-kubemark-e2e-gce-big	`cd89f94`	link	`/test pull-kubernetes-kubemark-e2e-gce-big`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

* Update exec plugin docs with TLS credentials kubernetes/kubernetes#61803 implements TLS client credential support for 1.11. * Copyedit * More copyedits for clarification * Additional copyedit * Change token->credential

The regular kubeconfig is fetched from metadata when CREATE_BOOTSTRAP_KUBECONFIG==false. We will experiment with an exec plugin that does TLS bootstrapping internally: kubernetes#61803

* Update exec plugin docs with TLS credentials kubernetes/kubernetes#61803 implements TLS client credential support for 1.11. * Copyedit * More copyedits for clarification * Additional copyedit * Change token->credential

* Seperate priority and preemption (#8144) * Doc about PID pressure condition. (#8211) * Doc about PID pressure condition. Signed-off-by: Da K. Ma <klaus1982.cn@gmail.com> * "so" -> "too" * Update version selector for 1.11 * StorageObjectInUseProtection is GA (#8291) * Feature gate: StorageObjectInUseProtection is GA Update feature gate reference for 1.11 * Trivial commit to re-trigger Netlify * CRIContainerLogRotation is Beta in 1.11 (#8665) * Seperate priority and preemption (#8144) * CRIContainerLogRotation is Beta in 1.11 xref: kubernetes/kubernetes#64046 * Bring StorageObjectInUseProtection feature to GA (#8159) * StorageObjectInUseProtection is GA (#8291) * Feature gate: StorageObjectInUseProtection is GA Update feature gate reference for 1.11 * Trivial commit to re-trigger Netlify * Bring StorageObjectInUseProtection feature to GA StorageObjectInUseProtection is Beta in K8s 1.10. It's brought to GA in K8s 1.11. * Fixed typo and added feature state tags. * Remove KUBE_API_VERSIONS doc (#8292) The support to the KUBER_API_VERSIONS environment variable is completely dropped (no deprecation). This PR removes the related doc in release-1.11. xref: kubernetes/kubernetes#63165 * Remove InitialResources from admission controllers (#8293) The feature (was experimental) is dropped in 1.11. xref: kubernetes/kubernetes#58784 * Remove docs related to in-tree support to GPU (#8294) * Remove docs related to in-tree support to GPU The in-tree support to GPU is completely removed in release 1.11. This PR removes the related docs in release-1.11 branch. xref: kubernetes/kubernetes#61498 * Update content updated by PR to Hugo syntax Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Update the doc about extra volume in kubeadm config (#8453) Signed-off-by: Xianglin Gao <xianglin.gxl@alibaba-inc.com> * Update CRD Subresources for 1.11 (#8519) * coredns: update notes in administer-cluster/coredns.md (#8697) CoreDNS is installed by default in 1.11. Add notes on how to install kube-dns instead. Update notes about CoreDNS->CoreDNS upgrades as in 1.11 the Corefile is retained. Add example on upgrading from kube-dns to CoreDNS. * kubeadm-alpha: CoreDNS related changes (#8727) Update note about CoreDNS feature gate. This change also updates a tab as a kubeadm sub-command will change. It looks for a new generated file: generated/kubeadm_alpha_phase_addon_coredns.md instead of: generated/kubeadm_alpha_phase_addon_kube-dns.md * Update cloud controller manager docs to beta 1.11 (#8756) * Update cloud controller manager docs to beta 1.11 * Use Hugo shortcode for feature state * kubeadm-upgrade: include new command `kubeadm upgrade diff` (#8617) Also: - Include note that this was added in 1.11. - Modify the note about upgrade guidance. * independent: update CoreDNS mentions for kubeadm (#8753) Give CoreDNS instead of kube-dns examples in: - docs/setup/independent/create-cluster-kubeadm.md - docs/setup/independent/troubleshooting-kubeadm.md * update 1.11 --server-print info (#8870) * update 1.11 --server-print info * Copyedit * Mark ExpandPersistentVolumes feature to beta (#8778) * Update version selector for 1.11 * Mark ExpandPersistentVolumes Beta xref: kubernetes/kubernetes#64288 * fix shortcode, add placeholder files to fix deploy failures (#8874) * declare ipvs ga (#8850) * kubeadm: update info about CoreDNS in kubeadm-init.md (#8728) Add info to install kube-dns instead of CoreDNS, as CoreDNS is the default DNS server in 1.11. Add notes that kubeadm config images can be used to list and pull the required images in 1.11. * kubeadm: update implementation-details.md about CoreDNS (#8829) - Replace examples from kube-dns to CoreDNS - Add notes about the CoreDNS feature gate status in 1.11 - Add note that the service name for CoreDNS is also called `kube-dns` * Update block device support for 1.11 (#8895) * Update block device support for 1.11 * Copyedits * Fix typo 'fiber channel' (#8957) Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * kubeadm-upgrade: add the 'node [config]' sub-command (#8960) - Add includes for the generated pages - Include placeholder generated pages * kubeadm-init: update the example for the MasterConfiguration (#8958) - include godocs link for MasterConfiguration - include example MasterConfiguration - add note that `kubeadm config print-default` can be used * kubeadm-config: include new commands (#8862) Add notes and includes for these new commands in 1.11: - kubeadm config print-default - kubeadm config migrate - kubeadm config images list - kubeadm config images pull Include placeholder generated files for the above. * administer-cluster/coredns: include more changes (#8985) It was requested that for this page a couple of methods should be outlined: - manual installation for CoreDNS explained at the Kubernetes section of the GitHub project for CoreDNS - installation and upgrade via kubeadm Make the above changes and also add a section "About CoreDNS". This commit also lowercases a section title. * Update CRD subresources doc for 1.11 (#8918) * Add docs for volume expansion and online resizing (#8896) * Add docs for volume expansion going beta * Copyedit * Address feedback * Update exec plugin docs with TLS credentials (#8826) * Update exec plugin docs with TLS credentials kubernetes/kubernetes#61803 implements TLS client credential support for 1.11. * Copyedit * More copyedits for clarification * Additional copyedit * Change token->credential * NodeRestriction admission prevents kubelet taint removal (#8911) * dns-custom-namerserver: break down the page into mutliple sections (#8900) * dns-custom-namerserver: break down the page into mutliple sections This page is currently about kube-dns and is a bit outdated. Introduce the heading `# Customizing kube-dns`. Introduce a separate section about CoreDNS. * Copyedits, fix headings for customizing DNS Hey Lubomir, I coypedited pretty heavily because this workflow is so much easier for docs and because I'm trying to help improve everything touching kubeadm as much as possible. But there's one outstanding issue wrt headings and intro content: you can't add a heading 1 to a topic to do what you wanted to do. The page title in the front matter is rendered as a heading 1 and everything else has to start at heading 2. (We still need to doc this better in the docs contributing content, I know.) Instead, I think we need to rewrite the top-of-page intro content to explain better the relationship between kube-dns and CoreDNS. I'm happy to write something, but I thought I'd push this commit first so you can see what I'm doing. Hope it's all clear -- ping here or on Slack with any questions ~ Jennifer * Interim fix for talking about CoreDNS * Fix CoreDNS details * PSP readOnly hostPath (#8898) * Add documentation for crictl (#8880) * Add documentation for crictl * Copyedit Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Final copyedit * VolumeSubpathEnvExpansion alpha feature (#8835) * Note that Heapster is deprecated (#8827) * Note that Heapster is deprecated This notes that Heapster is deprecated, and migrates the relevant docs to talk about metrics-server or other solutions by default. * Copyedits and improvements Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Address feedback * fix shortcode to troubleshoot deploy (#9057) * update dynamic kubelet config docs for v1.11 (#8766) * update dynamic kubelet config docs for v1.11 * Substantial copyedit * Address feedback * Reference doc for kubeadm (release-1.11) (#9044) * Reference doc for kubeadm (release-1.11) * fix shortcode to troubleshoot deploy (#9057) * Reference doc for kube-components (release-1.11) (#9045) * Reference doc for kube-components (release-1.11) * Update cloud-controller-manager.md * fix shortcode to troubleshoot deploy (#9057) * Documentation on lowercasing kubeadm init apiserver SANs (#9059) * Documentation on lowercasing kubeadm init apiserver SANs * fix shortcode to troubleshoot deploy (#9057) * Clarification in dynamic Kubelet config doc (#9061) * Promote sysctls to Beta (#8804) * Promote sysctls to Beta * Copyedits Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Review comments * Address feedback * More feedback * kubectl reference docs for 1.11 (#9080) * Update Kubernetes API 1.11 ref docs (#8977) * Update v1alpha1 to v1beta1. * Adjust left nav for 1.11 ref docs. * Trim list of old ref docs. * Update Federation API ref docs for 1.11. (#9064) * Update Federation API ref docs for 1.11. * Add titles. * Update definitions.html * CRD versioning Public Documentation (#8834) * CRD versioning Public Documentation * Copyedit Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com> * Address feedback * More rewrites * Address feedback * Update main CRD page in light of versioning * Reorg CRD docs * Further reorg * Tweak title * CSI documentation update for raw block volume support (#8927) * CSI documetation update for raw block volume support * minor edits for "CSI raw block volume support" Some small grammar and style nits. * minor CSIBlockVolume edits * Update kubectl component ref page for 1.11. (#9094) * Update kubectl component ref page for 1.11. * Add title. Replace stevepe with username. * crd versioning doc: fix nits (#9142) * Update `DynamicKubeletConfig` feature to beta (#9110) xref: kubernetes/kubernetes#64275 * Documentation for dynamic volume limits based on node type (#8871) * add cos for storage limits * Update docs specific for aws and gce * fix some minor things * Update storage-limits.md * Add k8s version to feature-state shortcode * The Doc update for ScheduleDaemonSetPods (#8842) Signed-off-by: Da K. Ma <klaus1982.cn@gmail.com> * Update docs related to PersistentVolumeLabel admission control (#9109) The said admission controller is disabled by default in 1.11 (kubernetes/kubernetes#64326) and scheduled to be removed in future release. * client exec auth: updates for 1.11 (#9154) * Updates HA kubeadm docs (#9066) * Updates HA kubeadm docs Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * kubeadm HA - Add stacked control plane steps * ssh instructions and some typos in the bash scripts Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * Fix typos and copypasta errors * Fix rebase issues * Integrate more changes Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * copyedits, layout and formatting fixes * final copyedits * Adds a sanity check for load balancer connection Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * formatting fixes, copyedits * fix typos, formatting * Document the Pod Ready++ feature (#9180) Closes: #9107 Xref: kubernetes/kubernetes#64057 * Mention 'KubeletPluginsWatcher' feature (#9177) * Mention 'KubeletPluginsWatcher' feature This feature is more developers oriented than users oriented, so simply mention it in the feature gate should be fine. In future, when the design doc is migrated from Google doc to the kubernetes/community repo, we can add links to it for users who want to dig deeper. Closes: #9108 Xref: kubernetes/kubernetes#63328, kubernetes/kubernetes#64605 * Copyedit * Amend dynamic volume list docs (#9181) The dynamic volume list feature has been documented but the feature gate related was not there yet. Closes: #9105 * Document for service account projection (#9182) This adds docs for the service account projection feature. Xref: kubernetes/kubernetes#63819, kubernetes/community#1973 Closes: #9102 * Update pod priority and preemption user docs (#9172) * Update pod priority and preemption user docs * Copyedit * Documentation on setting node name with Kubeadm (#8925) * Documentation on setting node name with Kubeadm * copyedit * Add kubeadm upgrade docs for 1.11 (#9089) * Add kubeadm upgrade docs for 1.11 * Initial docs review feedback * Add 1-11 to outline * Fix formatting on tab blocks * Move file to correct location * Add `kubeadm upgrade node config` step * Overzealous ediffing * copyedit, fix lists and headings * clarify --force flag for fixing bad state * Get TOML ready for 1.11 release * Blog post for 1.11 release (#9254) * Blog post for 1.11 release * Update 2018-06-26-kubernetes-1.11-release-announcement.md * Update 2018-06-26-kubernetes-1.11-release-announcement.md * Update 2018-06-26-kubernetes-1.11-release-announcement.md

k8s-ci-robot requested review from brendandburns, caesarxuchao, cjcullen, ericchiang and nikhiljindal March 28, 2018 01:19

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 28, 2018

mikedanese assigned mikedanese and ericchiang Mar 28, 2018

mikedanese requested review from mikedanese and removed request for caesarxuchao, brendandburns, cjcullen and nikhiljindal March 28, 2018 17:41

ericchiang reviewed Mar 28, 2018

View reviewed changes

mikedanese reviewed Apr 2, 2018

View reviewed changes

mikedanese reviewed Apr 3, 2018

View reviewed changes

awly force-pushed the client-auth-exec-tls branch from d5e6daf to 8b78a7e Compare April 3, 2018 17:20

ericchiang reviewed Apr 3, 2018

View reviewed changes

awly force-pushed the client-auth-exec-tls branch from ea21062 to b1e6898 Compare May 30, 2018 19:51

awly force-pushed the client-auth-exec-tls branch from b1e6898 to cd89f94 Compare May 30, 2018 21:04

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. status/approved-for-milestone labels May 30, 2018

k8s-github-robot removed the milestone/needs-approval label May 30, 2018

k8s-ci-robot assigned cblecker and sttts May 30, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 30, 2018

awly mentioned this pull request May 30, 2018

Update exec plugin docs with TLS credentials kubernetes/website#8826

Merged

k8s-github-robot merged commit f701b75 into kubernetes:master May 31, 2018

awly deleted the client-auth-exec-tls branch July 26, 2018 17:54

Add TLS support to exec authenticator plugin #61803

Add TLS support to exec authenticator plugin #61803

Conversation

awly commented Mar 28, 2018

ericchiang commented Mar 28, 2018

awly commented Mar 28, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikedanese Mar 28, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericchiang commented Mar 28, 2018

awly commented Mar 28, 2018

awly commented Mar 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awly commented Apr 3, 2018

mikedanese left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awly commented Apr 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericchiang Apr 3, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericchiang commented May 30, 2018

liggitt commented May 30, 2018

liggitt commented May 30, 2018

liggitt commented May 30, 2018

k8s-github-robot commented May 30, 2018

cblecker commented May 30, 2018

k8s-ci-robot commented May 30, 2018

liggitt commented May 30, 2018

awly commented May 30, 2018

awly commented May 30, 2018

k8s-github-robot commented May 31, 2018

k8s-ci-robot commented May 31, 2018

awly commented Mar 28, 2018 •

edited

mikedanese Mar 28, 2018 •

edited

ericchiang Apr 3, 2018 •

edited