Convert all component command-line flags to versioned configuration

[bgrant0607]

Forked from #1627, #5472, #246, #12201, and other issues.

We configure Kubernetes components like apiserver and controller-manager with command-line flags currently. Every time we change these flags, we break everyone's turnup automation, at least for anything not in our repo: hosted products, other repos, blog posts, ...

OTOH, the scheduler and kubectl read configuration files. We should be treating all configuration similarly, as versioned APIs, with backward compatibility, deprecation policies, documentation, etc. How we distribute the configuration is a separate issue -- #1627.

cc @davidopp @thockin @dchen1107 @lavalamp

[bgrant0607]

Scheduler config, as an example: https://github.com/kubernetes/kubernetes/tree/master/plugin/pkg/scheduler/api

[bgrant0607]

And Openshift:
https://github.com/openshift/origin/blob/master/pkg/cmd/server/api/v1/types.go

[bgrant0607]

And kubeconfig: https://github.com/kubernetes/kubernetes/tree/master/pkg/client/clientcmd/api

[stefwalter]

@mikedanese I started yesterday to convert KubeletServer (which is poorly named) to use a JSON settings file. I'd be happy to follow the lead of your work done on the scheduler config, as it does cover a couple things I missed.

[mikedanese]

I haven't done much with this yet. I've begun to add a component config api group in this commit:

a8b00ed

Is this a similar direction you are taking? Let's discuss how we want to redistribute the configuration work tomorrow.

[pmorie]

@mikedanese @stefwalter Can you guys confirm that you intend to combine this with the mechanism described in #6477 ?

[mikedanese]

@pmorie Yes it is. It is related but separate and can be done in parallel. The combination should close #1627.

[mikedanese]

@stefwalter can you please point me to the branch where you are working on this?

[bgrant0607]

All new configuration formats we create should go into new API groups #12951

[stefwalter]

@mikedanese I stopped working on it once I saw your work. I did about half an hour ... before posting what I was doing ... https://github.com/stefwalter/kubernetes/tree/kubelet-settings

I'd be happy to help out with the tedious bits or wherever necessary.

[bgrant0607]

Some additional motivation:

The current way in which we configure components via command-line flags is unnecessarily convoluted and hard to standardize, since some deployments run the components using containers, some use init, some use systemd, some use supervisord, etc.

Life of configuring a flag on GCE (if wrong, that's another indication of how convoluted it is; different for every cloud platform):

1. Set an environment variable (e.g., ADMISSION_CONTROL) in a config-default.sh script that is invoked by some script invoked indirectly by kube-up.sh: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-default.sh
2. Which is semi-manually translated to a YAML file and stuffed into VM metadata: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/debian/helper.sh
3. Which is used to generate a Salt pillar on the master node: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/configure-vm.sh
4. Which is used to template-substitute the apiserver's pod config: https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/kube-apiserver/kube-apiserver.manifest

[davidopp]

Not for 1.1

[kfox1111]

I'm not sure if this is relevant or not to the conversation, but if daemonsets are enhanced to support the linked feature, along with daemonset rolling upgrade support, maybe this will be useful:
#43112

[roberthbailey]

@mikedanese is going to be driving the component configuration standardization for 1.7.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[bgrant0607]

/remove-lifecycle stale
/lifecycle frozen

[mtaufen]

Guide on how to implement componentconfig: https://docs.google.com/document/d/1FdaEJUEh091qf5B98HM6_8MS764iXrxxigNIdwHYW9c/

[kfox1111]

doc is private

[mtaufen]

It is shared with the kubernetes-dev group.

…

On Wed, Mar 28, 2018, 3:44 PM kfox1111 ***@***.***> wrote: doc is private — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12245 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3JwRSjLc82GqL2q4I8Y0_Nk07k8CAqks5tjBJbgaJpZM4Flyil> .

[neolit123]

this topic has multiple stakeholding sigs, instead of cluster-lifecycle as the owner.
tagging sig-arch as the umbrella.

/sig architecture
/remove-sig cluster-lifecycle

the design idea is already in place and the component owners are graduating their config to v1.

• kube-schduler - v1
• kube-controller-manager - v1alpha1
• kube-apiserver - none yet
• kube-proxy - v1alpha1
• kubeadm - v1beta2
• kubelet - v1beta1

[zqzten]

Apologize for the bothering, I'm wondering if there's any update on the progress of this issue? Recently I found controller-runtime deprecating its ComponentConfig implementation, so I comed up with this issue in k/k, and wanted to know the community's current opinion and roadmap about this.

[neolit123]

Apologize for the bothering, I'm wondering if there's any update on the progress of this issue? Recently I found controller-runtime deprecating its ComponentConfig implementation, so I comed up with this issue in k/k, and wanted to know the community's current opinion and roadmap about this.

similarly to controller runtime, k/k componentconfig apis have made a rather slow progress into graduation to v1. some api versions are stale and should be higher - i.e. maturity is not accurately represented.

there has not been a discussion to drop the effort entirety. the apis are widely used, despite not beeing v1 - e.g. kubelet is still at v1beta1. removing such apis will be very disruptive to users. instead we should push them to graduation and maintain them.

[zqzten]

@neolit123 Thanks for the rapid reply! I would also be happy to see these configurations get maintained, is there anybody or group currently responsible for this?

Also, I've found that not every component of k/k has supported being configured by a versioned configration file, and for kubelet, it implemented the whole processing logic in its own code base which cannot be easily reused. So I'm wondering if there's any chance that we build a common mechanism (e.g. in the component-base) which can be reused across all the components of k/k, and even for other sub projects (such as the controller-runtime above). This way we can achieve a unified behavior as well as reducing the overall maintenance cost.

[neolit123]

@neolit123 Thanks for the rapid reply! I would also be happy to see these configurations get maintained, is there anybody or group currently responsible for this?

currently, different groups / sigs own components and their respective configs.

Also, I've found that not every component of k/k has supported being configured by a versioned configration file, and for kubelet, it implemented the whole processing logic in its own code base which cannot be easily reused. So I'm wondering if there's any chance that we build a common mechanism (e.g. in the component-base) which can be reused across all the components of k/k, and even for other sub projects (such as the controller-runtime above). This way we can achieve a unified behavior as well as reducing the overall maintenance cost.

a working group called "component standard" would have been the appropriate forum for this discussion but it disbanded. currently, each component has the freedom to decide on the versioned config implementation details. if you want to open the discussion the next suitable place would be the sig architecture regular zoom call.

https://github.com/kubernetes/community/blob/master/sig-architecture/README.md#meetings

please add the topic to the agenda doc there if you'd like.