Skip to content

Latest commit

 

History

History
183 lines (115 loc) · 9.65 KB

MachineConfigController.md

File metadata and controls

183 lines (115 loc) · 9.65 KB
Raw

MachineConfigController

Goals

  1. Coordinate upgrade of machines to desired configurations defined by a MachineConfig object.

  2. Provide options to control upgrade for sets of machines individually.

Non Goals

  1. MachineConfigController is not responsible for updating the machines directly.

  2. MachineConfigController is not responsible for serving Ignition configs to new machines trying to join the cluster.

Sub Controllers

  1. TemplateController is responsible for generating the MachineConfigs for pre-defined roles of machines from internal templates based on cluster configuration.

  2. UpdateController is responsible for upgrading machines to desired MachineConfig by coordinating with a daemon running on each machine.

  3. RenderController is responsible for discovering MachineConfigs for a Pool of Machines and generating the static MachineConfig.

  4. KubeletConfigController is responsible for wrapping custom Kubelet configurations within a CRD. The available options are documented within the KubeletConfiguration (https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go#L45).

MachineConfigPool

type MachineConfigPool struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec MachineConfigPoolSpec `json:"spec"`
    Status MachineConfigPoolStatus `json:"status"`
}

type MachineConfigPoolSpec struct {
    // Label selector for MachineConfigs.
    // Refer https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ on how label and selectors work.
    MachineConfigSelector *metav1.LabelSelector `json:"machineConfigSelector,omitempty"`

    // Label selector for Machines.
    NodeSelector *metav1.LabelSelector `json:"nodeSelector,omitempty"`

    // If true, changes to this machine config pool should be stopped.
    // This includes generating new desiredMachineConfig and update of machines.
    Paused bool `json:"paused"`

    // MaxUnavailable defines either an integer number or percentage
    // of nodes in the pool that can go Unavailable during an update.
    // This includes nodes Unavailable for any reason, including user
    // initiated cordons, failing nodes, etc. The default value is 1.
    //
    // A value larger than 1 will mean multiple nodes going unavailable during
    // the update, which may affect your workload stress on the remaining nodes.
    // You cannot set this value to 0 to stop updates (it will default back to 1);
    // to stop updates, use the 'paused' property instead. Drain will respect
    // Pod Disruption Budgets (PDBs) such as etcd quorum guards, even if
    // MaxUnavailable is greater than one.
    MaxUnavailable *intstr.IntOrString `json:"maxUnavailable"`
}

type MachineConfigPoolStatus struct {
    // The generation observed by the controller.
    ObservedGeneration int64 `json:"observedGeneration,omitempty"`

    // The current MachineConfig object for the machine config pool.
    CurrentMachineConfig string `json:"currentMachineConfig"`

    // Total number of machines in the machine config pool.
    MachineCount int32 `json:"machineCount"`

    // Total number of machines targeted by the pool that have the CurrentMachineConfig as their config.
    UpdatedMachineCount int32 `json:"updatingMachines"`

    // Total number of ready machines targeted by the pool.
    ReadyMachineCount int32 `json:"readyMachines"`

    // Total number of unavailable (non-ready) machines targeted by the pool.
    // A node is marked unavailable if it is in updating state or NodeReady condition is false.
    UnavailableMachineCount int32 `json:"unavailableMachines"`

    // Represents the latest available observations of current state.
    Conditions []MachineConfigPoolConditions `json:"conditions"`
}

MachineSets vs MachineConfigPool

  • MachineSets describe nodes with respect to cloud / machine provider. MachineConfigPool allows MachineConfigController components to define and provide status of machines in context of upgrades.

  • MachineConfigPool also allows users to configure how upgrades are rolled out to the machines in a pool.

  • NodeSelector can be replaced with reference to MachineSet.

TemplateController

The TemplateController uses internal templates and a configuration object to generate OpenShift-owned MachineConfig objects for pre-defined roles like master, worker etc. These templates are stored in the templates/ directory of this repository. For example, the templates/master/00-master directory will yield a MachineConfig object named 00-master for the master role. Template variables such as {{.EtcdCAData}} are filled in using the controllerconfig custom resource (e.g. controllerconfig/machine-config-controller).

  • The TemplateController constantly reconciles the MachineConfig objects in the cluster to match its internal state (which is essentially: baked-in templates + controllerconfig). The TemplateController will overwrite any user changes of its owned objects.

  • TemplateController watches changes to the controllerconfig to generate OpenShift-owned MachineConfig objects.

  • TemplateController adds OwnerReference or similar annotations on its objects to declare ownership.

RenderController

The RenderController generates the desired MachineConfig object based on the MachineConfigSelector defined in MachineConfigPool.

  • RenderController watches for changes on MachineConfigPool object to find all the MachineConfig objects using MachineConfigSelector and updating the CurrentMachineConfig with the generated MachineConfig.

  • RenderController watches for changes on all the MachineConfig objects and syncs all the MachineConfigPool objects with new CurrentMachineConfig.

Finding MachineConfigs

Use kubernetes Deployment behavior for LabelSelector to find Pods.

Generating desired MachineConfig

Use the merging behavior defined in MachineConfig design document here to create a single MachineConfig from all the MachineConfig objects that were selected above.

Ordering the MachineConfigs

The render controller sorts all the other MachineConfigs based on the lexicographically increasing order of their Name. It uses the first MachineConfig in the list as the base and appends the rest to the base MachineConfig.

UpdateController

The UpdateController coordinates upgrade for machines in a MachineConfigPool. UpdateController uses annotations on node objects to coordinate with the MachineConfigDaemon running on each machine to upgrade each machine to the desired Machine Configuration.

UpdateController watches for changes on MachineConfigPool and runs update if,

  1. If the .Status.CurrentMachineConfig has been updated.

  2. If new nodes can be updated to the current configuration as new Machines are available with old configuration if permitted by NodeLimit or the NodeLimit has increased allowing more nodes to be updated.

Historically the following annotations were used to coordinate between UpdateController and the MachineConfigDaemon,

  • node-configuration.v1.coreos.com/currentConfig
  • node-configuration.v1.coreos.com/targetConfig
  • node-configuration.v1.coreos.com/desiredConfig

With these three fields it becomes possible to determine the update progress of the machine:

  • desiredConfig == currentConfig: The machine is up-to-date.
  • desiredConfig != currentConfig && desiredConfig == targetConfig: The machine is not up-to-date, but is in the process of updating.
  • desiredConfig != currentConfig && desiredConfig != targetConfig: The machine is not up-to-date and is not in the process of updating.
  • Node is marked updated by UpdateController unless NodeReady is reported by kubelet.

UpdateController interface with MachineConfigDaemon

Following annotations on node object will be used by UpdateController to coordinate node update with MachineConfigDaemon.

  • machine-config-daemon.v1.openshift.com/currentConfig : defines the current MachineConfig applied by MachineConfigDaemon.
  • machine-config-daemon.v1.openshift.com/desiredConfig : defines the desired MachineConfig that need to be applied by MachineConfigDaemon
  • machine-config-daemon.v1.openshift.com/state : defines the state of the MachineConfigDaemon, It can be done, working and degraded.

With these three fields it becomes possible to determine the update progress of the machine:

a. desiredConfig == currentConfig : The machine is up-to-date.

b. desiredConfig != currentConfig && state == working : The machine is not up-to-date, but is in the process of updating.

c. desiredConfig != currentConfig && state == degraded : The machine is not up-to-date and MachineConfigDaemon cannot apply the desired configuration.

Node is marked updated by UpdateController only when NodeReady is reported by kubelet when case (a) is true.

KubeletConfig

The KubeletConfigController manages the KubeletConfig CRD allowing customers to manage their Feature Flags, Max Pods, and other Kubelet options.

The KubeletConfigController listens for changes within KubeletConfig CRDs and merges in changes from the user.

When the user creates a new KubeletConfig object they pass in the MachineConfigPoolSelector to target.

The MachineConfigController performs the following operations:

  1. Validates the user defined KubeletConfig
  2. Renders the current MachineConfig (storage.files.contents[kubelet.conf]) into the KubeletConfiguration structure
  3. Loads the user's KubeletConfig instance
  4. Uses mergo to merge the two structures
  5. Serializes the KubeletConfig to json
  6. Creates or Updates a MachineConfig (called 99-[role]-kubelet-managed) with a new /etc/kubernetes/kubelet.conf

The machine will subsequently reboot by the MachineConfigDaemon to apply the new config.