Reconfigure a Node's Kubelet in a Live Cluster

allows you to change the configuration of each kubelet in a running Kubernetes cluster, by deploying a ConfigMap and configuring each Node to use it.

Before you begin

You need to have a Kubernetes cluster. You also need kubectl, installed and configured to communicate with your cluster. Make sure that you are using a version of kubectl that is compatible with your cluster. Your Kubernetes server must be at or later than version v1.11. To check the version, enter kubectl version.

Some of the examples use the command line tool

For each node that you're reconfiguring, you must set the kubelet --dynamic-config-dir flag to a writable directory.

For more information on configuring the kubelet via a configuration file, see Set kubelet parameters via a config file).

Generate the configuration file

1. Choose a Node to reconfigure. In this example, the name of this Node is referred to as NODE_NAME.

2. Start the kubectl proxy in the background using the following command:

   kubectl proxy --port=8001 &
   

3. Run the following command to download and unpack the configuration from the configz endpoint. The command is long, so be careful when copying and pasting. If you use zsh, note that common zsh configurations add backslashes to escape the opening and closing curly braces around the variable name in the URL. For example: ${NODE_NAME} will be rewritten as $\{NODE_NAME\} during the paste. You must remove the backslashes before running the command, or the command will fail.

   NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}
   

Edit the configuration file

Using a text editor, change one of the parameters in the file generated by the previous procedure. For example, you might edit the parameter eventRecordQPS, that controls rate limiting for event recording.

Push the configuration file to the control plane

Push the edited configuration file to the control plane with the following command:

kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml

This is an example of a valid response:

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: 2017-09-14T20:23:33Z
  name: my-node-config-gkt4c2m4b2
  namespace: kube-system
  resourceVersion: "119980"
  uid: 946d785e-998a-11e7-a8dd-42010a800006
data:
  kubelet: |
    {...}

You created that ConfigMap inside the kube-system namespace because the kubelet is a Kubernetes system component.

The --append-hash option appends a short checksum of the ConfigMap contents to the name. This is convenient for an edit-then-push workflow, because it automatically, yet deterministically, generates new names for new resources. The name that includes this generated hash is referred to as CONFIG_MAP_NAME in the following examples.

Set the Node to use the new configuration

Edit the Node's reference to point to the new ConfigMap with the following command:

kubectl edit node ${NODE_NAME}

In your text editor, add the following YAML under spec:

configSource:
    configMap:
        name: CONFIG_MAP_NAME # replace CONFIG_MAP_NAME with the name of the ConfigMap
        namespace: kube-system
        kubeletConfigKey: kubelet

You must specify all three of name, namespace, and kubeletConfigKey. The kubeletConfigKey parameter shows the kubelet which key of the ConfigMap contains its config.

Observe that the Node begins using the new configuration

Retrieve the Node using the kubectl get node ${NODE_NAME} -o yaml command and inspect Node.Status.Config. The config sources corresponding to the active, assigned, and lastKnownGood configurations are reported in the status.

• The active configuration is the version the kubelet is currently running with.
• The assigned configuration is the latest version the kubelet has resolved based on Node.Spec.ConfigSource.
• The lastKnownGood configuration is the version the kubelet will fall back to if an invalid config is assigned in Node.Spec.ConfigSource.

ThelastKnownGood configuration might not be present if it is set to its default value, the local config deployed with the node. The status will update lastKnownGood to match a valid assigned config after the kubelet becomes comfortable with the config. The details of how the kubelet determines a config should become the lastKnownGood are not guaranteed by the API, but is currently implemented as a 10-minute grace period.

You can use the following command (using jq) to filter down to the config status:

kubectl get no ${NODE_NAME} -o json | jq '.status.config'

The following is an example response:

{
  "active": {
    "configMap": {
      "kubeletConfigKey": "kubelet",
      "name": "my-node-config-9mbkccg2cc",
      "namespace": "kube-system",
      "resourceVersion": "1326",
      "uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
    }
  },
  "assigned": {
    "configMap": {
      "kubeletConfigKey": "kubelet",
      "name": "my-node-config-9mbkccg2cc",
      "namespace": "kube-system",
      "resourceVersion": "1326",
      "uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
    }
  },
  "lastKnownGood": {
    "configMap": {
      "kubeletConfigKey": "kubelet",
      "name": "my-node-config-9mbkccg2cc",
      "namespace": "kube-system",
      "resourceVersion": "1326",
      "uid": "705ab4f5-6393-11e8-b7cc-42010a800002"
    }
  }
}

(if you do not have jq, you can look at the whole response and find Node.Status.Config by eye).

If an error occurs, the kubelet reports it in the Node.Status.Config.Error structure. Possible errors are listed in Understanding Node.Status.Config.Error messages. You can search for the identical text in the kubelet log for additional details and context about the error.

Make more changes

Follow the workflow above to make more changes and push them again. Each time you push a ConfigMap with new contents, the --append-hash kubectl option creates the ConfigMap with a new name. The safest rollout strategy is to first create a new ConfigMap, and then update the Node to use the new ConfigMap.

Reset the Node to use its local default configuration

To reset the Node to use the configuration it was provisioned with, edit the Node using kubectl edit node ${NODE_NAME} and remove the Node.Spec.ConfigSource field.

Observe that the Node is using its local default configuration

After removing this subfield, Node.Status.Config eventually becomes empty, since all config sources have been reset to nil, which indicates that the local default config is assigned, active, and lastKnownGood, and no error is reported.

kubectl patch example

You can change a Node's configSource using several different mechanisms. This example uses kubectl patch:

kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"

Understanding how the kubelet checkpoints config

When a new config is assigned to the Node, the kubelet downloads and unpacks the config payload as a set of files on the local disk. The kubelet also records metadata that locally tracks the assigned and last-known-good config sources, so that the kubelet knows which config to use across restarts, even if the API server becomes unavailable. After checkpointing a config and the relevant metadata, the kubelet exits if it detects that the assigned config has changed. When the kubelet is restarted by the OS-level service manager (such as systemd), it reads the new metadata and uses the new config.

The recorded metadata is fully resolved, meaning that it contains all necessary information to choose a specific config version - typically a UID and ResourceVersion. This is in contrast to Node.Spec.ConfigSource, where the intended config is declared via the idempotent namespace/name that identifies the target ConfigMap; the kubelet tries to use the latest version of this ConfigMap.

When you are debugging problems on a node, you can inspect the kubelet's config metadata and checkpoints. The structure of the kubelet's checkpointing directory is:

- --dynamic-config-dir (root for managing dynamic config)
| - meta
  | - assigned (encoded kubeletconfig/v1beta1.SerializedNodeConfigSource object, indicating the assigned config)
  | - last-known-good (encoded kubeletconfig/v1beta1.SerializedNodeConfigSource object, indicating the last-known-good config)
| - checkpoints
  | - uid1 (dir for versions of object identified by uid1)
    | - resourceVersion1 (dir for unpacked files from resourceVersion1 of object with uid1)
    | - ...
  | - ...

Understanding Node.Status.Config.Error messages

The following table describes error messages that can occur when using Dynamic Kubelet Config. You can search for the identical text in the Kubelet log for additional details and context about the error.

What's next

• Set kubelet parameters via a config file explains the supported way to configure a kubelet.
• See the reference documentation for Node, including the configSource field within the Node's .spec
• Learn more about kubelet configuration by checking the KubeletConfiguration reference.