Chapter 2. Performing Automated In-place Cluster Upgrades

Starting in OpenShift Container Platform 3.10, node configurations are now bootstrapped from the master. When node services are started, the node checks if a kubeconfig and other node configuration files exist before joining the cluster. If they do not, the node pulls the configuration from the master, then joins the cluster.

This process replaces administrators having to manually maintain the node configuration uniquely on each node host. Instead, the contents of a node host’s /etc/origin/node/node-config.yaml file are now provided by ConfigMaps sourced from the master.

To map which ConfigMap to use for which node host, node groups must be defined and then set for each host in the inventory file. These procedures are handled during the Preparing for an Automated Upgrade section.

In OpenShift Container Platform version 3.10, the openshift_hostname parameter was removed. If you used the openshift_hostname parameter in your inventory file in OpenShift Container Platform version 3.9, you must remove that parameter from your inventory file. Before you upgrade, you must set the openshift_kubelet_name_override parameter to the openshift_hostname that you used in previous versions.

Node ConfigMaps

The Configmaps for defining the node configurations must be available in the openshift-node project. ConfigMaps are also now the authoritative definition for node labels; the old openshift_node_labels value is effectively ignored.

The installer provides a playbook for creating the following default ConfigMaps:

The following ConfigMaps are also created, which label nodes into multiple roles:

Node Group Definitions

After updating the openshift-ansible package, you can view what the default set of node group definitions looks like in YAML format in the /usr/share/ansible/openshift-ansible/roles/openshift_facts/defaults/main.yml file:

openshift_node_groups:
  - name: node-config-master 1
    labels:
      - 'node-role.kubernetes.io/master=true' 2
    edits: [] 3
  - name: node-config-infra
    labels:
      - 'node-role.kubernetes.io/infra=true'
    edits: []
  - name: node-config-compute
    labels:
      - 'node-role.kubernetes.io/compute=true'
    edits: []
  - name: node-config-master-infra
    labels:
      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true'
    edits: []
  - name: node-config-all-in-one
    labels:
      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true,node-role.kubernetes.io/compute=true'
    edits: []

If you do not set the openshift_node_groups variable in your inventory file’s [OSEv3:vars] group, the defaults defined above will be used. However, if you want to deviate from these defaults, you must define the entire openshift_node_groups structure (including all desired node groups) in your inventory file.

The openshift_node_groups value is not merged with the defaults, and the YAML definition must first be translated into a Python dictionary. You can then use the edits field to modify any node configuration variables as desired by specifying the key-value pairs.

For example, the following entry in an inventory file defines groups named node-config-master, node-config-infra, and node-config-compute, and edits the ConfigMap for node-config-compute to set kubeletArguments.pods-per-core to 20:

openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true',]}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]

Whenever the openshift_node_group.yml playbook is run, the changes defined in the edits field will update the related ConfigMap (node-config-compute in this example), which will ultimately affect the node’s configuration file on the host.

The 3.9 to 3.10 control plane upgrade performs the following steps for you:

The 3.9 to 3.10 node upgrade performs a rolling update of nodes, which:

Automated upgrade playbooks are run via Ansible directly using the ansible-playbook command with an inventory file, similar to the advanced installation method. The same v3_10 upgrade playbooks can be used for either of the following scenarios:

During a cluster upgrade, and on every restart of any master, the default cluster roles are automatically reconciled to restore any missing permissions.

An OpenShift Container Platform cluster can be upgraded in one or more phases. You can choose whether to upgrade all hosts in one phase by running a single Ansible playbook, or upgrade the control plane (master components) and nodes in multiple phases using separate playbooks.

Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.10 Release.

When upgrading in separate phases, the control plane phase includes upgrading:

When upgrading only the nodes, the control plane must already be upgraded. The node phase includes upgrading:

Whether upgrading in a single or multiple phases, you can customize how the node portion of the upgrade progresses by passing certain Ansible variables to an upgrade playbook using the -e option.

The openshift_upgrade_nodes_serial variable can be set to an integer or percentage to control how many node hosts are upgraded at the same time. The default is 1, upgrading nodes one at a time.

For example, to upgrade 20 percent of the total number of detected nodes at a time:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="20%"

The openshift_upgrade_nodes_label variable allows you to specify that only nodes with a certain label are upgraded. This can also be combined with the openshift_upgrade_nodes_serial variable.

For example, to only upgrade nodes in the group1 region, two at a time:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="2" \
    -e openshift_upgrade_nodes_label="region=group1"

The openshift_upgrade_nodes_max_fail_percentage variable allows you to specify how many nodes may fail in each batch. The percentage of failure must exceed your value before the playbook aborts the upgrade.

The openshift_upgrade_nodes_drain_timeout variable allows you to specify the length of time to wait before giving up.

In this example, 10 nodes are upgraded at a time, the upgrade will abort if more than 20 percent of the nodes fail, and there is a 600-second wait to drain the node:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial=10 \
    -e openshift_upgrade_nodes_max_fail_percentage=20 \
    -e openshift_upgrade_nodes_drain_timeout=600

When upgrading OpenShift Container Platform, you can execute custom tasks during specific operations through a system called hooks. Hooks allow cluster administrators to provide files defining tasks to execute before and/or after specific areas during upgrades. This can be very helpful to validate or modify custom infrastructure when upgrading OpenShift Container Platform.

It is important to remember that when a hook fails, the operation fails. This means a good hook can run multiple times and provide the same results. A great hook is idempotent.

[OSEv3:vars]
openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
openshift_master_upgrade_hook=/usr/share/custom/master.yml
openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml

openshift_node_upgrade_pre_hook=/usr/share/custom/pre_node.yml
openshift_node_upgrade_hook=/usr/share/custom/node.yml
openshift_node_upgrade_post_hook=/usr/share/custom/post_node.yml

---
# Trivial example forcing an operator to ack the start of an upgrade
# file=/usr/share/custom/pre_master.yml

- name: note the start of a master upgrade
  debug:
      msg: "Master upgrade of {{ inventory_hostname }} is about to start"

- name: require an operator agree to start an upgrade
  pause:
      prompt: "Hit enter to start the master upgrade"