Chapter 2. Performing automated in-place cluster upgrades

During a cluster upgrade, and on every restart of any master, the default cluster roles are automatically reconciled to restore any missing permissions.

You can upgrade the OpenShift Container Platform cluster in one or more phases. You can choose to upgrade all hosts in one phase by running a single Ansible playbook or upgrade the control plane, or master components, and nodes in multiple phases using separate playbooks.

When upgrading in separate phases, the control plane phase includes upgrading:

If you upgrade only the nodes, you must first upgrade the control plane. The node phase includes upgrading:

Nodes that run master components are upgraded only during the control plane upgrade phase. This ensures that the node services and container engines on masters are not upgraded twice, once during the control plane phase and again during the node phase.

Whether you upgrade in a single or multiple phases, you can customize how the node portion of the upgrade progresses by passing certain Ansible variables to an upgrade playbook using the -e option.

When upgrading OpenShift Container Platform, you can execute custom tasks during specific operations through a system called hooks. Hooks allow cluster administrators to provide files defining tasks to execute before or after specific areas during upgrades. You can use hooks to validate or modify custom infrastructure when upgrading OpenShift Container Platform.

Because when a hook fails, the operation fail, design hooks that are idempotent, or can run multiple times and provide the same results.

[OSEv3:vars]
openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
openshift_master_upgrade_hook=/usr/share/custom/master.yml
openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml

openshift_node_upgrade_pre_hook=/usr/share/custom/pre_node.yml
openshift_node_upgrade_hook=/usr/share/custom/node.yml
openshift_node_upgrade_post_hook=/usr/share/custom/post_node.yml

---
# Trivial example forcing an operator to ack the start of an upgrade
# file=/usr/share/custom/pre_master.yml

- name: note the start of a master upgrade
  debug:
      msg: "Master upgrade of {{ inventory_hostname }} is about to start"

- name: require an operator agree to start an upgrade
  pause:
      prompt: "Hit enter to start the master upgrade"

If your OpenShift Container Platform cluster uses a mixed environment or gcePD storage, you need to take more steps before you upgrade it.

Before you upgrade a mixed environment, such as one with Red Hat Enterprise Linux (RHEL) and RHEL Atomic Host, set values in the inventory file for both the openshift_pkg_version and openshift_image_tag parameters. Setting these values ensures that all nodes in your cluster run the same version of OpenShift Container Platform. While this is a best practice for major updates, such as from OpenShift Container Platform 2 to OpenShift Container Platform 3, setting these values are mandatory for minor version upgrades.

For example, to upgrade from OpenShift Container Platform 3.9 to OpenShift Container Platform 3.10, set the following parameters and values:

openshift_pkg_version=-3.10.16
openshift_image_tag=v3.10.16

[OSEv3:vars]

openshift_hosted_registry_storage_provider=gcs
openshift_hosted_registry_storage_gcs_bucket=bucket01
openshift_hosted_registry_storage_gcs_keyfile=test.key
openshift_hosted_registry_storage_gcs_rootdirectory=/registry

To upgrade an existing EFK logging stack deployment, you review your parameters and run the openshift-logging/config.yml playbook.

The EFK upgrade also upgrades Elasticsearch from version 2 to version 5. For important information on changes in Elasticsearch 5, you should review the Elasticsearch breaking changes.

It is important to note that Elasticsearch 5 has some significant changes to the index structures. Previously, Elasticsearch permitted a dot character, ., in field names. In version 5, Elasticsearch interprets any dot in an Elasticsearch field name as nested structure. If you have a field with a dot, the string after the dot is interpreted as the type of field, leading to mapping conflicts during the upgrade.

To help identify potential conflicts, OpenShift Container Platform provides a script that examines your Elasticsearch fields to determine if any fields contain a dot in the name.

For example, the following fields were allowed in Elasticsearch 2:

{
  "field": 123  // "field" is of type number
}

// Any dot in field name is treated as any other valid character in the field name.
// It is just part of the field name.
{
  "field.name": "Bob"  // "field.name" is of type String

In Elasticsearch 5 and higher the field string would become the field and the name string would become a type for the field:

{
  "field": 123  // "field" is of type number
}

// Any dot in field name is always interpreted as nested structure.
{
  "field" : {  // "field" is of type Object
    "name": "Bob"  // "name" is of type String
  }
}

Upgrading in this case would result in the field field having two different types, which is not permitted.

If you need to keep these conflicting indices, you need to reindex the data and change the documents to get rid of conflicting data structure. For more information, see Upgrading fields with dots to 5.x.

oc exec -c elasticsearch -n $LOGGING_NS $pod -- es_util --query='_mapping?pretty&filter_path=**.mappings.*.properties' \
  | jq '.[].mappings[].properties | keys' \
  | jq .[] \
  | egrep -e "\."

To upgrade an existing cluster metrics deployment, you review your parameters and run the openshift-metrics/config.yml playbook.