Chapter 2. Performing Automated In-place Cluster Upgrades
2.1. Overview
If you installed using the advanced installation and the inventory file that was used is available, you can use upgrade playbooks to automate the OpenShift cluster upgrade process.
The OpenShift Container Platform 3.9 release includes a merge of features and fixes from Kubernetes 1.8 and 1.9. As a result, the upgrade process from OpenShift Container Platform 3.7 completes with the cluster fully upgraded to OpenShift Container Platform 3.9, seemingly "skipping" the 3.8 release. Technically, the OpenShift Container Platform 3.7 cluster is first upgraded to 3.8-versioned packages, and then the process immediately continues upgrading to OpenShift Container Platform 3.9 automatically. Your cluster should only remain at 3.8-versioned packages for as long as it takes to successfully complete the upgrade to OpenShift Container Platform 3.9.
As of OpenShift Container Platform 3.9, the quick installation method is deprecated. In a future release, it will be removed completely. In addition, using the quick installer to upgrade from version 3.7 to 3.9 is not supported.
The automated 3.7 to 3.9 control plane upgrade performs the following steps for you:
- A backup of all etcd data is taken for recovery purposes.
- The API and controllers are updated from 3.7 to 3.8.
- Internal data structures are updated to 3.8.
- A second backup of all etcd data is taken for recovery purposes.
- The API and controllers are updated from 3.8 to 3.9.
- Internal data structures are updated to 3.9.
- The default router, if one exists, is updated from 3.7 to 3.9.
- The default registry, if one exists, is updated from 3.7 to 3.9.
- The default image streams and InstantApp templates are updated.
The automated 3.7 to 3.9 node upgrade performs a rolling update of nodes, which:
- Marks a subset of nodes unschedulable and drains them of pods.
- Updates node components from 3.7 to 3.9 (including openvswitch and container runtime).
- Returns those nodes to service.
- Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.
- If you are using GlusterFS, see Special Considerations When Using Containerized GlusterFS before proceeding.
- If you are using GCE Persistent Disk (gcePD), see Special Considerations When Using gcePD before proceeding.
The day before the upgrade, validate OpenShift Container Platform storage migration to ensure potential issues are resolved prior to the outage window:
$ oc adm migrate storage --include=* --loglevel=2 --confirm --config /etc/origin/master/admin.kubeconfig
Automated upgrade playbooks are run via Ansible directly using the ansible-playbook
command with an inventory file, similar to the advanced installation method. The same v3_9 upgrade playbooks can be used for either of the following scenarios:
- Upgrading existing OpenShift Container Platform 3.7 clusters to 3.9
- Upgrading existing OpenShift Container Platform 3.9 clusters to the latest asynchronous errata updates
Running Ansible playbooks with the --tags
or --check
options is not supported by Red Hat.
2.2. Preparing for an Automated Upgrade
Before upgrading Upgrading your cluster to OpenShift Container Platform 3.9, the cluster must be already upgraded to the latest asynchronous release of version 3.7. If your cluster is at a version earlier than 3.7, you must first upgrade incrementally (e.g., 3.5 to 3.6, then 3.6 to 3.7).
Before attempting the upgrade, follow the guidance in Environment health checks to verify the cluster’s health. This will confirm that nodes are in the Ready state, running the expected starting version, and will ensure that there are no diagnostic errors or warnings.
If you are completing a large-scale upgrade, which involves at least 10 worker nodes and thousands of projects and pods, review Special considerations for large-scale upgrades to prevent upgrade failures.
To prepare for an automated upgrade:
Pull the latest subscription data from RHSM:
# subscription-manager refresh
If you are upgrading from OpenShift Container Platform 3.7 to 3.9:
Manually disable the 3.7 repository and enable both the 3.8 and 3.9 repositories on each master and node host. You must also enable the rhel-7-server-ansible-2.4-rpms repository, which is a new requirement starting with OpenShift Container Platform 3.9:
# subscription-manager repos --disable="rhel-7-server-ose-3.7-rpms" \ --enable="rhel-7-server-ose-3.8-rpms" \ --enable="rhel-7-server-ose-3.9-rpms" \ --enable="rhel-7-server-rpms" \ --enable="rhel-7-server-extras-rpms" \ --enable="rhel-7-server-ansible-2.4-rpms" \ --enable="rhel-7-fast-datapath-rpms" # yum clean all
Remove the
openshift_schedulable
parameter and its value from the entry for each master host in the Ansible inventory file.During upgrade, if a value is not provided for the
openshift_schedulable
parameter, it is set to true. If a value is provided foropenshift_schedulable
, it is not changed.In previous versions of OpenShift Container Platform, master hosts were marked unschedulable by default by the installer, meaning that no pods could be placed on the hosts. Starting with OpenShift Container Platform 3.9, however, masters must be marked schedulable.
During upgrade, masters are automatically labeled with the
master
node role, and non-master, non-infrastructure nodes are labeled with thecompute
node role.See the OpenShift Container Platform 3.9 Release Notes on the these notable technical changes and ensure that you understand their impact on your cluster:
-
You must disable swap memory in your cluster before upgrading to OpenShift Container Platform 3.9, otherwise the upgrade will fail. Whether swap memory was enabled using
openshift_disable_swap=false
in your Ansible inventory file or enabled manually per host, see Disabling Swap Memory in the Cluster Administration guide to disable it on each host.
For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package on each RHEL 7 system, which also updates the openshift-ansible-* packages:
# yum update atomic-openshift-utils
If you have applied manual configuration changes to your master or node configuration files since your last Ansible playbook run (whether that was initial installation or your most recent cluster upgrade), and you have not yet made the equivalent changes to your inventory file, review Configuring Ansible Inventory Files. For any variables that are relevant to the manual changes you made, apply the equivalent appropriate changes to your inventory files before running the upgrade. Otherwise, your manual changes may be overwritten by default values during the upgrade, which could cause pods to not run properly or other cluster stability issues.
In particular, if you made any changes to
admissionConfig
settings in your master configuration files, review theopenshift_master_admission_plugin_config
variable in Configuring Ansible Inventory Files. Failure to do so could cause pods to get stuck inPending
state if you hadClusterResourceOverride
settings manually configured previously (as described in Configuring Masters for Overcommitment).
After satisfying these steps, you can review the following sections for more information on how the upgrade process works and make decisions on additional upgrade customization options if you so choose. When you are prepared to run the upgrade, you can continue to Upgrading to the Latest OpenShift Container Platform 3.9 Release.
2.2.1. Upgrading the Control Plane and Nodes in Separate Phases
An OpenShift Container Platform cluster can be upgraded in one or more phases. You can choose whether to upgrade all hosts in one phase by running a single Ansible playbook, or upgrade the control plane (master components) and nodes in multiple phases using separate playbooks.
Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.9 Release.
If your OpenShift Container Platform cluster uses GlusterFS pods, you must perform the upgrade in multiple phases. See Special Considerations When Using Containerized GlusterFS for details on how to upgrade with GlusterFS.
When upgrading in separate phases, the control plane phase includes upgrading:
- master components
- node services running on masters
- Docker running on masters
- Docker running on any stand-alone etcd hosts
When upgrading only the nodes, the control plane must already be upgraded. The node phase includes upgrading:
- node services running on stand-alone nodes
- Docker running on stand-alone nodes
Nodes running master components are not included during the node upgrade phase, even though they have node services and Docker running on them. Instead, they are upgraded as part of the control plane upgrade phase. This ensures node services and Docker on masters are not upgraded twice (once during the control plane phase and again during the node phase).
2.2.2. Customizing Node Upgrades
Whether upgrading in a single or multiple phases, you can customize how the node portion of the upgrade progresses by passing certain Ansible variables to an upgrade playbook using the -e
option.
Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.7 Release.
The openshift_upgrade_nodes_serial
variable can be set to an integer or percentage to control how many node hosts are upgraded at the same time. The default is 1
, upgrading nodes one at a time.
For example, to upgrade 20 percent of the total number of detected nodes at a time:
$ ansible-playbook -i <path/to/inventory/file> \ </path/to/upgrade/playbook> \ -e openshift_upgrade_nodes_serial="20%"
The openshift_upgrade_nodes_label
variable allows you to specify that only nodes with a certain label are upgraded. This can also be combined with the openshift_upgrade_nodes_serial
variable.
For example, to only upgrade nodes in the group1 region, two at a time:
$ ansible-playbook -i <path/to/inventory/file> \ </path/to/upgrade/playbook> \ -e openshift_upgrade_nodes_serial="2" \ -e openshift_upgrade_nodes_label="region=group1"
See Managing Nodes for more on node labels.
The openshift_upgrade_nodes_max_fail_percentage
variable allows you to specify how many nodes may fail in each batch. The percentage of failure must exceed your value before the playbook aborts the upgrade.
The openshift_upgrade_nodes_drain_timeout
variable allows you to specify the length of time to wait before giving up.
In this example, 10 nodes are upgraded at a time, the upgrade will abort if more than 20 percent of the nodes fail, and there is a 600-second wait to drain the node:
$ ansible-playbook -i <path/to/inventory/file> \ </path/to/upgrade/playbook> \ -e openshift_upgrade_nodes_serial=10 \ -e openshift_upgrade_nodes_max_fail_percentage=20 \ -e openshift_upgrade_nodes_drain_timeout=600
2.2.3. Customizing Upgrades With Ansible Hooks
When upgrading OpenShift Container Platform, you can execute custom tasks during specific operations through a system called hooks. Hooks allow cluster administrators to provide files defining tasks to execute before and/or after specific areas during upgrades. This can be very helpful to validate or modify custom infrastructure when upgrading OpenShift Container Platform.
It is important to remember that when a hook fails, the operation fails. This means a good hook can run multiple times and provide the same results. A great hook is idempotent.
2.2.3.1. Limitations
- Hooks have no defined or versioned interface. They can use internal openshift-ansible variables, but there is no guarantee these will remain in future releases. In the future, hooks may be versioned, giving you advance warning that your hook needs to be updated to work with the latest openshift-ansible.
- Hooks have no error handling, so an error in a hook will halt the upgrade process. The problem will need to be addressed and the upgrade re-run.
2.2.3.2. Using Hooks
Hooks are defined in the hosts inventory file under the OSEv3:vars
section.
Each hook must point to a YAML file which defines Ansible tasks. This file will be used as an include, meaning that the file cannot be a playbook, but a set of tasks. Best practice suggests using absolute paths to the hook file to avoid any ambiguity.
Example Hook Definitions in an Inventory File
[OSEv3:vars] openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml openshift_master_upgrade_hook=/usr/share/custom/master.yml openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml
Example pre_master.yml Task
--- # Trivial example forcing an operator to ack the start of an upgrade # file=/usr/share/custom/pre_master.yml - name: note the start of a master upgrade debug: msg: "Master upgrade of {{ inventory_hostname }} is about to start" - name: require an operator agree to start an upgrade pause: prompt: "Hit enter to start the master upgrade"
2.2.3.3. Available Upgrade Hooks
openshift_master_upgrade_pre_hook
- Runs before each master is upgraded.
- This hook runs against each master in serial.
-
If a task must run against a different host, said task must use
delegate_to
orlocal_action
.
openshift_master_upgrade_hook
- Runs after each master is upgraded, but before its service or system restart.
- This hook runs against each master in serial.
-
If a task must run against a different host, said task must use
delegate_to
orlocal_action
.
openshift_master_upgrade_post_hook
- Runs after each master is upgraded and has had its service or system restart.
- This hook runs against each master in serial.
-
If a task must run against a different host, said task must use
delegate_to
orlocal_action
.
2.3. Upgrading to the Latest OpenShift Container Platform 3.9 Release
To upgrade an existing OpenShift Container Platform 3.7 or 3.9 cluster to the latest 3.9 release:
- Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.
-
Ensure the
openshift_deployment_type
parameter in your inventory file is set toopenshift-enterprise
. Starting with OpenShift Container Platform 3.9, the OpenShift Container Platform web console is deployed as a pod on masters during upgrade, and the
openshift_web_console_prefix
is introduced to deploy the web console with a customized image prefix. Thetemplate_service_broker_prefix
is updated to match other components. If you use a customized docker-registry for your installation instead of registry.access.redhat.com, you must explicitly specifyopenshift_web_console_prefix
andtemplate_service_broker_prefix
to point to the correct image prefix during upgrade:openshift_web_console_prefix=<registry_ip>:<port>/openshift3/ose- template_service_broker_prefix=<registry_ip>:<port>/openshift3/ose-
-
If you want to enable rolling, full system restarts of the hosts, you can set the
openshift_rolling_restart_mode
parameter in your inventory file tosystem
. Otherwise, the default valueservices
performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details. At this point, you can choose to run the upgrade in a single or multiple phases. See Upgrading the Control Plane and Nodes in Separate Phases for more details which components are upgraded in each phase.
If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the
-i
flag to specify its location. If you previously used theatomic-openshift-installer
command to run your installation, you can check ~/.config/openshift/hosts for the last inventory file that was used, if needed.Option A) Upgrade control plane and nodes in a single phase.
Run the upgrade.yml playbook to upgrade the cluster in a single phase using one playbook; the control plane is still upgraded first, then nodes in-place:
# ansible-playbook -i </path/to/inventory/file> \ /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.yml
Option B) Upgrade the control plane and nodes in separate phases.
To upgrade only the control plane, run the upgrade_control_plane.yaml playbook:
# ansible-playbook -i </path/to/inventory/file> \ /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml
To upgrade only the nodes, run the upgrade_nodes.yaml playbook:
# ansible-playbook -i </path/to/inventory/file> \ /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_nodes.yml \ [-e <customized_node_upgrade_variables>] 1
- 1
- See Customizing Node Upgrades for any desired
<customized_node_upgrade_variables>
.
If you are upgrading the nodes in groups as described in Customizing Node Upgrades, continue invoking the upgrade_nodes.yml playbook until all nodes have been successfully upgraded.
After all master and node upgrades have completed, reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features you have previously enabled.
Feature Next Step Aggregated Logging
Cluster Metrics
2.4. Upgrading the EFK Logging Stack
To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.
If you have any Elasticsearch SearchGuard indices in the following naming format, you need to delete and reseed the indices because the naming format has changed. Elastic search might not work as expected unless you update the indices:
.searchguard.logging-es-*
Run the following command to delete the SearchGuard indices:
# oc exec -c elasticsearch <pod> -- es_util --query=.searchguard* -XDELETE
Run the following command to reseed the SearchGuard indices:
# for pod in $(oc get pods -l component=es -o jsonpath={.items[*].metadata.name}); do oc exec -c elasticsearch $pod -- es_seed_acl; done
If you have not already done so, see Specifying Logging Ansible Variables in the Aggregating Container Logs topic and update your Ansible inventory file to at least set the following required variable within the
[OSEv3:vars]
section:[OSEv3:vars] openshift_logging_install_logging=true 1 openshift_logging_image_version=<tag> 2
-
Add any other
openshift_logging_*
variables that you want to specify to override the defaults, as described in Specifying Logging Ansible Variables. - When you have finished updating your inventory file, follow the instructions in Deploying the EFK Stack to run the openshift-logging/config.yml playbook and complete the logging deployment upgrade.
If your Fluentd DeploymentConfig and DaemonSet for the EFK components are already set with:
image: <image_name>:<vX.Y> imagePullPolicy: IfNotPresent
The latest version <image_name> might not be pulled if there is already one with the same <image_name:vX.Y> stored locally on the node where the pod is being re-deployed. If so, manually change the DeploymentConfig and DaemonSet to imagePullPolicy: Always
to make sure it is re-pulled.
2.5. Upgrading Cluster Metrics
To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-metrics/config.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.
If you have not already done so, see Specifying Metrics Ansible Variables in the Enabling Cluster Metrics topic and update your Ansible inventory file to at least set the following required variables within the
[OSEv3:vars]
section:[OSEv3:vars] openshift_metrics_install_metrics=true 1 openshift_metrics_image_version=<tag> 2 openshift_metrics_hawkular_hostname=<fqdn> 3 openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) 4
-
Add any other
openshift_metrics_*
variables that you want to specify to override the defaults, as described in Specifying Metrics Ansible Variables. - When you have finished updating your inventory file, follow the instructions in Deploying the Metrics Deployment to run the openshift-metrics/config.yml playbook and complete the metrics deployment upgrade.
2.6. Special Considerations for Large-scale Upgrades
For large-scale cluster upgrades, which involve at least 10 worker nodes and thousands of projects and pods, the API object storage migration should be performed prior to running the upgrade playbooks, and then again after the upgrade has successfully completed. Otherwise, the upgrade process will fail.
Refer to the Running the pre- and post- API server model object migration outside of the upgrade window section of the Recommendations for large-scale OpenShift upgrades for further guidance.
2.7. Special Considerations for Mixed Environments
Before you upgrade a mixed environment, such as one with Red Hat Enterprise Linux (RHEL) and RHEL Atomic Host, set values in the inventory file for both the openshift_pkg_version
and openshift_image_tag
parameters. Setting these values ensures that all nodes in your cluster run the same version of OpenShift Container Platform.
For example, to upgrade from OpenShift Container Platform 3.7 to OpenShift Container Platform 3.9, set the following parameters and values:
openshift_pkg_version=-3.9.74 openshift_image_tag=v3.9.74
These parameters can also be present in other, non-mixed, environments.
2.8. Special Considerations When Using Containerized GlusterFS
When upgrading OpenShift Container Platform, you must upgrade the set of nodes where GlusterFS pods are running.
Use care when upgrading these nodes, as drain
and unschedule
will not terminate and evacuate the GlusterFS pods because they are running as part of a daemonset.
There is also the potential for someone to run an upgrade on multiple nodes at the same time. If this occurs, it would lead to data availability issues if more than one node was hosting GlusterFS pods.
Even if a serial upgrade is running, there is no guarantee sufficient time will be given for GlusterFS to complete all of its healing operations before GlusterFS on the next node is terminated. This could leave the cluster in a bad or unknown state. Therefore, the following procedure is recommended.
- Upgrade the control plane (the master nodes and etcd nodes).
Upgrade standard
infra
nodes (router, registry, logging, and metrics).NoteIf any of the nodes in those groups are running GlusterFS, perform step 4 of this procedure at the same time. GlusterFS nodes must be upgraded along with other nodes in their class (
app
versusinfra
), one at a time.Upgrade standard nodes running application containers.
NoteIf any of the nodes in those groups are running GlusterFS, perform step 4 of this procedure at the same time. GlusterFS nodes must be upgraded along with other nodes in their class (
app
versusinfra
), one at a time.Upgrade the OpenShift Container Platform nodes running GlusterFS one at a time.
-
Run
oc get daemonset
to verify the label found underNODE-SELECTOR
. The default value isstoragenode=glusterfs
. -
Add a label (for example,
type=upgrade
) to the node you want to upgrade. Using the upgrade_nodes.yml playbook on the node where you terminated GlusterFS, run
-e openshift_upgrade_nodes_label="type=upgrade"
.NoteThe upgrade_nodes.yml playbook is located here:
# ansible-playbook -i </path/to/inventory/file> \ /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_nodes.yml \ [-e <customized_node_upgrade_variables>] 1
- 1
- See Customizing Node Upgrades for any desired
<customized_node_upgrade_variables>
.
- Wait for the GlusterFS pod to respawn and appear.
oc rsh
into the pod and verify all volumes are healed:$ oc rsh <GlusterFS_pod_name> $ for vol in `gluster volume list`; do gluster volume heal $vol info; done
Ensure all of the volumes are healed and there are no outstanding tasks. The
heal info
command lists all pending entries for a given volume’s heal process. A volume is considered healed whenNumber of entries
for that volume is0
.Remove the upgrade label and go to the next GlusterFS node:
$ oc label node <node_name> type-
-
Run
2.9. Special considerations when using gcePD
Because the default gcePD storage provider uses an RWO (Read-Write Only) access mode, you cannot perform a rolling upgrade on the registry or scale the registry to multiple pods. Therefore, when upgrading OpenShift Container Platform, you must specify the following environment variables in your Ansible inventory file:
[OSEv3:vars] openshift_hosted_registry_storage_provider=gcs openshift_hosted_registry_storage_gcs_bucket=bucket01 openshift_hosted_registry_storage_gcs_keyfile=test.key openshift_hosted_registry_storage_gcs_rootdirectory=/registry
2.10. Verifying the Upgrade
Ensure that the:
- cluster is healthy,
- services (master, node, and etcd) are running well,
-
the OpenShift Container Platform,
docker-registry
, and router versions are correct, - the original applications are still available and the new application can be created, and
-
running
oc adm diagnostics
produces no errors.
To verify the upgrade:
Check that all nodes are marked as Ready:
# oc get nodes NAME STATUS ROLES AGE VERSION master.example.com Ready master 7h v1.9.1+a0ce1bc657 node1.example.com Ready compute 7h v1.9.1+a0ce1bc657 node2.example.com Ready compute 7h v1.9.1+a0ce1bc657
Verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace
<tag>
withv3.9.102
for the latest version.# oc get -n default dc/docker-registry -o json | grep \"image\" "image": "openshift3/ose-docker-registry:<tag>", # oc get -n default dc/router -o json | grep \"image\" "image": "openshift3/ose-haproxy-router:<tag>",
Use the diagnostics tool on the master to look for common issues:
# oc adm diagnostics ... [Note] Summary of diagnostics execution: [Note] Completed with no errors or warnings seen.