Upgrading Clusters

When new versions of OpenShift Container Platform are released, you can upgrade your existing cluster to apply the latest enhancements and bug fixes. This includes upgrading from previous minor versions, such as release 3.6 to 3.7, and applying asynchronous errata updates within a minor version (3.7.z releases). See the OpenShift Container Platform 3.7 Release Notes to review the latest changes.

Unless noted otherwise, node and masters within a major version are forward and backward compatible across one minor version, so upgrading your cluster should go smoothly. However, you should not run mismatched versions longer than necessary to upgrade the entire cluster.

There are two methods available for performing OpenShift Container Platform cluster upgrades: automated or manual.

When using the automated upgrade method, there are two strategies you can take for performing the OpenShift Container Platform cluster upgrade: in-place upgrades or blue-green deployments. When using the manual upgrade method, an in-place upgrade is described.

If you installed using the advanced installation and the inventory file that was used is available, you can use the upgrade playbook to automate the OpenShift cluster upgrade process. If you installed using the quick installation method and a ~/.config/openshift/installer.cfg.yml file is available, you can use the quick installer to perform the automated upgrade.

The automated upgrade performs the following steps for you:

If you are completing a large-scale upgrade, which involves at least 10 worker nodes and thousands of projects and pods, review Special considerations for large-scale upgrades to prevent upgrade failures.

To prepare for an automated upgrade:

After satisfying these steps, there are two methods for running the automated upgrade:

Choose and follow one of these methods.

If you installed OpenShift Container Platform using the quick installation method, you should have an installation configuration file located at ~/.config/openshift/installer.cfg.yml. The quick installer requires this file to start an upgrade.

The quick installer supports upgrading between minor versions of OpenShift Container Platform (one minor version at a time, e.g., 3.5 to 3.6) as well as between asynchronous errata updates within a minor version (e.g., 3.6.z).

If you have an older format installation configuration file in ~/.config/openshift/installer.cfg.yml from an installation of a previous cluster version, the quick installer will attempt to upgrade the file to the new supported format. If you do not have an installation configuration file of any format, you can create one manually.

To start an upgrade with the quick installer:

You can run automated upgrade playbooks using Ansible directly, similar to the advanced installation method, if you have an inventory file. Playbooks can be run using the ansible-playbook command.

The same v3_7 upgrade playbooks can be used for either of the following scenarios:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="20%"

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="2" \
    -e openshift_upgrade_nodes_label="region=group1"

[OSEv3:vars]
openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
openshift_master_upgrade_hook=/usr/share/custom/master.yml
openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml

---
# Trivial example forcing an operator to ack the start of an upgrade
# file=/usr/share/custom/pre_master.yml

- name: note the start of a master upgrade
  debug:
      msg: "Master upgrade of {{ inventory_hostname }} is about to start"

- name: require an operator agree to start an upgrade
  pause:
      prompt: "Hit enter to start the master upgrade"

Starting with OpenShift Container Platform 3.7, the service catalog, OpenShift Ansible broker, and template service broker are enabled and deployed by default for new cluster installations. However, they are not deployed by default during the upgrade from OpenShift Container Platform 3.6 to 3.7, so you must run an individual component playbook separate post-upgrade.

To upgrade to these features:

To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.

        image: <image_name>:<vX.Y>
        imagePullPolicy: IfNotPresent

To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.

For large-scale cluster upgrades, which involve at least 10 worker nodes and thousands of projects and pods, the API object storage migration should be performed prior to running the upgrade playbooks, and then again after the upgrade has successfully completed. Otherwise, the upgrade process will fail.

Refer to the Running the pre- and post- API server model object migration outside of the upgrade window section of the Recommendations for large-scale OpenShift upgrades for further guidance.

Mixed environment upgrades (for example, those with Red Hat Enterprise Linux and Red Hat Enterprise Linux Atomic Host) require setting both openshift_pkg_version and openshift_image_tag. In mixed environments, if you only specify openshift_pkg_version, then that number is used for the packages for Red Hat Enterprise Linux and and the image for Red Hat Enterprise Linux Atomic Host.

When upgrading OpenShift Container Platform, you must upgrade the set of nodes where GlusterFS pods are running.

Special consideration must be taken when upgrading these nodes, as drain and unschedule will not terminate and evacuate the GlusterFS pods because they are running as part of a daemonset.

There is also the potential for someone to run an upgrade on multiple nodes at the same time, which would lead to data availability issues if more than one was hosting GlusterFS pods.

Even if a serial upgrade is running, there is no guarantee sufficient time will be given for GlusterFS to complete all of its healing operations before GlusterFS on the next node is terminated. This could leave the cluster in a bad or unknown state. Therefore, the following procedure is recommended.

Because the default gcePD storage provider uses an RWO (Read-Write Only) access mode, you cannot perform a rolling upgrade on the registry or scale the registry to multiple pods. Therefore, when upgrading OpenShift Container Platform, you must specify the following environment variables in your Ansible inventory file:

[OSEv3:vars]

openshift_hosted_registry_storage_provider=gcs
openshift_hosted_registry_storage_gcs_bucket=bucket01
openshift_hosted_registry_storage_gcs_keyfile=test.key
openshift_hosted_registry_storage_gcs_rootdirectory=/registry

To verify the upgrade:

As an alternative to performing an automated upgrade, you can manually upgrade your OpenShift Container Platform cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.

Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.

$ oc adm migrate storage --include=* --loglevel=2 --confirm --config /etc/origin/master/admin.kubeconfig

To prepare for a manual upgrade, follow these steps:

Before upgrading any stand-alone nodes, upgrade the master components (which provide the control plane for the cluster).

$ oc edit ep/kubernetes -n default

During a cluster upgrade, and on every restart of any master, the default cluster roles are automatically reconciled to restore any missing permissions.

After upgrading your masters, you can upgrade your nodes. When restarting the atomic-openshift-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.

One at at time for each node that is not also a master, you must disable scheduling and evacuate its pods to other nodes, then upgrade packages and restart services.

If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly available routing service.

Edit your router’s deployment configuration. For example, if it has the default router name:

# oc edit dc/router

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-haproxy-router:<tag> 1
        imagePullPolicy: IfNotPresent
        ...

You should see one router pod updated and then the next.

The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim or a host mount point, you may restart the registry without losing the contents of your registry. Storage for the Registry details how to configure persistent storage for the registry.

Edit your registry’s deployment configuration:

# oc edit dc/docker-registry

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-docker-registry:<tag> 1
        imagePullPolicy: IfNotPresent
        ...

If the registry console is deployed, edit its deployment configuration:

# oc edit dc/registry-console

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/registry-console:v3.7
        imagePullPolicy: IfNotPresent
        ...

By default, the quick and advanced installation methods automatically create default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under the /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/ directory.

Update the packages that provide the example JSON files. On a subscribed Red Hat Enterprise Linux 7 system where you can run the CLI as a user with cluster-admin permissions, install or update to the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible- packages:

# yum update atomic-openshift-utils

To persist /usr/share/openshift/examples/ on the first master:

$ scp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/* user@master1:/usr/share/openshift/examples/

To persist /usr/share/openshift/examples/ on all masters:

$ mkdir /usr/share/openshift/examples
$ scp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/* user@masterx:/usr/share/openshift/examples

The openshift-ansible-roles package provides the latest example JSON files.

Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.

After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run:

# oc import-image -n openshift <imagestream>

For example, get the list of all image streams in the default openshift project:

# oc get is -n openshift
NAME     DOCKER REPO                                                      TAGS                   UPDATED
mongodb  registry.access.redhat.com/openshift3/mongodb-24-rhel7           2.4,latest,v3.1.1.6    16 hours ago
mysql    registry.access.redhat.com/openshift3/mysql-55-rhel7             5.5,latest,v3.1.1.6    16 hours ago
nodejs   registry.access.redhat.com/openshift3/nodejs-010-rhel7           0.10,latest,v3.1.1.6   16 hours ago
...

Update each image stream one at a time:

# oc import-image -n openshift nodejs
The import completed successfully.

Name:			nodejs
Created:		10 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2016-07-05T19:20:30Z
Docker Pull Spec:	172.30.204.22:5000/openshift/nodejs

Tag	Spec								Created		PullSpec						Image
latest	4								9 seconds ago	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest	570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
4	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest		9 seconds ago	<same>							570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
0.10	registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest	9 seconds ago	<same>							a1ef33be788a28ec2bdd48a9a5d174ebcfbe11c8e986d2996b77f5bccaaa4774

Starting with OpenShift Container Platform 3.7, the service catalog, OpenShift Ansible broker, and template service broker are enabled and deployed by default for new cluster installations. However, they are not deployed by default during the upgrade from OpenShift Container Platform 3.6 to 3.7, so you must run an individual component playbook separate post-upgrade.

To upgrade to these features:

To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.

        image: <image_name>:<vX.Y>
        imagePullPolicy: IfNotPresent

To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.

Some OpenShift Container Platform releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster. This section will be updated over time as new asynchronous updates are released for OpenShift Container Platform 3.7.

See the OpenShift Container Platform 3.7 Release Notes to review the latest release notes.

To verify the upgrade:

The blue-green deployment upgrade method follows a similar flow to the in-place method: masters and etcd servers are still upgraded first, however a parallel environment is created for new node hosts instead of upgrading them in-place.

This method allows administrators to switch traffic from the old set of node hosts (e.g., the blue deployment) to the new set (e.g., the green deployment) after the new deployment has been verified. If a problem is detected, it is also then easy to rollback to the old deployment quickly.

While blue-green is a proven and valid strategy for deploying just about any software, there are always trade-offs. Not all environments have the same uptime requirements or the resources to properly perform blue-green deployments.

In an OpenShift Container Platform environment, the most suitable candidate for blue-green deployments are the node hosts. All user processes run on these systems and even critical pieces of OpenShift Container Platform infrastructure are self-hosted on these resources. Uptime is most important for these workloads and the additional complexity of blue-green deployments can be justified.

The exact implementation of this approach varies based on your requirements. Often the main challenge is having the excess capacity to facilitate such an approach.

Figure 4.1. Blue-Green Deployment

After you have upgraded your master and etcd hosts using method described for In-place Upgrades, use the following sections to prepare your environment for a blue-green upgrade of the remaining node hosts.

[new_nodes]
node4.example.com openshift_node_labels="{'region': 'primary', 'color':'green'}" openshift_schedulable=False
node5.example.com openshift_node_labels="{'region': 'primary', 'color':'green'}" openshift_schedulable=False
node6.example.com openshift_node_labels="{'region': 'primary', 'color':'green'}" openshift_schedulable=False
infra-node3.example.com openshift_node_labels="{'region': 'infra', 'color':'green'}" openshift_schedulable=False
infra-node4.example.com openshift_node_labels="{'region': 'infra', 'color':'green'}" openshift_schedulable=False

A common practice is to scale the registry and router pods until they are migrated to new (green) infrastructure node hosts. For these pods, a canary deployment approach is commonly used.

Scaling these pods up will make them immediately active on the new infrastructure nodes. Pointing their deployment configuration to the new image initiates a rolling update. However, because of node anti-affinity, and the fact that the blue nodes are still unschedulable, the deployments to the old nodes will fail.

At this point, the registry and router deployments can be scaled down to the original number of pods. At any given point, the original number of pods is still available so no capacity is lost and downtime should be avoided.

In order for pods to be migrated from the blue environment to the green, the required container images must be pulled. Network latency and load on the registry can cause delays if there is not sufficient capacity built in to the environment.

Often, the best way to minimize impact to the running system is to trigger new pod deployments that will land on the new nodes. Accomplish this by importing new image streams.

Major releases of OpenShift Container Platform (and sometimes asynchronous errata updates) introduce new image streams for builder images for users of Source-to-Image (S2I). Upon import, any builds or deployments configured with image change triggers are automatically created.

Another benefit of triggering the builds is that it does a fairly good job of fetching the majority of the ancillary images to all node hosts such as the various builder images, the pod infrastructure image, and deployers. The green nodes are then considered warmed (that is, ready for the expected load increase), and everything else can be migrated over using node evacuation in a later step, proceeding more quickly as a result.

When you are ready to continue with the upgrade process, follow these steps to warm the green nodes:

For larger deployments, it is possible to have other labels that help determine how evacuation can be coordinated. The most conservative approach for avoiding downtime is to evacuate one node host at a time.

If services are composed of pods using zone anti-affinity, then an entire zone can be evacuated at once. It is important to ensure that the storage volumes used are available in the new zone as this detail can vary among cloud providers.

In OpenShift Container Platform 3.2 and later, a node host evacuation is triggered whenever the node service is stopped. Node labeling is very important and can cause issues if nodes are mislabeled or commands are run on nodes with generalized labels. Exercise caution if master hosts are also labeled with color=blue.

When you are ready to continue with the upgrade process, follow these steps.

Updating the operating system (OS) on a host, by either upgrading across major releases or updating the system software for a minor release, can impact the OpenShift Container Platform software running on those machines. In particular, these updates can affect the iptables rules or ovs flows that OpenShift Container Platform requires to operate.

Use the following to safely upgrade the OS on a host:

Until OpenShift Container Platform 3.6, it was possible to deploy a cluster with an embedded etcd instance. This embedded etcd instance was deployed on your OpenShift Container Platform instance. Starting in OpenShift Container Platform version 3.7, this is no longer possible.

This migration process performs the following steps:

Additionally, the etcd API version since OpenShift Container Platform 3.6 defaults to v3. Also, since OpenShift Container Platform 3.7, v3 is the only version allowed. Therefore, older deployments with embedded etcd with the etcd API version v2 need to migrate to the external etcd first, followed by data migration, before they can be upgraded to OpenShift Container Platform 3.7.

Migration to external RPM etcd or external containerized etcd is currently supported.

A migration playbook is provided to automate all aspects of the process; this is the preferred method for performing the migration. You must have access to your existing inventory file with both the master and external etcd host defined in their separate groups.

In order to perform the migration on Red Hat Enterprise Linux Atomic Host, you must be running Atomic Host 7.4 or later.

Currently, manual migration is not recommended, as it requires a deployment of the new etcd cluster and re-deployment of etcd master certificates.

While etcd was updated from etcd v2 to v3 in a previous release, OpenShift Container Platform continued using an etcd v2 data model and API for both new and upgraded clusters. Starting with OpenShift Container Platform 3.6, new installations began using the v3 data model as well, providing improved performance and scalability.

For existing clusters that upgraded to OpenShift Container Platform 3.6, however, the etcd data must be migrated from v2 to v3 as a post-upgrade step. This must be performed using openshift-ansible version 3.6.173.0.21 or later.

Until OpenShift Container Platform 3.6, it was possible to deploy a cluster with an embedded etcd. As of OpenShift Container Platform 3.7, this is no longer possible. See Migrating Embedded etcd to External etcd.

The etcd v2 to v3 data migration is performed as an offline migration which means all etcd members and master services are stopped during the migration. Large clusters with up to 600MiB of etcd data can expect a 10 to 15 minute outage of the API, web console, and controllers.

This migration process performs the following steps:

You can only begin the etcd data migration process after upgrading to OpenShift Container Platform 3.6, as previous versions are not compatible with etcd v3 storage. Additionally, the upgrade to OpenShift Container Platform 3.6 reconfigures cluster DNS services to run on every node, rather than on the masters, which ensures that, even when master services are taken down, existing pods continue to function as expected.

Older deployments with embedded etcd with the etcd API version v2 need to migrate to the external etcd before migrating data. See Migrating Embedded etcd to External etcd.

A migration playbook is provided to automate all aspects of the process; this is the preferred method for performing the migration. You must have access to your existing inventory file with both masters and etcd hosts defined in their separate groups.

The following procedure describes the steps required to successfully migrate the cluster (implemented as part of the Ansible etcd migration playbook).

If you discover problems after the migration has completed, you may wish to restore from a backup:

Following an OpenShift Container Platform upgrade, it may be desirable in extreme cases to downgrade your cluster to a previous version. The following sections outline the required steps for each system in a cluster to perform such a downgrade for the OpenShift Container Platform 3.7 to 3.6 downgrade path.

The Ansible playbook used during the upgrade process should have created a backup of the master-config.yaml file and the etcd data directory. Ensure these exist on your masters and etcd members:

/etc/origin/master/master-config.yaml.<timestamp>
/var/lib/etcd/openshift-backup-pre-upgrade-<timestamp>

Also, back up the node-config.yaml file on each node (including masters, which have the node component on them) with a timestamp:

/etc/origin/node/node-config.yaml.<timestamp>

If you are using an external etcd cluster (versus the single embedded etcd), the backup is likely created on all etcd members, though only one is required for the recovery process.

The RPM downgrade process in a later step should create .rpmsave backups of the following files, but it may be a good idea to keep a separate copy regardless:

/etc/sysconfig/atomic-openshift-master
/etc/sysconfig/atomic-openshift-master-api
/etc/sysconfig/atomic-openshift-master-controller
/etc/etcd/etcd.conf 1

On all masters, nodes, and etcd members (if using an external etcd cluster), ensure the relevant services are stopped.

# systemctl stop atomic-openshift-master-api atomic-openshift-master-controllers

On all master and node hosts:

# systemctl stop atomic-openshift-node

On any external etcd hosts:

# systemctl stop etcd

Both OpenShift Container Platform 3.6 and 3.7 require Docker 1.12, so Docker does not need to be downgraded.

Disable the OpenShift Container Platform 3.7 repositories, and re-enable the 3.6 repositories:

# subscription-manager repos \
    --disable=rhel-7-server-ose-3.7-rpms \
    --enable=rhel-7-server-ose-3.6-rpms

On each master, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-clients \
    atomic-openshift-node \
    atomic-openshift-master-api \
    atomic-openshift-master-controllers \
    openvswitch \
    atomic-openshift-sdn-ovs \
    tuned-profiles-atomic-openshift-node \
    atomic-openshift-excluder \
    atomic-openshift-docker-excluder

On each node, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-node \
    openvswitch \
    atomic-openshift-sdn-ovs \
    tuned-profiles-atomic-openshift-node \
    atomic-openshift-excluder \
    atomic-openshift-docker-excluder

If using an external etcd cluster, install the following package on each etcd member:

# yum install etcd

The restore procedure for etcd configuration files replaces the appropriate files, then restarts the service.

If an etcd host has become corrupted and the /etc/etcd/etcd.conf file is lost, restore it using:

$ ssh master-0
# cp /backup/yesterday/master-0-files/etcd.conf /etc/etcd/etcd.conf
# restorecon -Rv /etc/etcd/etcd.conf
# systemctl restart etcd.service

In this example, the backup file is stored in the /backup/yesterday/master-0-files/etcd.conf path where it can be used as an external NFS share, S3 bucket, or other storage solution.

# etcdctl2 member list
5ee217d17301: name=master-0.example.com peerURLs=http://*localhost*:2380 clientURLs=https://192.168.55.8:2379 isLeader=true

After you finish your changes, bring OpenShift Container Platform back online.

Procedure

# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-node

When upgrading your OpenShift Container Platform cluster, there are two primary data migrations that take place. The first occurs while still running the current version of OpenShift Container Platform, and the second happens after the upgrade completes.

If the pre-upgrade migration fails, the upgrade will be halted and the cluster administrator must resolve any data consistency problems before attempting the upgrade again. If the post-upgrade migration fails, the upgrade will complete and the cluster administrator should investigate the data migration issues as soon as possible. This document catalogs known data consistency problems and how to resolve them.

NetworkPolicy objects have migration errors in OpenShift Container Platform versions 3.7.9 to 3.7.22, due to missing migration rules. (BZ#1570777)

# oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=*  --confirm
E0423 16:04:43.881409 error:     -n policy networkpolicies/allow-all: NetworkPolicy.networking.k8s.io "allow-all" is invalid: spec: Forbidden: updates to networkpolicy spec are forbidden.
summary: total=1061 errors=1 ignored=0 unchanged=1010 migrated=49
info: to rerun only failing resources, add --include=networkpolicies
error: 1 resources failed to migrate

The solution is to upgrade the binaries to 3.7.23 or beyond and then migrate the storage. If upgrading to 3.9 or beyond, upgrade to 3.7.23 first, migrate the storage and then do a normal upgrade.