Upgrading Clusters

Pull the latest subscription data from RHSM:

# subscription-manager refresh

If you are upgrading from OpenShift Container Platform 3.9 to 3.10:

If you have applied manual configuration changes to your master or node configuration files since your last Ansible playbook run (whether that was initial installation or your most recent cluster upgrade), and you have not yet made the equivalent changes to your inventory file, review Configuring Your Inventory File. For any variables that are relevant to the manual changes you made, apply the equivalent appropriate changes to your inventory files before running the upgrade. Otherwise, your manual changes may be overwritten by default values during the upgrade, which could cause pods to not run properly or other cluster stability issues.

Additionally, verify the following settings:

At this point, you can choose to run the upgrade in a single or multiple phases. See Upgrading the Control Plane and Nodes in Separate Phases for more details which components are upgraded in each phase.

If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify its location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts for the last inventory file that was used, if needed.

After all master and node upgrades have completed, reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features you have previously enabled.

If you have not already done so, see Specifying Logging Ansible Variables in the Aggregating Container Logs topic and update your Ansible inventory file to at least set the following required variable within the [OSEv3:vars] section:

[OSEv3:vars]

openshift_logging_install_logging=true 1

If you have not already done so, see Specifying Metrics Ansible Variables in the Enabling Cluster Metrics topic and update your Ansible inventory file to at least set the following required variables within the [OSEv3:vars] section:

[OSEv3:vars]

openshift_metrics_install_metrics=true 1
openshift_metrics_hawkular_hostname=<fqdn> 2
openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) 3

Upgrade standard infra nodes (router, registry, logging, and metrics).

Upgrade standard nodes running application containers.

Upgrade the OpenShift Container Platform nodes running GlusterFS one at a time.

Check that all nodes are marked as Ready:

# oc get nodes
NAME                     STATUS    ROLES        AGE       VERSION
master1.example.com      Ready     master       47d       v1.10.0+b81c8f8
master2.example.com      Ready     master       47d       v1.10.0+b81c8f8
master3.example.com      Ready     master       47d       v1.10.0+b81c8f8
infra-node1.example.com  Ready     infra        47d       v1.10.0+b81c8f8
infra-node2.example.com  Ready     infra        47d       v1.10.0+b81c8f8
node1.example.com        Ready     compute      47d       v1.10.0+b81c8f8
node2.example.com        Ready     compute      47d       v1.10.0+b81c8f8

Verify the static pods for the control plane are running:

# oc get pods -n kube-system
NAME                                 READY     STATUS    RESTARTS   AGE
master-api-master1.example.com           1/1       Running   4          1h
master-controllers-master1.example.com   1/1       Running   3          1h
master-etcd-master1.example.com          1/1       Running   6          5d
[...]

Verify that you are running the expected versions of the docker-registry and router images, if deployed.

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.10",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.10",

Use the diagnostics tool on the master to look for common issues:

# oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

Set the green nodes to schedulable so that new pods are deployed to them:

$ oc adm manage-node --schedulable=true --selector=color=green

Set the blue nodes to unschedulable so that no new pods run on them:

$ oc adm manage-node --schedulable=false --selector=color=blue

Update the node selectors for the registry and router deployment configurations to use the node-role.kubernetes.io/infra=true label. This change starts new deployments that place the registry and router pods on your new infrastructure nodes.

To monitor build progress across all namespaces (projects) in the cluster:

$ oc get events -w --all-namespaces

In large environments, builds rarely completely stop. However, you should see a large increase and decrease caused by the administrative image import.

Evacuate and delete all blue nodes with the following commands:

$ oc adm manage-node --selector=color=blue --evacuate
$ oc delete node --selector=color=blue

After the blue node hosts no longer contain pods and have been removed from OpenShift Container Platform, they are safe to power off. As a safety precaution, confirm that there are no issues with the upgrade before you power off the hosts.

Drain the node in preparation for maintenance:

$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

In order to protect sensitive packages that do not need to be updated, apply the exclude rules to the host:

# atomic-openshift-docker-excluder exclude
# atomic-openshift-excluder exclude

Update the host packages and reboot the host. A reboot ensures that the host is running the newest versions and means that the docker and OpenShift Container Platform processes have been restarted, which forces them to check that all of the rules in other services are correct.

 # yum update
 # reboot

However, instead of rebooting a node host, you can restart the services that are affected or preserve the iptables state. Both processes are described in the OpenShift Container Platform iptables topic. The ovs flow rules do not need to be saved, but restarting the OpenShift Container Platform node software fixes the flow rules.

Configure the host to be schedulable again:

$ oc adm uncordon <node_name>

The Ansible playbook used during the upgrade process should have created a backup of the master-config.yaml file. Ensure this and the scheduler.json file exist on your masters:

/etc/origin/master/master-config.yaml.<timestamp>
/etc/origin/master/scheduler.json

The procedure in Preparing for an Automated Upgrade instructed you to back up the following files before proceeding with an upgrade from OpenShift Container Platform 3.9 to 3.10; ensure that you still have these files available.

On master hosts:

/usr/lib/systemd/system/atomic-openshift-master-api.service
/usr/lib/systemd/system/atomic-openshift-master-controllers.service
/etc/sysconfig/atomic-openshift-master-api
/etc/sysconfig/atomic-openshift-master-controllers

On node and master hosts:

/usr/lib/systemd/system/atomic-openshift-*.service
/etc/origin/node/node-config.yaml

On etcd hosts, including masters that have etcd co-located on them:

/etc/etcd/etcd.conf
/backup/etcd-xxxxxx/backup.db

On all master and node hosts, stop the master and node services by removing the pod definition and rebooting the host:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/
# reboot

On all masters, nodes, and etcd members (if using a dedicated etcd cluster), remove the following packages:

# yum remove atomic-openshift \
    atomic-openshift-clients \
    atomic-openshift-node \
    atomic-openshift-master \
    atomic-openshift-sdn-ovs \
    atomic-openshift-excluder \
    atomic-openshift-docker-excluder \
    atomic-openshift-hyperkube

Verify the packages were removed successfully:

# rpm -qa | grep atomic-openshift

On control plane hosts (master and etcd hosts), move the static pod definitions:

# mkdir /etc/origin/node/pods-backup
# mv /etc/origin/node/pods/* /etc/origin/node/pods-backup/

Reboot each host:

# reboot

Disable the OpenShift Container Platform 3.10 repositories, and re-enable the 3.9 repositories:

# subscription-manager repos \
    --disable=rhel-7-server-ose-3.10-rpms \
    --enable=rhel-7-server-ose-3.9-rpms

On each master, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-clients \
    atomic-openshift-node \
    atomic-openshift-master \
    openvswitch \
    atomic-openshift-sdn-ovs \
    tuned-profiles-atomic-openshift-node \
    atomic-openshift-excluder \
    atomic-openshift-docker-excluder

On each node, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-node \
    openvswitch \
    atomic-openshift-sdn-ovs \
    tuned-profiles-atomic-openshift-node \
    atomic-openshift-excluder \
    atomic-openshift-docker-excluder

On each host, verify the packages were installed successfully:

# rpm -qa | grep atomic-openshift
# rpm -q openvswitch

On each OpenShift Container Platform master, restore your master and node configuration from backup and enable and restart all relevant services:

# cp ${MYBACKUPDIR}/etc/origin/node/pods/* /etc/origin/node/pods/
# cp ${MYBACKUPDIR}/etc/origin/master/master.env /etc/origin/master/master.env
# cp ${MYBACKUPDIR}/etc/origin/master/master-config.yaml.<timestamp> /etc/origin/master/master-config.yaml
# cp ${MYBACKUPDIR}/etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# cp ${MYBACKUPDIR}/etc/origin/master/scheduler.json.<timestamp> /etc/origin/master/scheduler.json
# cp ${MYBACKUPDIR}/usr/lib/systemd/system/atomic-openshift-master-api.service /usr/lib/systemd/system/atomic-openshift-master-api.service
# cp ${MYBACKUPDIR}/usr/lib/systemd/system/atomic-openshift-master-controllers.service /usr/lib/systemd/system/atomic-openshift-master-controllers.service
# rm /etc/systemd/system/atomic-openshift-node.service
# systemctl daemon-reload
# master-restart api
# master-restart controllers

On each OpenShift Container Platform node, update the node configuration maps as needed, and enable and restart the atomic-openshift-node service:

# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# rm /etc/systemd/system/atomic-openshift-node.service
# systemctl daemon-reload
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-node

To verify the downgrade, first check that all nodes are marked as Ready:

# oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Verify the successful downgrade of the registry and router, if deployed:

Use the diagnostics tool on the master to look for common issues and provide suggestions:

# oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.