Upgrading Clusters

Pull the latest subscription data from Red Hat Subscription Manager:

# subscription-manager refresh

If you are upgrading from OpenShift Container Platform 3.10 to 3.11:

Review and update your inventory file.

To enable rolling, full system restarts of the hosts, set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the service is restarted on masters, but the systems do not reboot.

See Configuring Cluster Variables for details.

If you modified the oreg_url parameter to change the cluster image registry location, you must run the imageconfig playbook to update the image location:

$ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook -i </path/to/inventory/file> \
    playbooks/openshift-node/imageconfig.yml

Upgrade your nodes.

If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify its location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts for the last inventory file that was used.

Upgrade standard infra nodes (router, registry, logging, and metrics).

Upgrade standard nodes running application containers.

Upgrade the OpenShift Container Platform nodes running GlusterFS one at a time.

Check that all nodes are marked as Ready:

# oc get nodes
NAME                     STATUS    ROLES        AGE       VERSION
master1.example.com      Ready     master       47d       v1.11.0+d4cacc0
master2.example.com      Ready     master       47d       v1.11.0+d4cacc0
master3.example.com      Ready     master       47d       v1.11.0+d4cacc0
infra-node1.example.com  Ready     infra        47d       v1.11.0+d4cacc0
infra-node2.example.com  Ready     infra        47d       v1.11.0+d4cacc0
node1.example.com        Ready     compute      47d       v1.11.0+d4cacc0
node2.example.com        Ready     compute      47d       v1.11.0+d4cacc0

Verify that the static pods for the control plane are running:

# oc get pods -n kube-system
NAME                                 READY     STATUS    RESTARTS   AGE
master-api-master1.example.com           1/1       Running   4          1h
master-controllers-master1.example.com   1/1       Running   3          1h
master-etcd-master1.example.com          1/1       Running   6          5d
[...]

Verify that you are running the expected versions of the docker-registry and router images, if deployed:

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.11.634",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.11.634",

Use the diagnostics tool on the master to look for common issues:

# oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

Set the green nodes to schedulable so that new pods are deployed to them:

$ oc adm manage-node --schedulable=true --selector=color=green

Set the blue nodes to unschedulable so that no new pods run on them:

$ oc adm manage-node --schedulable=false --selector=color=blue

Update the node selectors for the registry and router deployment configurations to use the node-role.kubernetes.io/infra=true label. This change starts new deployments that place the registry and router pods on your new infrastructure nodes.

To monitor build progress across all namespaces (projects) in the cluster:

$ oc get events -w --all-namespaces

In large environments, builds rarely completely stop. However, you should see a large increase and decrease caused by the administrative image import.

Evacuate and delete all blue nodes with the following commands:

$ oc adm manage-node --selector=color=blue --evacuate
$ oc delete node --selector=color=blue

After the blue node hosts no longer contain pods and have been removed from OpenShift Container Platform, they are safe to power off. As a safety precaution, confirm that there are no issues with the upgrade before you power off the hosts.

Drain the node in preparation for maintenance:

$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

In order to protect sensitive packages that do not need to be updated, apply the exclude rules to the host:

# atomic-openshift-docker-excluder exclude
# atomic-openshift-excluder exclude

A reboot ensures that the host is running the newest versions and means that the container engine and OpenShift Container Platform processes have been restarted, which forces them to check that all of the rules in other services are correct.

# yum update
# reboot

However, instead of rebooting a node host, you can restart the services that are affected or preserve the iptables state. Both processes are described in the OpenShift Container Platform iptables topic. The ovs flow rules do not need to be saved, but restarting the OpenShift Container Platform node software fixes the flow rules.

Configure the host to be schedulable again:

$ oc adm uncordon <node_name>

Ensure that a backup of the master-config.yaml file, scheduler.json file, and the etcd data directory exist on your masters:

/etc/origin/master/master-config.yaml.<timestamp>
/etc/origin/master/master.env
/etc/origin/master/scheduler.json
/var/lib/etcd/openshift-backup-xxxx

You save these files during the upgrade process.

Locate the copies of the following files that you created when you prepared for an upgrade.

On node and master hosts:

/etc/origin/node/node-config.yaml

On etcd hosts, including masters that have etcd co-located on them:

/etc/etcd/etcd.conf

On all master and node hosts, stop the master and node services by removing the pod definition and rebooting the host:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/
# reboot

On all masters, nodes, and etcd members (if using a dedicated etcd cluster), remove the following packages:

# yum remove atomic-openshift \
   atomic-openshift-excluder \
   atomic-openshift-hyperkube \
   atomic-openshift-node \
   atomic-openshift-docker-excluder \
   atomic-openshift-clients

Verify the packages were removed successfully:

# rpm -qa | grep atomic-openshift

On control plane hosts (master and etcd hosts), move the static pod definitions:

# mkdir /etc/origin/node/pods-backup
# mv /etc/origin/node/pods/* /etc/origin/node/pods-backup/

Reboot each host:

# reboot

Disable the OpenShift Container Platform 3.11 repositories, and re-enable the 3.10 repositories:

# subscription-manager repos \
    --disable=rhel-7-server-ose-3.11-rpms \
    --enable=rhel-7-server-ose-3.10-rpms

On each master and node host, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-node \
    atomic-openshift-docker-excluder \
    atomic-openshift-excluder \
    atomic-openshift-clients \
    atomic-openshift-hyperkube

On each host, verify the packages were installed successfully:

# rpm -qa | grep atomic-openshift

On each OpenShift Container Platform master, restore your master and node configuration from backup and enable and restart all relevant services:

# cp ${MYBACKUPDIR}/etc/origin/node/pods/* /etc/origin/node/pods/
# cp ${MYBACKUPDIR}/etc/origin/master/master.env /etc/origin/master/master.env
# cp ${MYBACKUPDIR}/etc/origin/master/master-config.yaml.<timestamp> /etc/origin/master/master-config.yaml
# cp ${MYBACKUPDIR}/etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# cp ${MYBACKUPDIR}/etc/origin/master/scheduler.json.<timestamp> /etc/origin/master/scheduler.json
# master-restart api
# master-restart controllers

On each OpenShift Container Platform node, update the node configuration maps as needed, and enable and restart the atomic-openshift-node service:

# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-node

Check that all nodes are marked as Ready:

# oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Verify the successful downgrade of the registry and router, if deployed:

Use the diagnostics tool on the master to look for common issues and provide suggestions:

# oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.