Chapter 5. Downgrading a cluster

Ensure that a backup of the master-config.yaml file, scheduler.json file, and the etcd data directory exist on your masters:

/etc/origin/master/master-config.yaml.<timestamp>
/etc/origin/master/master.env
/etc/origin/master/scheduler.json
/var/lib/etcd/openshift-backup-xxxx

You save these files during the upgrade process.

Locate the copies of the following files that you created when you prepared for an upgrade.

On node and master hosts:

/etc/origin/node/node-config.yaml

On etcd hosts, including masters that have etcd co-located on them:

/etc/etcd/etcd.conf

On all master and node hosts, stop the master and node services by removing the pod definition and rebooting the host:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/
# reboot

On all masters, nodes, and etcd members (if using a dedicated etcd cluster), remove the following packages:

# yum remove atomic-openshift \
   atomic-openshift-excluder \
   atomic-openshift-hyperkube \
   atomic-openshift-node \
   atomic-openshift-docker-excluder \
   atomic-openshift-clients

Verify the packages were removed successfully:

# rpm -qa | grep atomic-openshift

On control plane hosts (master and etcd hosts), move the static pod definitions:

# mkdir /etc/origin/node/pods-backup
# mv /etc/origin/node/pods/* /etc/origin/node/pods-backup/

Reboot each host:

# reboot

Disable the OpenShift Container Platform 3.11 repositories, and re-enable the 3.10 repositories:

# subscription-manager repos \
    --disable=rhel-7-server-ose-3.11-rpms \
    --enable=rhel-7-server-ose-3.10-rpms

On each master and node host, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-node \
    atomic-openshift-docker-excluder \
    atomic-openshift-excluder \
    atomic-openshift-clients \
    atomic-openshift-hyperkube

On each host, verify the packages were installed successfully:

# rpm -qa | grep atomic-openshift

On each OpenShift Container Platform master, restore your master and node configuration from backup and enable and restart all relevant services:

# cp ${MYBACKUPDIR}/etc/origin/node/pods/* /etc/origin/node/pods/
# cp ${MYBACKUPDIR}/etc/origin/master/master.env /etc/origin/master/master.env
# cp ${MYBACKUPDIR}/etc/origin/master/master-config.yaml.<timestamp> /etc/origin/master/master-config.yaml
# cp ${MYBACKUPDIR}/etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# cp ${MYBACKUPDIR}/etc/origin/master/scheduler.json.<timestamp> /etc/origin/master/scheduler.json
# master-restart api
# master-restart controllers

On each OpenShift Container Platform node, update the node configuration maps as needed, and enable and restart the atomic-openshift-node service:

# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-node

Check that all nodes are marked as Ready:

# oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Verify the successful downgrade of the registry and router, if deployed:

Use the diagnostics tool on the master to look for common issues and provide suggestions:

# oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.