SupportConsoleDevelopersStart a trialContact

Chapter 3. Upgrading

3.1. Overview

When new versions of OpenShift are released, you can upgrade your existing cluster to apply the latest enhancements and bug fixes. This includes upgrading from previous minor versions, such as release 3.1 to 3.2, and applying asynchronous errata updates within a minor version (3.2.z releases). See the OpenShift Enterprise 3.2 Release Notes to review the latest changes.

Note

Due to the core architectural changes between the major versions, OpenShift Enterprise 2 environments cannot be upgraded to OpenShift Enterprise 3 and require a fresh installation.

Unless noted otherwise, node and masters within a major version are forward and backward compatible, so upgrading your cluster should go smoothly. However, you should not run mismatched versions longer than necessary to upgrade the entire cluster.

If you installed using the quick or advanced installation and the ~/.config/openshift/installer.cfg.yml or inventory file that was used is available, you can perform an automated upgrade. Alternatively, you can upgrade OpenShift manually.

Important

Starting with RHBA-2016:1208, upgrades from OpenShift Enterprise 3.1 to 3.2 are supported for clusters using the containerized installation method. See Known Issues.

3.2. Performing Automated Cluster Upgrades

3.2.1. Overview

Starting with OpenShift 3.0.2, if you installed using the advanced installation and the inventory file that was used is available, you can use the upgrade playbook to automate the OpenShift cluster upgrade process. If you installed using the quick installation method and a ~/.config/openshift/installer.cfg.yml file is available, you can use the installer to perform the automated upgrade.

The automated upgrade performs the following steps for you:

  • Applies the latest configuration.
  • Upgrades and restart master services.
  • Upgrades and restart node services.
  • Applies the latest cluster policies.
  • Updates the default router if one exists.
  • Updates the default registry if one exists.
  • Updates default image streams and InstantApp templates.
Important

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

Important

Running Ansible playbooks with the --tags or --check options is not supported by Red Hat.

3.2.2. Preparing for an Automated Upgrade

Note

If you are on OpenShift Enterprise 3.0, you must first upgrade to 3.1 before upgrading to 3.2. Further, if you are currently using the Pacemaker HA method, you must first upgrade to the native HA method before upgrading to 3.2, as the Pacemaker method is no longer supported starting with 3.2. See the OpenShift Enterprise 3.1 upgrade documentation for instructions.

Important

Starting with RHBA-2016:1208, upgrades from OpenShift Enterprise 3.1 to 3.2 are supported for clusters using the containerized installation method. See Known Issues.

To prepare for an automated upgrade:

  1. If you are upgrading from OpenShift Enterprise 3.1 to 3.2, on each master and node host you must manually disable the 3.1 channel and enable the 3.2 channel: 

    # subscription-manager repos --disable="rhel-7-server-ose-3.1-rpms" \
    --enable="rhel-7-server-ose-3.2-rpms"\
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms"
# yum clean all

  2. For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible-* packages: 

    # yum update atomic-openshift-utils

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Enterprise version: 

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. You must be logged in as a cluster administrative user on the master host for the upgrade to succeed: 

    $ oc login

There are two methods for running the automated upgrade: using the installer or running the upgrade playbook directly. Choose and follow one method.

3.2.3. Using the Installer to Upgrade

If you installed OpenShift Enterprise using the quick installation method, you should have an installation configuration file located at ~/.config/openshift/installer.cfg.yml. The installer requires this file to start an upgrade.

The installer supports upgrading between minor versions of OpenShift Enterprise (e.g., 3.1 to 3.2) as well as between asynchronous errata updates within a minor version (e.g., 3.2.z).

If you have an older format installation configuration file in ~/.config/openshift/installer.cfg.yml from an existing OpenShift Enterprise 3.0 or 3.1 installation, the installer will attempt to upgrade the file to the new supported format. If you do not have an installation configuration file of any format, you can create one manually.

To start the upgrade, run the installer with the upgrade subcommand:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Run the following command on each host to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  3. Run the installer with the upgrade subcommand: 

    # atomic-openshift-installer upgrade
  4. Follow the on-screen instructions to upgrade to the latest release.

  5. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. Before rebooting, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude

    Then reboot all hosts.

  6. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

3.2.4. Running the Upgrade Playbook Directly

You can run the automated upgrade playbook using Ansible directly, similar to the advanced installation method, if you have an inventory file.

The same v3_2 upgrade playbook can be used to upgrade either of the following to the latest 3.2 release:

3.2.4.1. Upgrading to OpenShift Enterprise 3.2

Before running the upgrade, first ensure the deployment_type parameter in your inventory file is set to openshift-enterprise.

If you have multiple masters configured and want to enable rolling, full system restarts of the hosts, you can set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the default value services performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details.

Then, run the v3_2 upgrade playbook. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/.ansible/hosts for the last inventory file that was used, if needed. 

# ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml
Note

The upgrade playbook was previously located in a v3_1_to_v3_2 directory. Ensure you are using the latest playbooks per the Preparing for an Automated Upgrade section.

  1. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. Before rebooting, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude

    Then reboot all hosts.

  2. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

3.2.4.2. Upgrading to OpenShift Enterprise 3.2 Asynchronous Releases

To apply asynchronous errata updates to an existing OpenShift Enterprise 3.2 cluster, first upgrade the atomic-openshift-utils package on the Red Hat Enterprise Linux 7 system where you will be running Ansible: 

# yum update atomic-openshift-utils

Then, run the same v3_2 upgrade playbook that is used for upgrading to OpenShift Enterprise 3.2 from 3.1. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/.ansible/hosts for the last inventory file that was used, if needed. 

# ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml

  1. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. Before rebooting, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude

    Then reboot all hosts.

  2. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

3.2.5. Upgrading the EFK Logging Stack

If you have previously deployed the EFK logging stack and want to upgrade to the latest logging component images, the steps must be performed manually as shown in Manual Upgrades.

3.2.6. Upgrading Cluster Metrics

If you have previously deployed cluster metrics, you must manually update to the latest metric components.

3.2.7. Verifying the Upgrade

To verify the upgrade, first check that all nodes are marked as Ready: 

# oc get nodes
NAME                 LABELS                                                                STATUS
master.example.com   kubernetes.io/hostname=master.example.com,region=infra,zone=default   Ready
node1.example.com    kubernetes.io/hostname=node1.example.com,region=primary,zone=east     Ready

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed: 

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.2.1.31",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.2.1.31",

After upgrading, you can use the diagnostics tool on the master to look for common issues: 

# oadm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

3.3. Performing Manual Cluster Upgrades

3.3.1. Overview

As an alternative to performing an automated upgrade, you can manually upgrade your OpenShift cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.

Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.

Important

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

3.3.2. Preparing for a Manual Upgrade

Note

If you are on OpenShift Enterprise 3.0, you must first upgrade to 3.1 before upgrading to 3.2. Further, if you are currently using the Pacemaker HA method, you must first upgrade to the native HA method before upgrading to 3.2, as the Pacemaker method is no longer supported starting with 3.2. See the OpenShift Enterprise 3.1 upgrade documentation for instructions.

Important

Starting with RHBA-2016:1208, upgrades from OpenShift Enterprise 3.1 to 3.2 are supported for clusters using the containerized installation method. See Known Issues.

To prepare for a manual upgrade, follow these steps:

  1. If you are upgrading from OpenShift Enterprise 3.1 to 3.2, manually disable the 3.1 channel and enable the 3.2 channel on each host: 

    # subscription-manager repos --disable="rhel-7-server-ose-3.1-rpms" \
    --enable="rhel-7-server-ose-3.2-rpms" \
    --enable="rhel-7-server-extras-rpms"

    On RHEL 7 systems, also clear the yum cache: 

    # yum clean all

  2. Install or update to the latest available version of the atomic-openshift-utils package on each RHEL 7 system, which provides files that will be used in later sections: 

    # yum install atomic-openshift-utils

    Because RHEL Atomic Host 7 systems cannot use yum, you must download the atomic-openshift-utils package on a subscribed RHEL 7 system and copy the following file to any RHEL Atomic Host 7 systems to be used later: 

    /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/files/nuke_images.sh
    Note

    This file was previously located at /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/docker/files/nuke_images.sh.

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Enterprise version: 

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. Create an etcd backup on each master. The etcd package is required, even if using embedded etcd, for access to the etcdctl command to make the backup. The package is installed by default for RHEL Atomic Host 7 systems. If the master is a RHEL 7 system, ensure the package is installed: 

    # yum install etcd

    Then, create the backup: 

    # ETCD_DATA_DIR=/var/lib/origin 1
# etcdctl backup \
    --data-dir $ETCD_DATA_DIR \
    --backup-dir $ETCD_DATA_DIR.bak.<date> 2
    1
    This directory is for embedded etcd. If you use a separate etcd cluster, use /var/lib/etcd instead.
    2
    Use the date of the backup, or some unique identifier, for <date>. The command will not make a backup if the --backup-dir location already exists.

  5. For any upgrade path, ensure that you are running the latest kernel on each RHEL 7 system: 

    # yum update kernel

3.3.3. Upgrading Master Components

Upgrade your master hosts first:

  1. Run the following command on each master to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  2. Upgrade the atomic-openshift packages or related images.

    1. For masters using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages: 

      # yum upgrade atomic-openshift\*

    2. For masters using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter to the version you are upgrading to in the following files:

      • /etc/sysconfig/atomic-openshift-master (single master clusters only)
      • /etc/sysconfig/atomic-openshift-master-controllers (multi-master clusters only)
      • /etc/sysconfig/atomic-openshift-master-api (multi-master clusters only)
      • /etc/sysconfig/atomic-openshift-node

      • /etc/sysconfig/atomic-openshift-openvswitch

        For example: 

        IMAGE_VERSION=v3.2.1.31

  3. Restart the master service(s) on each master and review logs to ensure they restart successfully.

    For single master clusters: 

    # systemctl restart atomic-openshift-master
# journalctl -r -u atomic-openshift-master

    For multi-master clusters: 

    # systemctl restart atomic-openshift-master-controllers
# systemctl restart atomic-openshift-master-api
# journalctl -r -u atomic-openshift-master-controllers
# journalctl -r -u atomic-openshift-master-api

  4. Because masters also have node components running on them in order to be configured as part of the OpenShift SDN, restart the atomic-openshift-node and openvswitch services: 

    # systemctl restart atomic-openshift-node
# systemctl restart openvswitch
# journalctl -r -u openvswitch
# journalctl -r -u atomic-openshift-node

  5. Run the following command on each master to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude

Upgrade any external etcd hosts using the RPM-based method on a RHEL 7 system:

  1. Upgrade the etcd package: 

    # yum update etcd

  2. Restart the etcd service and review the logs to ensure it restarts successfully: 

    # systemctl restart etcd
# journalctl -r -u etcd

If you are performing a cluster upgrade that requires updating Docker to version 1.10, such as OpenShift Enterprise 3.1 to 3.2 or 3.2.0 to 3.2.1, you must also perform the following steps if you are not already on Docker 1.10:

Important

The node component on masters is set by default to unschedulable status during initial installation, so that pods are not deployed to them. However, it is possible to set them schedulable during the initial installation or manually thereafter. If any of your masters are also configured as a schedulable node, skip the following Docker upgrade steps for those masters and instead run all steps described in Upgrading Nodes when you get to that section for those hosts as well.

  1. Run the following script on each master and external etcd host to remove all containers and images, which is required to avoid a long upgrade process for older images after Docker is updated. Containers and images for pods backed by replication controllers will be recreated automatically: 

    # chmod u+x /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/files/nuke_images.sh
# /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/files/nuke_images.sh

  2. Upgrade Docker.

    1. For RHEL 7 systems: 

      # yum update docker

      Then, restart the docker service and review the logs to ensure it restarts successfully: 

      # systemctl restart docker
# journalctl -r -u docker

    2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

      Note

      If upgrading to RHEL Atomic Host 7.2.5, this upgrades Docker to version 1.10. See the OpenShift Enterprise 3.2.1.1 release notes for details and known issues. 

      # atomic host upgrade

      After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully: 

      # systemctl reboot
# journalctl -r -u docker

3.3.4. Updating Policy Definitions

After a cluster upgrade, the recommended default cluster roles may have been updated. To check if an update is recommended for your environment, you can run: 

# oadm policy reconcile-cluster-roles

This command outputs a list of roles that are out of date and their new proposed values. For example: 

# oadm policy reconcile-cluster-roles
apiVersion: v1
items:
- apiVersion: v1
  kind: ClusterRole
  metadata:
    creationTimestamp: null
    name: admin
  rules:
  - attributeRestrictions: null
    resources:
    - builds/custom
...
Note

Your output will vary based on the OpenShift version and any local customizations you have made. Review the proposed policy carefully.

You can either modify this output to re-apply any local policy changes you have made, or you can automatically apply the new policy using the following process:

  1. Reconcile the cluster roles: 

    # oadm policy reconcile-cluster-roles \
    --additive-only=true \
    --confirm

  2. Reconcile the cluster role bindings: 

    # oadm policy reconcile-cluster-role-bindings \
    --exclude-groups=system:authenticated \
    --exclude-groups=system:authenticated:oauth \
    --exclude-groups=system:unauthenticated \
    --exclude-users=system:anonymous \
    --additive-only=true \
    --confirm

  3. Reconcile security context constraints: 

    # oadm policy reconcile-sccs \
    --additive-only=true \
    --confirm

3.3.5. Upgrading Nodes

After upgrading your masters, you can upgrade your nodes. When restarting the atomic-openshift-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.

One at at time for each node that is not also a master, you must disable scheduling and evacuate its pods to other nodes, then upgrade packages and restart services.

  1. Run the following command on each node to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  2. As a user with cluster-admin privileges, disable scheduling for the node: 

    # oadm manage-node <node> --schedulable=false

  3. Evacuate pods on the node to other nodes:

    Important

    The --force option deletes any pods that are not backed by a replication controller. 

    # oadm manage-node <node> --evacuate --force

  4. Upgrade the node component packages or related images.

    1. For nodes using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages: 

      # yum upgrade atomic-openshift\*

    2. For nodes using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter in the /etc/sysconfig/atomic-openshift-node and /etc/sysconfig/openvswitch files to the version you are upgrading to. For example: 

      IMAGE_VERSION=v3.2.1.31

  5. Restart the atomic-openshift-node and openvswitch services and review the logs to ensure they restart successfully: 

    # systemctl restart atomic-openshift-node
# systemctl restart openvswitch
# journalctl -r -u atomic-openshift-node
# journalctl -r -u openvswitch

  6. If you are performing a cluster upgrade that requires updating Docker to version 1.10, such as OpenShift Enterprise 3.1 to 3.2 or 3.2.0 to 3.2.1, you must also perform the following steps if you are not already on Docker 1.10:

    1. Run the following script to remove all containers and images, which is required to avoid a long upgrade process for older images after Docker is updated. Containers and images for pods backed by replication controllers will be recreated automatically: 

      # chmod u+x /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/files/nuke_images.sh
# /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/files/nuke_images.sh

    2. Upgrade Docker.

      1. For RHEL 7 systems: 

        # yum update docker

        Then, restart the docker service and review the logs to ensure it restarts successfully: 

        # systemctl restart docker
# journalctl -r -u docker

        After Docker is restarted, restart the atomic-openshift-node service again and review the logs to ensure it restarts successfully: 

        # systemctl restart atomic-openshift-node
# journalctl -r -u atomic-openshift-node

      2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

        Note

        If upgrading to RHEL Atomic Host 7.2.5, this upgrades Docker to version 1.10. See the OpenShift Enterprise 3.2.1.1 release notes for details and known issues. 

        # atomic host upgrade

        After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully: 

        # systemctl reboot
# journalctl -r -u docker

  7. Re-enable scheduling for the node: 

    # oadm manage-node <node> --schedulable

  8. Run the following command on the node to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude
  9. Repeat these steps on the next node, and continue repeating these steps until all nodes have been upgraded.

  10. After all nodes have been upgraded, as a user with cluster-admin privileges, verify that all nodes are showing as Ready: 

    # oc get nodes
NAME                    LABELS                                        STATUS
master.example.com      kubernetes.io/hostname=master.example.com     Ready,SchedulingDisabled
node1.example.com       kubernetes.io/hostname=node1.example.com      Ready
node2.example.com       kubernetes.io/hostname=node2.example.com      Ready

3.3.6. Upgrading the Router

If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly-available routing service.

Edit your router’s deployment configuration. For example, if it has the default router name: 

# oc edit dc/router

Apply the following changes: 

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-haproxy-router:v3.2.1.31 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust the image version to match the version you are upgrading to.

You should see one router pod updated and then the next.

3.3.7. Upgrading the Registry

The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim or a host mount point, you may restart the registry without losing the contents of your registry. Deploying a Docker Registry details how to configure persistent storage for the registry.

Edit your registry’s deployment configuration: 

# oc edit dc/docker-registry

Apply the following changes: 

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-docker-registry:v3.2.1.31 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust the image version to match the version you are upgrading to.
Important

Images that are being pushed or pulled from the internal registry at the time of upgrade will fail and should be restarted automatically. This will not disrupt pods that are already running.

3.3.8. Updating the Default Image Streams and Templates

By default, the quick and advanced installation methods automatically create default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under the /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/ directory.

Note

Because RHEL Atomic Host 7 cannot use yum to update packages, the following steps must take place on a RHEL 7 system.

  1. Update the packages that provide the example JSON files. On a subscribed Red Hat Enterprise Linux 7 system where you can run the CLI as a user with cluster-admin permissions, install or update to the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible- packages: 

    # yum update atomic-openshift-utils

    The openshift-ansible-roles package provides the latest example JSON files.

  2. After a manual upgrade, get the latest templates from openshift-ansible-roles: 

    rpm -ql openshift-ansible-roles | grep examples | grep v1.2

    In this example, /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/image-streams-rhel7.json is the latest file that you want in the latest openshift-ansible-roles package.

    /usr/share/openshift/examples/image-streams/image-streams-rhel7.json is not owned by a package, but is updated by Ansible. If you are upgrading outside of Ansible. you need to get the latest .json files on the system where you are running oc, which can run anywhere that has access to the master.

  3. Install atomic-openshift-utils and its dependencies to install the new content into /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/.: 

    $ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/image-streams-rhel7.json
$ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/dotnet_imagestreams.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/image-streams-rhel7.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/dotnet_imagestreams.json

  4. Update the templates: 

    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/db-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/infrastructure-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-streams/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/db-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/infrastructure-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-streams/

    Errors are generated for items that already exist. This is expected behavior: 

    # oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/cakephp-mysql.json": templates "cakephp-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/cakephp.json": templates "cakephp-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/dancer-mysql.json": templates "dancer-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/dancer.json": templates "dancer-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/django-postgresql.json": templates "django-psql-example" already exists

Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.

3.3.9. Importing the Latest Images

After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run: 

# oc import-image -n openshift <imagestream>

For example, get the list of all image streams in the default openshift project: 

# oc get is -n openshift
NAME     DOCKER REPO                                                      TAGS                   UPDATED
mongodb  registry.access.redhat.com/openshift3/mongodb-24-rhel7           2.4,latest,v3.1.1.6    16 hours ago
mysql    registry.access.redhat.com/openshift3/mysql-55-rhel7             5.5,latest,v3.1.1.6    16 hours ago
nodejs   registry.access.redhat.com/openshift3/nodejs-010-rhel7           0.10,latest,v3.1.1.6   16 hours ago
...

Update each image stream one at a time: 

# oc import-image -n openshift nodejs
The import completed successfully.

Name:			nodejs
Created:		10 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2016-07-05T19:20:30Z
Docker Pull Spec:	172.30.204.22:5000/openshift/nodejs

Tag	Spec								Created		PullSpec						Image
latest	4								9 seconds ago	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest	570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
4	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest		9 seconds ago	<same>							570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
0.10	registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest	9 seconds ago	<same>							a1ef33be788a28ec2bdd48a9a5d174ebcfbe11c8e986d2996b77f5bccaaa4774
Important

In order to update your S2I-based applications, you must manually trigger a new build of those applications after importing the new images using oc start-build <app-name>.

3.3.10. Upgrading the EFK Logging Stack

Use the following to upgrade an already-deployed EFK logging stack.

Note

The following steps apply when upgrading from OpenShift Enterprise 3.1 to 3.2, or are applying an asynchronous update to 3.2. These steps pull the latest 3.2 logging images.

  1. Ensure you are working in the project where the EFK stack was previously deployed. For example, if the project is named logging: 

    $ oc project logging

  2. Scale down your Fluentd instances to 0: 

    $ oc scale dc/logging-fluentd --replicas=0

    Wait until they have terminated. This helps prevent loss of data by giving them time to properly flush their current buffer and send any logs they were processing to Elasticsearch.

  3. Scale down your Kibana instances: 

    $ oc scale dc/logging-kibana --replicas=0

    If you have an operations deployment, also run: 

    $ oc scale dc/logging-kibana-ops --replicas=0

  4. Once confirming your Fluentd and Kibana pods have been terminated, scale down the Elasticsearch pods: 

    $ oc scale dc/logging-es-<unique_name> --replicas=0

    If you have an operations deployment, also run: 

    $ oc scale dc/logging-es-ops-<unique_name> --replicas=0

  5. After confirming your Elasticsearch pods have been terminated, rerun the deployer to generate any missing or changed features.

    1. Follow the first step in Deploying the EFK Stack. After the deployer completes, re-attach the persistent volume claims you were previously using, then deploy a template that is created by the deployer: 

      $ oc process logging-support-template | oc apply -f -

  6. Deployment of logging components is intended to happen automatically based on tags being imported into the image streams created in the previous step. However, as not all tags are automatically imported, this mechanism has become unreliable as multiple versions are released. Therefore, manual importing may be necessary as follows.

    For each image stream logging-auth-proxy, logging-kibana, logging-elasticsearch, and logging-fluentd, manually import the tag corresponding to the IMAGE_VERSION specified (or defaulted) for the deployer. 

    $ oc import-image <name>:<version> --from <prefix><name>:<tag>

    For example: 

    $ oc import-image logging-auth-proxy:3.2.1 \
     --from registry.access.redhat.com/openshift3/logging-auth-proxy:3.2.1
$ oc import-image logging-kibana:3.2.1 \
     --from registry.access.redhat.com/openshift3/logging-kibana:3.2.1
$ oc import-image logging-elasticsearch:3.2.1 \
     --from registry.access.redhat.com/openshift3/logging-elasticsearch:3.2.1
$ oc import-image logging-fluentd:3.2.1 \
     --from registry.access.redhat.com/openshift3/logging-fluentd:3.2.1

  7. Next, scale Elasticsearch back up incrementally so that the cluster has time to rebuild.

    1. To begin, scale up to 1: 

      $ oc scale dc/logging-es-<unique_name> --replicas=1

      Follow the logs of the resulting pod to ensure that it is able to recover its indices correctly and that there are no errors: 

      $ oc logs -f <pod_name>

      If that is successful, you can then do the same for the operations cluster, if one was previously used.

    2. After all Elasticsearch nodes have recovered their indices, continue to scale it back up to the size it was prior to doing maintenance. Check the logs of the Elasticsearch members to verify that they have correctly joined the cluster and recovered.
  8. Now scale Kibana and Fluentd back up to their previous state. Because Fluentd was shut down and allowed to push its remaining records to Elasticsearch in the previous steps, it can now pick back up from where it left off with no loss of logs, provided any unread log files are still available on the node.

  9. In the latest version, Kibana will display indices differently now in order to prevent users from being able to access the logs of previously created projects that have been deleted.

    Due to this change, your old logs will not appear automatically. To migrate your old indices to the new format, rerun the deployer with -v MODE=migrate in addition to your prior flags. This should be run while your Elasticsearch cluster is running, as the script must connect to it to make changes.

    Note

    This only impacts non-operations logs. Operations logs will appear the same as in previous versions. There should be minimal performance impact to Elasticsearch while running this and it will not perform an install.

3.3.11. Upgrading Cluster Metrics

After upgrading an already-deployed Cluster Metrics install, you must update to a newer version of the metrics components.

  • The update process stops all the metrics containers, updates the metrics configuration files, and redeploys the newer components.
  • It does not change the metrics route.
  • It does not delete the metrics persistent volume claim. Metrics stored to persistent volumes before the update are available after the update completes.
Important

The update deletes all non-persisted metric values and overwrites local changes to the metrics configurations. For example, the number of instances in a replica set is not saved.

To update, follow the same steps as when the metrics components were first deployed, using the correct template, except this time, specify the MODE=refresh option: 

$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hm.example.com,MODE=refresh 1
1
In the original deployment command, there was no MODE=refresh.
Note

During the update, the metrics components do not run. Because of this, they cannot collect data and a gap normally appears in the graphs.

3.3.12. Additional Manual Steps Per Release

Some OpenShift releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster. Read through the following sections carefully depending on your upgrade path, as you may be required to perform certain steps at key points during the standard upgrade process described earlier in this topic.

See the OpenShift Enterprise 3.2 Release Notes to review the latest release notes.

3.3.12.1. OpenShift Enterprise 3.2.0

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.0 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.2. OpenShift Enterprise 3.2.1.1

The upgrade to OpenShift Enterprise 3.2.1.1 involves updating to Docker 1.10. The steps to properly upgrade Docker are higlighted and included inline in the Upgrading Master Components and Upgrading Nodes sections. No other additional manual steps are required for this release.

Important

See the Known Issues for OpenShift Enterprise 3.2.1.1 for more details on using OpenShift Enterprise and Docker 1.10.

3.3.12.3. OpenShift Enterprise 3.2.1.4

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.4 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.4. OpenShift Enterprise 3.2.1.9

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.9 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.5. OpenShift Enterprise 3.2.1.13

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.13 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.6. OpenShift Enterprise 3.2.1.15

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.15 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.7. OpenShift Enterprise 3.2.1.17

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.17 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.8. OpenShift Enterprise 3.2.1.21

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.21 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.9. OpenShift Enterprise 3.2.1.23

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.23 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.10. OpenShift Enterprise 3.2.1.26

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.26 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.11. OpenShift Enterprise 3.2.1.28

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.28 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.12. OpenShift Enterprise 3.2.1.30

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.30 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.13. OpenShift Enterprise 3.2.1.31-2

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.31-2 that are not already mentioned inline during the standard manual upgrade process.

3.3.12.14. OpenShift Enterprise 3.2.1.31-4

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.2.1.31-4 that are not already mentioned inline during the standard manual upgrade process.

3.3.13. Verifying the Upgrade

To verify the upgrade, first check that all nodes are marked as Ready: 

# oc get nodes
NAME                 LABELS                                                                STATUS
master.example.com   kubernetes.io/hostname=master.example.com,region=infra,zone=default   Ready
node1.example.com    kubernetes.io/hostname=node1.example.com,region=primary,zone=east     Ready

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed: 

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.2.1.31",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.2.1.31",

After upgrading, you can use the diagnostics tool on the master to look for common issues: 

# oadm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.