Chapter 3. Performing Manual In-place Cluster Upgrades
3.1. Overview
As of OpenShift Container Platform 3.9, manual upgrades are not supported. In a future release, this process will be removed.
As an alternative to performing an automated upgrade, you can manually upgrade your OpenShift Container Platform cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.
Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.
Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.
If you are using GlusterFS, see Special Considerations When Using Containerized GlusterFS before proceeding.
The day before the upgrade, validate OpenShift Container Platform storage migration to ensure potential issues are resolved prior to the outage window:
$ oc adm migrate storage --include=* --loglevel=2 --confirm --config /etc/origin/master/admin.kubeconfig
3.2. Preparing for a Manual Upgrade
Before upgrading your cluster to OpenShift Container Platform 3.9, the cluster must be already upgraded to the latest asynchronous release of version 3.7. If your cluster is at a version earlier than 3.7, you must first upgrade incrementally (e.g., 3.5 to 3.6, then 3.6 to 3.7).
To prepare for a manual upgrade, follow these steps:
- Follow the steps in Verifying the Upgrade to verify the cluster’s health (if you are upgrading from 3.7 to 3.9, check the command outputs for the relevant 3.7 version instead of the stated 3.9). This will confirm that nodes are in the Ready state, running the expected starting version, and will ensure that there are no diagnostic errors or warnings.
Pull the latest subscription data from Red Hat Subscription Manager (RHSM):
# subscription-manager refresh
If you are upgrading from OpenShift Container Platform 3.7 to 3.9:
Manually disable the 3.7 repository and enable both the 3.8 and 3.9 repositories on each master and node host. You must also enable the rhel-7-server-ansible-2.4-rpms repository, which is a new requirement starting with OpenShift Container Platform 3.9:
# subscription-manager repos --disable="rhel-7-server-ose-3.7-rpms" \ --enable="rhel-7-server-ose-3.8-rpms" \ --enable="rhel-7-server-ose-3.9-rpms" \ --enable="rhel-7-server-extras-rpms" \ --enable="rhel-7-server-ansible-2.4-rpms" \ --enable="rhel-7-fast-datapath-rpms" # yum clean all
-
You must disable swap memory in your cluster before upgrading to OpenShift Container Platform 3.9, otherwise the upgrade will fail. Whether swap memory was enabled using
openshift_disable_swap=false
in your Ansible inventory file or enabled manually per host, see Disabling Swap Memory in the Cluster Administration guide to disable it on each host.
Install or update to the latest available version of the atomic-openshift-utils package on each RHEL 7 system, which provides files that will be used in later sections:
# yum install atomic-openshift-utils
Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Container Platform version:
# yum install atomic-openshift-excluder atomic-openshift-docker-excluder
These packages add entries to the
exclude
directive in the host’s /etc/yum.conf file.- Create an etcd backup on each etcd host.
For any upgrade path, ensure that you are running the latest kernel on each RHEL 7 system:
# yum update kernel
3.3. Upgrading Master Components
Before upgrading any stand-alone nodes, upgrade the master components (which provide the control plane for the cluster).
Run the following command on each master to remove the atomic-openshift packages from the list of yum excludes on the host:
# atomic-openshift-excluder unexclude
Upgrade etcd on all master hosts and any external etcd hosts.
For RHEL 7 systems using the RPM-based method:
Upgrade the etcd package:
# yum update etcd
Restart the etcd service and review the logs to ensure it restarts successfully:
# systemctl restart etcd # journalctl -r -u etcd
For RHEL Atomic Host 7 systems and RHEL 7 systems using the containerized method:
Pull the latest rhel7/etcd image:
# docker pull registry.access.redhat.com/rhel7/etcd
Restart the etcd_container service and review the logs to ensure it restarts successfully:
# systemctl restart etcd_container # journalctl -r -u etcd_container
Configure Dnsmasq. You must perform the following steps to configure dnsmasq as part of the upgrade.
Create a /etc/origin/node/node-dnsmasq.conf node configuration file.
server=/in-addr.arpa/127.0.0.1 server=/cluster.local/127.0.0.1
Edit the /etc/dnsmasq.d/origin-dns.conf file as follows:
no-resolv domain-needed no-negcache max-cache-ttl=1 enable-dbus dns-forward-max=5000 cache-size=5000 bind-dynamic except-interface=lo
Edit the /etc/dnsmasq.d/origin-upstream-dns.conf file as follows:
server=<dns_server1_ip_address> server=<dns_server2_ip_address>
Edit the /etc/origin/node/node-config.yaml as follows:
dnsBindAddress: 127.0.0.1:53 dnsRecursiveResolvConf: /etc/origin/node/resolv.conf dnsDomain: cluster.local dnsIP: <node_ip_address> 1
- 1
- This is the IP address of the node host.
Update the /etc/systemd/system/atomic-openshift-node.service node systemd unit file:
[Unit] Description=OpenShift Node After=docker.service Wants=openvswitch.service After=ovsdb-server.service After=ovs-vswitchd.service Wants=docker.service Documentation=https://github.com/openshift/origin Requires=dnsmasq.service After=dnsmasq.service [Service] Type=notify EnvironmentFile=/etc/sysconfig/atomic-openshift-node Environment=GOTRACEBACK=crash ExecStartPre=/usr/bin/cp /etc/origin/node/node-dnsmasq.conf /etc/dnsmasq.d/ ExecStartPre=/usr/bin/dbus-send --system --dest=uk.org.thekelleys.dnsmasq /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetDomainServers array:string:/in-addr.arpa/127.0.0.1,/cluster.local/127.0.0.1 ExecStopPost=/usr/bin/rm /etc/dnsmasq.d/node-dnsmasq.conf ExecStopPost=/usr/bin/dbus-send --system --dest=uk.org.thekelleys.dnsmasq /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetDomainServers array:string: ExecStart=/usr/bin/openshift start node --config=${CONFIG_FILE} $OPTIONS LimitNOFILE=65536 LimitCORE=infinity WorkingDirectory=/var/lib/origin/ SyslogIdentifier=atomic-openshift-node Restart=always RestartSec=5s TimeoutStartSec=300 OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target
Reload systemd and restart node service.
# systemctl daemon-reload # systemctl restart atomic-openshift-node dnsmaq
Restart dnsmasq:
# systemctl restart dnsmasq
On each master host, upgrade the atomic-openshift packages or related images.
For masters using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages and the openvswitch package:
# yum upgrade atomic-openshift\* openvswitch
For masters using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the
IMAGE_VERSION
parameter to the version you are upgrading to in the following files:- /etc/sysconfig/atomic-openshift-master-controllers
- /etc/sysconfig/atomic-openshift-master-api
- /etc/sysconfig/atomic-openshift-node
- /etc/sysconfig/atomic-openshift-openvswitch
For example:
IMAGE_VERSION=<tag>
Replace
<tag>
withv3.9.102
for the latest version.
Modify host labels and parameters.
In previous versions of OpenShift Container Platform, master hosts were marked unschedulable by default by the installer, meaning that no pods could be placed on the hosts. Starting with OpenShift Container Platform 3.9, however, the web console is removed from the master services and deployed as a pod on the masters, requiring that masters be marked schedulable. This change also requires hosts to have node role labels, either
master
orcompute
.
You can see the OpenShift Container Platform 3.9 Release Notes for more information on how these notable technical changes relate in the context of supported automated upgrades:
If you are upgrading from OpenShift Container Platform 3.7 to 3.9:
Add the following label to each master host:
$ oc label node <hostname> node-role.kubernetes.io/master=true
This assigns them the
master
node role.Add the following label to each non-master, non-infrastructure (e.g.,
region=infra
labeled) host:$ oc label node <hostname> node-role.kubernetes.io/compute=true
This assigns them the
compute
node role.The default node selector must also now be set to ensure that only the web console pods will be scheduled to the masters. If you already have this set to a custom value via
projectConfig.defaultNodeSelector
in the /etc/origin/master/master-config.yaml file, you can skip this step. If you have not previously set it, set it to the following:projectConfig: defaultNodeSelector: node-role.kubernetes.io/compute=true
Restart the master service(s) on each master and review logs to ensure they restart successfully.
# systemctl restart atomic-openshift-master-controllers # systemctl restart atomic-openshift-master-api # journalctl -r -u atomic-openshift-master-controllers # journalctl -r -u atomic-openshift-master-api
Because masters also have node components running on them in order to be configured as part of the OpenShift SDN, restart the atomic-openshift-node and openvswitch services:
# systemctl restart openvswitch # systemctl restart atomic-openshift-node # journalctl -r -u openvswitch # journalctl -r -u atomic-openshift-node
If you are upgrading from OpenShift Container Platform 3.7 to 3.9, run the following for each master host to mark them schedulable:
$ oc adm manage-node <hostname> --schedulable=true
- If you are performing a cluster upgrade that requires updating Docker to version 1.13, you must also perform the following steps if you are not already on Docker 1.13:
Upgrade the docker package.
For RHEL 7 systems:
# yum update docker
Then, restart the docker service and review the logs to ensure it restarts successfully:
# systemctl restart docker # journalctl -r -u docker
For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:
# atomic host upgrade
After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:
# systemctl reboot # journalctl -r -u docker
Remove the following file, which is no longer required:
# rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf
Run the following command on each master to add the atomic-openshift packages back to the list of yum excludes on the host:
# atomic-openshift-excluder exclude
During the cluster upgrade, it can sometimes be useful to take a master out of rotation since some DNS client libraries will not properly to the other masters for cluster DNS. In addition to stopping the master and controller services, you can remove the EndPoint from the Kubernetes service’s subsets.addresses
.
$ oc edit ep/kubernetes -n default
When the master is restarted, the Kubernetes service will be automatically updated.
3.4. Updating Policy Definitions
During a cluster upgrade, and on every restart of any master, the default cluster roles are automatically reconciled to restore any missing permissions.
If you customized default cluster roles and want to ensure a role reconciliation does not modify them, protect each role from reconciliation:
$ oc annotate clusterrole.rbac <role_name> --overwrite rbac.authorization.kubernetes.io/autoupdate=false
WarningYou must manually update the roles that contain this setting to include any new or required permissions after upgrading.
Generate a default bootstrap policy template file:
$ oc adm create-bootstrap-policy-file --filename=policy.json
NoteThe contents of the file vary based on the OpenShift Container Platform version, but the file contains only the default policies.
- Update the policy.json file to include any cluster role customizations.
Use the policy file to automatically reconcile roles and role bindings that are not reconcile protected:
$ oc process --local -f policy.json | oc auth reconcile -f policy.json
Reconcile security context constraints:
# oc adm policy reconcile-sccs \ --additive-only=true \ --confirm
3.5. Upgrading Nodes
After upgrading your masters, you can upgrade your nodes. When restarting the atomic-openshift-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.
You can alternatively use the blue-green deployment method at this point to create a parallel environment for new nodes instead of upgrading them in place.
One at at time for each node that is not also a master, you must disable scheduling and evacuate its pods to other nodes, then upgrade packages and restart services.
Run the following command on each node to remove the atomic-openshift packages from the list of yum excludes on the host:
# atomic-openshift-excluder unexclude
As a user with cluster-admin privileges, disable scheduling for the node:
# oc adm manage-node <node> --schedulable=false
Evacuate pods on the node to other nodes:
ImportantThe
--force
option deletes any pods that are not backed by a replication controller.# oc adm drain <node> --force --delete-local-data --ignore-daemonsets
Upgrade the node component packages or related images.
For nodes using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages and the openvswitch package:
# yum upgrade atomic-openshift\* openvswitch
For nodes using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the
IMAGE_VERSION
parameter in the /etc/sysconfig/atomic-openshift-node and /etc/sysconfig/openvswitch files to the version you are upgrading to. For example:IMAGE_VERSION=<tag>
Replace
<tag>
withv3.9.102
for the latest version.
Restart the atomic-openshift-node and openvswitch services and review the logs to ensure they restart successfully:
# systemctl restart openvswitch # systemctl restart atomic-openshift-node # journalctl -r -u atomic-openshift-node # journalctl -r -u openvswitch
If you are performing a cluster upgrade that requires updating Docker to version 1.13, you must also perform the following steps if you are not already on Docker 1.13:
Upgrade the docker package.
For RHEL 7 systems:
# yum update docker
Then, restart the docker service and review the logs to ensure it restarts successfully:
# systemctl restart docker # journalctl -r -u docker
After Docker is restarted, restart the atomic-openshift-node service again and review the logs to ensure it restarts successfully:
# systemctl restart atomic-openshift-node # journalctl -r -u atomic-openshift-node
For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:
# atomic host upgrade
After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:
# systemctl reboot # journalctl -r -u docker
Remove the following file, which is no longer required:
# rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf
Re-enable scheduling for the node:
# oc adm manage-node <node> --schedulable
Run the following command on each node to add the atomic-openshift packages back to the list of yum excludes on the host:
# atomic-openshift-excluder exclude
- Repeat the previous steps on the next node, and continue repeating these steps until all nodes have been upgraded.
After all nodes have been upgraded, as a user with cluster-admin privileges, verify that all nodes are showing as Ready:
# oc get nodes NAME STATUS ROLES AGE VERSION master.example.com Ready master 7h v1.9.1+a0ce1bc657 node1.example.com Ready compute 7h v1.9.1+a0ce1bc657 node2.example.com Ready compute 7h v1.9.1+a0ce1bc657
3.6. Upgrading the Router
If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly available routing service.
Edit your router’s deployment configuration. For example, if it has the default router name:
# oc edit dc/router
Apply the following changes:
...
spec:
template:
spec:
containers:
- env:
...
image: registry.access.redhat.com/openshift3/ose-haproxy-router:<tag> 1
imagePullPolicy: IfNotPresent
...
- 1
- Adjust
<tag>
to match the version you are upgrading to (usev3.9.102
for the latest version).
You should see one router pod updated and then the next.
3.7. Upgrading the Registry
The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim
or a host mount point, you may restart the registry without losing the contents of your registry. Storage for the Registry details how to configure persistent storage for the registry.
Edit your registry’s deployment configuration:
# oc edit dc/docker-registry
Apply the following changes:
...
spec:
template:
spec:
containers:
- env:
...
image: registry.access.redhat.com/openshift3/ose-docker-registry:<tag> 1
imagePullPolicy: IfNotPresent
...
- 1
- Adjust
<tag>
to match the version you are upgrading to (usev3.9.102
for the latest version).
If the registry console is deployed, edit its deployment configuration:
# oc edit dc/registry-console
Apply the following changes:
... spec: template: spec: containers: - env: ... image: registry.access.redhat.com/openshift3/registry-console:v3.9 imagePullPolicy: IfNotPresent ...
Images that are being pushed or pulled from the internal registry at the time of upgrade will fail and should be restarted automatically. This will not disrupt pods that are already running.
3.8. Updating the Default Image Streams and Templates
By default, the advanced installation method automatically creates default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under the /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/ directory.
Because RHEL Atomic Host 7 cannot use yum to update packages, the following steps must take place on a RHEL 7 system.
Update the packages that provide the example JSON files. On a subscribed Red Hat Enterprise Linux 7 system where you can run the CLI as a user with cluster-admin permissions, install or update to the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible- packages:
# yum update atomic-openshift-utils
To persist /usr/share/openshift/examples/ on the first master:
$ scp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/* user@master1:/usr/share/openshift/examples/
To persist /usr/share/openshift/examples/ on all masters:
$ mkdir /usr/share/openshift/examples $ scp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/* user@masterx:/usr/share/openshift/examples
The openshift-ansible-roles package provides the latest example JSON files.
After a manual upgrade, get the latest templates from openshift-ansible-roles:
# rpm -ql openshift-ansible-roles | grep examples | grep v3.9
In this example, /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/image-streams/image-streams-rhel7.json is the latest file that you want in the latest openshift-ansible-roles package.
/usr/share/openshift/examples/image-streams/image-streams-rhel7.json is not owned by a package, but is updated by Ansible. If you are upgrading outside of Ansible. you need to get the latest .json files on the system where you are running
oc
, which can run anywhere that has access to the master.Install atomic-openshift-utils and its dependencies to install the new content into /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/.:
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/image-streams/image-streams-rhel7.json $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/image-streams/dotnet_imagestreams.json $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/image-streams/image-streams-rhel7.json $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/image-streams/dotnet_imagestreams.json
Update the templates:
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/ $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/db-templates/ $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/infrastructure-templates/ $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/xpaas-templates/ $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/xpaas-streams/ $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/ $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/db-templates/ $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/infrastructure-templates/ $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/xpaas-templates/ $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/xpaas-streams/
Errors are generated for items that already exist. This is expected behavior:
# oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/ Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/cakephp-mysql.json": templates "cakephp-mysql-example" already exists Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/cakephp.json": templates "cakephp-example" already exists Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/dancer-mysql.json": templates "dancer-mysql-example" already exists Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/dancer.json": templates "dancer-example" already exists Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.9/quickstart-templates/django-postgresql.json": templates "django-psql-example" already exists
Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.
3.9. Importing the Latest Images
After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run:
# oc import-image -n openshift <imagestream>
For example, get the list of all image streams in the default openshift project:
# oc get is -n openshift NAME DOCKER REPO TAGS UPDATED mongodb registry.access.redhat.com/openshift3/mongodb-24-rhel7 2.4,latest,v3.1.1.6 16 hours ago mysql registry.access.redhat.com/openshift3/mysql-55-rhel7 5.5,latest,v3.1.1.6 16 hours ago nodejs registry.access.redhat.com/openshift3/nodejs-010-rhel7 0.10,latest,v3.1.1.6 16 hours ago ...
Update each image stream one at a time:
# oc import-image -n openshift nodejs The import completed successfully. Name: nodejs Created: 10 seconds ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2016-07-05T19:20:30Z Docker Pull Spec: 172.30.204.22:5000/openshift/nodejs Tag Spec Created PullSpec Image latest 4 9 seconds ago registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest 570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05 4 registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest 9 seconds ago <same> 570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05 0.10 registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest 9 seconds ago <same> a1ef33be788a28ec2bdd48a9a5d174ebcfbe11c8e986d2996b77f5bccaaa4774
In order to update your S2I-based applications, you must manually trigger a new build of those applications after importing the new images using oc start-build <app-name>
.
3.10. Upgrading the Web Console
Manual upgrade steps for the web console are not available.
Starting with OpenShift Container Platform 3.9, the web console is deployed as a separate component rather than as part of the master. The console will be installed by default as part of an automated upgrade unless you set the openshift_web_console_install
variable to false
.
Alternatively, you can install the web console separately if needed, for example as part of a manual upgrade:
- See the following section in the Advanced Installation topic and update your inventory file accordingly.
Run the following playbook:
# ansible-playbook -i </path/to/inventory/file> \ /usr/share/ansible/openshift-ansible/playbooks/openshift-web-console/config.yml
The way the web console loads extensions has changed in OpenShift Container Platform 3.9. If you are upgrading from an earlier version, you must host your extension scripts and stylesheets at a URL instead of on the master file system. See the Loading Extension Scripts and Stylesheets topic for details.
3.11. Upgrading the Service Catalog
Manual upgrade steps for the service catalog and service brokers are not available.
To upgrade the service catalog:
See the following three sections in the Advanced Installation topic and update your inventory file accordingly:
Run the following playbook:
# ansible-playbook -i </path/to/inventory/file> \ /usr/share/ansible/openshift-ansible/playbooks/openshift-service-catalog/config.yml
3.12. Upgrading the EFK Logging Stack
Manual upgrade steps for logging deployments are no longer available starting in OpenShift Container Platform 3.5.
To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.
If you have any Elasticsearch SearchGuard indices in the following naming format, you need to delete and reseed the indices because the naming format has changed. Elastic search might not work as expected unless you update the indices:
.searchguard.logging-es-*
Run the following command to delete the SearchGuard indices:
# oc exec -c elasticsearch <pod> -- es_util --query=.searchguard* -XDELETE
Run the following command to reseed the SearchGuard indices:
# for pod in $(oc get pods -l component=es -o jsonpath={.items[*].metadata.name}); do oc exec -c elasticsearch $pod -- es_seed_acl; done
If you have not already done so, see Specifying Logging Ansible Variables in the Aggregating Container Logs topic and update your Ansible inventory file to at least set the following required variable within the
[OSEv3:vars]
section:[OSEv3:vars] openshift_logging_install_logging=true 1 openshift_logging_image_version=<tag> 2
-
Add any other
openshift_logging_*
variables that you want to specify to override the defaults, as described in Specifying Logging Ansible Variables. - When you have finished updating your inventory file, follow the instructions in Deploying the EFK Stack to run the openshift-logging/config.yml playbook and complete the logging deployment upgrade.
If your Fluentd DeploymentConfig and DaemonSet for the EFK components are already set with:
image: <image_name>:<vX.Y> imagePullPolicy: IfNotPresent
The latest version <image_name> might not be pulled if there is already one with the same <image_name:vX.Y> stored locally on the node where the pod is being re-deployed. If so, manually change the DeploymentConfig and DaemonSet to imagePullPolicy: Always
to make sure it is re-pulled.
3.13. Upgrading Cluster Metrics
Manual upgrade steps for metrics deployments are no longer available starting in OpenShift Container Platform 3.5.
To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-metrics/config.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.
If you have not already done so, see Specifying Metrics Ansible Variables in the Enabling Cluster Metrics topic and update your Ansible inventory file to at least set the following required variables within the
[OSEv3:vars]
section:[OSEv3:vars] openshift_metrics_install_metrics=true 1 openshift_metrics_image_version=<tag> 2 openshift_metrics_hawkular_hostname=<fqdn> 3 openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) 4
-
Add any other
openshift_metrics_*
variables that you want to specify to override the defaults, as described in Specifying Metrics Ansible Variables. - When you have finished updating your inventory file, follow the instructions in Deploying the Metrics Deployment to run the openshift-metrics/config.yml playbook and complete the metrics deployment upgrade.
3.14. Additional Manual Steps Per Release
Some OpenShift Container Platform releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster. This section will be updated over time as new asynchronous updates are released for OpenShift Container Platform 3.9.
See the OpenShift Container Platform 3.9 Release Notes to review the latest release notes.
3.15. Verifying the Upgrade
To verify the upgrade:
Check that all nodes are marked as Ready:
# oc get nodes NAME STATUS ROLES AGE VERSION master.example.com Ready master 7h v1.9.1+a0ce1bc657 node1.example.com Ready compute 7h v1.9.1+a0ce1bc657 node2.example.com Ready compute 7h v1.9.1+a0ce1bc657
Verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace
<tag>
withv3.9.102
for the latest version.# oc get -n default dc/docker-registry -o json | grep \"image\" "image": "openshift3/ose-docker-registry:<tag>", # oc get -n default dc/router -o json | grep \"image\" "image": "openshift3/ose-haproxy-router:<tag>",
Use the diagnostics tool on the master to look for common issues:
# oc adm diagnostics ... [Note] Summary of diagnostics execution: [Note] Completed with no errors or warnings seen.