红帽全球峰会此内容没有您所选择的语言版本。
Chapter 6. Non-Director Environments: Upgrading Individual OpenStack Services (Live Compute) in a High Availability Environment
This chapter describes the steps you should follow to upgrade your cloud deployment by updating one service at a time with live compute in a High Availability (HA) environment. This scenario upgrades from Red Hat OpenStack Platform 9 to Red Hat OpenStack Platform 10 in environments that do not use the director.
A live Compute upgrade minimizes interruptions to your Compute service, with only a few minutes for the smaller services, and a longer migration interval for the workloads moving to newly-upgraded Compute hosts. Existing workloads can run indefinitely, and you do not need to wait for a database migration.
Due to certain package dependencies, upgrading the packages for one OpenStack service might cause Python libraries to upgrade before other OpenStack services upgrade. This might cause certain services to fail prematurely. In this situation, continue upgrading the remaining services. All services should be operational upon completion of this scenario.
This method may require additional hardware resources to bring up the Compute nodes.
The procedures in this chapter follow the architectural naming convention followed by all Red Hat OpenStack Platform documentation. If you are unfamiliar with this convention, refer to Architecture Guide available at Red Hat OpenStack Platform Documentation Suite before proceeding.
6.1. Pre-Upgrade Tasks
On each node, change to the Red Hat OpenStack Platform 10 repository using the subscription-manager command:
# subscription-manager repos --disable=rhel-7-server-openstack-9-rpms
# subscription-manager repos --enable=rhel-7-server-openstack-10-rpms
Upgrade the openstack-selinux package:
# yum upgrade openstack-selinuxThis is necessary to ensure that the upgraded services will run correctly on a system with SELinux enabled.
6.2. Upgrading MariaDB
Perform the follow steps on each host running MariaDB. Complete the steps on one host before starting the process on another host.
Stop the service from running on the local node:
# pcs resource ban galera-master $(crm_node -n)Wait until
pcs statusshows that the service is no longer running on the local node. This may take a few minutes. The local node transitions to slave mode:Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-1 overcloud-controller-2 ] Slaves: [ overcloud-controller-0 ]The node eventually transitions to stopped:
Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-controller-0 ]Upgrade the relevant packages:
# yum upgrade '*mariadb*' '*galera*'Allow Pacemaker to schedule the
galeraresource on the local node:# pcs resource clear galera-masterWait until
pcs statusshows that the galera resource is running on the local node as a master. Thepcs statuscommand should provide output similar to the following:Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Perform this procedure on each node individually until the MariaDB cluster completes a full upgrade.
6.3. Upgrading MongoDB
This procedure upgrades MongoDB, which acts as the backend database for the OpenStack Telemetry service.
Remove the
mongodresource from Pacemaker’s control:# pcs resource unmanage mongod-cloneStop the service on all Controller nodes. On each Controller node, run the following:
# systemctl stop mongodUpgrade the relevant packages:
# yum upgrade 'mongodb*' 'python-pymongo*'Reload
systemdto account for updated unit files:# systemctl daemon-reloadRestart the
mongodservice on your controllers by running, on each controller:# systemctl start mongodClean up the resource:
# pcs resource cleanup mongod-cloneReturn the resource to Pacemaker control:
# pcs resource manage mongod-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.4. Upgrading WSGI Services
This procedure upgrades the packages for the WSGI services on all Controller nodes simultaneously. This includes OpenStack Identity (keystone) and OpenStack Dashboard (horizon).
Remove the service from Pacemaker’s control:
# pcs resource unmanage httpd-cloneStop the
httpdservice by running the following on each Controller node:# systemctl stop httpdUpgrade the relevant packages:
# yum -d1 -y upgrade \*keystone\* # yum -y upgrade \*horizon\* \*openstack-dashboard\* httpd # yum -d1 -y upgrade \*horizon\* \*python-django\*Reload
systemdto account for updated unit files on each Controller node:# systemctl daemon-reloadEarlier versions of the installer may not have configured your system to automatically purge expired Keystone token, it is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
Flush expired tokens from the database to alleviate the problem. Run the
keystone-managecommand before running the Identity database upgrade.# keystone-manage token_flushThis flushes expired tokens from the database. You can arrange to run this command periodically (e.g., daily) using
cron.Update the Identity service database schema:
# su -s /bin/sh -c "keystone-manage db_sync" keystoneRestart the service by running the following on each Controller node:
# systemctl start httpdClean up the Identity service using Pacemaker:
# pcs resource cleanup httpd-cloneReturn the resource to Pacemaker control:
# pcs resource manage httpd-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.5. Upgrading Image service (glance)
This procedure upgrades the packages for the Image service on all Controller nodes simultaneously.
Stop the Image service resources in Pacemaker:
# pcs resource disable openstack-glance-registry-clone # pcs resource disable openstack-glance-api-clone-
Wait until the output of
pcs statusshows that both services have stopped running. Upgrade the relevant packages:
# yum upgrade '*glance*'Reload
systemdto account for updated unit files:# systemctl daemon-reloadUpdate the Image service database schema:
# su -s /bin/sh -c "glance-manage db_sync" glanceClean up the Image service using Pacemaker:
# pcs resource cleanup openstack-glance-api-clone # pcs resource cleanup openstack-glance-registry-cloneRestart Image service resources in Pacemaker:
# pcs resource enable openstack-glance-api-clone # pcs resource enable openstack-glance-registry-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.6. Upgrading Block Storage service (cinder)
This procedure upgrades the packages for the Block Storage service on all Controller nodes simultaneously.
Stop all Block Storage service resources in Pacemaker:
# pcs resource disable openstack-cinder-api-clone # pcs resource disable openstack-cinder-scheduler-clone # pcs resource disable openstack-cinder-volume-
Wait until the output of
pcs statusshows that the above services have stopped running. Upgrade the relevant packages:
# yum upgrade '*cinder*'Reload
systemdto account for updated unit files:# systemctl daemon-reloadUpdate the Block Storage service database schema:
# su -s /bin/sh -c "cinder-manage db sync" cinderClean up the Block Storage service using Pacemaker:
# pcs resource cleanup openstack-cinder-volume # pcs resource cleanup openstack-cinder-scheduler-clone # pcs resource cleanup openstack-cinder-api-cloneRestart all Block Storage service resources in Pacemaker:
# pcs resource enable openstack-cinder-volume # pcs resource enable openstack-cinder-scheduler-clone # pcs resource enable openstack-cinder-api-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.7. Upgrading Orchestration (heat)
This procedure upgrades the packages for the Orchestration service on all Controller nodes simultaneously.
Stop Orchestration resources in Pacemaker:
# pcs resource disable openstack-heat-api-clone # pcs resource disable openstack-heat-api-cfn-clone # pcs resource disable openstack-heat-api-cloudwatch-clone # pcs resource disable openstack-heat-engine-clone-
Wait until the output of
pcs statusshows that the above services have stopped running. Upgrade the relevant packages:
# yum upgrade '*heat*'Reload
systemdto account for updated unit files:# systemctl daemon-reloadUpdate the Orchestration database schema:
# su -s /bin/sh -c "heat-manage db_sync" heatClean up the Orchestration service using Pacemaker:
# pcs resource cleanup openstack-heat-clone # pcs resource cleanup openstack-heat-api-cloudwatch-clone # pcs resource cleanup openstack-heat-api-cfn-clone # pcs resource cleanup openstack-heat-api-cloneRestart Orchestration resources in Pacemaker:
# pcs resource enable openstack-heat-clone # pcs resource enable openstack-heat-api-cloudwatch-clone # pcs resource enable openstack-heat-api-cfn-clone # pcs resource enable openstack-heat-api-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.8. Upgrading Telemetry (ceilometer)
This procedure upgrades the packages for the Telemetry service on all Controller nodes simultaneously.
This component has some additional upgrade procedures detailed in Chapter 7, Additional Procedures for Non-Director Environments. These additional procedures are optional for manual environments but help align with the current OpenStack Platform recommendations.
Stop all Telemetry resources in Pacemaker:
# pcs resource disable openstack-ceilometer-api-clone # pcs resource disable openstack-ceilometer-collector-clone # pcs resource disable openstack-ceilometer-notification-clone # pcs resource disable openstack-ceilometer-central-clone # pcs resource disable openstack-aodh-evaluator-clone # pcs resource disable openstack-aodh-listener-clone # pcs resource disable openstack-aodh-notifier-clone # pcs resource disable openstack-gnocchi-metricd-clone # pcs resource disable openstack-gnocchi-statsd-clone # pcs resource disable delay-clone-
Wait until the output of
pcs statusshows that the above services have stopped running. Upgrade the relevant packages:
# yum upgrade '*ceilometer*' '*aodh*' '*gnocchi*'Reload
systemdto account for updated unit files:# systemctl daemon-reloadUse the following command to update Telemetry database schema.
# ceilometer-dbsync # aodh-dbsync # gnocchi-upgradeClean up the Telemetry service using Pacemaker:
# pcs resource cleanup delay-clone # pcs resource cleanup openstack-ceilometer-api-clone # pcs resource cleanup openstack-ceilometer-collector-clone # pcs resource cleanup openstack-ceilometer-notification-clone # pcs resource cleanup openstack-ceilometer-central-clone # pcs resource cleanup openstack-aodh-evaluator-clone # pcs resource cleanup openstack-aodh-listener-clone # pcs resource cleanup openstack-aodh-notifier-clone # pcs resource cleanup openstack-gnocchi-metricd-clone # pcs resource cleanup openstack-gnocchi-statsd-cloneRestart all Telemetry resources in Pacemaker:
# pcs resource enable delay-clone # pcs resource enable openstack-ceilometer-api-clone # pcs resource enable openstack-ceilometer-collector-clone # pcs resource enable openstack-ceilometer-notification-clone # pcs resource enable openstack-ceilometer-central-clone # pcs resource enable openstack-aodh-evaluator-clone # pcs resource enable openstack-aodh-listener-clone # pcs resource enable openstack-aodh-notifier-clone # pcs resource enable openstack-gnocchi-metricd-clone # pcs resource enable openstack-gnocchi-statsd-clone-
Wait until the output of
pcs statusshows that the above resources are running.
Previous versions of the Telemetry service used an value for the rpc_backend parameter that is now deprecated. Check the rpc_backend parameter in the /etc/ceilometer/ceilometer.conf file is set to the following:
rpc_backend=rabbitThis procedure upgrades the packages for the Compute service on all Controller nodes simultaneously.
Stop all Compute resources in Pacemaker:
# pcs resource disable openstack-nova-novncproxy-clone # pcs resource disable openstack-nova-consoleauth-clone # pcs resource disable openstack-nova-conductor-clone # pcs resource disable openstack-nova-api-clone # pcs resource disable openstack-nova-scheduler-clone-
Wait until the output of
pcs statusshows that the above services have stopped running. Upgrade the relevant packages:
# yum upgrade '*nova*'Reload
systemdto account for updated unit files:# systemctl daemon-reloadUpdate the Compute database schema:
# su -s /bin/sh -c "nova-manage api_db sync" nova # su -s /bin/sh -c "nova-manage db sync" novaIf you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Mitaka and Newton environments.
Before starting Compute services on Controller or Compute nodes, set the
computeoption in the[upgrade_levels]section ofnova.confto the previous Red Hat OpenStack Platform version (mitaka):# crudini --set /etc/nova/nova.conf upgrade_levels compute mitakaThis ensures the Controller node can still communicate to the Compute nodes, which are still using the previous version.
You will need to first unmanage the Compute resources by running
pcs resource unmanageon one Controller node:# pcs resource unmanage openstack-nova-novncproxy-clone # pcs resource unmanage openstack-nova-consoleauth-clone # pcs resource unmanage openstack-nova-conductor-clone # pcs resource unmanage openstack-nova-api-clone # pcs resource unmanage openstack-nova-scheduler-cloneRestart all the services on all controllers:
# openstack-service restart novaYou should return control to the Pacemaker after upgrading all of your compute hosts to Red Hat OpenStack Platform 10.
# pcs resource manage openstack-nova-scheduler-clone # pcs resource manage openstack-nova-api-clone # pcs resource manage openstack-nova-conductor-clone # pcs resource manage openstack-nova-consoleauth-clone # pcs resource manage openstack-nova-novncproxy-cloneClean up all Compute resources in Pacemaker:
# pcs resource cleanup openstack-nova-scheduler-clone # pcs resource cleanup openstack-nova-api-clone # pcs resource cleanup openstack-nova-conductor-clone # pcs resource cleanup openstack-nova-consoleauth-clone # pcs resource cleanup openstack-nova-novncproxy-cloneRestart all Compute resources in Pacemaker:
# pcs resource enable openstack-nova-scheduler-clone # pcs resource enable openstack-nova-api-clone # pcs resource enable openstack-nova-conductor-clone # pcs resource enable openstack-nova-consoleauth-clone # pcs resource enable openstack-nova-novncproxy-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.10. Upgrading Clustering service (sahara)
This procedure upgrades the packages for the Clustering service on all Controller nodes simultaneously.
Stop all Clustering service resources in Pacemaker:
# pcs resource disable openstack-sahara-api-clone # pcs resource disable openstack-sahara-engine-clone-
Wait until the output of
pcs statusshows that the above services have stopped running. Upgrade the relevant packages:
# yum upgrade '*sahara*'Reload
systemdto account for updated unit files:# systemctl daemon-reloadUpdate the Clustering service database schema:
# su -s /bin/sh -c "sahara-db-manage upgrade heads" saharaClean up the Clustering service using Pacemaker:
# pcs resource cleanup openstack-sahara-api-clone # pcs resource cleanup openstack-sahara-engine-cloneRestart all Block Storage service resources in Pacemaker:
# pcs resource enable openstack-sahara-api-clone # pcs resource enable openstack-sahara-engine-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.11. Upgrading OpenStack Networking (neutron)
This procedure upgrades the packages for the Networking service on all Controller nodes simultaneously.
Prevent Pacemaker from triggering the OpenStack Networking cleanup scripts:
# pcs resource unmanage neutron-ovs-cleanup-clone # pcs resource unmanage neutron-netns-cleanup-cloneStop OpenStack Networking resources in Pacemaker:
# pcs resource disable neutron-server-clone # pcs resource disable neutron-openvswitch-agent-clone # pcs resource disable neutron-dhcp-agent-clone # pcs resource disable neutron-l3-agent-clone # pcs resource disable neutron-metadata-agent-cloneUpgrade the relevant packages:
# yum upgrade 'openstack-neutron*' 'python-neutron*'Update the OpenStack Networking database schema:
# su -s /bin/sh -c "neutron-db-manage upgrade heads" neutronClean up OpenStack Networking resources in Pacemaker:
# pcs resource cleanup neutron-metadata-agent-clone # pcs resource cleanup neutron-l3-agent-clone # pcs resource cleanup neutron-dhcp-agent-clone # pcs resource cleanup neutron-openvswitch-agent-clone # pcs resource cleanup neutron-server-cloneRestart OpenStack Networking resources in Pacemaker:
# pcs resource enable neutron-metadata-agent-clone # pcs resource enable neutron-l3-agent-clone # pcs resource enable neutron-dhcp-agent-clone # pcs resource enable neutron-openvswitch-agent-clone # pcs resource enable neutron-server-cloneReturn the cleanup agents to Pacemaker control:
# pcs resource manage neutron-ovs-cleanup-clone # pcs resource manage neutron-netns-cleanup-clone-
Wait until the output of
pcs statusshows that the above resources are running.
6.12. Upgrading Compute (nova) Nodes
This procedure upgrades the packages for on a single Compute node. Run this procedure on each Compute node individually.
If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Mitaka and Newton environments.
Before starting Compute services on Controller or Compute nodes, set the compute option in the [upgrade_levels] section of nova.conf to the previous Red Hat OpenStack Platform version (mitaka):
# crudini --set /etc/nova/nova.conf upgrade_levels compute mitaka
Before updating, take a systemd snapshot of the OpenStack Platform services.
# systemctl snapshot openstack-servicesThis ensures the Controller node can still communicate to the Compute nodes, which are still using the previous version.
Stop all OpenStack services on the host:
# systemctl stop 'openstack*' '*nova*'Upgrade all packages:
# yum upgradeStart all OpenStack services on the host:
# openstack-service startAfter you have upgraded all of your hosts, remove the API limits configured in the previous step. On all of your hosts:
# crudini --del /etc/nova/nova.conf upgrade_levels computeRestart all OpenStack services on the host:
# systemctl isolate openstack-services.snapshot
6.13. Post-Upgrade Tasks
After completing all of your individual service upgrades, you should perform a complete package upgrade on all nodes:
# yum upgradeThis will ensure that all packages are up-to-date. You may want to schedule a restart of your OpenStack hosts at a future date in order to ensure that all running processes are using updated versions of the underlying binaries.
Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat OpenStack Platform 10 version of the service.
New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service, see Configuration Reference available from Red Hat OpenStack Platform Documentation Suite.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}