Upgrading OpenStack

Procedure 3.1. Upgrading OpenStack components on a host

1. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
2. Check that the openstack-utils package is installed:

   # yum install openstack-utils

   Copy to Clipboard Toggle word wrap
3. Take down all OpenStack services on all the nodes. This step depends on how your services are distributed among your nodes.
   • In a non-HA environment:
     To stop all the OpenStack services running on a node, login to the node and run:

     # openstack-service stop

     Copy to Clipboard Toggle word wrap
   • In an HA environment:
     1. To stop all the OpenStack services running on a node, login to the node and run:

        # openstack-service stop

        Copy to Clipboard Toggle word wrap
     2. Disable all Pacemaker-managed resources by setting the stop-all-resources property on the cluster. Run the following on a single member of your Pacemaker cluster:

        # pcs property set stop-all-resources=true

        Copy to Clipboard Toggle word wrap
        Then wait until the output of pcs status shows that all resources have stopped.
4. Perform a complete upgrade of all packages, and then flush expired tokens in the Identity service (might decrease the time required to synchronize the database):

   # yum upgrade

   Copy to Clipboard Toggle word wrap
5. Perform the necessary configuration updates on each of your services.
   1. Identity service
      In the RHEL OpenStack Platform 7 (Kilo) release, the location of the token persistence backends has changed. You need to update the driver option in the [token] section of the keystone.conf. To do this, replace any instance of keystone.token.backends with keystone.token.persistence.backends.

      # sed -i 's/keystone.token.backends/keystone.token.persistence.backends/g' \
      /etc/keystone/keystone.conf

      Copy to Clipboard Toggle word wrap
      Package updates may include new systemd unit files, so confirm that systemd is aware of any updated files.

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   2. OpenStack Networking service
      Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
      To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace:

      dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq

      Copy to Clipboard Toggle word wrap
      with:

      dnsmasq: CommandFilter, dnsmasq, root

      Copy to Clipboard Toggle word wrap
6. Upgrade the database schema for each service that uses the database. To do so, login to the node hosting the service and run:

   # openstack-db --service SERVICENAME --update

   Copy to Clipboard Toggle word wrap
   Use the service's project name as the SERVICENAME. For example, to upgrade the database schema of the Identity service:

   # openstack-db --service keystone --update

   Copy to Clipboard Toggle word wrap
   Expand

   Table 3.1.  Project name of each OpenStack service that uses the database

   Service	Project name
   Identity	keystone
   Block Storage	cinder
   Image Service	glance
   Compute	nova
   Networking	neutron
   Orchestration	heat

   Show more
   Certain services require additional database maintenance as part of the Juno to Kilo upgrade that is not covered by the openstack-db command:
   1. Identity service
      Earlier versions of the installer may not have configured your system to automatically purge expired Keystone tokens. It is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      You can alleviate this problem by running the following command before beginning the Keystone database upgrade process:

      # keystone-manage token_flush

      Copy to Clipboard Toggle word wrap
      This will flush expired tokens from the database. You should arrange to run this command periodically (for example, daily) using cron.
   2. Compute
      After fully upgrading to Kilo (that is, all nodes running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user:

      # runuser -u nova -- nova-manage db migrate_flavor_data

      Copy to Clipboard Toggle word wrap
7. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
   New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.
8. If the package upgrades you performed require a reboot (for example, if a new kernel was installed as part of the upgrade), reboot the affected hosts now while the OpenStack service is still disabled.
   • In a non-HA environment:
     To restart the OpenStack service, run the following command on each node:

     # openstack-service start

     Copy to Clipboard Toggle word wrap
   • In a HA environment:
     1. Allow Pacemaker to restart your resources by resetting the stop-all-resources property. On a single member of your Pacemaker cluster, run:

        # pcs property set stop-all-resources=false

        Copy to Clipboard Toggle word wrap
        Then wait until the output of pcs status shows that all resources have started (this may take several minutes).
     2. Restart OpenStack services on the compute nodes. On each compute node, run:

        # openstack-service start

        Copy to Clipboard Toggle word wrap

1. Pre-upgrade tasks:
   On all of your hosts:
   1. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
   2. Upgrade the openstack-selinux package, if available:

      # yum upgrade openstack-selinux

      Copy to Clipboard Toggle word wrap
      This is necessary to ensure that the upgraded services will run correctly on a system with SELinux enabled.
2. Upgrade each of your services:
   The following steps provide specific instructions for each service, and the order in which they should be upgraded.
   1. Identity (keystone)
      Earlier versions of the installer may not have configured your system to automatically purge expired Keystone token, it is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      To flush expired tokens from the database and alleviate the problem, the keystone-manage command can be used before running the Identity database upgrade.
      This will flush expired tokens from the database. You can arrange to run this command periodically (e.g., daily) using cron.
      On your Identity host, run:

      # openstack-service stop keystone
      # yum -d1 -y upgrade \*keystone\*
      # keystone-manage token_flush
      # openstack-db --service keystone --update
      # openstack-service start keystone

      Copy to Clipboard Toggle word wrap
   2. Object Storage (swift)
      On your Object Storage hosts, run:

      # openstack-service stop swift
      # yum -d1 -y upgrade \*swift\*
      # openstack-service start swift

      Copy to Clipboard Toggle word wrap
   3. Image Service (glance)
      On your Image Service host, run:

      # openstack-service stop glance
      # yum -d1 -y upgrade \*glance\*
      # openstack-db --service glance --update
      # openstack-service start glance

      Copy to Clipboard Toggle word wrap
   4. Block Storage (cinder)
      On your Block Storage host, run:

      # openstack-service stop cinder
      # yum -d1 -y upgrade \*cinder\*
      # openstack-db --service cinder --update
      # openstack-service start cinder

      Copy to Clipboard Toggle word wrap
   5. Orchestration (heat)
      On your Orchestration host, run:

      # openstack-service stop heat
      # yum -d1 -y upgrade \*heat\*
      # openstack-db --service heat --update
      # openstack-service start heat

      Copy to Clipboard Toggle word wrap
   6. Telemetry (ceilometer)
      1. On all nodes hosting Telemetry component services, run:

         # openstack-service stop ceilometer
         # yum -d1 -y upgrade \*ceilometer\*

         Copy to Clipboard Toggle word wrap
      2. On the controller node, where database is installed, run:

         # ceilometer-dbsync

         Copy to Clipboard Toggle word wrap
         This command allows you to configure MySQL as a back-end for Telemetry service.
         For a list of Telemetry component services, refer to Launch the Telemetry API and Agents.
      3. After completing the package upgrade, restart the Telemetry service by running the following command on all nodes hosting Telemetry component services:

         # openstack-service start ceilometer

         Copy to Clipboard Toggle word wrap
   7. Compute (nova)
      1. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
         Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno:

         # crudini --set /etc/nova/nova.conf upgrade_levels compute juno

         Copy to Clipboard Toggle word wrap
         You need to make this change on your controllers and on your compute hosts.
         You should undo this operation after upgrading all of your compute hosts to OpenStack Kilo.
      2. On your Compute host, run:

         # openstack-service stop nova
         # yum -d1 -y upgrade \*nova\*
         # openstack-db --service nova --update

         Copy to Clipboard Toggle word wrap
      3. After fully upgrading to Kilo (i.e. all nodes are running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user:

         # runuser -u nova -- nova-manage db migrate_flavor_data

         Copy to Clipboard Toggle word wrap
      4. After you have upgraded all of your hosts to Kilo, you will want to remove the API limits configured in the previous step. On all of your hosts:

         # crudini --del /etc/nova/nova.conf upgrade_levels compute

         Copy to Clipboard Toggle word wrap
      5. Restart the Compute service on all the compute hosts and controllers:

         # openstack-service start nova

         Copy to Clipboard Toggle word wrap
   8. OpenStack Networking (neutron)
      1. On your OpenStack Networking host, run:

         # openstack-service stop neutron
         # yum -d1 -y upgrade \*neutron\*
         # openstack-db --service neutron --update

         Copy to Clipboard Toggle word wrap
      2. Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
         To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace:

         dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq

         Copy to Clipboard Toggle word wrap
         with:

         dnsmasq: CommandFilter, dnsmasq, root

         Copy to Clipboard Toggle word wrap
      3. Restart the OpenStack Networking service:

         # openstack-service start neutron

         Copy to Clipboard Toggle word wrap
   9. Dashboard (horizon)
      On your Dashboard host, run:

      # yum -y upgrade \*horizon\* \*openstack-dashboard\*
      # yum -d1 -y upgrade \*horizon\* \*python-django\*
      # systemctl restart httpd

      Copy to Clipboard Toggle word wrap
3. Post-upgrade tasks:
   1. After completing all of your individual service upgrades, you should perform a complete package upgrade on all of your systems:

      # yum upgrade

      Copy to Clipboard Toggle word wrap
      This will ensure that all packages are up-to-date. You may want to schedule a restart of your OpenStack hosts at a future date in order to ensure that all running processes are using updated versions of the underlying binaries.
   2. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
      New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.

1. Pre-upgrade tasks:
   On all of your hosts:
   1. If you are running Puppet as configured by Staypuft, you must disable it:

      # systemctl stop puppet
      # systemctl disable puppet

      Copy to Clipboard Toggle word wrap
      This ensures that the Staypuft-configured puppet will not revert changes made as part of the upgrade process.
   2. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
   3. Manually upgrade all the python packages.

      # yum upgrade python*

      Copy to Clipboard Toggle word wrap
   4. Upgrade the openstack-selinux package, if available:

      # yum upgrade openstack-selinux

      Copy to Clipboard Toggle word wrap
      This is necessary to ensure that the upgraded services will run correctly on a system with SELinux enabled.
2. Service upgrades:
   Upgrade each of your services. The following is a reasonable order in which to perform the upgrades on your controllers:
   Upgrade MariaDB:
   Perform the follow steps on each host running MariaDB. Complete the steps on one host before starting the process on another host.
   1. Stop the service from running on the local node:

      # pcs resource ban galera-master $(crm_node -n)

      Copy to Clipboard Toggle word wrap
   2. Wait until pcs status shows that the service is no longer running on the local node. This may take a few minutes. The local node will first transition to slave mode:

      Master/Slave Set: galera-master [galera]
      Masters: [ pcmk-mac525400aeb753 pcmk-mac525400bab8ae ]
      Slaves: [ pcmk-mac5254004bd62f ]

      Copy to Clipboard Toggle word wrap
      It will eventually transition to stopped:

      Master/Slave Set: galera-master [galera]
      Masters: [ pcmk-mac525400aeb753 pcmk-mac525400bab8ae ]
      Stopped: [ pcmk-mac5254004bd62f ]

      Copy to Clipboard Toggle word wrap
   3. Upgrade the relevant packages.

      # yum upgrade '*mariadb*' '*galera*'

      Copy to Clipboard Toggle word wrap
   4. Allow Pacemaker to schedule the galera resource on the local node:

      # pcs resource clear galera-master

      Copy to Clipboard Toggle word wrap
   5. Wait until pcs status shows that the galera resource is running on the local node as a master. The output from pcs status should include something like:

      Master/Slave Set: galera-master [galera]
      Masters: [ pcmk-mac5254004bd62f pcmk-mac525400aeb753
      pcmk-mac525400bab8ae ]

      Copy to Clipboard Toggle word wrap
   Upgrade MongoDB:
   1. Remove the mongod resource from Pacemaker's control:

      # pcs resource unmanage mongod-clone

      Copy to Clipboard Toggle word wrap
   2. Stop the service on all of your controllers. On each controller, run:

      # systemctl stop mongod

      Copy to Clipboard Toggle word wrap
   3. Upgrade the relevant packages:

      #  yum upgrade 'mongodb*' 'python-pymongo*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. Restart the mongod service on your controllers by running, on each controller:

      # systemctl start mongod

      Copy to Clipboard Toggle word wrap
   6. Clean up the resource to Pacemaker control:

      # pcs resource cleanup mongod-clone

      Copy to Clipboard Toggle word wrap
   7. Return the resource to Pacemaker control:

      # pcs resource manage mongod-clone

      Copy to Clipboard Toggle word wrap
   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Identity service (keystone):
   1. Remove Identity service from Pacemaker's control:

      # pcs resource unmanage keystone-clone

      Copy to Clipboard Toggle word wrap
   2. Stop the Identity service by running the following on each of your controllers:

      # systemctl stop openstack-keystone

      Copy to Clipboard Toggle word wrap
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-keystone*' 'python-keystone*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. In the RHEL OpenStack Platform 7 (Kilo) release, the location of the token persistence backends has changed. You need to update the driver option in the [token] section of the keystone.conf. To do this, replace any instance of keystone.token.backends with keystone.token.persistence.backends.

      # sed -i 's/keystone.token.backends/keystone.token.persistence.backends/g' \
      /etc/keystone/keystone.conf

      Copy to Clipboard Toggle word wrap
      Package updates may include new systemd unit files, so ensure that systemd is aware of any updated files.

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   6. Earlier versions of the installer may not have configured your system to automatically purge expired Keystone token, it is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      To flush expired tokens from the database and alleviate the problem, the keystone-manage command can be used before running the Identity database upgrade.
      This will flush expired tokens from the database. You can arrange to run this command periodically (e.g., daily) using cron.

      # keystone-manage token_flush

      Copy to Clipboard Toggle word wrap
   7. Update the Identity service database schema:

      # openstack-db --service keystone --update

      Copy to Clipboard Toggle word wrap
   8. Restart the service by running the following on each of your controllers:

      # systemctl start openstack-keystone

      Copy to Clipboard Toggle word wrap
   9. Clean up the Identity service using Pacemaker:

      # pcs resource cleanup keystone-clone

      Copy to Clipboard Toggle word wrap
   10. Return the resource to Pacemaker control:

       # pcs resource manage keystone-clone

       Copy to Clipboard Toggle word wrap
   11. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Image service (glance):
   1. Stop the Image service resources in Pacemaker:

      # pcs resource disable glance-registry-clone
      # pcs resource disable glance-api-clone

      Copy to Clipboard Toggle word wrap
   2. Wait until the output of pcs status shows that both services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-glance*' 'python-glance*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. Update the Image service database schema:

      # openstack-db --service glance --update

      Copy to Clipboard Toggle word wrap
   6. Clean up the Image service using Pacemaker:

      # pcs resource cleanup glance-api-clone
      # pcs resource cleanup glance-registry-clone

      Copy to Clipboard Toggle word wrap
   7. Restart Image service resources in Pacemaker:

      # pcs resource enable glance-api-clone
      # pcs resource enable glance-registry-clone

      Copy to Clipboard Toggle word wrap
   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Block Storage service (cinder):
   1. Stop all Block Storage service resources in Pacemaker:

      # pcs resource disable cinder-api-clone
      # pcs resource disable cinder-scheduler-clone
      # pcs resource disable cinder-volume

      Copy to Clipboard Toggle word wrap
   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-cinder*' 'python-cinder*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. Update the Block Storage service database schema:

      # openstack-db --service cinder --update

      Copy to Clipboard Toggle word wrap
   6. Clean up the Block Storage service using Pacemaker:

      # pcs resource cleanup cinder-volume
      # pcs resource cleanup cinder-scheduler-clone
      # pcs resource cleanup cinder-api-clone

      Copy to Clipboard Toggle word wrap
   7. Restart all Block Storage service resources in Pacemaker:

      # pcs resource enable cinder-volume
      # pcs resource enable cinder-scheduler-clone
      # pcs resource enable cinder-api-clone

      Copy to Clipboard Toggle word wrap
   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Orchestration (heat):
   1. Stop Orchestration resources in Pacemaker:

      # pcs resource disable heat-api-clone
      # pcs resource disable heat-api-cfn-clone
      # pcs resource disable heat-api-cloudwatch-clone
      # pcs resource disable heat

      Copy to Clipboard Toggle word wrap
   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-heat*' 'python-heat*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. Update the Orchestration database schema:

      # openstack-db --service heat --update

      Copy to Clipboard Toggle word wrap
   6. Clean up the Orchestration service using Pacemaker:

      # pcs resource cleanup heat
      # pcs resource cleanup heat-api-cloudwatch-clone
      # pcs resource cleanup heat-api-cfn-clone
      # pcs resource cleanup heat-api-clone

      Copy to Clipboard Toggle word wrap
   7. Restart Orchestration resources in Pacemaker:

      # pcs resource enable heat
      # pcs resource enable heat-api-cloudwatch-clone
      # pcs resource enable heat-api-cfn-clone
      # pcs resource enable heat-api-clone

      Copy to Clipboard Toggle word wrap
   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Telemetry (ceilometer):
   1. Stop all Telemetry resources in Pacemaker:

      # pcs resource disable openstack-ceilometer-central
      # pcs resource disable openstack-ceilometer-api-clone
      # pcs resource disable openstack-ceilometer-alarm-evaluator-clone
      # pcs resource disable openstack-ceilometer-collector-clone
      # pcs resource disable openstack-ceilometer-notification-clone
      # pcs resource disable openstack-ceilometer-alarm-notifier-clone
      # pcs resource disable ceilometer-delay-clone

      Copy to Clipboard Toggle word wrap
   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-ceilometer*' 'python-ceilometer*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. If you are using the MySQL backend for Telemetry, update the Telemetry database schema.

      # openstack-db --service ceilometer --update

      Copy to Clipboard Toggle word wrap

      Note

      This step is not necessary of you are using the MongoDB backend.
   6. Clean up the Telemetry service using Pacemaker:

      # pcs resource cleanup ceilometer-delay-clone
      # pcs resource cleanup openstack-ceilometer-alarm-notifier-clone
      # pcs resource cleanup openstack-ceilometer-notification-clone
      # pcs resource cleanup openstack-ceilometer-collector-clone
      # pcs resource cleanup openstack-ceilometer-alarm-evaluator-clone
      # pcs resource cleanup openstack-ceilometer-api-clone
      # pcs resource cleanup openstack-ceilometer-central

      Copy to Clipboard Toggle word wrap
   7. Restart all Telemetry resources in Pacemaker:

      # pcs resource enable ceilometer-delay-clone
      # pcs resource enable openstack-ceilometer-alarm-notifier-clone
      # pcs resource enable openstack-ceilometer-notification-clone
      # pcs resource enable openstack-ceilometer-collector-clone
      # pcs resource enable openstack-ceilometer-alarm-evaluator-clone
      # pcs resource enable openstack-ceilometer-api-clone
      # pcs resource enable openstack-ceilometer-central

      Copy to Clipboard Toggle word wrap
   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Compute (nova):
   1. Stop all Compute resources in Pacemaker:

      # pcs resource disable openstack-nova-novncproxy-clone
      # pcs resource disable openstack-nova-consoleauth-clone
      # pcs resource disable openstack-nova-conductor-clone
      # pcs resource disable openstack-nova-api-clone
      # pcs resource disable openstack-nova-scheduler-clone

      Copy to Clipboard Toggle word wrap
   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-nova*' 'python-nova*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. Update the Compute database schema:

      # openstack-db --service nova --update

      Copy to Clipboard Toggle word wrap
      After fully upgrading to Kilo (i.e. all nodes are running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user:

      # runuser -u nova -- nova-manage db migrate_flavor_data

      Copy to Clipboard Toggle word wrap
   6. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
      Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno:

      # crudini --set /etc/nova/nova.conf upgrade_levels compute juno

      Copy to Clipboard Toggle word wrap
      You will need to first unmanage the Compute resources by running pcs resource unmanage on one of your controllers:

      # pcs resource unmanage openstack-nova-novncproxy-clone
      # pcs resource unmanage openstack-nova-consoleauth-clone
      # pcs resource unmanage openstack-nova-conductor-clone
      # pcs resource unmanage openstack-nova-api-clone
      # pcs resource unmanage openstack-nova-scheduler-clone

      Copy to Clipboard Toggle word wrap
      Restart all the services on all controllers:

      # openstack-service restart nova

      Copy to Clipboard Toggle word wrap
      You should return control to the Pacemaker after upgrading all of your compute hosts to OpenStack Kilo.

      # pcs resource manage openstack-nova-scheduler-clone
      # pcs resource manage openstack-nova-api-clone
      # pcs resource manage openstack-nova-conductor-clone
      # pcs resource manage openstack-nova-consoleauth-clone
      # pcs resource manage openstack-nova-novncproxy-clone

      Copy to Clipboard Toggle word wrap
   7. Clean up all Compute resources in Pacemaker:

      # pcs resource cleanup openstack-nova-scheduler-clone
      # pcs resource cleanup openstack-nova-api-clone
      # pcs resource cleanup openstack-nova-conductor-clone
      # pcs resource cleanup openstack-nova-consoleauth-clone
      # pcs resource cleanup openstack-nova-novncproxy-clone

      Copy to Clipboard Toggle word wrap
   8. Restart all Compute resources in Pacemaker:

      # pcs resource enable openstack-nova-scheduler-clone
      # pcs resource enable openstack-nova-api-clone
      # pcs resource enable openstack-nova-conductor-clone
      # pcs resource enable openstack-nova-consoleauth-clone
      # pcs resource enable openstack-nova-novncproxy-clone

      Copy to Clipboard Toggle word wrap
   9. Wait until the output of pcs status shows that the above resources are running.
   Upgrade OpenStack Networking (neutron):
   1. Prevent Pacemaker from triggering the OpenStack Networking cleanup scripts:

      # pcs resource unmanage neutron-ovs-cleanup-clone
      # pcs resource unmanage neutron-netns-cleanup-clone

      Copy to Clipboard Toggle word wrap
   2. Stop OpenStack Networking resources in Pacemaker:

      # pcs resource disable neutron-server-clone
      # pcs resource disable neutron-openvswitch-agent-clone
      # pcs resource disable neutron-dhcp-agent-clone
      # pcs resource disable neutron-l3-agent-clone
      # pcs resource disable neutron-metadata-agent-clone

      Copy to Clipboard Toggle word wrap
   3. Upgrade the relevant packages

      # yum upgrade 'openstack-neutron*' 'python-neutron*'

      Copy to Clipboard Toggle word wrap
   4. Install packages for the advanced Openstack Networking services enabled in the neutron.conf file, for example, openstack-neutron-vpnaas, openstack-neutron-fwaas and openstack-neutron-lbaas.

      # yum install openstack-neutron-vpnaas
      # yum install openstack-neutron-fwaas
      # yum install openstack-neutron-lbaas

      Copy to Clipboard Toggle word wrap
      Installing these packages will create the corresponding configuration files.
   5. For the VPNaaS, LBaaS service entries in the neutron.conf file, copy the service_provider entries to the corresponding neutron-*aas.conf file located in /etc/neutron and comment these entries from the neutron.conf file.
      For the FWaaS service entry, the service_provider parameters should remain in the neutron.conf file.
   6. On every node that runs the LBaaS agents, install the openstack-neutron-lbaas package.

      # yum install openstack-neutron-lbaas

      Copy to Clipboard Toggle word wrap
   7. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   8. Update the OpenStack Networking database schema:

      # openstack-db --service neutron --update

      Copy to Clipboard Toggle word wrap
   9. Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
      To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace:

      dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq

      Copy to Clipboard Toggle word wrap
      with

      dnsmasq: CommandFilter, dnsmasq, root

      Copy to Clipboard Toggle word wrap
   10. Clean up OpenStack Networking resources in Pacemaker:

       # pcs resource cleanup neutron-metadata-agent-clone
       # pcs resource cleanup neutron-l3-agent-clone
       # pcs resource cleanup neutron-dhcp-agent-clone
       # pcs resource cleanup neutron-openvswitch-agent-clone
       # pcs resource cleanup neutron-server-clone

       Copy to Clipboard Toggle word wrap
   11. Restart OpenStack Networking resources in Pacemaker:

       # pcs resource enable neutron-metadata-agent-clone
       # pcs resource enable neutron-l3-agent-clone
       # pcs resource enable neutron-dhcp-agent-clone
       # pcs resource enable neutron-openvswitch-agent-clone
       # pcs resource enable neutron-server-clone

       Copy to Clipboard Toggle word wrap
   12. Return the cleanup agents to Pacemaker control:

       # pcs resource manage neutron-ovs-cleanup-clone
       # pcs resource manage neutron-netns-cleanup-clone

       Copy to Clipboard Toggle word wrap
   13. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Dashboard (horizon):
   1. Stop the Dashboard resource in Pacemaker:

      # pcs resource disable horizon-clone

      Copy to Clipboard Toggle word wrap
   2. Wait until the output of pcs status shows that the service has stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade httpd 'openstack-dashboard*' 'python-django*'

      Copy to Clipboard Toggle word wrap
   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

      Copy to Clipboard Toggle word wrap
   5. Correct the Dashboard configuration:
      Fix Apache Configuration:
      The openstack-dashboard package installs /etc/httpd/conf.d/openstack-dashboard.conf file, but the Staypuft installer replaces this with the /etc/httpd/conf.d/15-horizon_vhost.conf file. After upgrading horizon, you will have the following configuration files:
      • 15-horizon_vhost.conf
      • openstack-dashboard.conf
      • openstack-dashboard.conf.rpmnew
      Ensure you make the following changes:
      • Remove the openstack-dashboard.conf.rpmnew file:

        # rm openstack-dashboard.conf.rpmnew

        Copy to Clipboard Toggle word wrap
      • Modify the 15-horizon_vhost.conf file by replacing:

        Alias /static "/usr/share/openstack-dashboard/static"

        Copy to Clipboard Toggle word wrap
        with

        Alias /dashboard/static "/usr/share/openstack-dashboard/static"

        Copy to Clipboard Toggle word wrap
      Fix Dashboard Configuration:
      The openstack-dashboard package installs the /etc/openstack-dashboard/local_settings file. After an upgrade, you will find the following configuration files:
      • /etc/openstack-dashboard/local_settings
      • /etc/openstack-dashboard/local_settings.rpmnew
      Ensure you make the following changes:
      • Backup your existing local_settings file:

        # cp local_settings local_settings.old

        Copy to Clipboard Toggle word wrap
      • Rename the local_settings.rpmnew file to local_settings file:

        # mv local_settings.rpmnew local_settings

        Copy to Clipboard Toggle word wrap
      • Replace the following configuration options with the corresponding value from your local_settings.old file:
        • ALLOWED_HOSTS
        • SECRET_KEY
        • CACHES
        • OPENSTACK_KEYSTONE_URL
      • Restart the web server on all your controllers to apply all changes:

        # service httpd restart

        Copy to Clipboard Toggle word wrap
   6. Clean up the Dashboard resource in Pacemaker:

      #  pcs resource cleanup horizon-clone

      Copy to Clipboard Toggle word wrap
   7. Restart the Dashboard resource in Pacemaker:

      #  pcs resource enable horizon-clone

      Copy to Clipboard Toggle word wrap
   8. Wait until the output of pcs status shows that the above resource is running.
   Upgrade Compute hosts (nova):
   On each compute host:
   1. Stop all OpenStack services on the host:

      # openstack-service stop

      Copy to Clipboard Toggle word wrap
   2. Upgrade all packages:

      # yum upgrade

      Copy to Clipboard Toggle word wrap
   3. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
      Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno:

      # crudini --set /etc/nova/nova.conf upgrade_levels compute juno

      Copy to Clipboard Toggle word wrap
      You need to make this change on your controllers and on your compute hosts.
   4. Start all openstack services on the host:

      # openstack-service start

      Copy to Clipboard Toggle word wrap
   5. After you have upgraded all of your hosts to Kilo, you will want to remove the API limits configured in the previous step. On all of your hosts:

      # crudini --del /etc/nova/nova.conf upgrade_levels compute

      Copy to Clipboard Toggle word wrap
   Post-upgrade tasks:
   1. After completing all of your individual service upgrades, you should perform a complete package upgrade on all of your systems:

      # yum upgrade

      Copy to Clipboard Toggle word wrap
      This will ensure that all packages are up-to-date. You may want to schedule a restart of your OpenStack hosts at a future date in order to ensure that all running processes are using updated versions of the underlying binaries.
   2. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
      New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.