High Availability for Compute Instances

1. At the time of failure, the IPMI agent performs first-layer fencing and physically resets the node to ensure that it is powered off. Evacuating instances from online Compute nodes might result in data corruption or in multiple identical instances running on the overcloud. When the node is powered off, it is considered fenced.

2. After the physical IPMI fencing, the fence-nova agent performs second-layer fencing and marks the fenced node with the “evacuate=yes” cluster per-node attribute. To do this, the agent runs the following command:

   $ attrd_updater -n evacuate -A name="evacuate" host="FAILEDHOST" value="yes"

   Copy to Clipboard Toggle word wrap

   Where FAILEDHOST is the hostname of the failed Compute node.

3. The nova-evacuate agent continually runs in the background, periodically checking the cluster for nodes with the “evacuate=yes” attribute. When nova-evacuate detects that the fenced node contains this attribute, the agent starts evacuating the node using the process described in Evacuate Instances.
4. While the failed node boots up from the IPMI reset, the nova-compute process on that node starts automatically. Because the node was fenced earlier, it does not run any new instances until Pacemaker unfences it.
5. When Pacemaker sees that the Compute node is online again, it tries to start the compute-unfence-trigger resource on the node, reverting the force-down API call and setting the node as enabled again.

Configure the Instance HA role, flavor, and profile

1. Add the ComputeInstanceHA role to your roles data file and regenerate the file. For example:

   $ openstack overcloud roles generate -o ~/my_roles_data.yaml Controller Compute ComputeInstanceHA

   Copy to Clipboard Toggle word wrap

   The ComputeInstanceHA role includes all the services in the default Compute role as well as the ComputeInstanceHA and the PacemakerRemote services. For general information about custom roles and about the roles-data.yaml, see the Roles section in the Advanced Overcloud Customization guide.

2. Create the compute-instance-ha flavor to tag Compute nodes that you want to designate for Instance HA. For example:

   $ source ~/stackrc
   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-instance-ha
   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="compute-instance-ha" compute-instance-ha

   Copy to Clipboard Toggle word wrap

3. Tag each Compute node that you want to designate for Instance HA with the compute-instance-ha profile.

   $ openstack baremetal node set --property capabilities='profile:compute-instance-ha,boot_option:local' <NODE UUID>

   Copy to Clipboard Toggle word wrap

4. Map the ComputeInstanceHA role to the compute-instance-ha flavor by creating an environment file with the following content:

   parameter_defaults:
     OvercloudComputeInstanceHAFlavor: compute-instance-ha

   Copy to Clipboard Toggle word wrap

Enable fencing

1. Enable fencing on all Controller and Compute nodes in the overcloud by creating an environment file with fencing information. Make sure to create the environment file in an accessible location, such as ~/templates. For example:

   parameter_defaults:
     EnableFencing: true
     FencingConfig:
       devices:
       - agent: fence_ipmilan
         host_mac: 00:ec:ad:cb:3c:c7
         params:
           login: admin
           ipaddr: 192.168.24.1
           ipport: 6230
           passwd: password
           lanplus: 1
       - agent: fence_ipmilan
         host_mac: 00:ec:ad:cb:3c:cb
         params:
           login: admin
           ipaddr: 192.168.24.1
           ipport: 6231
           passwd: password
           lanplus: 1
       - agent: fence_ipmilan
         host_mac: 00:ec:ad:cb:3c:cf
         params:
           login: admin
           ipaddr: 192.168.24.1
           ipport: 6232
           passwd: password
           lanplus: 1
       - agent: fence_ipmilan
         host_mac: 00:ec:ad:cb:3c:d3
         params:
           login: admin
           ipaddr: 192.168.24.1
           ipport: 6233
           passwd: password
           lanplus: 1
       - agent: fence_ipmilan
         host_mac: 00:ec:ad:cb:3c:d7
         params:
           login: admin
           ipaddr: 192.168.24.1
           ipport: 6234
           passwd: password
           lanplus: 1

   Copy to Clipboard Toggle word wrap

   For more information about fencing and STONITH configuration, see the Fencing Controller Nodes with STONITH section of the High Availability Deployment and Usage guide.

2. Instance HA uses shared storage by default. If shared storage is not configured for your Compute instance, then add the following parameter to the environment file that you created in the previous step:

   parameter_defaults:
     ExtraConfig:
       tripleo::instanceha::no_shared_storage: true

   Copy to Clipboard Toggle word wrap

   See Section 3.1, “Considerations for Shared Storage” for details on how to boot from an OpenStack Block Storage (cinder) volume rather than configuring shared storage for the disk image of instances.

1. Boot one or more instances on the overcloud before crashing the Compute node that hosts the instances to test.

   stack@director $ . overcloudrc
   stack@director $ nova boot --image cirros --flavor 2 test-failover
   stack@director $ nova list --fields name,status,host

   Copy to Clipboard Toggle word wrap

2. Log in to the Compute node that hosts the instances and change to the root user. Replace compute-n with the name of the Compute node:

   stack@director $ . stackrc
   stack@director $ ssh -l heat-admin compute-n
   heat-admin@compute-n $ su -

   Copy to Clipboard Toggle word wrap

3. Crash the Compute node.

   root@compute-n $ echo c > /proc/sysrq-trigger

   Copy to Clipboard Toggle word wrap

4. Wait a few minutes and then verify that these instances re-spawned on another Compute nodes.

   stack@director $ nova list --fields name,status,host
   stack@director $ nova service-list

   Copy to Clipboard Toggle word wrap