Chapter 4. Fencing Controller nodes with STONITH

Procedure

1. Log in to each Controller node as the heat-admin user.

2. Verify that the cluster is running:

   $ sudo pcs status

   Example output:

   Cluster name: openstackHA
   Last updated: Wed Jun 24 12:40:27 2015
   Last change: Wed Jun 24 11:36:18 2015
   Stack: corosync
   Current DC: lb-c1a2 (2) - partition with quorum
   Version: 1.1.12-a14efad
   3 Nodes configured
   141 Resources configured

3. Verify that STONITH is disabled:

   $ sudo pcs property show

   Example output:

   Cluster Properties:
   cluster-infrastructure: corosync
   cluster-name: openstackHA
   dc-version: 1.1.12-a14efad
   have-watchdog: false
   stonith-enabled: false

4. Depending on the fencing agent that you want to use, choose one of the following options:

   • If you use the IPMI or RHV fencing agent, generate the fencing.yaml environment file:

     $ openstack overcloud generate fencing --output fencing.yaml nodes.json

     Note

     This command converts ilo and drac power management details to IPMI equivalents.

   • If you use a different fencing agent, such as STONITH Block Device (SBD), fence_kdump, or Redfish, or if you use pre-provisioned nodes, create the fencing.yaml file manually.

5. SBD fencing only: Add the following parameter to the fencing.yaml file:

   parameter_defaults:
     ExtraConfig:
       pacemaker::corosync::enable_sbd: true

   Note

   This step is applicable to initial overcloud deployments only. For more information about how to enable SBD fencing on an existing overcloud, see Enabling sbd fencing in RHEL 7 and 8.

6. Multi-layered fencing only: Add the level-specific parameters to the generated fencing.yaml file:

   parameter_defaults:
     EnableFencing: true
     FencingConfig:
       devices:
         level1:
         - agent: [VALUE]
           host_mac: aa:bb:cc:dd:ee:ff
           params:
             <parameter>: <value>
         level2:
         - agent: fence_agent2
           host_mac: aa:bb:cc:dd:ee:ff
           params:
             <parameter>: <value>

   Replace <parameter> and <value> with the actual parameters and values that the fencing agent requires.

7. Run the overcloud deploy command and include the fencing.yaml file and any other environment files that are relevant for your deployment:

   openstack overcloud deploy --templates \
   -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
   -e ~/templates/network-environment.yaml \
   -e ~/templates/storage-environment.yaml --ntp-server pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan \
   -e fencing.yaml

8. SBD fencing only: Set the watchdog timer device interval and check that the interval is set correctly.

   # pcs property set stonith-watchdog-timeout=<interval>
   # pcs property show

Verification

1. Log in to the overcloud as the stack user and check that Pacemaker is configured as the resource manager:

   $ source stackrc
   $ openstack server list | grep controller
   $ ssh heat-admin@<controller-x_ip>
   $ sudo pcs status | grep fence
   stonith-overcloud-controller-x (stonith:fence_ipmilan): Started overcloud-controller-y

   In this example, Pacemaker is configured to use a STONITH resource for each of the Controller nodes that are specified in the fencing.yaml file.

   Note

   You must not configure the fence-resource process on the same node that it controls.

2. Check the fencing resource attributes. The STONITH attribute values must match the values in the fencing.yaml file:

   $ sudo pcs stonith show <stonith-resource-controller-x>

Procedure

1. Log in to a Controller node as the stack user and source the credentials file:

   $ source stackrc
   $ openstack server list | grep controller
   $ ssh heat-admin@<controller-x_ip>

2. Change to the root user and close all connections to the Controller node:

   $ sudo -i
   iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT &&
   iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT &&
   iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 5016 -j ACCEPT &&
   iptables -A INPUT -p udp -m state --state NEW -m udp --dport 5016 -j ACCEPT &&
   iptables -A INPUT ! -i lo -j REJECT --reject-with icmp-host-prohibited &&
   iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT &&
   iptables -A OUTPUT -p tcp --sport 5016 -j ACCEPT &&
   iptables -A OUTPUT -p udp --sport 5016 -j ACCEPT &&
   iptables -A OUTPUT ! -o lo -j REJECT --reject-with icmp-host-prohibited

3. From a different Controller node, locate the fencing event in the Pacemaker log file:

   $ ssh heat-admin@<controller-x_ip>
   $ less /var/log/cluster/corosync.log
   (less): /fenc*

   If the STONITH service performed the fencing action on the Controller, the log file shows a fencing event.

4. Wait a few minutes and then verify that the restarted Controller node is running in the cluster again by running the pcs status command. If you can see the Controller node that you restarted in the output, fencing functions correctly.