Chapter 4. Fencing Controller nodes with STONITH

Procedure

1. Log in to each Controller node as the tripleo-admin user.

2. Verify that the cluster is running:

   $ sudo pcs status

   Copy to Clipboard Toggle word wrap

   Example output:

   Cluster name: openstackHA
   Last updated: Wed Jun 24 12:40:27 2015
   Last change: Wed Jun 24 11:36:18 2015
   Stack: corosync
   Current DC: lb-c1a2 (2) - partition with quorum
   Version: 1.1.12-a14efad
   3 Nodes configured
   141 Resources configured

   Copy to Clipboard Toggle word wrap

3. Verify that STONITH is disabled:

   $ sudo pcs property show

   Copy to Clipboard Toggle word wrap

   Example output:

   Cluster Properties:
   cluster-infrastructure: corosync
   cluster-name: openstackHA
   dc-version: 1.1.12-a14efad
   have-watchdog: false
   stonith-enabled: false

   Copy to Clipboard Toggle word wrap

4. Depending on the fencing agent that you want to use, choose one of the following options:

   • If you use the IPMI or RHV fencing agent, generate the fencing.yaml environment file:

     (undercloud) $ openstack overcloud generate fencing --output fencing.yaml nodes.json

     Copy to Clipboard Toggle word wrap
     Note

     This command converts ilo and drac power management details to IPMI equivalents.

   • If you use a different fencing agent, such as STONITH Block Device (SBD), fence_kdump, or Redfish, or if you use pre-provisioned nodes, create the fencing.yaml file manually.

5. SBD fencing only: Add the following parameter to the fencing.yaml file:

   parameter_defaults:
     ExtraConfig:
       pacemaker::corosync::enable_sbd: true

   Copy to Clipboard Toggle word wrap
   Note

   This step is applicable to initial overcloud deployments only. For more information about how to enable SBD fencing on an existing overcloud, see Enabling sbd fencing in RHEL 7 and 8.

6. Multi-layered fencing only: Add the level-specific parameters to the generated fencing.yaml file:

   parameter_defaults:
     EnableFencing: true
     FencingConfig:
       devices:
         level1:
         - agent: [VALUE]
           host_mac: "aa:bb:cc:dd:ee:ff"
           params:
             <parameter>: <value>
         level2:
         - agent: fence_agent2
           host_mac: "aa:bb:cc:dd:ee:ff"
           params:
             <parameter>: <value>

   Copy to Clipboard Toggle word wrap

   Replace <parameter> and <value> with the actual parameters and values that the fencing agent requires.

7. Run the overcloud deploy command and include the fencing.yaml file and any other environment files that are relevant for your deployment:

   openstack overcloud deploy --templates \
   -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
   -e ~/templates/network-environment.yaml \
   -e ~/templates/storage-environment.yaml --ntp-server pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan \
   -e fencing.yaml

   Copy to Clipboard Toggle word wrap

8. SBD fencing only: Set the watchdog timer device interval and check that the interval is set correctly.

   # pcs property set stonith-watchdog-timeout=<interval>
   # pcs property show

   Copy to Clipboard Toggle word wrap
9. Ensure that all of the created fencing agents are in the "running" and not the "stopped" state. Fencing agents in the "stopped" state have been incorrectly configured.

Verification

1. Log in to the overcloud as the tripleo-admin user and ensure that Pacemaker is configured as the resource manager:

   $ source stackrc
   $ metalsmith list | grep controller
   $ ssh tripleo-admin@<controller-x_ip>
   $ sudo pcs status | grep fence
   stonith-overcloud-controller-x (stonith:fence_ipmilan): Started overcloud-controller-y

   Copy to Clipboard Toggle word wrap

   In this example, Pacemaker is configured to use a STONITH resource for each of the Controller nodes that are specified in the fencing.yaml file.

   Note

   You must not configure the fence-resource process on the same node that it controls.

2. Check the fencing resource attributes. The STONITH attribute values must match the values in the fencing.yaml file:

   $ sudo pcs stonith show <stonith-resource-controller-x>

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Controller node as the stack user and source the credentials file:

   $ source stackrc
   $ metalsmith list | grep controller
   $ ssh tripleo-admin@<controller-x_ip>

   Copy to Clipboard Toggle word wrap

2. Change to the root user and close all connections to the Controller node:

   $ sudo -i
   iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT &&
   iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT &&
   iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 5016 -j ACCEPT &&
   iptables -A INPUT -p udp -m state --state NEW -m udp --dport 5016 -j ACCEPT &&
   iptables -A INPUT ! -i lo -j REJECT --reject-with icmp-host-prohibited &&
   iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT &&
   iptables -A OUTPUT -p tcp --sport 5016 -j ACCEPT &&
   iptables -A OUTPUT -p udp --sport 5016 -j ACCEPT &&
   iptables -A OUTPUT ! -o lo -j REJECT --reject-with icmp-host-prohibited

   Copy to Clipboard Toggle word wrap

3. From a different Controller node, locate the fencing event in the Pacemaker log file:

   $ ssh tripleo-admin@<controller-x_ip>
   $ less /var/log/cluster/corosync.log
   (less): /fenc*

   Copy to Clipboard Toggle word wrap

   If the STONITH service performed the fencing action on the Controller, the log file shows a fencing event.

4. Wait a few minutes and then verify that the restarted Controller node is running in the cluster again by running the pcs status command. If you can see the Controller node that you restarted in the output, fencing functions correctly.