Chapter 4. Configuring a Red Hat High Availability cluster on AWS

Procedure

1. Use SSH, Telnet, HTTP, or whatever remote protocol is used to connect to the device to manually log in and test the fence device or see what output is given. For example, if you will be configuring fencing for an IPMI-enabled device,then try to log in remotely with ipmitool. Take note of the options used when logging in manually because those options might be needed when using the fencing agent.

   If you are unable to log in to the fence device, verify that the device is pingable, there is nothing such as a firewall configuration that is preventing access to the fence device, remote access is enabled on the fencing device, and the credentials are correct.

2. Run the fence agent manually, using the fence agent script. This does not require that the cluster services are running, so you can perform this step before the device is configured in the cluster. This can ensure that the fence device is responding properly before proceeding.

   Note

   These examples use the fence_ipmilan fence agent script for an iLO device. The actual fence agent you will use and the command that calls that agent will depend on your server hardware. You should consult the man page for the fence agent you are using to determine which options to specify. You will usually need to know the login and password for the fence device and other information related to the fence device.

   The following example shows the format you would use to run the fence_ipmilan fence agent script with -o status parameter to check the status of the fence device interface on another node without actually fencing it. This allows you to test the device and get it working before attempting to reboot the node. When running this command, you specify the name and password of an iLO user that has power on and off permissions for the iLO device.

   # fence_ipmilan -a ipaddress -l username -p password -o status

   Copy to Clipboard Toggle word wrap

   The following example shows the format you would use to run the fence_ipmilan fence agent script with the -o reboot parameter. Running this command on one node reboots the node managed by this iLO device.

   # fence_ipmilan -a ipaddress -l username -p password -o reboot

   Copy to Clipboard Toggle word wrap

   If the fence agent failed to properly do a status, off, on, or reboot action, you should check the hardware, the configuration of the fence device, and the syntax of your commands. In addition, you can run the fence agent script with the debug output enabled. The debug output is useful for some fencing agents to see where in the sequence of events the fencing agent script is failing when logging into the fence device.

   # fence_ipmilan -a ipaddress -l username -p password -o status -D /tmp/$(hostname)-fence_agent.debug

   Copy to Clipboard Toggle word wrap

   When diagnosing a failure that has occurred, you should ensure that the options you specified when manually logging in to the fence device are identical to what you passed on to the fence agent with the fence agent script.

   For fence agents that support an encrypted connection, you may see an error due to certificate validation failing, requiring that you trust the host or that you use the fence agent’s ssl-insecure parameter. Similarly, if SSL/TLS is disabled on the target device, you may need to account for this when setting the SSL parameters for the fence agent.

   Note

   If the fence agent that is being tested is a fence_drac, fence_ilo, or some other fencing agent for a systems management device that continues to fail, then fall back to trying fence_ipmilan. Most systems management cards support IPMI remote login and the only supported fencing agent is fence_ipmilan.

3. Once the fence device has been configured in the cluster with the same options that worked manually and the cluster has been started, test fencing with the pcs stonith fence command from any node (or even multiple times from different nodes), as in the following example. The pcs stonith fence command reads the cluster configuration from the CIB and calls the fence agent as configured to execute the fence action. This verifies that the cluster configuration is correct.

   # pcs stonith fence node_name

   Copy to Clipboard Toggle word wrap

   If the pcs stonith fence command works properly, that means the fencing configuration for the cluster should work when a fence event occurs. If the command fails, it means that cluster management cannot invoke the fence device through the configuration it has retrieved. Check for the following issues and update your cluster configuration as needed.

   • Check your fence configuration. For example, if you have used a host map you should ensure that the system can find the node using the host name you have provided.
   • Check whether the password and user name for the device include any special characters that could be misinterpreted by the bash shell. Making sure that you enter passwords and user names surrounded by quotation marks could address this issue.
   • Check whether you can connect to the device using the exact IP address or host name you specified in the pcs stonith command. For example, if you give the host name in the stonith command but test by using the IP address, that is not a valid test.

   • If the protocol that your fence device uses is accessible to you, use that protocol to try to connect to the device. For example many agents use ssh or telnet. You should try to connect to the device with the credentials you provided when configuring the device, to see if you get a valid prompt and can log in to the device.

     If you determine that all your parameters are appropriate but you still have trouble connecting to your fence device, you can check the logging on the fence device itself, if the device provides that, which will show if the user has connected and what command the user issued. You can also search through the /var/log/messages file for instances of stonith and error, which could give some idea of what is transpiring, but some agents can provide additional information.

4. Once the fence device tests are working and the cluster is up and running, test an actual failure. To do this, take an action in the cluster that should initiate a token loss.

   • Take down a network. How you take a network depends on your specific configuration. In many cases, you can physically pull the network or power cables out of the host. For information about simulating a network failure, see the Red Hat Knowledgebase solution What is the proper way to simulate a network failure on a RHEL Cluster?.

     Note

     Disabling the network interface on the local host rather than physically disconnecting the network or power cables is not recommended as a test of fencing because it does not accurately simulate a typical real-world failure.

   • Block corosync traffic both inbound and outbound using the local firewall.

     The following example blocks corosync, assuming the default corosync port is used, firewalld is used as the local firewall, and the network interface used by corosync is in the default firewall zone:

     # firewall-cmd --direct --add-rule ipv4 filter OUTPUT 2 -p udp --dport=5405 -j DROP
     # firewall-cmd --add-rich-rule='rule family="ipv4" port port="5405" protocol="udp" drop

     Copy to Clipboard Toggle word wrap

   • Simulate a crash and panic your machine with sysrq-trigger. Note, however, that triggering a kernel panic can cause data loss; it is recommended that you disable your cluster resources first.

     # echo c > /proc/sysrq-trigger

     Copy to Clipboard Toggle word wrap

Procedure

1. Add a fencing level of 1 for fence device my_ilo on node rh7-2.

   # pcs stonith level add 1 rh7-2 my_ilo

   Copy to Clipboard Toggle word wrap

2. Add a fencing level of 2 for fence device my_apc on node rh7-2.

   # pcs stonith level add 2 rh7-2 my_apc

   Copy to Clipboard Toggle word wrap

3. List the currently configured fencing levels.

   # pcs stonith level
    Node: rh7-2
     Level 1 - my_ilo
     Level 2 - my_apc

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the first fence device.

   # pcs stonith create apc1 fence_apc_snmp ipaddr=apc1.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1;node2.example.com:2"

   Copy to Clipboard Toggle word wrap

2. Create the second fence device.

   # pcs stonith create apc2 fence_apc_snmp ipaddr=apc2.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1;node2.example.com:2"

   Copy to Clipboard Toggle word wrap

3. Specify that both devices are required to fence the node.

   # pcs stonith level add 1 node1.example.com apc1,apc2
   # pcs stonith level add 1 node2.example.com apc1,apc2

   Copy to Clipboard Toggle word wrap

Procedure

1. Reboot the node and start the BIOS CMOS Setup Utility program.
2. Navigate to the Power menu (or equivalent power management menu).

3. At the Power menu, set the Soft-Off by PWR-BTTN function (or equivalent) to Instant-Off (or the equivalent setting that turns off the node by means of the power button without delay). The BIOS CMOS Setup Utiliy example below shows a Power menu with ACPI Function set to Enabled and Soft-Off by PWR-BTTN set to Instant-Off.

   Note

   The equivalents to ACPI Function, Soft-Off by PWR-BTTN, and Instant-Off may vary among computers. However, the objective of this procedure is to configure the BIOS so that the computer is turned off by means of the power button without delay.

4. Exit the BIOS CMOS Setup Utility program, saving the BIOS configuration.
5. Verify that the node turns off immediately when fenced. For information about testing a fence device, see Testing a fence device.

Procedure

1. Define the following configuration in the /etc/systemd/logind.conf file:

   HandlePowerKey=ignore

   Copy to Clipboard Toggle word wrap

2. Restart the systemd-logind service:

   # systemctl restart systemd-logind.service

   Copy to Clipboard Toggle word wrap

Verification

1. Verify that the node turns off immediately when fenced. For information about testing a fence device, see Testing a fence device.

Procedure

1. Use the --args option in combination with the --update-kernel option of the grubby tool to change the grub.cfg file of each cluster node as follows:

   # grubby --args=acpi=off --update-kernel=ALL

   Copy to Clipboard Toggle word wrap
2. Reboot the node.

Verification

1. Verify that the node turns off immediately when fenced. For information about testing a fence device, see Testing a fence device

Procedure

1. Add the two resources to the same group that you have already created to enforce order and colocation constraints.

2. Install the resource-agents package:

   # dnf install resource-agents

   Copy to Clipboard Toggle word wrap

3. Create an elastic IP address:

   [root@ip-10-0-0-48 ~]# aws ec2 allocate-address --domain vpc --output text
   eipalloc-4c4a2c45   vpc 35.169.153.122

   Copy to Clipboard Toggle word wrap

4. Optional: Display the description of awseip. This shows the options and default operations for this agent.

   # pcs resource describe awseip

   Copy to Clipboard Toggle word wrap

5. Create the Secondary Elastic IP address resource with the allocated IP address in the 2nd step:

   # pcs resource create <resource_id> awseip elastic_ip=<elastic_ip_address> allocation_id=<elastic_ip_association_id> --group networking-group

   Copy to Clipboard Toggle word wrap

   Example:

   # pcs resource create elastic awseip elastic_ip=35.169.153.122 allocation_id=eipalloc-4c4a2c45 --group networking-group

   Copy to Clipboard Toggle word wrap

Verification

1. Verify the cluster status to ensure resources are available:

   [root@ip-10-0-0-58 ~]# pcs status
   
   Cluster name: newcluster
   Stack: corosync
   Current DC: ip-10-0-0-58 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
   Last updated: Mon Mar  5 16:27:55 2018
   Last change: Mon Mar  5 15:57:51 2018 by root via cibadmin on ip-10-0-0-46
   
   3 nodes configured
   4 resources configured
   
   Online: [ ip-10-0-0-46 ip-10-0-0-48 ip-10-0-0-58 ]
   
   Full list of resources:
   
    clusterfence   (stonith:fence_aws):    Started ip-10-0-0-46
    Resource Group: networking-group
        vip (ocf::heartbeat:IPaddr2): Started ip-10-0-0-48
        elastic (ocf::heartbeat:awseip): Started ip-10-0-0-48
   
   Daemon Status:
     corosync: active/disabled
     pacemaker: active/disabled
     pcsd: active/enabled

   Copy to Clipboard Toggle word wrap

   In this example, newcluster is an active cluster where resources such as vip and elastic are part of the networking-group resource group.

2. Launch an SSH session from your local workstation to the elastic IP address that you have already created:

   $ ssh -l ec2-user -i ~/.ssh/cluster-admin.pem 35.169.153.122

   Copy to Clipboard Toggle word wrap
3. Verify that the SSH connected host is same as the host with the elastic resources.

Procedure

1. Install the resource-agents package.

   # dnf install resource-agents

   Copy to Clipboard Toggle word wrap

2. Optional: View the options and default operations for awsvip:

   # pcs resource describe awsvip

   Copy to Clipboard Toggle word wrap

3. Create a Secondary Private IP address with an unused private IP address in the VPC CIDR block:

   [root@ip-10-0-0-48 ~]# pcs resource create privip awsvip secondary_private_ip=10.0.0.68 --group networking-group

   Copy to Clipboard Toggle word wrap

   Here, secondary private IP address is a part of gets included in a resource group

4. Create a virtual IP resource with the vip resource ID and the networking-group group name:

   root@ip-10-0-0-48 ~]# pcs resource create vip IPaddr2 ip=10.0.0.68 --group networking-group

   Copy to Clipboard Toggle word wrap

   This is a Virtual Private Cloud (VPC) IP address that maps from the fence node to the failover node, masking the failure of the fence node within the subnet. Ensure that the virtual IP belongs to the same resource group as the Secondary Private IP address you created in the last step.

Procedure

1. Install the resource-agents package:

   # dnf install resource-agents

   Copy to Clipboard Toggle word wrap

2. Optional: View the options and default operations for awsvip:

   # pcs resource describe aws-vpc-move-ip

   Copy to Clipboard Toggle word wrap

3. Set up an OverlayIPAgent IAM policy for the IAM user.

   1. In the AWS console, navigate to Services IAM Policies Create OverlayIPAgent Policy

   2. Input the following configuration, and change the <region>, <account_id>, and <cluster_route_table_id> values to correspond with your cluster.

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Sid": "Stmt1424870324000",
                  "Effect": "Allow",
                  "Action":  "ec2:DescribeRouteTables",
                  "Resource": "*"
              },
              {
                  "Sid": "Stmt1424860166260",
                  "Action": [
                      "ec2:CreateRoute",
                      "ec2:ReplaceRoute"
                  ],
                  "Effect": "Allow",
                  "Resource": "arn:aws:ec2:_<region>_:_<account_id>_:route-table/_<cluster_route_table_id>_"
              }
          ]
      }

      Copy to Clipboard Toggle word wrap

4. In the AWS console, disable the Source/Destination Check function on all nodes in the cluster.

   To do this, right-click each node Networking Change Source/Destination Checks. In the pop-up message that is displayed, click Yes, Disable.

5. Create a route table for the cluster. To do so, use the following command on one node in the cluster:

   # aws ec2 create-route --route-table-id <cluster_route_table_id> --destination-cidr-block <new_cidr_block_ip/net_mask> --instance-id <cluster_node_id>

   Copy to Clipboard Toggle word wrap

   In the command, replace values as follows:

   • ClusterRouteTableID: The route table ID for the existing cluster Virtual Private Cloud (VPC) route table.
   • NewCIDRblockIP: A new IP address and netmask outside of the VPC classless inter-domain routing (CIDR) block. For example, if the VPC CIDR block is 172.31.0.0/16, the new IP address or netmask can be 192.168.0.15/32.
   • ClusterNodeID: The instance ID for another node in the cluster.

6. On one of the nodes in the cluster, create a aws-vpc-move-ip resource that uses a free IP address that is accessible to the client. The following example creates a resource named vpcip that uses IP 192.168.0.15.

   # pcs resource create vpcip aws-vpc-move-ip ip=192.168.0.15 interface=eth0 routing_table=<cluster_route_table_id>

   Copy to Clipboard Toggle word wrap

7. On all nodes in the cluster, edit the /etc/hosts/ file, and add a line with the IP address of the newly created resource. For example:

   192.168.0.15 vpcip

   Copy to Clipboard Toggle word wrap

Verification

1. Test the failover ability of the new aws-vpc-move-ip resource:

   # pcs resource move vpcip

   Copy to Clipboard Toggle word wrap

2. If the failover succeeded, remove the automatically created constraint after the move of the vpcip resource:

   # pcs resource clear vpcip

   Copy to Clipboard Toggle word wrap