Chapter 4. Configuring a Red Hat High Availability cluster on AWS

Procedure

1. Launch the AWS Console.
2. Click on your AWS Account ID to display the drop-down menu and select My Security Credentials.
3. Click Users.
4. Select the user and open the Summary screen.
5. Click the Security credentials tab.
6. Click Create access key.
7. Download the .csv file (or save both keys). You need to enter these keys when creating the fencing device.

Procedure

1. Install the AWS command line tools by using the yum command.

   # yum install awscli

2. Use the aws --version command to verify that you installed the AWS CLI.

   $ aws --version
   aws-cli/1.19.77 Python/3.6.15 Linux/5.14.16-201.fc34.x86_64 botocore/1.20.77

3. Configure the AWS command line client according to your AWS access details.

   $ aws configure
   AWS Access Key ID [None]:
   AWS Secret Access Key [None]:
   Default region name [None]:
   Default output format [None]:

Procedure

1. From the AWS EC2 Dashboard, select Images and then AMIs.
2. Right-click on your image and select Launch.

3. Choose an Instance Type that meets or exceeds the requirements of your workload. Depending on your HA application, each instance may need to have higher capacity.

   See Amazon EC2 Instance Types for information about instance types.

4. Click Next: Configure Instance Details.

   1. Enter the Number of instances you want to create for the cluster. This example procedure uses three cluster nodes.

      Note

      Do not launch into an Auto Scaling Group.

   2. For Network, select the VPC you created in Set up the AWS environment. Select the subnet for the instance to create a new subnet.

   3. Select Enable for Auto-assign Public IP. These are the minimum selections you need to make for Configure Instance Details. Depending on your specific HA application, you may need to make additional selections.

      Note

      These are the minimum configuration options necessary to create a basic instance. Review additional options based on your HA application requirements.

5. Click Next: Add Storage and verify that the default storage is sufficient. You do not need to modify these settings unless your HA application requires other storage options.

6. Click Next: Add Tags.

   Note

   Tags can help you manage your AWS resources. See Tagging Your Amazon EC2 Resources for information about tagging.

7. Click Next: Configure Security Group. Select the existing security group you created in Setting up the AWS environment.
8. Click Review and Launch and verify your selections.
9. Click Launch. You are prompted to select an existing key pair or create a new key pair. Select the key pair you created when Setting up the AWS environment.
10. Click Launch Instances.

11. Click View Instances. You can name the instance(s).

    Note

    Alternatively, you can launch instances by using the AWS CLI. See Launching, Listing, and Terminating Amazon EC2 Instances in the Amazon documentation for more information.

Procedure

1. Move the key file from the Downloads directory to your Home directory or to your ~/.ssh directory.

2. Change the permissions of the key file so that only the root user can read it.

   # chmod 400 KeyName.pem

Procedure

1. Launch the AWS Console and select the EC2 instance.
2. Click Connect and select A standalone SSH client.
3. From your SSH terminal session, connect to the instance by using the AWS example provided in the pop-up window. Add the correct path to your KeyName.pem file if the path is not shown in the example.

Procedure

1. Remove the AWS Red Hat Update Infrastructure (RHUI) client.

   $ sudo -i
   # yum -y remove rh-amazon-rhui-client*

2. Register the VM with Red Hat.

   # subscription-manager register --auto-attach

3. Disable all repositories.

   # subscription-manager repos --disable=*

4. Enable the RHEL 8 Server HA repositories.

   # subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms

5. Update the RHEL AWS instance.

   # yum update -y

6. Install the Red Hat High Availability Add-On software packages, along with the AWS fencing agent from the High Availability channel.

   # yum install pcs pacemaker fence-agents-aws

7. The user hacluster was created during the pcs and pacemaker installation in the previous step. Create a password for hacluster on all cluster nodes. Use the same password for all nodes.

   # passwd hacluster

8. Add the high availability service to the RHEL Firewall if firewalld.service is installed.

   # firewall-cmd --permanent --add-service=high-availability
   # firewall-cmd --reload

9. Start the pcs service and enable it to start on boot.

   # systemctl start pcsd.service
   # systemctl enable pcsd.service

10. Edit /etc/hosts and add RHEL host names and internal IP addresses. For more information, see the Red Hat Knowledgebase solution How should the /etc/hosts file be set up on RHEL cluster nodes?.

Procedure

1. On one of the nodes, enter the following command to authenticate the pcs user hacluster. In the command, specify the name of each node in the cluster.

   # pcs host auth <hostname1> <hostname2> <hostname3>

   Example:

   [root@node01 clouduser]# pcs host auth node01 node02 node03
   Username: hacluster
   Password:
   node01: Authorized
   node02: Authorized
   node03: Authorized

2. Create the cluster.

   # pcs cluster setup <cluster_name> <hostname1> <hostname2> <hostname3>

   Example:

   [root@node01 clouduser]# pcs cluster setup new_cluster node01 node02 node03
   
   [...]
   
   Synchronizing pcsd certificates on nodes node01, node02, node03...
   node02: Success
   node03: Success
   node01: Success
   Restarting pcsd on the nodes in order to reload the certificates...
   node02: Success
   node03: Success
   node01: Success

Verification

1. Enable the cluster.

   [root@node01 clouduser]# pcs cluster enable --all
   node02: Cluster Enabled
   node03: Cluster Enabled
   node01: Cluster Enabled

2. Start the cluster.

   [root@node01 clouduser]# pcs cluster start --all
   node02: Starting Cluster...
   node03: Starting Cluster...
   node01: Starting Cluster...

Standard procedure

1. Enter the following AWS metadata query to get the Instance ID for each node. You need these IDs to configure the fence device. See Instance Metadata and User Data for additional information.

   # echo $(curl -s http://169.254.169.254/latest/meta-data/instance-id)

   Example:

   [root@ip-10-0-0-48 ~]# echo $(curl -s http://169.254.169.254/latest/meta-data/instance-id) i-07f1ac63af0ec0ac6

2. Enter the following command to configure the fence device. Use the pcmk_host_map command to map the RHEL host name to the Instance ID. Use the AWS Access Key and AWS Secret Access Key that you previously set up.

   # pcs stonith \
       create <name> fence_aws access_key=access-key secret_key=<secret-access-key> \
       region=<region> pcmk_host_map="rhel-hostname-1:Instance-ID-1;rhel-hostname-2:Instance-ID-2;rhel-hostname-3:Instance-ID-3" \
       power_timeout=240 pcmk_reboot_timeout=480 pcmk_reboot_retries=4

   Example:

   [root@ip-10-0-0-48 ~]# pcs stonith \
   create clusterfence fence_aws access_key=AKIAI123456MRMJA secret_key=a75EYIG4RVL3hdsdAslK7koQ8dzaDyn5yoIZ/ \
   region=us-east-1 pcmk_host_map="ip-10-0-0-48:i-07f1ac63af0ec0ac6;ip-10-0-0-46:i-063fc5fe93b4167b2;ip-10-0-0-58:i-08bd39eb03a6fd2c7" \
   power_timeout=240 pcmk_reboot_timeout=480 pcmk_reboot_retries=4

3. To ensure immediate and complete fencing, disable ACPI Soft-Off on all cluster nodes. For information about disabling ACPI Soft-Off, see Disabling ACPI for use with integrated fence device.

Alternate procedure

1. Obtain the VPC ID of the cluster.

   # aws ec2 describe-vpcs --output text --filters "Name=tag:Name,Values=<clustername>-vpc" --query 'Vpcs[*].VpcId'
   vpc-06bc10ac8f6006664

2. By using the VPC ID of the cluster, obtain the VPC instances.

   $ aws ec2 describe-instances --output text --filters "Name=vpc-id,Values=vpc-06bc10ac8f6006664" --query 'Reservations[*].Instances[*].{Name:Tags[? Key==Name]|[0].Value,Instance:InstanceId}' | grep "\-node[a-c]"
   i-0b02af8927a895137     <clustername>-nodea-vm
   i-0cceb4ba8ab743b69     <clustername>-nodeb-vm
   i-0502291ab38c762a5     <clustername>-nodec-vm

3. Use the obtained instance IDs to configure fencing on each node on the cluster. For example, to configure a fencing device on all nodes in a cluster:

   [root@nodea ~]# CLUSTER=<clustername> && pcs stonith create fence${CLUSTER} fence_aws access_key=XXXXXXXXXXXXXXXXXXXX pcmk_host_map=$(for NODE \
   in node{a..c}; do ssh ${NODE} "echo -n \${HOSTNAME}:\$(curl -s http://169.254.169.254/latest/meta-data/instance-id)\;"; done) \
   pcmk_reboot_retries=4 pcmk_reboot_timeout=480 power_timeout=240 region=xx-xxxx-x secret_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

   For information about specific parameters for creating fencing devices, see the fence_aws man page or the Configuring and managing high availability clusters guide.

4. To ensure immediate and complete fencing, disable ACPI Soft-Off on all cluster nodes. For information about disabling ACPI Soft-Off, see Disabling ACPI for use with integrated fence device.

Verification

1. Display the configured fencing devices and their parameters on your nodes:

   [root@nodea ~]# pcs stonith config fence${CLUSTER}
   
   Resource: <clustername> (class=stonith type=fence_aws)
   Attributes: access_key=XXXXXXXXXXXXXXXXXXXX pcmk_host_map=nodea:i-0b02af8927a895137;nodeb:i-0cceb4ba8ab743b69;nodec:i-0502291ab38c762a5;
   pcmk_reboot_retries=4 pcmk_reboot_timeout=480 power_timeout=240 region=xx-xxxx-x secret_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   Operations: monitor interval=60s (<clustername>-monitor-interval-60s)

2. Test the fencing agent for one of the cluster nodes.

   # pcs stonith fence <awsnodename>

   Note

   The command response may take several minutes to display. If you watch the active terminal session for the node being fenced, you see that the terminal connection is immediately terminated after you enter the fence command.

   Example:

   [root@ip-10-0-0-48 ~]# pcs stonith fence ip-10-0-0-58
   
   Node: ip-10-0-0-58 fenced

3. Check the status to verify that the node is fenced.

   # pcs status

   Example:

   [root@ip-10-0-0-48 ~]# pcs status
   
   Cluster name: newcluster
   Stack: corosync
   Current DC: ip-10-0-0-46 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
   Last updated: Fri Mar  2 19:55:41 2018
   Last change: Fri Mar  2 19:24:59 2018 by root via cibadmin on ip-10-0-0-46
   
   3 nodes configured
   1 resource configured
   
   Online: [ ip-10-0-0-46 ip-10-0-0-48 ]
   OFFLINE: [ ip-10-0-0-58 ]
   
   Full list of resources:
   clusterfence  (stonith:fence_aws):    Started ip-10-0-0-46
   
   Daemon Status:
   corosync: active/disabled
   pacemaker: active/disabled
   pcsd: active/enabled

4. Start the node that was fenced in the previous step.

   # pcs cluster start <awshostname>

5. Check the status to verify the node started.

   # pcs status

   Example:

   [root@ip-10-0-0-48 ~]# pcs status
   
   Cluster name: newcluster
   Stack: corosync
   Current DC: ip-10-0-0-46 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
   Last updated: Fri Mar  2 20:01:31 2018
   Last change: Fri Mar  2 19:24:59 2018 by root via cibadmin on ip-10-0-0-48
   
   3 nodes configured
   1 resource configured
   
   Online: [ ip-10-0-0-46 ip-10-0-0-48 ip-10-0-0-58 ]
   
   Full list of resources:
   
     clusterfence  (stonith:fence_aws):    Started ip-10-0-0-46
   
   Daemon Status:
     corosync: active/disabled
     pacemaker: active/disabled
     pcsd: active/enabled

Procedure

1. Install the AWS CLI. For instructions, see Installing the AWS CLI.

2. Verify that the AWS CLI is configured properly. The instance IDs and instance names should display.

   Example:

   [root@ip-10-0-0-48 ~]# aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,Tags[?Key==Name].Value]'
   i-07f1ac63af0ec0ac6
   ip-10-0-0-48
   i-063fc5fe93b4167b2
   ip-10-0-0-46
   i-08bd39eb03a6fd2c7
   ip-10-0-0-58

Procedure

1. Create a shared block volume by using the AWS command create-volume.

   $ aws ec2 create-volume --availability-zone <availability_zone> --no-encrypted --size 1024 --volume-type io1 --iops 51200 --multi-attach-enabled

   For example, the following command creates a volume in the us-east-1a availability zone.

   $ aws ec2 create-volume --availability-zone us-east-1a --no-encrypted --size 1024 --volume-type io1 --iops 51200 --multi-attach-enabled
   
   {
       "AvailabilityZone": "us-east-1a",
       "CreateTime": "2020-08-27T19:16:42.000Z",
       "Encrypted": false,
       "Size": 1024,
       "SnapshotId": "",
       "State": "creating",
       "VolumeId": "vol-042a5652867304f09",
       "Iops": 51200,
       "Tags": [ ],
       "VolumeType": "io1"
   }

   Note

   You need the VolumeId in the next step.

2. For each instance in your cluster, attach a shared block volume by using the AWS command attach-volume. Use your <instance_id> and <volume_id>.

   $ aws ec2 attach-volume --device /dev/xvdd --instance-id <instance_id> --volume-id <volume_id>

   For example, the following command attaches a shared block volume vol-042a5652867304f09 to instance i-0eb803361c2c887f2.

   $ aws ec2 attach-volume --device /dev/xvdd --instance-id i-0eb803361c2c887f2 --volume-id vol-042a5652867304f09
   
   {
       "AttachTime": "2020-08-27T19:26:16.086Z",
       "Device": "/dev/xvdd",
       "InstanceId": "i-0eb803361c2c887f2",
       "State": "attaching",
       "VolumeId": "vol-042a5652867304f09"
   }

Verification

1. For each instance in your cluster, verify that the block device is available by using the ssh command with your instance <ip_address>.

   # ssh <ip_address> "hostname ; lsblk -d | grep ' 1T '"

   For example, the following command lists details including the host name and block device for the instance IP 198.51.100.3.

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T '"
   
   nodea
   nvme2n1 259:1    0   1T  0 disk

2. Use the ssh command to verify that each instance in your cluster uses the same shared disk.

   # ssh <ip_address> "hostname ; lsblk -d | grep ' 1T ' | awk '{print \$1}' | xargs -i udevadm info --query=all --name=/dev/{} | grep '^E: ID_SERIAL='"

   For example, the following command lists details including the host name and shared disk volume ID for the instance IP address 198.51.100.3.

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T ' | awk '{print \$1}' | xargs -i udevadm info --query=all --name=/dev/{} | grep '^E: ID_SERIAL='"
   
   nodea
   E: ID_SERIAL=Amazon Elastic Block Store_vol0fa5342e7aedf09f7