Deploying and managing RHEL on Amazon Web Services

Procedure

1. Install Python 3 and the pip tool:

   # dnf install python3 python3-pip

   Copy to Clipboard Toggle word wrap

2. Install the AWS command-line tools with pip:

   # pip3 install awscli

   Copy to Clipboard Toggle word wrap

3. Set your profile. The terminal prompts you to provide your credentials, region, and output format:

   $ aws configure
   AWS Access Key ID [None]:
   AWS Secret Access Key [None]:
   Default region name [None]:
   Default output format [None]:

   Copy to Clipboard Toggle word wrap

4. Define a name for your bucket and create a bucket:

   $ BUCKET=bucketname
   $ aws s3 mb s3://$BUCKET

   Copy to Clipboard Toggle word wrap

   Replace bucketname with the actual bucket name. It must be a globally unique name. As a result, your bucket is created.

5. To grant permission to access the S3 bucket, create a vmimport S3 Role in the AWS Identity and Access Management (IAM), if you have not already done so in the past:

   1. Create a trust-policy.json file with the trust policy configuration in JSON format. For example:

      {
          "Version": "2022-10-17",
          "Statement": [{
              "Effect": "Allow",
              "Principal": {
                  "Service": "vmie.amazonaws.com"
              },
              "Action": "sts:AssumeRole",
              "Condition": {
                  "StringEquals": {
                      "sts:Externalid": "vmimport"
                  }
              }
          }]
      }

      Copy to Clipboard Toggle word wrap

   2. Create a role-policy.json file with the role policy configuration, in the JSON format. For example:

      {
          "Version": "2012-10-17",
          "Statement": [{
              "Effect": "Allow",
              "Action": ["s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket"],
              "Resource": ["arn:aws:s3:::%s", "arn:aws:s3:::%s/"] }, { "Effect": "Allow", "Action": ["ec2:ModifySnapshotAttribute", "ec2:CopySnapshot", "ec2:RegisterImage", "ec2:Describe"],
              "Resource": "*"
          }]
      }
      $BUCKET $BUCKET

      Copy to Clipboard Toggle word wrap

   3. Create a role for your Amazon Web Services account, by using the trust-policy.json file:

      $ aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json

      Copy to Clipboard Toggle word wrap

   4. Embed an inline policy document by using the role-policy.json file:

      $ aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json

      Copy to Clipboard Toggle word wrap

Procedure

1. Build the image:

   # image-builder build ami \
   --blueprint blueprint-name \
   --aws-region us-east-1 \
   --aws-bucket <example-bucket> \
   --aws-ami-name <image-name> \

   Copy to Clipboard Toggle word wrap

2. Upload the image to AWS:

   # image-builder upload <image-name> \

   Copy to Clipboard Toggle word wrap

Verification

1. Confirm that the image upload was successful by accessing EC2 on the menu and selecting the correct region in the AWS console. The image must have the available status to indicate that it was successfully uploaded.
2. On the dashboard, select your image and click Launch.

Procedure

1. In the RHEL image builder dashboard, click the blueprint name that you previously created.
2. Select the tab Images.

3. Click Create Image to create your customized image.

   The Create Image window opens.

   1. From the Type drop-down menu list, select Amazon Machine Image Disk (.raw).
   2. Check the Upload to AWS checkbox to upload your image to the AWS Cloud and click Next.

   3. To authenticate your access to AWS, type your AWS access key ID and AWS secret access key in the corresponding fields. Click Next.

      Note

      You can view your AWS secret access key only when you create a new Access Key ID. If you do not know your Secret Key, generate a new Access Key ID.

   4. Type the name of the image in the Image name field, type the Amazon bucket name in the Amazon S3 bucket name field, and type the AWS region field for the bucket you are going to add your customized image to. Click Next.

   5. Review the information and click Finish.

      Optionally, click Back to modify any incorrect details.

      Note

      You must have the correct IAM settings for the bucket you are going to send your customized image to. This procedure uses the IAM Import and Export, so you have to set up a policy for your bucket before you are able to upload images to it. For more information, see Required Permissions for IAM Users.

4. A pop-up on the upper right informs you of the saving progress. It also informs that the image creation has been initiated, the progress of this image creation, and the subsequent upload to the AWS Cloud.

   After the process is complete, you can see the Image build complete status.

5. In a browser, access Service→EC2.

   1. On the AWS console dashboard menu, choose the correct region. The image must have the Available status to indicate that it is uploaded.
   2. On the AWS dashboard, select your image and click Launch.
6. A new window opens. Choose an instance type according to the resources you need to start your image. Click Review and Launch.
7. Review your instance start details. You can edit each section if you need to make any changes. Click Launch.

8. Before you start the instance, select a public key to access it.

   You can either use the key pair you already have or you can create a new key pair.

   Follow the next steps to create a new key pair in EC2 and attach it to the new instance.

   1. From the drop-down menu list, select Create a new key pair.
   2. Enter the name of the new key pair. It generates a new key pair.
   3. Click Download Key Pair to save the new key pair on your local system.

9. Then, you can click Launch Instance to start your instance.

   You can check the status of the instance, which displays as Initializing.

10. After the instance status is running, the Connect button becomes available.

11. Click Connect. A window is displayed with instructions on how to connect by using SSH.

    1. Select A standalone SSH client as the preferred connection method to open a terminal.

    2. In the location where you store your private key, ensure that your key is publicly viewable for SSH to work. To do so, run the command:

       $ chmod 400 <your_instance_name>.pem

       Copy to Clipboard Toggle word wrap

    3. Connect to your instance by using its Public DNS:

       $ ssh -i <your-instance_name>.pem ec2-user@<your-instance-IP-address>

       Copy to Clipboard Toggle word wrap

    4. Type yes to confirm that you want to continue connecting.

       As a result, you are connected to your instance over SSH.

Procedure

1. Configure the Red Hat Enterprise Linux (RHEL) VM:

   1. To install from the command line (CLI), ensure that you set the default memory, network interfaces, and CPUs according to your requirement for the VM. For details, see Creating virtual machines by using the command line
   2. To install from the web console, see Creating virtual machines by using the web console

2. When the installation starts:

   1. Create a root password.
   2. Create an administrative user account.
3. After the installation completes, reboot the VM and log in to the root account.
4. After logging in as root, you can configure the image.

5. Register the VM and enable the RHEL repository:

   # subscription-manager register

   Copy to Clipboard Toggle word wrap

6. For AMD64 or Intel 64 (x86_64) VMs, install the nvme, xen-netfront, and xen-blkfront drivers:

   # dracut -f --add-drivers "nvme xen-netfront xen-blkfront"

   Copy to Clipboard Toggle word wrap

7. For ARM 64 (aarch64) VMs, install the nvme driver:

   # dracut -f --add-drivers "nvme"

   Copy to Clipboard Toggle word wrap

   Including these drivers prevents a dracut time-out.

   Alternatively, you can add the drivers to /etc/dracut.conf.d/ and then enter dracut -f to overwrite the existing initramfs file.

Procedure

1. Remove the AWS Red Hat Update Infrastructure (RHUI) client.

   $ sudo -i
   # dnf -y remove rh-amazon-rhui-client

   Copy to Clipboard Toggle word wrap

2. Register the VM with Red Hat.

   # subscription-manager register

   Copy to Clipboard Toggle word wrap

3. Disable all repositories.

   # subscription-manager repos --disable=

   Copy to Clipboard Toggle word wrap

4. Enable the RHEL 10 Server HA repositories.

   # subscription-manager repos --enable=rhel-10-for-x86_64-highavailability-rpms

   Copy to Clipboard Toggle word wrap

5. Update the RHEL AWS instance.

   # dnf update -y

   Copy to Clipboard Toggle word wrap

6. Install the Red Hat High Availability Add-On software packages, along with the AWS fencing agent from the High Availability channel.

   # dnf install pcs pacemaker fence-agents-aws

   Copy to Clipboard Toggle word wrap

7. The user hacluster was created during the pcs and pacemaker installation in the earlier step. Create a password for hacluster on all cluster nodes. Use the same password for all nodes.

   # passwd hacluster

   Copy to Clipboard Toggle word wrap

8. Add the high availability service to the RHEL Firewall if firewalld.service is installed.

   # firewall-cmd --permanent --add-service=high-availability
   # firewall-cmd --reload

   Copy to Clipboard Toggle word wrap

9. Start the pcs service and enable it to start on boot.

   # systemctl start pcsd.service
   # systemctl enable pcsd.service

   Copy to Clipboard Toggle word wrap
10. Edit /etc/hosts and add Red Hat Enterprise Linux (RHEL) host names and internal IP addresses. See How should the /etc/hosts file be set up on RHEL cluster nodes? for details.

Procedure

1. Authenticate the pcs user hacluster for each node in the cluster on the node from which you will be running pcs.

   The following command authenticates user hacluster on z1.example.com for both of the nodes in a two-node cluster that will consist of z1.example.com and z2.example.com.

   [root@z1 ~]# pcs host auth z1.example.com z2.example.com
   Username: hacluster
   Password:
   z1.example.com: Authorized
   z2.example.com: Authorized

   Copy to Clipboard Toggle word wrap

2. Execute the following command from z1.example.com to create the two-node cluster my_cluster that consists of nodes z1.example.com and z2.example.com. This will propagate the cluster configuration files to both nodes in the cluster. This command includes the --start option, which will start the cluster services on both nodes in the cluster.

   [root@z1 ~]# pcs cluster setup my_cluster --start z1.example.com z2.example.com

   Copy to Clipboard Toggle word wrap

3. Enable the cluster services to run on each node in the cluster when the node is booted.

   Note

   For your particular environment, you can skip this step by keeping the cluster services disabled. If enabled and a node goes down, any issues with your cluster or your resources are resolved before the node rejoins the cluster. If you keep the cluster services disabled, you need to manually start the services when you reboot a node by executing the pcs cluster start command on that node.

   [root@z1 ~]# pcs cluster enable --all

   Copy to Clipboard Toggle word wrap

4. Display the status of the cluster you created with the pcs cluster status command. Because there could be a slight delay before the cluster is up and running when you start the cluster services with the --start option of the pcs cluster setup command, you should ensure that the cluster is up and running before performing any subsequent actions on the cluster and its configuration.

   [root@z1 ~]# pcs cluster status
   Cluster Status:
    Stack: corosync
    Current DC: z2.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum
    Last updated: Thu Oct 11 16:11:18 2018
    Last change: Thu Oct 11 16:11:00 2018 by hacluster via crmd on z2.example.com
    2 Nodes configured
    0 Resources configured
   
   ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a shared block volume in the us-east-1a availability zone:

   $ aws ec2 create-volume --availability-zone <us_east_1a> --no-encrypted --size <1024> --volume-type <io1> --iops <51200> --multi-attach-enabled
   
   {
       "AvailabilityZone": "us-east-1a",
       "CreateTime": "2020-08-27T19:16:42.000Z",
       "Encrypted": false,
       "Size": 1024,
       "SnapshotId": "",
       "State": "creating",
       "VolumeId": "vol-042a5652867304f09",
       "Iops": 51200,
       "Tags": [ ],
       "VolumeType": "io1"
   }

   Copy to Clipboard Toggle word wrap

2. For each instance in your cluster, attach a shared block volume. Use your <instance_id> and <volume_id>, for example vol-042a5652867304f09 to instance i-0eb803361c2c887f2:

   $ aws ec2 attach-volume --device /dev/xvdd --instance-id i-0eb803361c2c887f2 --volume-id vol-042a5652867304f09
   
   {
       "AttachTime": "2020-08-27T19:26:16.086Z",
       "Device": "/dev/xvdd",
       "InstanceId": "i-0eb803361c2c887f2",
       "State": "attaching",
       "VolumeId": "vol-042a5652867304f09"
   }

   Copy to Clipboard Toggle word wrap

Verification

1. Verify the block device is available by using the ssh command with your instance IP 198.51.100.3 for each instance in the cluster:

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T '"
   
   node a
   nvme2n1 259:1    0   1T  0 disk

   Copy to Clipboard Toggle word wrap

2. Use the ssh command to verify that each instance in your cluster uses the same shared disk:

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T ' | awk '{print \$1}' | xargs -i udevadm info --query=all --name=/dev/{} | grep '^E: ID_SERIAL='"
   
   nodea
   E: ID_SERIAL=Amazon Elastic Block Store_vol0fa5342e7aedf09f7

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the platform status of the RHEL Marketplace AMI instance:

   $ mokutil --sb-state
   
   SecureBoot disabled
   Platform is in Setup Mode

   Copy to Clipboard Toggle word wrap

   The setup mode allows updating the Secure Boot UEFI variables within the instance.

2. Create a new random universally unique identifier (UUID) and store it in a system-generated text file:

   $ uuidgen --random > GUID.txt

   Copy to Clipboard Toggle word wrap

3. Generate a new PK.key RSA private key and a self-signed PK.cer X.509 certificate for the Platform Key database:

   $ openssl req -quiet \
   -newkey rsa:3072 \
   -nodes -keyout PK.key \
   -new -x509 -sha256 \
   -days 3650 \
   -subj "/CN=Platform key/" \
   -outform DER -out PK.cer

   Copy to Clipboard Toggle word wrap

   The openssl utility generates a common name Platform key for the certificate by setting output format to Distinguished Encoding Rules (DER).

4. Generate a new KEK.key RSA private key and a self-signed KEK.cer X.509 certificate for the Key Exchange Key database:

   $ openssl req -quiet \
   -newkey rsa:3072 \
   -nodes -keyout KEK.key \
   -new -x509 -sha256 \
   -days 3650 \
   -subj "CN=Key Exchange Key/" \
   -outform DER -out KEK.cer

   Copy to Clipboard Toggle word wrap

5. Generate a custom_db.cer custom certificate:

   $ openssl req -quiet \
   -newkey rsa:3072 \
   -nodes -keyout custom_db.key \
   -new -x509 -sha256 \
   -days 3650 \
   -subj "/CN=Signature Database key/" \
   --outform DER -out custom_db.cer

   Copy to Clipboard Toggle word wrap

6. Download the Microsoft Corporation UEFI CA 2011 Certificate:

   $ wget https://go.microsoft.com/fwlink/p/?linkid=321194 --user-agent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" -O MicCorUEFCA2011_2011-06-27.crt

   Copy to Clipboard Toggle word wrap

7. Download the updated UEFI Revocation List File of forbidden signatures (dbx) for x64 bits system:

   $ wget https://uefi.org/sites/default/files/resources/x64_DBXUpdate.bin

   Copy to Clipboard Toggle word wrap

8. Generate UEFI variables file using the virt-fw-vars utility:

   $ virt-fw-vars --set-pk "$(< GUID.txt)" PK.cer \
   --add-kek "$(< GUID.txt)" KEK.cer \
   --add-db "$(< GUID.txt)" custom_db.cer \
   --add-db 77fa9abd-0359-4d32-bd60-28f4e78f784b MicCorUEFCA2011_2011-06-27.crt \
   --set-dbx DBXUpdate.bin -i /usr/share/edk2/ovmf/OVMF_VARS.secboot.fd \
   --output VARS

   Copy to Clipboard Toggle word wrap

   For details, see the virt-fw-vars(1) man page on your system.

9. Convert UEFI variables to the Extensible Firmware Interface (EFI) Signature List (ESL) format:

   $ python3 /usr/share/doc/python3-virt-firmware/experimental/authfiles.py \
   --input VARS \
   --outdir .
   $ for f in PK KEK db dbx; do tail -c +41 $f.auth > $f.esl; done

   Copy to Clipboard Toggle word wrap
   Note

   Each GUID is an assigned value and represents an EFI parameter

   • 8be4df61-93ca-11d2-aa0d-00e098032b8c: EFI_GLOBAL_VARIABLE_GUID
   • d719b2cb-3d3a-4596-a3bc-dad00e67656f: EFI_IMAGE_SECURITY_DATABASE_GUID

   The EFI_GLOBAL_VARIABLE_GUID parameter maintains settings of the bootable devices and boot managers, while the EFI_IMAGE_SECURITY_DATABASE_GUID parameter represents the image security database for Secure Boot variables db, dbx, and storage of required keys and certificates.

10. Transfer the database certificates to the target instance, use the efivar utility to manage UEFI environment variables.

    1. To transfer PK.esl, enter:

       # efivar -w -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-PK -f PK.esl

       Copy to Clipboard Toggle word wrap

    2. To transfer KEK.esl, enter:

       # efivar -w -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-KEK -f KEK.esl

       Copy to Clipboard Toggle word wrap

    3. To transfer db.esl, enter:

       # efivar -w -n d719b2cb-3d3a-4596-a3bc-dad00e67656f-db -f db.esl

       Copy to Clipboard Toggle word wrap

    4. To transfer the dbx.esl UEFI revocation list file for x64 architecture, enter:

       # efivar -w -n d719b2cb-3d3a-4596-a3bc-dad00e67656f-dbx -f dbx.esl

       Copy to Clipboard Toggle word wrap
11. Reboot the instance from the AWS console.

Verification

1. Verify if Secure Boot is enabled:

   $ mokutil --sb-state
   
   SecureBoot enabled

   Copy to Clipboard Toggle word wrap

2. Use the keyctl utility to verify the kernel keyring for the custom certificate:

   $ sudo keyctl list %:.platform
   
   4 keys in keyring:
   786569360: ---lswrv     0     0 asymmetric: Signature Database key: 5856827178d376838611787277dc1d090c575759
   ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new random universally unique identifier (UUID) and store it in a system-generated text file:

   $ uuidgen --random > GUID.txt

   Copy to Clipboard Toggle word wrap

2. Generate a new RSA private key PK.key and a self-signed X.509 certificate PK.cer for the platform key database:

   $ openssl req -quiet \
   -newkey rsa:3072 \
   -nodes -keyout PK.key \
   -new -x509 -sha256 \
   -days 3650 \
   -outform DER -out PK.cer

   Copy to Clipboard Toggle word wrap

   The openssl utility generates a common name platform key for the certificate by setting output format to Distinguished Encoding Rules (DER).

3. Generate a new RSA private key KEK.key and a self-signed X.509 certificate KEK.cer for the Key Exchange Key database:

   $ openssl req -quiet \
   -newkey rsa:3072 \
   -nodes -keyout KEK.key \
   -new -x509 -sha256 \
   -days 3650 \
   -subj "/CN=Key Exchange Key/" \
   -outform DER -out KEK.cer

   Copy to Clipboard Toggle word wrap

4. Generate a custom certificate custom_db.cer:

   $ openssl req -quiet \
   -newkey rsa:3072 \
   -nodes -keyout custom_db.key \
   -new -x509 -sha256 \
   -days 3650 -subj "/CN=Signature Database key/" \
   --outform DER -out custom_db.cer

   Copy to Clipboard Toggle word wrap

5. Download the updated UEFI Revocation List File of forbidden signatures (dbx) for 64 bit system:

   $ wget https://uefi.org/sites/default/files/resources/x64_DBXUpdate.bin

   Copy to Clipboard Toggle word wrap

6. Use the virt-fw-vars utility to generate the aws_blob.bin binary file from keys, database certificates, and the UEFI variable store:

   $ virt-fw-vars --output-aws aws_blob.bin \
   --set-pk "$(< GUID.txt)" PK.cer \
   --add-kek "$(< GUID.txt)" KEK.cer \
   --add-db "$(< GUID.txt)" custom_db.cer \
   --add-db 77fa9abd-0359-4d32-bd60-28f4e78f784b MicCorUEFCA2011_2011-06-27.crt \
   --set-dbx x64_DBXUpdate.bin

   Copy to Clipboard Toggle word wrap

   The customized blob consists of:

   • PK.cer with a self-signed X.509 certificate
   • KEK.cer and custom_db.cer with owner group GUID and Privacy Enhanced Mail (pem) format
   • x64_DBXUpdate.bin list downloaded from database of excluded signatures (dbx).
   • The 77fa9abd-0359-4d32-bd60-28f4e78f784b UUID is for MicCorUEFCA2011_2011-06-27.crt Microsoft Corporation UEFI Certification Authority 2011.

7. Use the awscli2 utility to create and register the AMI from a disk snapshot with the required Secure Boot variables:

   $ aws ec2 register-image \
   --name rhel-10.0-secure-boot \
   --architecture x86_64 \
   --virtualization-type hvm \
   --root-device-name "/dev/sda1" \
   --block-device-mappings "{\"DeviceName\": \"/dev/sda1\",\"Ebs\": {\"SnapshotId\": \"<snap-02d4db3813ff9b98e>\"}}" \
   --ena-support --boot-mode uefi \
   --region eu-central-1 \
   --uefi-data $(cat aws_blob.bin)
   {
       "ImageId": "<ami-09125e5689d5fdf2b>"
   }

   Copy to Clipboard Toggle word wrap
8. Reboot the instance from the AWS Console.

Verification

1. Verify Secure Boot functionality:

   $ mokutil --sb-state
   SecureBoot enabled

   Copy to Clipboard Toggle word wrap

2. Use the keyctl utility to verify the kernel keyring for the custom certificate:

   $ sudo keyctl list %:.platform
   
   4 keys in keyring:
   216534498: ---lswrv     0     0 asymmetric: Signature Database key: 5856827178d376838611787277dc1d090c575759
   ...

   Copy to Clipboard Toggle word wrap

Procedure

1. UEFI Firmware: The boot process starts from the Unified Extensible Firmware Interface (UEFI) firmware. For boot, Red Hat Enterprise Linux (RHEL) UKI requires UEFI firmware, because legacy basic input/output system (BIOS) firmware is not supported.
2. Shim boot loader: Use the shim boot loader for booting rather than directly booting the UKI (a PE binary) from the UEFI firmware. shim includes additional security mechanisms such as Machine Owner Key (MOK) and Secure Boot Advanced Targeting (SBAT).
3. Signature verification (Secure Boot UEFI mechanism): During boot, shim reads the UKI binary and the secure boot UEFI mechanism verifies the signature of UKI against trusted keys stored in the Secure Boot Allowed Signature Database (db) of the system, the MOK database, and the built-in database of the shim binary. If the signature key is valid, the verification passes.

4. SBAT verification: Immediately after signature verification, the shim boot loader verifies the SBAT rules at startup.

   During SBAT verification, the system compares generation numbers, for components such as systemd.rhel or linux.rhel, embedded in the UKI by using the .sbat section against values in the shim boot loader. If the generation number for a component in the shim is higher than the generation number in the UKI, the binary gets automatically discarded, even if it was signed by a trusted key.

   Note that the generation number is a version identifier for UEFI applications, such as shim and grub.

5. Unpacking and Execution: If verification passes, control passes from shim to the systemd-stub code in the UKI to continue the boot process.

6. systemd-stub add-ons: During execution, systemd-stub unpacks and extracts the contents of the .cmdline section, the plain text kernel command line, and the .initrd section, the temporary root file system, for the boot process.

   Note that systemd-stub reads UKI add-ons, which are also PE binaries, and verifies their signatures to safely extend the kernel command line of UKI by appending the .cmdline content from add-ons. systemd-stub reads add-ons from two locations:

   • Global (UKI-independent) add-ons from the /loader/addons/ directory on the Extensible Firmware Interface (EFI) System Partition (ESP).
   • Per-UKI add-ons from the /EFI/Linux/<UKI-name>.extra.d/ directory on the ESP.

7. Control passes from systemd-stub to the Linux kernel and the operating system boot process continues.

   From this point, secure boot with UKI mechanisms follows the standard kernel boot process.