Chapter 12. Managing control plane machines

Procedure

1. In the web portal or console for your cloud provider, take the following actions:

   1. Locate and delete the appropriate load balancer component:

      • For AWS, delete the external load balancer. The API DNS entry in the private zone already points to the internal load balancer, which uses an identical configuration, so you do not need to modify the internal load balancer.
   2. Delete the api.$clustername.$yourdomain DNS entry in the public zone.

2. Remove the external load balancers by deleting the following lines in the control plane machine set custom resource:

   providerSpec:
     value:
       loadBalancers:
       - name: lk4pj-ext 1
         type: network 2
       - name: lk4pj-int
         type: network

   1 2

   Delete this line.

Procedure

1. Edit the following line under the providerSpec field:

   providerSpec:
     value:
       ...
       instanceType: <compatible_aws_instance_type> 1

   1

   Specify a larger AWS instance type with the same base as the previous selection. For example, you can change m6i.xlarge to m6i.2xlarge or m6i.4xlarge.

2. Save your changes.

Procedure

1. In a text editor, open the YAML file for an existing machine set or create a new one.

2. Edit the following lines under the providerSpec field:

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   # ...
   spec:
     template:
       spec:
         providerSpec:
           value:
             instanceType: <supported_instance_type> 1
             networkInterfaceType: EFA 2
             placement:
               availabilityZone: <zone> 3
               region: <region> 4
             placementGroupName: <placement_group> 5
   # ...

   1

   Specify an instance type that supports EFAs.

   2

   Specify the EFA network interface type.

   3

   Specify the zone, for example, us-east-1a.

   4

   Specify the region, for example, us-east-1.

   5

   Specify the name of the existing AWS placement group to deploy machines in.

Procedure

1. In the web portal or console for your cloud provider, take the following actions:

   1. Locate and delete the appropriate load balancer component:

      • For Azure, delete the api-internal rule for the load balancer.
   2. Delete the api.$clustername.$yourdomain DNS entry in the public zone.

2. Remove the external load balancers by deleting the following lines in the control plane machine set custom resource:

   providerSpec:
     value:
       loadBalancers:
       - name: lk4pj-ext 1
         type: network 2
       - name: lk4pj-int
         type: network

   1 2

   Delete this line.

Procedure

1. Display all of the available OpenShift Container Platform images by running one of the following commands:

   • North America:

     $  az vm image list --all --offer rh-ocp-worker --publisher redhat -o table

     Example output

     Offer          Publisher       Sku                 Urn                                                             Version
     -------------  --------------  ------------------  --------------------------------------------------------------  -----------------
     rh-ocp-worker  RedHat          rh-ocp-worker       RedHat:rh-ocp-worker:rh-ocp-worker:413.92.2023101700            413.92.2023101700
     rh-ocp-worker  RedHat          rh-ocp-worker-gen1  RedHat:rh-ocp-worker:rh-ocp-worker-gen1:413.92.2023101700       413.92.2023101700

   • EMEA:

     $  az vm image list --all --offer rh-ocp-worker --publisher redhat-limited -o table

     Example output

     Offer          Publisher       Sku                 Urn                                                                     Version
     -------------  --------------  ------------------  --------------------------------------------------------------          -----------------
     rh-ocp-worker  redhat-limited  rh-ocp-worker       redhat-limited:rh-ocp-worker:rh-ocp-worker:413.92.2023101700            413.92.2023101700
     rh-ocp-worker  redhat-limited  rh-ocp-worker-gen1  redhat-limited:rh-ocp-worker:rh-ocp-worker-gen1:413.92.2023101700       413.92.2023101700

   Note

   Regardless of the version of OpenShift Container Platform that you install, the correct version of the Azure Marketplace image to use is 4.13. If required, your VMs are automatically upgraded as part of the installation process.

2. Inspect the image for your offer by running one of the following commands:

   • North America:

     $ az vm image show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

   • EMEA:

     $ az vm image show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

3. Review the terms of the offer by running one of the following commands:

   • North America:

     $ az vm image terms show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

   • EMEA:

     $ az vm image terms show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

4. Accept the terms of the offering by running one of the following commands:

   • North America:

     $ az vm image terms accept --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

   • EMEA:

     $ az vm image terms accept --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

5. Record the image details of your offer, specifically the values for publisher, offer, sku, and version.

6. Add the following parameters to the providerSpec section of your machine set YAML file using the image details for your offer:

   Sample providerSpec image values for Azure Marketplace machines

   providerSpec:
     value:
       image:
         offer: rh-ocp-worker
         publisher: redhat
         resourceID: ""
         sku: rh-ocp-worker
         type: MarketplaceWithPlan
         version: 413.92.2023101700

1. Using the secureBoot field.
2. Using the virtualizedTrustedPlatformModule field.

Procedure

1. In a text editor, open the YAML file for an existing machine set or create a new one.

2. Edit the following section under the providerSpec field to provide a valid configuration:

   Sample valid configuration with UEFI Secure Boot and vTPM enabled

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   # ...
   spec:
     template:
       spec:
         providerSpec:
           value:
             securityProfile:
               settings:
                 securityType: TrustedLaunch 1
                 trustedLaunch:
                   uefiSettings: 2
                     secureBoot: Enabled 3
                     virtualizedTrustedPlatformModule: Enabled 4
   # ...

   1

   Enables the use of trusted launch for Azure virtual machines. This value is required for all valid configurations.

   2

   Specifies which UEFI security features to use. This section is required for all valid configurations.

   3

   Enables UEFI Secure Boot.

   4

   Enables the use of a vTPM.

Procedure

1. In a text editor, open the YAML file for an existing machine set or create a new one.

2. Edit the following section under the providerSpec field:

   Sample configuration

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   # ...
   spec:
     template:
       spec:
         providerSpec:
           value:
             osDisk:
               # ...
               managedDisk:
                 securityProfile: 1
                   securityEncryptionType: VMGuestStateOnly 2
               # ...
             securityProfile: 3
               settings:
                   securityType: ConfidentialVM 4
                   confidentialVM:
                     uefiSettings: 5
                       secureBoot: Disabled 6
                       virtualizedTrustedPlatformModule: Enabled 7
             vmSize: Standard_DC16ads_v5 8
   # ...

   1

   Specifies security profile settings for the managed disk when using a confidential VM.

   2

   Enables encryption of the Azure VM Guest State (VMGS) blob. This setting requires the use of vTPM.

   3

   Specifies security profile settings for the confidential VM.

   4

   Enables the use of confidential VMs. This value is required for all valid configurations.

   5

   Specifies which UEFI security features to use. This section is required for all valid configurations.

   6

   Disables UEFI Secure Boot.

   7

   Enables the use of a vTPM.

   8

   Specifies an instance type that supports confidential VMs.

Procedure

1. In a text editor, open the YAML file for an existing machine set or create a new one.

2. Edit the following section under the providerSpec field:

   Sample configuration

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   # ...
   spec:
     template:
       machines_v1beta1_machine_openshift_io:
         spec:
           providerSpec:
             value:
               capacityReservationGroupID: <capacity_reservation_group> 1
   # ...

   1

   Specify the ID of the Capacity Reservation group that you want the machine set to deploy machines on.

Procedure

1. In a text editor, open the YAML file for an existing machine set or create a new one.

2. Edit the following line under the providerSpec field:

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   ...
   spec:
     template:
       spec:
         providerSpec:
           value:
             disks:
               type: pd-ssd 1

   1

   Control plane nodes must use the pd-ssd disk type.

Procedure

1. In a text editor, open the YAML file for an existing machine set or create a new one.

2. Edit the following section under the providerSpec field:

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   ...
   spec:
     template:
       spec:
         providerSpec:
           value:
             confidentialCompute: Enabled 1
             onHostMaintenance: Terminate 2
             machineType: n2d-standard-8 3
   ...

   1

   Specify whether Confidential VM is enabled. Valid values are Disabled or Enabled.

   2

   Specify the behavior of the VM during a host maintenance event, such as a hardware or software update. For a machine that uses Confidential VM, this value must be set to Terminate, which stops the VM. Confidential VM does not support live VM migration.

   3

   Specify a machine type that supports Confidential VM. Confidential VM supports the N2D and C2D series of machine types.

Procedure

1. In a text editor, open the YAML file for an existing machine set or create a new one.

2. Edit the following section under the providerSpec field:

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   # ...
   spec:
     template:
       spec:
         providerSpec:
           value:
             shieldedInstanceConfig: 1
               integrityMonitoring: Enabled 2
               secureBoot: Disabled 3
               virtualizedTrustedPlatformModule: Enabled 4
   # ...

   1

   In this section, specify any Shielded VM options that you want.

   2

   Specify whether integrity monitoring is enabled. Valid values are Disabled or Enabled.

   Note

   When integrity monitoring is enabled, you must not disable virtual trusted platform module (vTPM).

   3

   Specify whether UEFI Secure Boot is enabled. Valid values are Disabled or Enabled.

   4

   Specify whether vTPM is enabled. Valid values are Disabled or Enabled.

Procedure

1. To allow a specific service account to use your KMS key and to grant the service account the correct IAM role, run the following command with your KMS key name, key ring name, and location:

   $ gcloud kms keys add-iam-policy-binding <key_name> \
     --keyring <key_ring_name> \
     --location <key_ring_location> \
     --member "serviceAccount:service-<project_number>@compute-system.iam.gserviceaccount.com” \
     --role roles/cloudkms.cryptoKeyEncrypterDecrypter

2. Configure the encryption key under the providerSpec field in your machine set YAML file. For example:

   apiVersion: machine.openshift.io/v1
   kind: ControlPlaneMachineSet
   ...
   spec:
     template:
       spec:
         providerSpec:
           value:
             disks:
             - type:
               encryptionKey:
                 kmsKey:
                   name: machine-encryption-key 1
                   keyRing: openshift-encrpytion-ring 2
                   location: global 3
                   projectID: openshift-gcp-project 4
                 kmsKeyServiceAccount: openshift-service-account@openshift-gcp-project.iam.gserviceaccount.com 5

   1

   The name of the customer-managed encryption key that is used for the disk encryption.

   2

   The name of the KMS key ring that the KMS key belongs to.

   3

   The GCP location in which the KMS key ring exists.

   4

   Optional: The ID of the project in which the KMS key ring exists. If a project ID is not set, the machine set projectID in which the machine set was created is used.

   5

   Optional: The service account that is used for the encryption request for the given KMS key. If a service account is not set, the Compute Engine default service account is used.

   When a new machine is created by using the updated providerSpec object configuration, the disk encryption key is encrypted with the KMS key.

Procedure

1. Edit the following line under the providerSpec field:

   providerSpec:
     value:
   # ...
       flavor: m1.xlarge 1

   1

   Specify a RHOSP flavor type that has the same base as the existing selection. For example, you can change m6i.xlarge to m6i.2xlarge or m6i.4xlarge. You can choose larger or smaller flavors depending on your vertical scaling needs.

2. Save your changes.

1. Nutanix has a failure domain concept, but OpenShift Container Platform 4.14 does not include support for this feature.

1. A control plane machine is slated for deletion.
2. The control plane machine enters the Deleting phase.

3. To satisfy the preDrain lifecycle hook, the etcd Operator takes the following actions:

   1. The etcd Operator waits until a fourth control plane machine is added to the cluster as an etcd member. This new etcd member has a state of Running but not ready until it receives the full database update from the etcd leader.
   2. When the new etcd member receives the full database update, the etcd Operator promotes the new etcd member to a voting member and removes the old etcd member from the cluster.

   After this transition is complete, it is safe for the old etcd pod and its data to be removed, so the preDrain lifecycle hook is removed.

4. The control plane machine status condition Drainable is set to True.

5. The machine controller attempts to drain the node that is backed by the control plane machine.

   • If draining fails, Drained is set to False and the machine controller attempts to drain the node again.
   • If draining succeeds, Drained is set to True.
6. The control plane machine status condition Drained is set to True.
7. If no other Operators have added a preTerminate lifecycle hook, the control plane machine status condition Terminable is set to True.
8. The machine controller removes the instance from the infrastructure provider.
9. The machine controller deletes the Node object.

Procedure

1. For all control plane machines, edit the provider spec for all control plane machines that match the environment. For example, to edit the machine master-0, enter the following command:

   $ oc edit machine/<cluster_id>-master-0 -n openshift-machine-api

   where:

   <cluster_id>

   Specifies the ID of the upgraded cluster.

2. In the provider spec, set the value of the property rootVolume.availabilityZone to the volume of the availability zone you want to use.

   An example RHOSP provider spec

   providerSpec:
     value:
       apiVersion: machine.openshift.io/v1alpha1
       availabilityZone: az0
         cloudName: openstack
       cloudsSecret:
         name: openstack-cloud-credentials
         namespace: openshift-machine-api
       flavor: m1.xlarge
       image: rhcos-4.14
       kind: OpenstackProviderSpec
       metadata:
         creationTimestamp: null
       networks:
       - filter: {}
         subnets:
         - filter:
             name: refarch-lv7q9-nodes
             tags: openshiftClusterID=refarch-lv7q9
       rootVolume:
           availabilityZone: nova 1
           diskSize: 30
           sourceUUID: rhcos-4.12
           volumeType: fast-0
       securityGroups:
       - filter: {}
         name: refarch-lv7q9-master
       serverGroupName: refarch-lv7q9-master
       serverMetadata:
         Name: refarch-lv7q9-master
         openshiftClusterID: refarch-lv7q9
       tags:
       - openshiftClusterID=refarch-lv7q9
       trunk: true
       userDataSecret:
         name: master-user-data

   1

   Set the zone name as this value.

   Note

   If you edited or recreated machine resources after your initial cluster deployment, you might have to adapt these steps for your configuration.

   In your RHOSP cluster, find the availability zone of the root volumes for your machines and use that as the value.

3. Run the following command to retrieve information about the control plane machine set resource:

   $ oc describe controlplanemachineset.machine.openshift.io/cluster --namespace openshift-machine-api

4. Run the following command to edit the resource:

   $ oc edit controlplanemachineset.machine.openshift.io/cluster --namespace openshift-machine-api

5. For that resource, set the value of the spec.state property to Active to activate control plane machine sets for your cluster.

Procedure

1. For the master-1 and master-2 control plane machines, open the provider specs for editing. For example, to edit the first machine, enter the following command:

   $ oc edit machine/<cluster_id>-master-1 -n openshift-machine-api

   where:

   <cluster_id>

   Specifies the ID of the upgraded cluster.

2. For the master-1 and master-2 control plane machines, edit the value of the serverGroupName property in their provider specs to match that of the machine master-0.

   An example RHOSP provider spec

   providerSpec:
     value:
       apiVersion: machine.openshift.io/v1alpha1
       availabilityZone: az0
         cloudName: openstack
       cloudsSecret:
         name: openstack-cloud-credentials
         namespace: openshift-machine-api
       flavor: m1.xlarge
       image: rhcos-4.14
       kind: OpenstackProviderSpec
       metadata:
         creationTimestamp: null
       networks:
       - filter: {}
         subnets:
         - filter:
             name: refarch-lv7q9-nodes
             tags: openshiftClusterID=refarch-lv7q9
       securityGroups:
       - filter: {}
         name: refarch-lv7q9-master
       serverGroupName: refarch-lv7q9-master-az0 1
       serverMetadata:
         Name: refarch-lv7q9-master
         openshiftClusterID: refarch-lv7q9
       tags:
       - openshiftClusterID=refarch-lv7q9
       trunk: true
       userDataSecret:
         name: master-user-data

   1

   This value must match for machines master-0, master-1, and master-3.

   Note

   If you edited or recreated machine resources after your initial cluster deployment, you might have to adapt these steps for your configuration.

   In your RHOSP cluster, find the server group that your control plane instances are in and use that as the value.

3. Run the following command to retrieve information about the control plane machine set resource:

   $ oc describe controlplanemachineset.machine.openshift.io/cluster --namespace openshift-machine-api

4. Run the following command to edit the resource:

   $ oc edit controlplanemachineset.machine.openshift.io/cluster --namespace openshift-machine-api

5. For that resource, set the value of the spec.state property to Active to activate control plane machine sets for your cluster.