Chapter 12. Managing control plane machines

The control plane machine set concept of a failure domain is analogous to existing AWS concept of an Availability Zone (AZ). The ControlPlaneMachineSet CR spreads control plane machines across multiple failure domains when possible.

When configuring AWS failure domains in the control plane machine set, you must specify the availability zone name and the subnet to use.

failureDomains:
  aws:
  - placement:
      availabilityZone: <aws_zone_a> 1
    subnet: 2
      filters:
      - name: tag:Name
        values:
        - <cluster_id>-private-<aws_zone_a> 3
      type: Filters 4
  - placement:
      availabilityZone: <aws_zone_b> 5
    subnet:
      filters:
      - name: tag:Name
        values:
        - <cluster_id>-private-<aws_zone_b> 6
      type: Filters
  platform: AWS 7

When you create a control plane machine set for an existing cluster, the provider specification must match the providerSpec configuration in the control plane machine CR that is created by the installation program. You can omit any field that is set in the failure domain section of the CR.

In the following example, <cluster_id> is the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

providerSpec:
  value:
    ami:
      id: ami-<ami_id_string> 1
    apiVersion: machine.openshift.io/v1beta1
    blockDevices:
    - ebs: 2
        encrypted: true
        iops: 0
        kmsKey:
          arn: ""
        volumeSize: 120
        volumeType: gp3
    credentialsSecret:
      name: aws-cloud-credentials 3
    deviceIndex: 0
    iamInstanceProfile:
      id: <cluster_id>-master-profile 4
    instanceType: m6i.xlarge 5
    kind: AWSMachineProviderConfig 6
    loadBalancers: 7
    - name: <cluster_id>-int
      type: network
    - name: <cluster_id>-ext
      type: network
    metadata:
      creationTimestamp: null
    metadataServiceOptions: {}
    placement: 8
      region: <region> 9
    securityGroups:
    - filters:
      - name: tag:Name
        values:
        - <cluster_id>-master-sg 10
    subnet: {} 11
    userDataSecret:
      name: master-user-data 12

The control plane machine set concept of a failure domain is analogous to existing Azure concept of an Azure availability zone. The ControlPlaneMachineSet CR spreads control plane machines across multiple failure domains when possible.

When configuring Azure failure domains in the control plane machine set, you must specify the availability zone name.

failureDomains:
  azure: 1
  - zone: "1"
  - zone: "2"
  - zone: "3"
  platform: Azure 2

When you create a control plane machine set for an existing cluster, the provider specification must match the providerSpec configuration in the control plane Machine CR that is created by the installation program. You can omit any field that is set in the failure domain section of the CR.

In the following example, <cluster_id> is the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

providerSpec:
  value:
    acceleratedNetworking: true
    apiVersion: machine.openshift.io/v1beta1
    credentialsSecret:
      name: azure-cloud-credentials 1
      namespace: openshift-machine-api
    diagnostics: {}
    image: 2
      offer: ""
      publisher: ""
      resourceID: /resourceGroups/<cluster_id>-rg/providers/Microsoft.Compute/galleries/gallery_<cluster_id>/images/<cluster_id>-gen2/versions/412.86.20220930 3
      sku: ""
      version: ""
    internalLoadBalancer: <cluster_id>-internal 4
    kind: AzureMachineProviderSpec 5
    location: <region> 6
    managedIdentity: <cluster_id>-identity
    metadata:
      creationTimestamp: null
      name: <cluster_id>
    networkResourceGroup: <cluster_id>-rg
    osDisk: 7
      diskSettings: {}
      diskSizeGB: 1024
      managedDisk:
        storageAccountType: Premium_LRS
      osType: Linux
    publicIP: false
    publicLoadBalancer: <cluster_id> 8
    resourceGroup: <cluster_id>-rg
    subnet: <cluster_id>-master-subnet 9
    userDataSecret:
      name: master-user-data 10
    vmSize: Standard_D8s_v3
    vnet: <cluster_id>-vnet
    zone: "" 11

When you create a control plane machine set for an existing cluster, the provider specification must match the providerSpec configuration in the control plane machine CR that is created by the installation program.

providerSpec:
  value:
    apiVersion: machine.openshift.io/v1beta1
    credentialsSecret:
      name: vsphere-cloud-credentials 1
    diskGiB: 120 2
    kind: VSphereMachineProviderSpec 3
    memoryMiB: 16384 4
    metadata:
      creationTimestamp: null
    network: 5
      devices:
      - networkName: <vm_network_name>
    numCPUs: 4 6
    numCoresPerSocket: 4 7
    snapshot: ""
    template: <vm_template_name> 8
    userDataSecret:
      name: master-user-data 9
    workspace:
      datacenter: <vcenter_datacenter_name> 10
      datastore: <vcenter_datastore_name> 11
      folder: <path_to_vcenter_vm_folder> 12
      resourcePool: <vsphere_resource_pool> 13
      server: <vcenter_server_ip> 14

The RollingUpdate update strategy automatically propagates changes to your control plane configuration. This update strategy is the default configuration for the control plane machine set.

For clusters that use the RollingUpdate update strategy, the Operator creates a replacement control plane machine with the configuration that is specified in the CR. When the replacement control plane machine is ready, the Operator deletes the control plane machine that is marked for replacement. The replacement machine then joins the control plane.

If multiple control plane machines are marked for replacement, the Operator protects etcd health during replacement by repeating this replacement process one machine at a time until it has replaced each machine.

You can use the OnDelete update strategy to propagate changes to your control plane configuration by replacing machines manually. Manually replacing machines allows you to test changes to your configuration on a single machine before applying the changes more broadly.

For clusters that are configured to use the OnDelete update strategy, the Operator creates a replacement control plane machine when you delete an existing machine. When the replacement control plane machine is ready, the etcd Operator allows the existing machine to be deleted. The replacement machine then joins the control plane.

If multiple control plane machines are deleted, the Operator creates all of the required replacement machines simultaneously. The Operator maintains etcd health by preventing more than one machine being removed from the control plane at once.

After you deploy a cluster to Amazon Web Services (AWS), you can reconfigure the API server to use only the private zone.

You can change the Amazon Web Services (AWS) instance type that your control plane machines use by updating the specification in the control plane machine set custom resource (CR).

You can use machine sets to create machines that use a specific version of the Amazon EC2 Instance Metadata Service (IMDS). Machine sets can create machines that allow the use of both IMDSv1 and IMDSv2 or machines that require the use of IMDSv2.

You can create a machine set running on AWS that deploys machines as Dedicated Instances. Dedicated Instances run in a virtual private cloud (VPC) on hardware that is dedicated to a single customer. These Amazon EC2 instances are physically isolated at the host hardware level. The isolation of Dedicated Instances occurs even if the instances belong to different AWS accounts that are linked to a single payer account. However, other instances that are not dedicated can share hardware with Dedicated Instances if they belong to the same AWS account.

Instances with either public or dedicated tenancy are supported by the Machine API. Instances with public tenancy run on shared hardware. Public tenancy is the default tenancy. Instances with dedicated tenancy run on single-tenant hardware.

After you deploy a cluster to Microsoft Azure, you can reconfigure the API server to use only the private zone.

You can create a machine set running on Azure that deploys machines that use the Azure Marketplace offering. To use this offering, you must first obtain the Azure Marketplace image. When obtaining your image, consider the following:

You can enable boot diagnostics on Azure machines that your machine set creates.

You can create a machine set running on Azure that deploys machines with ultra disks. Ultra disks are high-performance storage that are intended for use with the most demanding data workloads.

StorageAccountType UltraSSD_LRS can be used only when additionalCapabilities.ultraSSDEnabled is set.

failed to create vm <machine_name>: failure sending request for machine <machine_name>: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="Storage Account type 'UltraSSD_LRS' is not supported <more_information_about_why>."

You can supply an encryption key to Azure to encrypt data on managed disks at rest. You can enable server-side encryption with customer-managed keys by using the Machine API.

An Azure Key Vault, a disk encryption set, and an encryption key are required to use a customer-managed key. The disk encryption set must be in a resource group where the Cloud Credential Operator (CCO) has granted permissions. If not, an additional reader role is required to be granted on the disk encryption set.

Accelerated Networking uses single root I/O virtualization (SR-IOV) to provide Microsoft Azure VMs with a more direct path to the switch. This enhances network performance. This feature can be enabled after installation.

The control plane machine set concept of a failure domain is analogous to existing concepts on cloud providers. Not all platforms support the use of failure domains.

The failure domain configuration in the control plane machine set custom resource (CR) is platform-specific. For more information about failure domain parameters in the CR, see the sample failure domain configuration for your provider.

The control plane machine set balances control plane machines across the failure domains that are specified in the custom resource (CR).

When possible, the control plane machine set uses each failure domain equally to ensure appropriate fault tolerance. If there are fewer failure domains than control plane machines, failure domains are selected for reuse alphabetically by name. For clusters with no failure domains specified, all control plane machines are placed within a single failure domain.

Some changes to the failure domain configuration cause the control plane machine set to rebalance the control plane machines. For example, if you add failure domains to a cluster with fewer failure domains than control plane machines, the control plane machine set rebalances the machines across all available failure domains.

When a control plane machine is replaced on a cluster that uses a control plane machine set, the cluster temporarily has four control plane machines. When the fourth control plane node joins the cluster, the etcd Operator starts a new etcd member on the replacement node. When the etcd Operator observes that the old control plane machine is marked for deletion, it stops the etcd member on the old node and promotes the replacement etcd member to join the quorum of the cluster.

The control plane machine Deleting phase proceeds in the following order:

apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  ...
spec:
  lifecycleHooks:
    preDrain:
    - name: EtcdQuorumOperator 1
      owner: clusteroperator/etcd 2
  ...