Machine management

Specify a valid Red Hat Enterprise Linux CoreOS (RHCOS) AMI for your AWS zone for your OpenShift Container Platform nodes. If you want to use an AWS Marketplace image, you must complete the OpenShift Container Platform subscription from the AWS Marketplace to obtain an AMI ID for your region.

Specify the zone, for example, us-east-1a.

Specify the region, for example, us-east-1.

Specify the infrastructure ID and zone.

2.1.3. Creating a machine set
Copy link

In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

Deploy an OpenShift Container Platform cluster.
Install the OpenShift CLI (
```
oc
```
).
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

Create a new YAML file that contains the machine set custom resource (CR) sample and is named

<file_name>.yaml

.

Ensure that you set the

<clusterID>

and

<role>

parameter values.

If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

$ oc get machinesets -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

Check values of a specific machine set:

$ oc get machineset <machineset_name> -n \
     openshift-machine-api -o yaml

Example output

...
template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: agl030519-vplxk

1


        machine.openshift.io/cluster-api-machine-role: worker

2


        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

1: The cluster ID.
2: A default node label.

Create the new

MachineSet

CR:

$ oc create -f <file_name>.yaml

View the list of machine sets:

$ oc get machineset -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

When the new machine set is available, the

DESIRED

and

CURRENT

values match. If the machine set is not available, wait a few minutes and run the command again.

Next steps

If you need machine sets in other availability zones, repeat this process to create more machine sets.

2.1.4. Machine sets that deploy machines as Spot Instances
Copy link

You can save on costs by creating a machine set running on AWS that deploys machines as non-guaranteed Spot Instances. Spot Instances utilize unused AWS EC2 capacity and are less expensive than On-Demand Instances. You can use Spot Instances for workloads that can tolerate interruptions, such as batch or stateless, horizontally scalable workloads.

AWS EC2 can terminate a Spot Instance at any time. AWS gives a two-minute warning to the user when an interruption occurs. OpenShift Container Platform begins to remove the workloads from the affected instances when AWS issues the termination warning.

Interruptions can occur when using Spot Instances for the following reasons:

The instance price exceeds your maximum price
The demand for Spot Instances increases
The supply of Spot Instances decreases

When AWS terminates an instance, a termination handler running on the Spot Instance node deletes the machine resource. To satisfy the machine set

replicas

quantity, the machine set creates a machine that requests a Spot Instance.

2.1.5. Creating Spot Instances by using machine sets
Copy link

You can launch a Spot Instance on AWS by adding

spotMarketOptions

to your machine set YAML file.

Procedure

Add the following line under the
```
providerSpec
```
field:
```
providerSpec:
  value:
    spotMarketOptions: {}
```
You can optionally set the
```
spotMarketOptions.maxPrice
```
field to limit the cost of the Spot Instance. For example you can set
```
maxPrice: '2.50'
```
.
If the
```
maxPrice
```
is set, this value is used as the hourly maximum spot price. If it is not set, the maximum price defaults to charge up to the On-Demand Instance price.
Note
It is strongly recommended to use the default On-Demand price as the
maxPrice
value and to not set the maximum price for Spot Instances.

2.1.6. Machine sets that deploy machines as Dedicated Instances
Copy link

You can create a machine set running on AWS that deploys machines as Dedicated Instances. Dedicated Instances run in a virtual private cloud (VPC) on hardware that is dedicated to a single customer. These Amazon EC2 instances are physically isolated at the host hardware level. The isolation of Dedicated Instances occurs even if the instances belong to different AWS accounts that are linked to a single payer account. However, other instances that are not dedicated can share hardware with Dedicated Instances if they belong to the same AWS account.

Instances with either public or dedicated tenancy are supported by the Machine API. Instances with public tenancy run on shared hardware. Public tenancy is the default tenancy. Instances with dedicated tenancy run on single-tenant hardware.

2.1.7. Creating Dedicated Instances by using machine sets
Copy link

You can run a machine that is backed by a Dedicated Instance by using Machine API integration. Set the

tenancy

field in your machine set YAML file to launch a Dedicated Instance on AWS.

Procedure

Specify a dedicated tenancy under the

providerSpec

field:

providerSpec:
  placement:
    tenancy: dedicated

2.2. Creating a machine set on Azure
Copy link

You can create a different machine set to serve a specific purpose in your OpenShift Container Platform cluster on Microsoft Azure. For example, you might create infrastructure machine sets and related machines so that you can move supporting workloads to the new machines.

Important

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

2.2.1. Machine API overview
Copy link

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

For OpenShift Container Platform 4.8 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.8 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.

The two primary resources are:

Machines

A fundamental unit that describes the host for a node. A machine has a providerSpec specification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.

Machine sets

MachineSet

resources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.

Warning

Control plane machines cannot be managed by machine sets.

The following custom resources add more capabilities to your cluster:

Machine autoscaler: The MachineAutoscaler resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator object.
Cluster autoscaler: This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
Machine health check: The MachineHealthCheck resource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.

In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.

2.2.2. Sample YAML for a machine set custom resource on Azure
Copy link

This sample YAML defines a machine set that runs in the

Microsoft Azure zone in a region and creates nodes that are labeled with

node-role.kubernetes.io/<role>: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<role>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


    machine.openshift.io/cluster-api-machine-role: <role>

2


    machine.openshift.io/cluster-api-machine-type: <role>

3


  name: <infrastructure_id>-<role>-<region>

4


  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

5


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>-<region>

6


  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

7


        machine.openshift.io/cluster-api-machine-role: <role>

8


        machine.openshift.io/cluster-api-machine-type: <role>

9


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>-<region>

10


    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/<role>: ""

11


      providerSpec:
        value:
          apiVersion: azureproviderconfig.openshift.io/v1beta1
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image:

12


            offer: ""
            publisher: ""
            resourceID: /resourceGroups/<infrastructure_id>-rg/providers/Microsoft.Compute/images/<infrastructure_id>

13


            sku: ""
            version: ""
          internalLoadBalancer: ""
          kind: AzureMachineProviderSpec
          location: <region>

14


          managedIdentity: <infrastructure_id>-identity

15


          metadata:
            creationTimestamp: null
          natRule: null
          networkResourceGroup: ""
          osDisk:
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS
            osType: Linux
          publicIP: false
          publicLoadBalancer: ""
          resourceGroup: <infrastructure_id>-rg

16


          sshPrivateKey: ""
          sshPublicKey: ""
          subnet: <infrastructure_id>-<role>-subnet

17

18


          userDataSecret:
            name: worker-user-data

19


          vmSize: Standard_DS4_v2
          vnet: <infrastructure_id>-vnet

20


          zone: "1"

21

1 5 7 15 16 17 20

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

You can obtain the subnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.subnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1

You can obtain the vnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.vnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1

2 3 8 9 11 18 19

Specify the node label to add.

4 6 10

Specify the infrastructure ID, node label, and region.

Specify the image details for your machine set. If you want to use an Azure Marketplace image, see "Selecting an Azure Marketplace image".

Specify an image that is compatible with your instance type. The Hyper-V generation V2 images created by the installation program have a -gen2 suffix, while V1 images have the same name without the suffix.

Specify the region to place machines on.

Specify the zone within your region to place machines on. Be sure that your region supports the zone that you specify.

2.2.3. Creating a machine set
Copy link

In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

Deploy an OpenShift Container Platform cluster.
Install the OpenShift CLI (
```
oc
```
).
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

Create a new YAML file that contains the machine set custom resource (CR) sample and is named

<file_name>.yaml

.

Ensure that you set the

<clusterID>

and

<role>

parameter values.

If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

$ oc get machinesets -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

Check values of a specific machine set:

$ oc get machineset <machineset_name> -n \
     openshift-machine-api -o yaml

Example output

...
template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: agl030519-vplxk

1


        machine.openshift.io/cluster-api-machine-role: worker

2


        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

1: The cluster ID.
2: A default node label.

Create the new

MachineSet

CR:

$ oc create -f <file_name>.yaml

View the list of machine sets:

$ oc get machineset -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

When the new machine set is available, the

DESIRED

and

CURRENT

values match. If the machine set is not available, wait a few minutes and run the command again.

2.2.4. Selecting an Azure Marketplace image
Copy link

You can create a machine set running on Azure that deploys machines that use the Azure Marketplace offering. To use this offering, you must first obtain the Azure Marketplace image. When obtaining your image, consider the following:

While the images are the same, the Azure Marketplace publisher is different depending on your region. If you are located in North America, specify
```
redhat
```
as the publisher. If you are located in EMEA, specify
```
redhat-limited
```
as the publisher.
The offer includes a
```
rh-ocp-worker
```
SKU and a
```
rh-ocp-worker-gen1
```
SKU. The
```
rh-ocp-worker
```
SKU represents a Hyper-V generation version 2 VM image. The default instance types used in OpenShift Container Platform are version 2 compatible. If you are going to use an instance type that is only version 1 compatible, use the image associated with the
```
rh-ocp-worker-gen1
```
SKU. The
```
rh-ocp-worker-gen1
```
SKU represents a Hyper-V version 1 VM image.

Prerequisites

You have installed the Azure CLI client
```
(az)
```
.
Your Azure account is entitled for the offer and you have logged into this account with the Azure CLI client.

Procedure

Display all of the available OpenShift Container Platform images by running one of the following commands:

North America:

$  az vm image list --all --offer rh-ocp-worker --publisher redhat -o table

Example output

Offer          Publisher       Sku                 Urn                                                             Version
-------------  --------------  ------------------  --------------------------------------------------------------  --------------
rh-ocp-worker  RedHat          rh-ocp-worker       RedHat:rh-ocp-worker:rh-ocpworker:4.8.2021122100               4.8.2021122100
rh-ocp-worker  RedHat          rh-ocp-worker-gen1  RedHat:rh-ocp-worker:rh-ocp-worker-gen1:4.8.2021122100         4.8.2021122100

EMEA:

$  az vm image list --all --offer rh-ocp-worker --publisher redhat-limited -o table

Example output

Offer          Publisher       Sku                 Urn                                                             Version
-------------  --------------  ------------------  --------------------------------------------------------------  --------------
rh-ocp-worker  redhat-limited  rh-ocp-worker       redhat-limited:rh-ocp-worker:rh-ocp-worker:4.8.2021122100       4.8.2021122100
rh-ocp-worker  redhat-limited  rh-ocp-worker-gen1  redhat-limited:rh-ocp-worker:rh-ocp-worker-gen1:4.8.2021122100  4.8.2021122100

Note

Regardless of the version of OpenShift Container Platform you are installing, the correct version of the Azure Marketplace image to use is 4.8.x. If required, as part of the installation process, your VMs are automatically upgraded.

Inspect the image for your offer by running one of the following commands:

North America:

$ az vm image show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

EMEA:

$ az vm image show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

Review the terms of the offer by running one of the following commands:

North America:

$ az vm image terms show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

EMEA:

$ az vm image terms show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

Accept the terms of the offering by running one of the following commands:

North America:

$ az vm image terms accept --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

EMEA:

$ az vm image terms accept --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

Record the image details of your offer, specifically the values for
```
publisher
```
,
```
offer
```
,
```
sku
```
, and
```
version
```
.

Add the following parameters to the

providerSpec

section of your machine set YAML file using the image details for your offer:

Sample providerSpec image values for Azure Marketplace compute machines

providerSpec:
  value:
    image:
      offer: rh-ocp-worker
      publisher: redhat
      resourceID: ""
      sku: rh-ocp-worker
      type: MarketplaceWithPlan
      version: 4.8.2021122100

2.2.5. Machine sets that deploy machines as Spot VMs
Copy link

You can save on costs by creating a machine set running on Azure that deploys machines as non-guaranteed Spot VMs. Spot VMs utilize unused Azure capacity and are less expensive than standard VMs. You can use Spot VMs for workloads that can tolerate interruptions, such as batch or stateless, horizontally scalable workloads.

Azure can terminate a Spot VM at any time. Azure gives a 30-second warning to the user when an interruption occurs. OpenShift Container Platform begins to remove the workloads from the affected instances when Azure issues the termination warning.

Interruptions can occur when using Spot VMs for the following reasons:

The instance price exceeds your maximum price
The supply of Spot VMs decreases
Azure needs capacity back

When Azure terminates an instance, a termination handler running on the Spot VM node deletes the machine resource. To satisfy the machine set

replicas

quantity, the machine set creates a machine that requests a Spot VM.

2.2.6. Creating Spot VMs by using machine sets
Copy link

You can launch a Spot VM on Azure by adding

spotVMOptions

to your machine set YAML file.

Procedure

Add the following line under the
```
providerSpec
```
field:
```
providerSpec:
  value:
    spotVMOptions: {}
```
You can optionally set the
```
spotVMOptions.maxPrice
```
field to limit the cost of the Spot VM. For example you can set
```
maxPrice: '0.98765'
```
. If the
```
maxPrice
```
is set, this value is used as the hourly maximum spot price. If it is not set, the maximum price defaults to
```
-1
```
and charges up to the standard VM price.
Azure caps Spot VM prices at the standard price. Azure will not evict an instance due to pricing if the instance is set with the default
```
maxPrice
```
. However, an instance can still be evicted due to capacity restrictions.

Note

It is strongly recommended to use the default standard VM price as the

maxPrice

value and to not set the maximum price for Spot VMs.

2.2.7. Enabling customer-managed encryption keys for a machine set
Copy link

You can supply an encryption key to Azure to encrypt data on managed disks at rest. You can enable server-side encryption with customer-managed keys by using the Machine API.

An Azure Key Vault, a disk encryption set, and an encryption key are required to use a customer-managed key. The disk encryption set must preside in a resource group where the Cloud Credential Operator (CCO) has granted permissions. If not, an additional reader role is required to be granted on the disk encryption set.

Prerequisites

Procedure

Configure the disk encryption set under the

providerSpec

field in your machine set YAML file. For example:

...
providerSpec:
  value:
    ...
    osDisk:
      diskSizeGB: 128
      managedDisk:
        diskEncryptionSet:
          id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Compute/diskEncryptionSets/<disk_encryption_set_name>
        storageAccountType: Premium_LRS
...

2.3. Creating a machine set on GCP
Copy link

You can create a different machine set to serve a specific purpose in your OpenShift Container Platform cluster on Google Cloud Platform (GCP). For example, you might create infrastructure machine sets and related machines so that you can move supporting workloads to the new machines.

Important

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

2.3.1. Machine API overview
Copy link

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

For OpenShift Container Platform 4.8 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.8 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.

The two primary resources are:

Machines

A fundamental unit that describes the host for a node. A machine has a providerSpec specification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.

Machine sets

MachineSet

resources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.

Warning

Control plane machines cannot be managed by machine sets.

The following custom resources add more capabilities to your cluster:

Machine autoscaler: The MachineAutoscaler resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator object.
Cluster autoscaler: This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
Machine health check: The MachineHealthCheck resource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.

In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.

2.3.2. Sample YAML for a machine set custom resource on GCP
Copy link

This sample YAML defines a machine set that runs in Google Cloud Platform (GCP) and creates nodes that are labeled with

node-role.kubernetes.io/<role>: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<role>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


  name: <infrastructure_id>-w-a
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: <role>

2


        machine.openshift.io/cluster-api-machine-type: <role>
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/<role>: ""
      providerSpec:
        value:
          apiVersion: gcpprovider.openshift.io/v1beta1
          canIPForward: false
          credentialsSecret:
            name: gcp-cloud-credentials
          deletionProtection: false
          disks:
          - autoDelete: true
            boot: true
            image: <path_to_image>

3


            labels: null
            sizeGb: 128
            type: pd-ssd
          gcpMetadata:

4


          - key: <custom_metadata_key>
            value: <custom_metadata_value>
          kind: GCPMachineProviderSpec
          machineType: n1-standard-4
          metadata:
            creationTimestamp: null
          networkInterfaces:
          - network: <infrastructure_id>-network
            subnetwork: <infrastructure_id>-worker-subnet
          projectID: <project_name>

5


          region: us-central1
          serviceAccounts:
          - email: <infrastructure_id>-w@<project_name>.iam.gserviceaccount.com
            scopes:
            - https://www.googleapis.com/auth/cloud-platform
          tags:
            - <infrastructure_id>-worker
          userDataSecret:
            name: worker-user-data
          zone: us-central1-a

1

For <infrastructure_id>, specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2

For <node>, specify the node label to add.

3

Specify the path to the image that is used in current compute machine sets. If you have the OpenShift CLI installed, you can obtain the path to the image by running the following command:

$ oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.disks[0].image}{"\n"}' \
    get machineset/<infrastructure_id>-worker-a

To use a GCP Marketplace image, specify the offer to use:

OpenShift Container Platform:

https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-ocp-48-x86-64-202210040145

OpenShift Platform Plus:

https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-opp-48-x86-64-202206140145

OpenShift Kubernetes Engine:

https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-oke-48-x86-64-202206140145

4

Optional: Specify custom metadata in the form of a key:value pair. For example use cases, see the GCP documentation for setting custom metadata.

For <project_name>, specify the name of the GCP project that you use for your cluster.

2.3.3. Creating a machine set
Copy link

In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

Deploy an OpenShift Container Platform cluster.
Install the OpenShift CLI (
```
oc
```
).
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

Create a new YAML file that contains the machine set custom resource (CR) sample and is named

<file_name>.yaml

.

Ensure that you set the

<clusterID>

and

<role>

parameter values.

If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

$ oc get machinesets -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

Check values of a specific machine set:

$ oc get machineset <machineset_name> -n \
     openshift-machine-api -o yaml

Example output

...
template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: agl030519-vplxk

1


        machine.openshift.io/cluster-api-machine-role: worker

2


        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

1: The cluster ID.
2: A default node label.

Create the new

MachineSet

CR:

$ oc create -f <file_name>.yaml

View the list of machine sets:

$ oc get machineset -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

When the new machine set is available, the

DESIRED

and

CURRENT

values match. If the machine set is not available, wait a few minutes and run the command again.

2.3.4. Machine sets that deploy machines as preemptible VM instances
Copy link

You can save on costs by creating a machine set running on GCP that deploys machines as non-guaranteed preemptible VM instances. Preemptible VM instances utilize excess Compute Engine capacity and are less expensive than normal instances. You can use preemptible VM instances for workloads that can tolerate interruptions, such as batch or stateless, horizontally scalable workloads.

GCP Compute Engine can terminate a preemptible VM instance at any time. Compute Engine sends a preemption notice to the user indicating that an interruption will occur in 30 seconds. OpenShift Container Platform begins to remove the workloads from the affected instances when Compute Engine issues the preemption notice. An ACPI G3 Mechanical Off signal is sent to the operating system after 30 seconds if the instance is not stopped. The preemptible VM instance is then transitioned to a

TERMINATED

state by Compute Engine.

Interruptions can occur when using preemptible VM instances for the following reasons:

There is a system or maintenance event
The supply of preemptible VM instances decreases
The instance reaches the end of the allotted 24-hour period for preemptible VM instances

When GCP terminates an instance, a termination handler running on the preemptible VM instance node deletes the machine resource. To satisfy the machine set

replicas

quantity, the machine set creates a machine that requests a preemptible VM instance.

2.3.5. Creating preemptible VM instances by using machine sets
Copy link

You can launch a preemptible VM instance on GCP by adding

preemptible

to your machine set YAML file.

Procedure

Add the following line under the
```
providerSpec
```
field:
```
providerSpec:
  value:
    preemptible: true
```
If
```
preemptible
```
is set to
```
true
```
, the machine is labelled as an
```
interruptable-instance
```
after the instance is launched.

2.3.6. Enabling customer-managed encryption keys for a machine set
Copy link

Google Cloud Platform (GCP) Compute Engine allows users to supply an encryption key to encrypt data on disks at rest. The key is used to encrypt the data encryption key, not to encrypt the customer’s data. By default, Compute Engine encrypts this data by using Compute Engine keys.

You can enable encryption with a customer-managed key by using the Machine API. You must first create a KMS key and assign the correct permissions to a service account. The KMS key name, key ring name, and location are required to allow a service account to use your key.

Note

If you do not want to use a dedicated service account for the KMS encryption, the Compute Engine default service account is used instead. You must grant the default service account permission to access the keys if you do not use a dedicated service account. The Compute Engine default service account name follows the

service-<project_number>@compute-system.iam.gserviceaccount.com

pattern.

Procedure

Run the following command with your KMS key name, key ring name, and location to allow a specific service account to use your KMS key and to grant the service account the correct IAM role:

gcloud kms keys add-iam-policy-binding <key_name> \
  --keyring <key_ring_name> \
  --location <key_ring_location> \
  --member "serviceAccount:service-<project_number>@compute-system.iam.gserviceaccount.com” \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

Configure the encryption key under the
```
providerSpec
```
field in your machine set YAML file. For example:
```
providerSpec:
  value:
    # ...
    disks:
    - type:
      # ...
      encryptionKey:
        kmsKey:
          name: machine-encryption-key 
```
1
```
          keyRing: openshift-encrpytion-ring 
```
2
```
          location: global 
```
3
```
          projectID: openshift-gcp-project 
```
4
```
        kmsKeyServiceAccount: openshift-service-account@openshift-gcp-project.iam.gserviceaccount.com 
```
5
1
The name of the customer-managed encryption key that is used for the disk encryption.
2
The name of the KMS key ring that the KMS key belongs to.
3
The GCP location in which the KMS key ring exists.
4
Optional: The ID of the project in which the KMS key ring exists. If a project ID is not set, the machine set projectID in which the machine set was created is used.
5
Optional: The service account that is used for the encryption request for the given KMS key. If a service account is not set, the Compute Engine default service account is used.
After a new machine is created by using the updated
```
providerSpec
```
object configuration, the disk encryption key is encrypted with the KMS key.

2.4. Creating a machine set on OpenStack
Copy link

You can create a different machine set to serve a specific purpose in your OpenShift Container Platform cluster on Red Hat OpenStack Platform (RHOSP). For example, you might create infrastructure machine sets and related machines so that you can move supporting workloads to the new machines.

Important

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

2.4.1. Machine API overview
Copy link

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

For OpenShift Container Platform 4.8 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.8 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.

The two primary resources are:

Machines

A fundamental unit that describes the host for a node. A machine has a providerSpec specification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.

Machine sets

MachineSet

resources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.

Warning

Control plane machines cannot be managed by machine sets.

The following custom resources add more capabilities to your cluster:

Machine autoscaler: The MachineAutoscaler resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator object.
Cluster autoscaler: This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
Machine health check: The MachineHealthCheck resource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.

In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.

2.4.2. Sample YAML for a machine set custom resource on RHOSP
Copy link

This sample YAML defines a machine set that runs on Red Hat OpenStack Platform (RHOSP) and creates nodes that are labeled with

node-role.kubernetes.io/<role>: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<role>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


    machine.openshift.io/cluster-api-machine-role: <role>

2


    machine.openshift.io/cluster-api-machine-type: <role>

3


  name: <infrastructure_id>-<role>

4


  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas>
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

5


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

6


  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

7


        machine.openshift.io/cluster-api-machine-role: <role>

8


        machine.openshift.io/cluster-api-machine-type: <role>

9


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

10


    spec:
      providerSpec:
        value:
          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: <nova_flavor>
          image: <glance_image_name_or_location>
          serverGroupID: <optional_UUID_of_server_group>

11


          kind: OpenstackProviderSpec
          networks:

12


          - filter: {}
            subnets:
            - filter:
                name: <subnet_name>
                tags: openshiftClusterID=<infrastructure_id>

13


          primarySubnet: <rhosp_subnet_UUID>

14


          securityGroups:
          - filter: {}
            name: <infrastructure_id>-worker

15


          serverMetadata:
            Name: <infrastructure_id>-worker

16


            openshiftClusterID: <infrastructure_id>

17


          tags:
          - openshiftClusterID=<infrastructure_id>

18


          trunk: true
          userDataSecret:
            name: worker-user-data

19


          availabilityZone: <optional_openstack_availability_zone>

1 5 7 13 15 16 17 18

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2 3 8 9 19

Specify the node label to add.

4 6 10

Specify the infrastructure ID and node label.

To set a server group policy for the MachineSet, enter the value that is returned from creating a server group. For most deployments, anti-affinity or soft-anti-affinity policies are recommended.

Required for deployments to multiple networks. To specify multiple networks, add another entry in the networks array. Also, you must include the network that is used as the primarySubnet value.

Specify the RHOSP subnet that you want the endpoints of nodes to be published on. Usually, this is the same subnet that is used as the value of machinesSubnet in the install-config.yaml file.

2.4.3. Sample YAML for a machine set custom resource that uses SR-IOV on RHOSP
Copy link

If you configured your cluster for single-root I/O virtualization (SR-IOV), you can create machine sets that use that technology.

This sample YAML defines a machine set that uses SR-IOV networks. The nodes that it creates are labeled with

node-role.openshift.io/<node_role>: ""

In this sample,

infrastructure_id

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

node_role

is the node label to add.

The sample assumes two SR-IOV networks that are named "radio" and "uplink". The networks are used in port definitions in the

spec.template.spec.providerSpec.value.ports

list.

Note

Only parameters that are specific to SR-IOV deployments are described in this sample. To review a more general sample, see "Sample YAML for a machine set custom resource on RHOSP".

An example machine set that uses SR-IOV networks

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>
    machine.openshift.io/cluster-api-machine-role: <node_role>
    machine.openshift.io/cluster-api-machine-type: <node_role>
  name: <infrastructure_id>-<node_role>
  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas>
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<node_role>
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: <node_role>
        machine.openshift.io/cluster-api-machine-type: <node_role>
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<node_role>
    spec:
      metadata:
      providerSpec:
        value:
          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: <nova_flavor>
          image: <glance_image_name_or_location>
          serverGroupID: <optional_UUID_of_server_group>
          kind: OpenstackProviderSpec
          networks:
            - subnets:
              - UUID: <machines_subnet_UUID>
          ports:
            - networkID: <radio_network_UUID>

1


              nameSuffix: radio
              fixedIPs:
                - subnetID: <radio_subnet_UUID>

2


              tags:
                - sriov
                - radio
              vnicType: direct

3


              portSecurity: false

4


            - networkID: <uplink_network_UUID>

5


              nameSuffix: uplink
              fixedIPs:
                - subnetID: <uplink_subnet_UUID>

6


              tags:
                - sriov
                - uplink
              vnicType: direct

7


              portSecurity: false

8


          primarySubnet: <machines_subnet_UUID>
          securityGroups:
          - filter: {}
            name: <infrastructure_id>-<node_role>
          serverMetadata:
            Name: <infrastructure_id>-<node_role>
            openshiftClusterID: <infrastructure_id>
          tags:
          - openshiftClusterID=<infrastructure_id>
          trunk: true
          userDataSecret:
            name: <node_role>-user-data
          availabilityZone: <optional_openstack_availability_zone>
          configDrive: true

9

1 5: Enter a network UUID for each port.
2 6: Enter a subnet UUID for each port.
3 7: The value of the vnicType parameter must be direct for each port.
4 8: The value of the portSecurity parameter must be false for each port.
You cannot set security groups and allowed address pairs for ports when port security is disabled. Setting security groups on the instance applies the groups to all ports that are attached to it.
9: The value of the configDrive parameter must be true.

Note

Trunking is enabled for ports that are created by entries in the networks and subnets lists. The name of ports that are created from these lists follow the pattern

<machine_name>-<nameSuffix>

. The

nameSuffix

field is required in port definitions.

Trunking is not enabled for ports that are defined in the ports list.

Optionally, you can add tags to ports as part of their

tags

lists.

2.4.4. Sample YAML for SR-IOV deployments where port security is disabled
Copy link

To create single-root I/O virtualization (SR-IOV) ports on a network that has port security disabled, define a machine set that includes the ports as items in the

spec.template.spec.providerSpec.value.ports

list. This difference from the standard SR-IOV machine set is due to the automatic security group and allowed address pair configuration that occurs for ports that are created by using the network and subnet interfaces.

Ports that you define for machines subnets require:

Allowed address pairs for the API and ingress virtual IP ports
The compute security group
Attachment to the machines network and subnet

Note

Only parameters that are specific to SR-IOV deployments where port security is disabled are described in this sample. To review a more general sample, see Sample YAML for a machine set custom resource that uses SR-IOV on RHOSP".

An example machine set that uses SR-IOV networks and has port security disabled

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>
    machine.openshift.io/cluster-api-machine-role: <node_role>
    machine.openshift.io/cluster-api-machine-type: <node_role>
  name: <infrastructure_id>-<node_role>
  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas>
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<node_role>
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: <node_role>
        machine.openshift.io/cluster-api-machine-type: <node_role>
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<node_role>
    spec:
      metadata: {}
      providerSpec:
        value:
          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: <nova_flavor>
          image: <glance_image_name_or_location>
          kind: OpenstackProviderSpec
          ports:
            - allowedAddressPairs:

1


              - ipAddress: <API_VIP_port_IP>
              - ipAddress: <ingress_VIP_port_IP>
              fixedIPs:
                - subnetID: <machines_subnet_UUID>

2


              nameSuffix: nodes
              networkID: <machines_network_UUID>

3


              securityGroups:
                  - <compute_security_group_UUID>

4


            - networkID: <SRIOV_network_UUID>
              nameSuffix: sriov
              fixedIPs:
                - subnetID: <SRIOV_subnet_UUID>
              tags:
                - sriov
              vnicType: direct
              portSecurity: False
          primarySubnet: <machines_subnet_UUID>
          serverMetadata:
            Name: <infrastructure_ID>-<node_role>
            openshiftClusterID: <infrastructure_id>
          tags:
          - openshiftClusterID=<infrastructure_id>
          trunk: false
          userDataSecret:
            name: worker-user-data
          configDrive: True

1: Specify allowed address pairs for the API and ingress ports.
2 3: Specify the machines network and subnet.
4: Specify the compute machines security group.

Note

Trunking is enabled for ports that are created by entries in the networks and subnets lists. The name of ports that are created from these lists follow the pattern

<machine_name>-<nameSuffix>

. The

nameSuffix

field is required in port definitions.

Trunking is not enabled for ports that are defined in the ports list.

Optionally, you can add tags to ports as part of their

tags

lists.

If your cluster uses Kuryr and the RHOSP SR-IOV network has port security disabled, the primary port for compute machines must have:

The value of the

spec.template.spec.providerSpec.value.networks.portSecurityEnabled

parameter set to

false

.

For each subnet, the value of the

spec.template.spec.providerSpec.value.networks.subnets.portSecurityEnabled

parameter set to

false

.

The value of

spec.template.spec.providerSpec.value.securityGroups

set to empty:

[]

.

An example section of a machine set for a cluster on Kuryr that uses SR-IOV and has port security disabled

...
          networks:
            - subnets:
              - uuid: <machines_subnet_UUID>
                portSecurityEnabled: false
              portSecurityEnabled: false
          securityGroups: []
...

In that case, you can apply the compute security group to the primary VM interface after the VM is created. For example, from a command line:

$ openstack port set --enable-port-security --security-group <infrastructure_id>-<node_role> <main_port_ID>

2.4.5. Creating a machine set
Copy link

In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

Deploy an OpenShift Container Platform cluster.
Install the OpenShift CLI (
```
oc
```
).
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

Create a new YAML file that contains the machine set custom resource (CR) sample and is named

<file_name>.yaml

.

Ensure that you set the

<clusterID>

and

<role>

parameter values.

If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

$ oc get machinesets -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

Check values of a specific machine set:

$ oc get machineset <machineset_name> -n \
     openshift-machine-api -o yaml

Example output

...
template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: agl030519-vplxk

1


        machine.openshift.io/cluster-api-machine-role: worker

2


        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

1: The cluster ID.
2: A default node label.

Create the new

MachineSet

CR:

$ oc create -f <file_name>.yaml

View the list of machine sets:

$ oc get machineset -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

When the new machine set is available, the

DESIRED

and

CURRENT

values match. If the machine set is not available, wait a few minutes and run the command again.

2.5. Creating a machine set on RHV
Copy link

You can create a different machine set to serve a specific purpose in your OpenShift Container Platform cluster on Red Hat Virtualization (RHV). For example, you might create infrastructure machine sets and related machines so that you can move supporting workloads to the new machines.

Important

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

2.5.1. Machine API overview
Copy link

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

For OpenShift Container Platform 4.8 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.8 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.

The two primary resources are:

Machines

A fundamental unit that describes the host for a node. A machine has a providerSpec specification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.

Machine sets

MachineSet

resources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.

Warning

Control plane machines cannot be managed by machine sets.

The following custom resources add more capabilities to your cluster:

Machine autoscaler: The MachineAutoscaler resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator object.
Cluster autoscaler: This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
Machine health check: The MachineHealthCheck resource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.

In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.

2.5.2. Sample YAML for a machine set custom resource on RHV
Copy link

This sample YAML defines a machine set that runs on RHV and creates nodes that are labeled with

node-role.kubernetes.io/<node_role>: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<role>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


    machine.openshift.io/cluster-api-machine-role: <role>

2


    machine.openshift.io/cluster-api-machine-type: <role>

3


  name: <infrastructure_id>-<role>

4


  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas>

5


  selector:

6


    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

7


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

8


  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

9


        machine.openshift.io/cluster-api-machine-role: <role>

10


        machine.openshift.io/cluster-api-machine-type: <role>

11


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

12


    spec:
      metadata:
        labels:
          node-role.kubernetes.io/<role>: ""

13


      providerSpec:
        value:
          apiVersion: ovirtproviderconfig.machine.openshift.io/v1beta1
          cluster_id: <ovirt_cluster_id>

14


          template_name: <ovirt_template_name>

15


          instance_type_id: <instance_type_id>

16


          cpu:

17


            sockets: <number_of_sockets>

18


            cores: <number_of_cores>

19


            threads: <number_of_threads>

20


          memory_mb: <memory_size>

21


          os_disk:

22


            size_gb: <disk_size>

23


          network_interfaces:

24


            vnic_profile_id:  <vnic_profile_id>

25


          credentialsSecret:
            name: ovirt-credentials

26


          kind: OvirtMachineProviderSpec
          type: <workload_type>

27


          userDataSecret:
            name: worker-user-data
          affinityGroupsNames:
            - compute

28

1 7 9

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2 3 10 11 13

Specify the node label to add.

4 8 12

Specify the infrastructure ID and node label. These two strings together cannot be longer than 35 characters.

Specify the number of machines to create.

Selector for the machines.

Specify the UUID for the RHV cluster to which this VM instance belongs.

Specify the RHV VM template to use to create the machine.

Optional: Specify the VM instance type.

Warning

The

instance_type_id

field is deprecated and will be removed in a future release.

If you include this parameter, you do not need to specify the hardware parameters of the VM including CPU and memory because this parameter overrides all hardware parameters.

17

Optional: The CPU field contains the CPU’s configuration, including sockets, cores, and threads.

18

Optional: Specify the number of sockets for a VM.

19

Optional: Specify the number of cores per socket.

20

Optional: Specify the number of threads per core.

Optional: Specify the size of a VM’s memory in MiB.

22

Optional: Root disk of the node.

23

Optional: Specify the size of the bootable disk in GiB.

24

Optional: List of the network interfaces of the VM. If you include this parameter, OpenShift Container Platform discards all network interfaces from the template and creates new ones.

25

Optional: Specify the vNIC profile ID.

26

Specify the name of the secret that holds the RHV credentials.

27

Optional: Specify the workload type for which the instance is optimized. This value affects the RHV VM parameter. Supported values: desktop, server (default), high_performance. high_performance improves performance on the VM, but there are limitations. For example, you cannot access the VM with a graphical console. For more information see Configuring High Performance Virtual Machines, Templates, and Pools in the Virtual Machine Management Guide.

28

A list of affinity group names that should be applied to the VMs. The affinity groups must exist in oVirt.

Note

Because RHV uses a template when creating a VM, if you do not specify a value for an optional parameter, RHV uses the value for that parameter that is specified in the template.

2.5.3. Creating a machine set
Copy link

In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

Deploy an OpenShift Container Platform cluster.
Install the OpenShift CLI (
```
oc
```
).
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

Create a new YAML file that contains the machine set custom resource (CR) sample and is named

<file_name>.yaml

.

Ensure that you set the

<clusterID>

and

<role>

parameter values.

If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

$ oc get machinesets -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

Check values of a specific machine set:

$ oc get machineset <machineset_name> -n \
     openshift-machine-api -o yaml

Example output

...
template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: agl030519-vplxk

1


        machine.openshift.io/cluster-api-machine-role: worker

2


        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

1: The cluster ID.
2: A default node label.

Create the new

MachineSet

CR:

$ oc create -f <file_name>.yaml

View the list of machine sets:

$ oc get machineset -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

When the new machine set is available, the

DESIRED

and

CURRENT

values match. If the machine set is not available, wait a few minutes and run the command again.

2.6. Creating a machine set on vSphere
Copy link

You can create a different machine set to serve a specific purpose in your OpenShift Container Platform cluster on VMware vSphere. For example, you might create infrastructure machine sets and related machines so that you can move supporting workloads to the new machines.

Important

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

2.6.1. Machine API overview
Copy link

The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.

For OpenShift Container Platform 4.8 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.8 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.

The two primary resources are:

Machines

A fundamental unit that describes the host for a node. A machine has a providerSpec specification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata.

Machine sets

MachineSet

resources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.

Warning

Control plane machines cannot be managed by machine sets.

The following custom resources add more capabilities to your cluster:

Machine autoscaler: The MachineAutoscaler resource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. The MachineAutoscaler object takes effect after a ClusterAutoscaler object exists. Both ClusterAutoscaler and MachineAutoscaler resources are made available by the ClusterAutoscalerOperator object.
Cluster autoscaler: This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
Machine health check: The MachineHealthCheck resource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.

In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.

2.6.2. Sample YAML for a machine set custom resource on vSphere
Copy link

This sample YAML defines a machine set that runs on VMware vSphere and creates nodes that are labeled with

node-role.kubernetes.io/<role>: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<role>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


  name: <infrastructure_id>-<role>

2


  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

3


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

4


  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

5


        machine.openshift.io/cluster-api-machine-role: <role>

6


        machine.openshift.io/cluster-api-machine-type: <role>

7


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

8


    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/<role>: ""

9


      providerSpec:
        value:
          apiVersion: vsphereprovider.openshift.io/v1beta1
          credentialsSecret:
            name: vsphere-cloud-credentials
          diskGiB: 120
          kind: VSphereMachineProviderSpec
          memoryMiB: 8192
          metadata:
            creationTimestamp: null
          network:
            devices:
            - networkName: "<vm_network_name>"

10


          numCPUs: 4
          numCoresPerSocket: 1
          snapshot: ""
          template: <vm_template_name>

11


          userDataSecret:
            name: worker-user-data
          workspace:
            datacenter: <vcenter_datacenter_name>

12


            datastore: <vcenter_datastore_name>

13


            folder: <vcenter_vm_folder_path>

14


            resourcepool: <vsphere_resource_pool>

15


            server: <vcenter_server_ip>

16

1 3 5

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2 4 8

Specify the infrastructure ID and node label.

6 7 9

Specify the node label to add.

Specify the vSphere VM network to deploy the compute machine set to. This VM network must be where other compute machines reside in the cluster.

Specify the vSphere VM template to use, such as user-5ddjd-rhcos.

Specify the vCenter Datacenter to deploy the compute machine set on.

Specify the vCenter Datastore to deploy the compute machine set on.

Specify the path to the vSphere VM folder in vCenter, such as /dc1/vm/user-inst-5ddjd.

Specify the vSphere resource pool for your VMs.

Specify the vCenter server IP or fully qualified domain name.

2.6.3. Creating a machine set
Copy link

In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

Deploy an OpenShift Container Platform cluster.
Install the OpenShift CLI (
```
oc
```
).
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.
Create a tag inside your vCenter instance based on the cluster API name. This tag is utilized by the machine set to associate the OpenShift Container Platform nodes to the provisioned virtual machines (VM). For directions on creating tags in vCenter, see the VMware documentation for vSphere Tags and Attributes.
Have the necessary permissions to deploy VMs in your vCenter instance and have the required access to the datastore specified.

Procedure

Create a new YAML file that contains the machine set custom resource (CR) sample and is named

<file_name>.yaml

.

Ensure that you set the

<clusterID>

and

<role>

parameter values.

If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

$ oc get machinesets -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

Check values of a specific machine set:

$ oc get machineset <machineset_name> -n \
     openshift-machine-api -o yaml

Example output

...
template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: agl030519-vplxk

1


        machine.openshift.io/cluster-api-machine-role: worker

2


        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

1: The cluster ID.
2: A default node label.

Create the new

MachineSet

CR:

$ oc create -f <file_name>.yaml

View the list of machine sets:

$ oc get machineset -n openshift-machine-api

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

When the new machine set is available, the

DESIRED

and

CURRENT

values match. If the machine set is not available, wait a few minutes and run the command again.

Chapter 3. Manually scaling a machine set
Copy link

You can add or remove an instance of a machine in a machine set.

Note

If you need to modify aspects of a machine set outside of scaling, see Modifying a machine set.

3.1. Prerequisites
Copy link

If you enabled the cluster-wide proxy and scale up workers not included in
```
networking.machineNetwork[].cidr
```
from the installation configuration, you must add the workers to the Proxy object’s noProxy field to prevent connection issues.

Important

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

3.2. Scaling a machine set manually
Copy link

To add or remove an instance of a machine in a machine set, you can manually scale the machine set.

This guidance is relevant to fully automated, installer-provisioned infrastructure installations. Customized, user-provisioned infrastructure installations do not have machine sets.

Prerequisites

Install an OpenShift Container Platform cluster and the
```
oc
```
command line.
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

View the machine sets that are in the cluster:
```
$ oc get machinesets -n openshift-machine-api
```
The machine sets are listed in the form of
```
<clusterid>-worker-<aws-region-az>
```
.

View the machines that are in the cluster:

$ oc get machine -n openshift-machine-api

Set the annotation on the machine that you want to delete:

$ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/cluster-api-delete-machine="true"

Cordon and drain the node that you want to delete:

$ oc adm cordon <node_name>
$ oc adm drain <node_name>

Scale the machine set:

$ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

Or:

$ oc edit machineset <machineset> -n openshift-machine-api

Tip

You can alternatively apply the following YAML to scale the machine set:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: <machineset>
  namespace: openshift-machine-api
spec:
  replicas: 2

You can scale the machine set up or down. It takes several minutes for the new machines to be available.

Verification

Verify the deletion of the intended machine:
```
$ oc get machines
```

3.3. The machine set deletion policy
Copy link

Random

,

Newest

, and

Oldest

are the three supported deletion options. The default is

Random

, meaning that random machines are chosen and deleted when scaling machine sets down. The deletion policy can be set according to the use case by modifying the particular machine set:

spec:
  deletePolicy: <delete_policy>
  replicas: <desired_replica_count>

Specific machines can also be prioritized for deletion by adding the annotation

machine.openshift.io/cluster-api-delete-machine=true

to the machine of interest, regardless of the deletion policy.

Important

By default, the OpenShift Container Platform router pods are deployed on workers. Because the router is required to access some cluster resources, including the web console, do not scale the worker machine set to

unless you first relocate the router pods.

Note

Custom machine sets can be used for use cases requiring that services run on specific nodes and that those services are ignored by the controller when the worker machine sets are scaling down. This prevents service disruption.

Chapter 4. Modifying a machine set
Copy link

You can modify a machine set, such as adding labels, changing the instance type, or changing block storage.

On Red Hat Virtualization (RHV), you can also change a machine set to provision new nodes on a different storage domain.

Note

If you need to scale a machine set without making other changes, see Manually scaling a machine set.

4.1. Modifying a machine set
Copy link

To make changes to a machine set, edit the

MachineSet

YAML. Then, remove all machines associated with the machine set by deleting each machine or scaling down the machine set to

replicas. Then, scale the replicas back to the desired number. Changes you make to a machine set do not affect existing machines.

If you need to scale a machine set without making other changes, you do not need to delete the machines.

Note

By default, the OpenShift Container Platform router pods are deployed on workers. Because the router is required to access some cluster resources, including the web console, do not scale the worker machine set to

unless you first relocate the router pods.

Prerequisites

Install an OpenShift Container Platform cluster and the
```
oc
```
command line.
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

Edit the machine set:

$ oc edit machineset <machineset> -n openshift-machine-api

Scale down the machine set to

:

$ oc scale --replicas=0 machineset <machineset> -n openshift-machine-api

Or:

$ oc edit machineset <machineset> -n openshift-machine-api

Tip

You can alternatively apply the following YAML to scale the machine set:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: <machineset>
  namespace: openshift-machine-api
spec:
  replicas: 0

Wait for the machines to be removed.

Scale up the machine set as needed:

$ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

Or:

$ oc edit machineset <machineset> -n openshift-machine-api

Tip

You can alternatively apply the following YAML to scale the machine set:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: <machineset>
  namespace: openshift-machine-api
spec:
  replicas: 2

Wait for the machines to start. The new machines contain changes you made to the machine set.

4.2. Migrating nodes to a different storage domain on RHV
Copy link

You can migrate the OpenShift Container Platform control plane and compute nodes to a different storage domain in a Red Hat Virtualization (RHV) cluster.

4.2.1. Migrating compute nodes to a different storage domain in RHV
Copy link

Prerequisites

You are logged in to the Manager.
You have the name of the target storage domain.

Procedure

Identify the virtual machine template:

$ oc get -o jsonpath='{.items[0].spec.template.spec.providerSpec.value.template_name}{"\n"}' machineset -A

Create a new virtual machine in the Manager, based on the template you identified. Leave all other settings unchanged. For details, see Creating a Virtual Machine Based on a Template in the Red Hat Virtualization Virtual Machine Management Guide.
Tip
You do not need to start the new virtual machine.
Create a new template from the new virtual machine. Specify the target storage domain under Target. For details, see Creating a Template in the Red Hat Virtualization Virtual Machine Management Guide.
Add a new machine set to the OpenShift Container Platform cluster with the new template.
1. Get the details of the current machine set:
  $ oc get machineset -o yaml
2. Use these details to create a machine set. For more information see Creating a machine set.
  Enter the new virtual machine template name in the template_name field. Use the same template name you used in the New template dialog in the Manager.
3. Note the names of both the old and new machine sets. You need to refer to them in subsequent steps.
Migrate the workloads.
1. Scale up the new machine set. For details on manually scaling machine sets, see Scaling a machine set manually.
  OpenShift Container Platform moves the pods to an available worker when the old machine is removed.
2. Scale down the old machine set.

Remove the old machine set:

$ oc delete machineset <machineset-name>

4.2.2. Migrating control plane nodes to a different storage domain on RHV
Copy link

OpenShift Container Platform does not manage control plane nodes, so they are easier to migrate than compute nodes. You can migrate them like any other virtual machine on Red Hat Virtualization (RHV).

Perform this procedure for each node separately.

Prerequisites

You are logged in to the Manager.
You have identified the control plane nodes. They are labeled master in the Manager.

Procedure

Select the virtual machine labeled master.
Shut down the virtual machine.
Click the Disks tab.
Click the virtual machine’s disk.
Click More Actions and select Move.
Select the target storage domain and wait for the migration process to complete.
Start the virtual machine.
Verify that the OpenShift Container Platform cluster is stable:
```
$ oc get nodes
```
The output should display the node with the status
```
Ready
```
.
Repeat this procedure for each control plane node.

Chapter 5. Deleting a machine
Copy link

You can delete a specific machine.

5.1. Deleting a specific machine
Copy link

You can delete a specific machine.

Note

You cannot delete a control plane machine.

Prerequisites

Install an OpenShift Container Platform cluster.
Install the OpenShift CLI (
```
oc
```
).
Log in to
```
oc
```
as a user with
```
cluster-admin
```
permission.

Procedure

View the machines that are in the cluster and identify the one to delete:
```
$ oc get machine -n openshift-machine-api
```
The command output contains a list of machines in the
```
<clusterid>-worker-<cloud_region>
```
format.
Delete the machine:
```
$ oc delete machine <machine> -n openshift-machine-api
```
Important
By default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured pod disruption budget, the drain operation might not be able to succeed in preventing the machine from being deleted. You can skip draining the node by annotating "machine.openshift.io/exclude-node-draining" in a specific machine. If the machine being deleted belongs to a machine set, a new machine is immediately created to satisfy the specified number of replicas.

Chapter 6. Applying autoscaling to an OpenShift Container Platform cluster
Copy link

Applying autoscaling to an OpenShift Container Platform cluster involves deploying a cluster autoscaler and then deploying machine autoscalers for each machine type in your cluster.

Important

You can configure the cluster autoscaler only in clusters where the machine API is operational.

6.1. About the cluster autoscaler
Copy link

The cluster autoscaler adjusts the size of an OpenShift Container Platform cluster to meet its current deployment needs. It uses declarative, Kubernetes-style arguments to provide infrastructure management that does not rely on objects of a specific cloud provider. The cluster autoscaler has a cluster scope, and is not associated with a particular namespace.

The cluster autoscaler increases the size of the cluster when there are pods that fail to schedule on any of the current worker nodes due to insufficient resources or when another node is necessary to meet deployment needs. The cluster autoscaler does not increase the cluster resources beyond the limits that you specify.

The cluster autoscaler computes the total memory, CPU, and GPU on all nodes the cluster, even though it does not manage the control plane nodes. These values are not single-machine oriented. They are an aggregation of all the resources in the entire cluster. For example, if you set the maximum memory resource limit, the cluster autoscaler includes all the nodes in the cluster when calculating the current memory usage. That calculation is then used to determine if the cluster autoscaler has the capacity to add more worker resources.

Important

Ensure that the

maxNodesTotal

value in the

ClusterAutoscaler

resource definition that you create is large enough to account for the total possible number of machines in your cluster. This value must encompass the number of control plane machines and the possible number of compute machines that you might scale to.

Every 10 seconds, the cluster autoscaler checks which nodes are unnecessary in the cluster and removes them. The cluster autoscaler considers a node for removal if the following conditions apply:

The sum of CPU and memory requests of all pods running on the node is less than 50% of the allocated resources on the node.
The cluster autoscaler can move all pods running on the node to the other nodes.
The cluster autoscaler does not have scale down disabled annotation.

If the following types of pods are present on a node, the cluster autoscaler will not remove the node:

Pods with restrictive pod disruption budgets (PDBs).
Kube-system pods that do not run on the node by default.
Kube-system pods that do not have a PDB or have a PDB that is too restrictive.
Pods that are not backed by a controller object such as a deployment, replica set, or stateful set.
Pods with local storage.
Pods that cannot be moved elsewhere because of a lack of resources, incompatible node selectors or affinity, matching anti-affinity, and so on.

Unless they also have a

"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"

annotation, pods that have a

"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

annotation.

For example, you set the maximum CPU limit to 64 cores and configure the cluster autoscaler to only create machines that have 8 cores each. If your cluster starts with 30 cores, the cluster autoscaler can add up to 4 more nodes with 32 cores, for a total of 62.

If you configure the cluster autoscaler, additional usage restrictions apply:

Do not modify the nodes that are in autoscaled node groups directly. All nodes within the same node group have the same capacity and labels and run the same system pods.
Specify requests for your pods.
If you have to prevent pods from being deleted too quickly, configure appropriate PDBs.
Confirm that your cloud provider quota is large enough to support the maximum node pools that you configure.
Do not run additional node group autoscalers, especially the ones offered by your cloud provider.

The horizontal pod autoscaler (HPA) and the cluster autoscaler modify cluster resources in different ways. The HPA changes the deployment’s or replica set’s number of replicas based on the current CPU load. If the load increases, the HPA creates new replicas, regardless of the amount of resources available to the cluster. If there are not enough resources, the cluster autoscaler adds resources so that the HPA-created pods can run. If the load decreases, the HPA stops some replicas. If this action causes some nodes to be underutilized or completely empty, the cluster autoscaler deletes the unnecessary nodes.

The cluster autoscaler takes pod priorities into account. The Pod Priority and Preemption feature enables scheduling pods based on priorities if the cluster does not have enough resources, but the cluster autoscaler ensures that the cluster has resources to run all pods. To honor the intention of both features, the cluster autoscaler includes a priority cutoff function. You can use this cutoff to schedule "best-effort" pods, which do not cause the cluster autoscaler to increase resources but instead run only when spare resources are available.

Pods with priority lower than the cutoff value do not cause the cluster to scale up or prevent the cluster from scaling down. No new nodes are added to run the pods, and nodes running these pods might be deleted to free resources.

6.2. About the machine autoscaler
Copy link

The machine autoscaler adjusts the number of Machines in the machine sets that you deploy in an OpenShift Container Platform cluster. You can scale both the default

worker

machine set and any other machine sets that you create. The machine autoscaler makes more Machines when the cluster runs out of resources to support more deployments. Any changes to the values in

MachineAutoscaler

resources, such as the minimum or maximum number of instances, are immediately applied to the machine set they target.

Important

You must deploy a machine autoscaler for the cluster autoscaler to scale your machines. The cluster autoscaler uses the annotations on machine sets that the machine autoscaler sets to determine the resources that it can scale. If you define a cluster autoscaler without also defining machine autoscalers, the cluster autoscaler will never scale your cluster.

6.3. Configuring the cluster autoscaler
Copy link

First, deploy the cluster autoscaler to manage automatic resource scaling in your OpenShift Container Platform cluster.

Note

Because the cluster autoscaler is scoped to the entire cluster, you can make only one cluster autoscaler for the cluster.

6.3.1. ClusterAutoscaler resource definition
Copy link

This

ClusterAutoscaler

resource definition shows the parameters and sample values for the cluster autoscaler.

apiVersion: "autoscaling.openshift.io/v1"
kind: "ClusterAutoscaler"
metadata:
  name: "default"
spec:
  podPriorityThreshold: -10

1


  resourceLimits:
    maxNodesTotal: 24

2


    cores:
      min: 8

3


      max: 128

4


    memory:
      min: 4

5


      max: 256

6


    gpus:
      - type: nvidia.com/gpu

7


        min: 0

8


        max: 16

9


      - type: amd.com/gpu
        min: 0
        max: 4
  scaleDown:

10


    enabled: true

11


    delayAfterAdd: 10m

12


    delayAfterDelete: 5m

13


    delayAfterFailure: 30s

14


    unneededTime: 5m

15

1: Specify the priority that a pod must exceed to cause the cluster autoscaler to deploy additional nodes. Enter a 32-bit integer value. The podPriorityThreshold value is compared to the value of the PriorityClass that you assign to each pod.
2: Specify the maximum number of nodes to deploy. This value is the total number of machines that are deployed in your cluster, not just the ones that the autoscaler controls. Ensure that this value is large enough to account for all of your control plane and compute machines and the total number of replicas that you specify in your MachineAutoscaler resources.
3: Specify the minimum number of cores to deploy in the cluster.
4: Specify the maximum number of cores to deploy in the cluster.
5: Specify the minimum amount of memory, in GiB, in the cluster.
6: Specify the maximum amount of memory, in GiB, in the cluster.
7: Optionally, specify the type of GPU node to deploy. Only nvidia.com/gpu and amd.com/gpu are valid types.
8: Specify the minimum number of GPUs to deploy in the cluster.
9: Specify the maximum number of GPUs to deploy in the cluster.
10: In this section, you can specify the period to wait for each action by using any valid ParseDuration interval, including ns, us, ms, s, m, and h.
11: Specify whether the cluster autoscaler can remove unnecessary nodes.
12: Optionally, specify the period to wait before deleting a node after a node has recently been added. If you do not specify a value, the default value of 10m is used.
13: Specify the period to wait before deleting a node after a node has recently been deleted. If you do not specify a value, the default value of 10s is used.
14: Specify the period to wait before deleting a node after a scale down failure occurred. If you do not specify a value, the default value of 3m is used.
15: Specify the period before an unnecessary node is eligible for deletion. If you do not specify a value, the default value of 10m is used.

Note

When performing a scaling operation, the cluster autoscaler remains within the ranges set in the

ClusterAutoscaler

resource definition, such as the minimum and maximum number of cores to deploy or the amount of memory in the cluster. However, the cluster autoscaler does not correct the current values in your cluster to be within those ranges.

The minimum and maximum CPUs, memory, and GPU values are determined by calculating those resources on all nodes in the cluster, even if the cluster autoscaler does not manage the nodes. For example, the control plane nodes are considered in the total memory in the cluster, even though the cluster autoscaler does not manage the control plane nodes.

6.3.2. Deploying the cluster autoscaler
Copy link

To deploy the cluster autoscaler, you create an instance of the

ClusterAutoscaler

resource.

Procedure

Create a YAML file for the
```
ClusterAutoscaler
```
resource that contains the customized resource definition.
Create the resource in the cluster:
```
$ oc create -f <filename>.yaml 
```
1
1
<filename> is the name of the resource file that you customized.

6.4. Next steps
Copy link

After you configure the cluster autoscaler, you must configure at least one machine autoscaler.

6.5. Configuring the machine autoscalers
Copy link

After you deploy the cluster autoscaler, deploy

MachineAutoscaler

resources that reference the machine sets that are used to scale the cluster.

Important

You must deploy at least one

MachineAutoscaler

resource after you deploy the

ClusterAutoscaler

resource.

Note

You must configure separate resources for each machine set. Remember that machine sets are different in each region, so consider whether you want to enable machine scaling in multiple regions. The machine set that you scale must have at least one machine in it.

6.5.1. MachineAutoscaler resource definition
Copy link

This

MachineAutoscaler

resource definition shows the parameters and sample values for the machine autoscaler.

apiVersion: "autoscaling.openshift.io/v1beta1"
kind: "MachineAutoscaler"
metadata:
  name: "worker-us-east-1a"

1


  namespace: "openshift-machine-api"
spec:
  minReplicas: 1

2


  maxReplicas: 12

3


  scaleTargetRef:

4


    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet

5


    name: worker-us-east-1a

6

1

Specify the machine autoscaler name. To make it easier to identify which machine set this machine autoscaler scales, specify or include the name of the machine set to scale. The machine set name takes the following form: <clusterid>-<machineset>-<region>.

2

Specify the minimum number machines of the specified type that must remain in the specified zone after the cluster autoscaler initiates cluster scaling. If running in AWS, GCP, Azure, RHOSP, or vSphere, this value can be set to 0. For other providers, do not set this value to 0.

You can save on costs by setting this value to

for use cases such as running expensive or limited-usage hardware that is used for specialized workloads, or by scaling a machine set with extra large machines. The cluster autoscaler scales the machine set down to zero if the machines are not in use.

Important

Do not set the

spec.minReplicas

value to

for the three compute machine sets that are created during the OpenShift Container Platform installation process for an installer provisioned infrastructure.

3

Specify the maximum number machines of the specified type that the cluster autoscaler can deploy in the specified zone after it initiates cluster scaling. Ensure that the maxNodesTotal value in the ClusterAutoscaler resource definition is large enough to allow the machine autoscaler to deploy this number of machines.

4

In this section, provide values that describe the existing machine set to scale.

The kind parameter value is always MachineSet.

The name value must match the name of an existing machine set, as shown in the metadata.name parameter value.

6.5.2. Deploying the machine autoscaler
Copy link

To deploy the machine autoscaler, you create an instance of the

MachineAutoscaler

resource.

Procedure

Create a YAML file for the
```
MachineAutoscaler
```
resource that contains the customized resource definition.
Create the resource in the cluster:
```
$ oc create -f <filename>.yaml 
```
1
1
<filename> is the name of the resource file that you customized.

Chapter 7. Creating infrastructure machine sets
Copy link

Important

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

You can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.

In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. Both OpenShift Logging and Red Hat OpenShift Service Mesh deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. This configuration requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

7.1. OpenShift Container Platform infrastructure components
Copy link

The following infrastructure workloads do not incur OpenShift Container Platform worker subscriptions:

Kubernetes and OpenShift Container Platform control plane services that run on masters
The default router
The integrated container image registry
The HAProxy-based Ingress Controller
The cluster metrics collection, or monitoring service, including components for monitoring user-defined projects
Cluster aggregated logging
Service brokers
Red Hat Quay
Red Hat OpenShift Container Storage
Red Hat Advanced Cluster Manager
Red Hat Advanced Cluster Security for Kubernetes
Red Hat OpenShift GitOps
Red Hat OpenShift Pipelines

Any node that runs any other container, pod, or component is a worker node that your subscription must cover.

To create an infrastructure node, you can use a machine set, label the node, or use a machine config pool.

7.2. Creating infrastructure machine sets for production environments
Copy link

In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. Both OpenShift Logging and Red Hat OpenShift Service Mesh deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. A configuration like this requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

7.2.1. Creating machine sets for different clouds
Copy link

Use the sample machine set for your cloud.

7.2.1.1. Sample YAML for a machine set custom resource on AWS
Copy link

This sample YAML defines a machine set that runs in the

us-east-1a

Amazon Web Services (AWS) zone and creates nodes that are labeled with

node-role.kubernetes.io/infra: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<infra>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


  name: <infrastructure_id>-infra-<zone>

2


  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

3


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<zone>

4


  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

5


        machine.openshift.io/cluster-api-machine-role: <infra>

6


        machine.openshift.io/cluster-api-machine-type: <infra>

7


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<zone>

8


    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""

9


      taints:

10


        - key: node-role.kubernetes.io/infra
          effect: NoSchedule
      providerSpec:
        value:
          ami:
            id: ami-046fe691f52a953f9

11


          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
            - ebs:
                iops: 0
                volumeSize: 120
                volumeType: gp2
          credentialsSecret:
            name: aws-cloud-credentials
          deviceIndex: 0
          iamInstanceProfile:
            id: <infrastructure_id>-worker-profile

12


          instanceType: m4.large
          kind: AWSMachineProviderConfig
          placement:
            availabilityZone: <zone>

13


            region: <region>

14


          securityGroups:
            - filters:
                - name: tag:Name
                  values:
                    - <infrastructure_id>-worker-sg

15


          subnet:
            filters:
              - name: tag:Name
                values:
                  - <infrastructure_id>-private-<zone>

16


          tags:
            - name: kubernetes.io/cluster/<infrastructure_id>

17


              value: owned
          userDataSecret:
            name: worker-user-data

1 3 5 12 15 17

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2 4 8

Specify the infrastructure ID, <infra> node label, and zone.

6 7 9

Specify the <infra> node label.

Specify a taint to prevent user workloads from being scheduled on infra nodes.

Specify a valid Red Hat Enterprise Linux CoreOS (RHCOS) AMI for your AWS zone for your OpenShift Container Platform nodes. If you want to use an AWS Marketplace image, you must complete the OpenShift Container Platform subscription from the AWS Marketplace to obtain an AMI ID for your region.

$ oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.ami.id}{"\n"}' \
    get machineset/<infrastructure_id>-worker-<zone>

Specify the zone, for example, us-east-1a.

Specify the region, for example, us-east-1.

Specify the infrastructure ID and zone.

Machine sets running on AWS support non-guaranteed Spot Instances. You can save on costs by using Spot Instances at a lower price compared to On-Demand Instances on AWS. Configure Spot Instances by adding

spotMarketOptions

to the

MachineSet

YAML file.

7.2.1.2. Sample YAML for a machine set custom resource on Azure
Copy link

This sample YAML defines a machine set that runs in the

Microsoft Azure zone in a region and creates nodes that are labeled with

node-role.kubernetes.io/infra: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<infra>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


    machine.openshift.io/cluster-api-machine-role: <infra>

2


    machine.openshift.io/cluster-api-machine-type: <infra>

3


  name: <infrastructure_id>-infra-<region>

4


  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

5


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<region>

6


  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

7


        machine.openshift.io/cluster-api-machine-role: <infra>

8


        machine.openshift.io/cluster-api-machine-type: <infra>

9


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<region>

10


    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/infra: ""

11


      providerSpec:
        value:
          apiVersion: azureproviderconfig.openshift.io/v1beta1
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image:

12


            offer: ""
            publisher: ""
            resourceID: /resourceGroups/<infrastructure_id>-rg/providers/Microsoft.Compute/images/<infrastructure_id>

13


            sku: ""
            version: ""
          internalLoadBalancer: ""
          kind: AzureMachineProviderSpec
          location: <region>

14


          managedIdentity: <infrastructure_id>-identity

15


          metadata:
            creationTimestamp: null
          natRule: null
          networkResourceGroup: ""
          osDisk:
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS
            osType: Linux
          publicIP: false
          publicLoadBalancer: ""
          resourceGroup: <infrastructure_id>-rg

16


          sshPrivateKey: ""
          sshPublicKey: ""
          subnet: <infrastructure_id>-<role>-subnet

17

18


          userDataSecret:
            name: worker-user-data

19


          vmSize: Standard_DS4_v2
          vnet: <infrastructure_id>-vnet

20


          zone: "1"

21


      taints:

22


      - key: node-role.kubernetes.io/infra
        effect: NoSchedule

1 5 7 15 16 17 20

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

You can obtain the subnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.subnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1

You can obtain the vnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.vnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1

2 3 8 9 11 18 19

Specify the <infra> node label.

4 6 10

Specify the infrastructure ID, <infra> node label, and region.

Specify the image details for your machine set. If you want to use an Azure Marketplace image, see "Selecting an Azure Marketplace image".

Specify an image that is compatible with your instance type. The Hyper-V generation V2 images created by the installation program have a -gen2 suffix, while V1 images have the same name without the suffix.

Specify the region to place machines on.

Specify the zone within your region to place machines on. Be sure that your region supports the zone that you specify.

22

Specify a taint to prevent user workloads from being scheduled on infra nodes.

Machine sets running on Azure support non-guaranteed Spot VMs. You can save on costs by using Spot VMs at a lower price compared to standard VMs on Azure. You can configure Spot VMs by adding

spotVMOptions

to the

MachineSet

YAML file.

7.2.1.3. Sample YAML for a machine set custom resource on GCP
Copy link

This sample YAML defines a machine set that runs in Google Cloud Platform (GCP) and creates nodes that are labeled with

node-role.kubernetes.io/infra: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<infra>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


  name: <infrastructure_id>-w-a
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: <infra>

2


        machine.openshift.io/cluster-api-machine-type: <infra>
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""
      providerSpec:
        value:
          apiVersion: gcpprovider.openshift.io/v1beta1
          canIPForward: false
          credentialsSecret:
            name: gcp-cloud-credentials
          deletionProtection: false
          disks:
          - autoDelete: true
            boot: true
            image: <path_to_image>

3


            labels: null
            sizeGb: 128
            type: pd-ssd
          gcpMetadata:

4


          - key: <custom_metadata_key>
            value: <custom_metadata_value>
          kind: GCPMachineProviderSpec
          machineType: n1-standard-4
          metadata:
            creationTimestamp: null
          networkInterfaces:
          - network: <infrastructure_id>-network
            subnetwork: <infrastructure_id>-worker-subnet
          projectID: <project_name>

5


          region: us-central1
          serviceAccounts:
          - email: <infrastructure_id>-w@<project_name>.iam.gserviceaccount.com
            scopes:
            - https://www.googleapis.com/auth/cloud-platform
          tags:
            - <infrastructure_id>-worker
          userDataSecret:
            name: worker-user-data
          zone: us-central1-a
      taints:

6


      - key: node-role.kubernetes.io/infra
        effect: NoSchedule

1

For <infrastructure_id>, specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2

For <infra>, specify the <infra> node label.

3

Specify the path to the image that is used in current compute machine sets. If you have the OpenShift CLI installed, you can obtain the path to the image by running the following command:

$ oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.disks[0].image}{"\n"}' \
    get machineset/<infrastructure_id>-worker-a

To use a GCP Marketplace image, specify the offer to use:

OpenShift Container Platform:

https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-ocp-48-x86-64-202210040145

OpenShift Platform Plus:

https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-opp-48-x86-64-202206140145

OpenShift Kubernetes Engine:

https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-oke-48-x86-64-202206140145

4

Optional: Specify custom metadata in the form of a key:value pair. For example use cases, see the GCP documentation for setting custom metadata.

For <project_name>, specify the name of the GCP project that you use for your cluster.

Specify a taint to prevent user workloads from being scheduled on infra nodes.

Machine sets running on GCP support non-guaranteed preemptible VM instances. You can save on costs by using preemptible VM instances at a lower price compared to normal instances on GCP. You can configure preemptible VM instances by adding

preemptible

to the

MachineSet

YAML file.

7.2.1.4. Sample YAML for a machine set custom resource on RHOSP
Copy link

This sample YAML defines a machine set that runs on Red Hat OpenStack Platform (RHOSP) and creates nodes that are labeled with

node-role.kubernetes.io/infra: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<infra>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


    machine.openshift.io/cluster-api-machine-role: <infra>

2


    machine.openshift.io/cluster-api-machine-type: <infra>

3


  name: <infrastructure_id>-infra

4


  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas>
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

5


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra

6


  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

7


        machine.openshift.io/cluster-api-machine-role: <infra>

8


        machine.openshift.io/cluster-api-machine-type: <infra>

9


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra

10


    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/infra: ""
      taints:

11


      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
      providerSpec:
        value:
          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: <nova_flavor>
          image: <glance_image_name_or_location>
          serverGroupID: <optional_UUID_of_server_group>

12


          kind: OpenstackProviderSpec
          networks:

13


          - filter: {}
            subnets:
            - filter:
                name: <subnet_name>
                tags: openshiftClusterID=<infrastructure_id>

14


          primarySubnet: <rhosp_subnet_UUID>

15


          securityGroups:
          - filter: {}
            name: <infrastructure_id>-worker

16


          serverMetadata:
            Name: <infrastructure_id>-worker

17


            openshiftClusterID: <infrastructure_id>

18


          tags:
          - openshiftClusterID=<infrastructure_id>

19


          trunk: true
          userDataSecret:
            name: worker-user-data

20


          availabilityZone: <optional_openstack_availability_zone>

1 5 7 14 16 17 18 19

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2 3 8 9 20

Specify the <infra> node label.

4 6 10

Specify the infrastructure ID and <infra> node label.

Specify a taint to prevent user workloads from being scheduled on infra nodes.

To set a server group policy for the MachineSet, enter the value that is returned from creating a server group. For most deployments, anti-affinity or soft-anti-affinity policies are recommended.

Required for deployments to multiple networks. If deploying to multiple networks, this list must include the network that is used as the primarySubnet value.

Specify the RHOSP subnet that you want the endpoints of nodes to be published on. Usually, this is the same subnet that is used as the value of machinesSubnet in the install-config.yaml file.

7.2.1.5. Sample YAML for a machine set custom resource on RHV
Copy link

This sample YAML defines a machine set that runs on RHV and creates nodes that are labeled with

node-role.kubernetes.io/<node_role>: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<role>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


    machine.openshift.io/cluster-api-machine-role: <role>

2


    machine.openshift.io/cluster-api-machine-type: <role>

3


  name: <infrastructure_id>-<role>

4


  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas>

5


  selector:

6


    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

7


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

8


  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

9


        machine.openshift.io/cluster-api-machine-role: <role>

10


        machine.openshift.io/cluster-api-machine-type: <role>

11


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>

12


    spec:
      metadata:
        labels:
          node-role.kubernetes.io/<role>: ""

13


      providerSpec:
        value:
          apiVersion: ovirtproviderconfig.machine.openshift.io/v1beta1
          cluster_id: <ovirt_cluster_id>

14


          template_name: <ovirt_template_name>

15


          instance_type_id: <instance_type_id>

16


          cpu:

17


            sockets: <number_of_sockets>

18


            cores: <number_of_cores>

19


            threads: <number_of_threads>

20


          memory_mb: <memory_size>

21


          os_disk:

22


            size_gb: <disk_size>

23


          network_interfaces:

24


            vnic_profile_id:  <vnic_profile_id>

25


          credentialsSecret:
            name: ovirt-credentials

26


          kind: OvirtMachineProviderSpec
          type: <workload_type>

27


          userDataSecret:
            name: worker-user-data
          affinityGroupsNames:
            - compute

28

1 7 9

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2 3 10 11 13

Specify the node label to add.

4 8 12

Specify the infrastructure ID and node label. These two strings together cannot be longer than 35 characters.

Specify the number of machines to create.

Selector for the machines.

Specify the UUID for the RHV cluster to which this VM instance belongs.

Specify the RHV VM template to use to create the machine.

Optional: Specify the VM instance type.

Warning

The

instance_type_id

field is deprecated and will be removed in a future release.

If you include this parameter, you do not need to specify the hardware parameters of the VM including CPU and memory because this parameter overrides all hardware parameters.

17

Optional: The CPU field contains the CPU’s configuration, including sockets, cores, and threads.

18

Optional: Specify the number of sockets for a VM.

19

Optional: Specify the number of cores per socket.

20

Optional: Specify the number of threads per core.

Optional: Specify the size of a VM’s memory in MiB.

22

Optional: Root disk of the node.

23

Optional: Specify the size of the bootable disk in GiB.

24

Optional: List of the network interfaces of the VM. If you include this parameter, OpenShift Container Platform discards all network interfaces from the template and creates new ones.

25

Optional: Specify the vNIC profile ID.

26

Specify the name of the secret that holds the RHV credentials.

27

Optional: Specify the workload type for which the instance is optimized. This value affects the RHV VM parameter. Supported values: desktop, server (default), high_performance. high_performance improves performance on the VM, but there are limitations. For example, you cannot access the VM with a graphical console. For more information see Configuring High Performance Virtual Machines, Templates, and Pools in the Virtual Machine Management Guide.

28

A list of affinity group names that should be applied to the VMs. The affinity groups must exist in oVirt.

Note

Because RHV uses a template when creating a VM, if you do not specify a value for an optional parameter, RHV uses the value for that parameter that is specified in the template.

7.2.1.6. Sample YAML for a machine set custom resource on vSphere
Copy link

This sample YAML defines a machine set that runs on VMware vSphere and creates nodes that are labeled with

node-role.kubernetes.io/infra: ""

.

In this sample,

<infrastructure_id>

is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and

<infra>

is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>

1


  name: <infrastructure_id>-infra

2


  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>

3


      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra

4


  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>

5


        machine.openshift.io/cluster-api-machine-role: <infra>

6


        machine.openshift.io/cluster-api-machine-type: <infra>

7


        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra

8


    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/infra: ""

9


      taints:

10


      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
      providerSpec:
        value:
          apiVersion: vsphereprovider.openshift.io/v1beta1
          credentialsSecret:
            name: vsphere-cloud-credentials
          diskGiB: 120
          kind: VSphereMachineProviderSpec
          memoryMiB: 8192
          metadata:
            creationTimestamp: null
          network:
            devices:
            - networkName: "<vm_network_name>"

11


          numCPUs: 4
          numCoresPerSocket: 1
          snapshot: ""
          template: <vm_template_name>

12


          userDataSecret:
            name: worker-user-data
          workspace:
            datacenter: <vcenter_datacenter_name>

13


            datastore: <vcenter_datastore_name>

14


            folder: <vcenter_vm_folder_path>

15


            resourcepool: <vsphere_resource_pool>

16


            server: <vcenter_server_ip>

17

1 3 5

Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

2 4 8

Specify the infrastructure ID and <infra> node label.

6 7 9

Specify the <infra> node label.

Specify a taint to prevent user workloads from being scheduled on infra nodes.

Specify the vSphere VM network to deploy the machine set to. This VM network must be where other compute machines reside in the cluster.

Specify the vSphere VM template to use, such as user-5ddjd-rhcos.

Specify the vCenter Datacenter to deploy the machine set on.

Specify the vCenter Datastore to deploy the machine set on.

Specify the path to the vSphere VM folder in vCenter, such as /dc1/vm/user-inst-5ddjd.