Machine management

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

4. If you need compute machine sets in other availability zones, repeat this process to create more compute machine sets.

Procedure

1. View the existing nodes, machines, and machine sets by running the following command. Note that each node is an instance of a machine definition with a specific AWS region and OpenShift Container Platform role.

   $ oc get nodes

   Example output

   NAME                                        STATUS   ROLES                  AGE     VERSION
   ip-10-0-52-50.us-east-2.compute.internal    Ready    worker                 3d17h   v1.25.4+86bd4ff
   ip-10-0-58-24.us-east-2.compute.internal    Ready    control-plane,master   3d17h   v1.25.4+86bd4ff
   ip-10-0-68-148.us-east-2.compute.internal   Ready    worker                 3d17h   v1.25.4+86bd4ff
   ip-10-0-68-68.us-east-2.compute.internal    Ready    control-plane,master   3d17h   v1.25.4+86bd4ff
   ip-10-0-72-170.us-east-2.compute.internal   Ready    control-plane,master   3d17h   v1.25.4+86bd4ff
   ip-10-0-74-50.us-east-2.compute.internal    Ready    worker                 3d17h   v1.25.4+86bd4ff

2. View the machines and machine sets that exist in the openshift-machine-api namespace by running the following command. Each compute machine set is associated with a different availability zone within the AWS region. The installer automatically load balances compute machines across availability zones.

   $ oc get machinesets -n openshift-machine-api

   Example output

   NAME                                        DESIRED   CURRENT   READY   AVAILABLE   AGE
   preserve-dsoc12r4-ktjfc-worker-us-east-2a   1         1         1       1           3d11h
   preserve-dsoc12r4-ktjfc-worker-us-east-2b   2         2         2       2           3d11h

3. View the machines that exist in the openshift-machine-api namespace by running the following command. At this time, there is only one compute machine per machine set, though a compute machine set could be scaled to add a node in a particular region and zone.

   $ oc get machines -n openshift-machine-api | grep worker

   Example output

   preserve-dsoc12r4-ktjfc-worker-us-east-2a-dts8r      Running   m5.xlarge   us-east-2   us-east-2a   3d11h
   preserve-dsoc12r4-ktjfc-worker-us-east-2b-dkv7w      Running   m5.xlarge   us-east-2   us-east-2b   3d11h
   preserve-dsoc12r4-ktjfc-worker-us-east-2b-k58cw      Running   m5.xlarge   us-east-2   us-east-2b   3d11h

4. Make a copy of one of the existing compute MachineSet definitions and output the result to a JSON file by running the following command. This will be the basis for the GPU-enabled compute machine set definition.

   $ oc get machineset preserve-dsoc12r4-ktjfc-worker-us-east-2a -n openshift-machine-api -o json > <output_file.json>

5. Edit the JSON file and make the following changes to the new MachineSet definition:

   • Replace worker with gpu. This will be the name of the new machine set.

   • Change the instance type of the new MachineSet definition to g4dn, which includes an NVIDIA Tesla T4 GPU. To learn more about AWS g4dn instance types, see Accelerated Computing.

     $ jq .spec.template.spec.providerSpec.value.instanceType preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a.json
     
     "g4dn.xlarge"

     The <output_file.json> file is saved as preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a.json.

6. Update the following fields in preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a.json:

   • .metadata.name to a name containing gpu.
   • .spec.selector.matchLabels["machine.openshift.io/cluster-api-machineset"] to match the new .metadata.name.
   • .spec.template.metadata.labels["machine.openshift.io/cluster-api-machineset"] to match the new .metadata.name.
   • .spec.template.spec.providerSpec.value.instanceType to g4dn.xlarge.

7. To verify your changes, perform a diff of the original compute definition and the new GPU-enabled node definition by running the following command:

   $ oc -n openshift-machine-api get preserve-dsoc12r4-ktjfc-worker-us-east-2a -o json | diff preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a.json -

   Example output

   10c10
   
   < "name": "preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a",
   ---
   > "name": "preserve-dsoc12r4-ktjfc-worker-us-east-2a",
   
   21c21
   
   < "machine.openshift.io/cluster-api-machineset": "preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a"
   ---
   > "machine.openshift.io/cluster-api-machineset": "preserve-dsoc12r4-ktjfc-worker-us-east-2a"
   
   31c31
   
   < "machine.openshift.io/cluster-api-machineset": "preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a"
   ---
   > "machine.openshift.io/cluster-api-machineset": "preserve-dsoc12r4-ktjfc-worker-us-east-2a"
   
   60c60
   
   < "instanceType": "g4dn.xlarge",
   ---
   > "instanceType": "m5.xlarge",

8. Create the GPU-enabled compute machine set from the definition by running the following command:

   $ oc create -f preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a.json

   Example output

   machineset.machine.openshift.io/preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a created

Verification

1. View the machine set you created by running the following command:

   $ oc -n openshift-machine-api get machinesets | grep gpu

   The MachineSet replica count is set to 1 so a new Machine object is created automatically.

   Example output

   preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a   1         1         1       1           4m21s

2. View the Machine object that the machine set created by running the following command:

   $ oc -n openshift-machine-api get machines | grep gpu

   Example output

   preserve-dsoc12r4-ktjfc-worker-gpu-us-east-2a    running    g4dn.xlarge   us-east-2   us-east-2a  4m36s

Procedure

1. Install the Node Feature Discovery Operator from OperatorHub in the OpenShift Container Platform console.
2. After installing the NFD Operator into OperatorHub, select Node Feature Discovery from the installed Operators list and select Create instance. This installs the nfd-master and nfd-worker pods, one nfd-worker pod for each compute node, in the openshift-nfd namespace.

3. Verify that the Operator is installed and running by running the following command:

   $ oc get pods -n openshift-nfd

   Example output

   NAME                                       READY    STATUS     RESTARTS   AGE
   
   nfd-controller-manager-8646fcbb65-x5qgk    2/2      Running 7  (8h ago)   1d

4. Browse to the installed Oerator in the console and select Create Node Feature Discovery.
5. Select Create to build a NFD custom resource. This creates NFD pods in the openshift-nfd namespace that poll the OpenShift Container Platform nodes for hardware resources and catalogue them.

Verification

1. After a successful build, verify that a NFD pod is running on each nodes by running the following command:

   $ oc get pods -n openshift-nfd

   Example output

   NAME                                       READY   STATUS      RESTARTS        AGE
   nfd-controller-manager-8646fcbb65-x5qgk    2/2     Running     7 (8h ago)      12d
   nfd-master-769656c4cb-w9vrv                1/1     Running     0               12d
   nfd-worker-qjxb2                           1/1     Running     3 (3d14h ago)   12d
   nfd-worker-xtz9b                           1/1     Running     5 (3d14h ago)   12d

   The NFD Operator uses vendor PCI IDs to identify hardware in a node. NVIDIA uses the PCI ID 10de.

2. View the NVIDIA GPU discovered by the NFD Operator by running the following command:

   $ oc describe node ip-10-0-132-138.us-east-2.compute.internal | egrep 'Roles|pci'

   Example output

   Roles: worker
   
   feature.node.kubernetes.io/pci-1013.present=true
   
   feature.node.kubernetes.io/pci-10de.present=true
   
   feature.node.kubernetes.io/pci-1d0f.present=true

   10de appears in the node feature list for the GPU-enabled node. This mean the NFD Operator correctly identified the node from the GPU-enabled MachineSet.

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Display all of the available OpenShift Container Platform images by running one of the following commands:

   • North America:

     $  az vm image list --all --offer rh-ocp-worker --publisher redhat -o table

     Example output

     Offer          Publisher       Sku                 Urn                                                             Version
     -------------  --------------  ------------------  --------------------------------------------------------------  --------------
     rh-ocp-worker  RedHat          rh-ocp-worker       RedHat:rh-ocp-worker:rh-ocpworker:4.8.2021122100               4.8.2021122100
     rh-ocp-worker  RedHat          rh-ocp-worker-gen1  RedHat:rh-ocp-worker:rh-ocp-worker-gen1:4.8.2021122100         4.8.2021122100

   • EMEA:

     $  az vm image list --all --offer rh-ocp-worker --publisher redhat-limited -o table

     Example output

     Offer          Publisher       Sku                 Urn                                                             Version
     -------------  --------------  ------------------  --------------------------------------------------------------  --------------
     rh-ocp-worker  redhat-limited  rh-ocp-worker       redhat-limited:rh-ocp-worker:rh-ocp-worker:4.8.2021122100       4.8.2021122100
     rh-ocp-worker  redhat-limited  rh-ocp-worker-gen1  redhat-limited:rh-ocp-worker:rh-ocp-worker-gen1:4.8.2021122100  4.8.2021122100

   Note

   Regardless of the version of OpenShift Container Platform that you install, the correct version of the Azure Marketplace image to use is 4.8. If required, your VMs are automatically upgraded as part of the installation process.

2. Inspect the image for your offer by running one of the following commands:

   • North America:

     $ az vm image show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

   • EMEA:

     $ az vm image show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

3. Review the terms of the offer by running one of the following commands:

   • North America:

     $ az vm image terms show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

   • EMEA:

     $ az vm image terms show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

4. Accept the terms of the offering by running one of the following commands:

   • North America:

     $ az vm image terms accept --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>

   • EMEA:

     $ az vm image terms accept --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>

5. Record the image details of your offer, specifically the values for publisher, offer, sku, and version.

6. Add the following parameters to the providerSpec section of your machine set YAML file using the image details for your offer:

   Sample providerSpec image values for Azure Marketplace machines

   providerSpec:
     value:
       image:
         offer: rh-ocp-worker
         publisher: redhat
         resourceID: ""
         sku: rh-ocp-worker
         type: MarketplaceWithPlan
         version: 4.8.2021122100

Procedure

1. View the machines and machine sets that exist in the openshift-machine-api namespace by running the following command. Each compute machine set is associated with a different availability zone within the Azure region. The installer automatically load balances compute machines across availability zones.

   $ oc get machineset -n openshift-machine-api

   Example output

   NAME                              DESIRED   CURRENT   READY   AVAILABLE   AGE
   myclustername-worker-centralus1   1         1         1       1           6h9m
   myclustername-worker-centralus2   1         1         1       1           6h9m
   myclustername-worker-centralus3   1         1         1       1           6h9m

2. Make a copy of one of the existing compute MachineSet definitions and output the result to a YAML file by running the following command. This will be the basis for the GPU-enabled compute machine set definition.

   $ oc get machineset -n openshift-machine-api myclustername-worker-centralus1 -o yaml > machineset-azure.yaml

3. View the content of the machineset:

   $ cat machineset-azure.yaml

   Example machineset-azure.yaml file

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     annotations:
       machine.openshift.io/GPU: "0"
       machine.openshift.io/memoryMb: "16384"
       machine.openshift.io/vCPU: "4"
     creationTimestamp: "2023-02-06T14:08:19Z"
     generation: 1
     labels:
       machine.openshift.io/cluster-api-cluster: myclustername
       machine.openshift.io/cluster-api-machine-role: worker
       machine.openshift.io/cluster-api-machine-type: worker
     name: myclustername-worker-centralus1
     namespace: openshift-machine-api
     resourceVersion: "23601"
     uid: acd56e0c-7612-473a-ae37-8704f34b80de
   spec:
     replicas: 1
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: myclustername
         machine.openshift.io/cluster-api-machineset: myclustername-worker-centralus1
     template:
       metadata:
         labels:
           machine.openshift.io/cluster-api-cluster: myclustername
           machine.openshift.io/cluster-api-machine-role: worker
           machine.openshift.io/cluster-api-machine-type: worker
           machine.openshift.io/cluster-api-machineset: myclustername-worker-centralus1
       spec:
         lifecycleHooks: {}
         metadata: {}
         providerSpec:
           value:
             acceleratedNetworking: true
             apiVersion: machine.openshift.io/v1beta1
             credentialsSecret:
               name: azure-cloud-credentials
               namespace: openshift-machine-api
             diagnostics: {}
             image:
               offer: ""
               publisher: ""
               resourceID: /resourceGroups/myclustername-rg/providers/Microsoft.Compute/galleries/gallery_myclustername_n6n4r/images/myclustername-gen2/versions/latest
               sku: ""
               version: ""
             kind: AzureMachineProviderSpec
             location: centralus
             managedIdentity: myclustername-identity
             metadata:
               creationTimestamp: null
             networkResourceGroup: myclustername-rg
             osDisk:
               diskSettings: {}
               diskSizeGB: 128
               managedDisk:
                 storageAccountType: Premium_LRS
               osType: Linux
             publicIP: false
             publicLoadBalancer: myclustername
             resourceGroup: myclustername-rg
             spotVMOptions: {}
             subnet: myclustername-worker-subnet
             userDataSecret:
               name: worker-user-data
             vmSize: Standard_D4s_v3
             vnet: myclustername-vnet
             zone: "1"
   status:
     availableReplicas: 1
     fullyLabeledReplicas: 1
     observedGeneration: 1
     readyReplicas: 1
     replicas: 1

4. Make a copy of the machineset-azure.yaml file by running the following command:

   $ cp machineset-azure.yaml machineset-azure-gpu.yaml

5. Update the following fields in machineset-azure-gpu.yaml:

   • Change .metadata.name to a name containing gpu.
   • Change .spec.selector.matchLabels["machine.openshift.io/cluster-api-machineset"] to match the new .metadata.name.
   • Change .spec.template.metadata.labels["machine.openshift.io/cluster-api-machineset"] to match the new .metadata.name.

   • Change .spec.template.spec.providerSpec.value.vmSize to Standard_NC4as_T4_v3.

     Example machineset-azure-gpu.yaml file

     apiVersion: machine.openshift.io/v1beta1
     kind: MachineSet
     metadata:
       annotations:
         machine.openshift.io/GPU: "1"
         machine.openshift.io/memoryMb: "28672"
         machine.openshift.io/vCPU: "4"
       creationTimestamp: "2023-02-06T20:27:12Z"
       generation: 1
       labels:
         machine.openshift.io/cluster-api-cluster: myclustername
         machine.openshift.io/cluster-api-machine-role: worker
         machine.openshift.io/cluster-api-machine-type: worker
       name: myclustername-nc4ast4-gpu-worker-centralus1
       namespace: openshift-machine-api
       resourceVersion: "166285"
       uid: 4eedce7f-6a57-4abe-b529-031140f02ffa
     spec:
       replicas: 1
       selector:
         matchLabels:
           machine.openshift.io/cluster-api-cluster: myclustername
           machine.openshift.io/cluster-api-machineset: myclustername-nc4ast4-gpu-worker-centralus1
       template:
         metadata:
           labels:
             machine.openshift.io/cluster-api-cluster: myclustername
             machine.openshift.io/cluster-api-machine-role: worker
             machine.openshift.io/cluster-api-machine-type: worker
             machine.openshift.io/cluster-api-machineset: myclustername-nc4ast4-gpu-worker-centralus1
         spec:
           lifecycleHooks: {}
           metadata: {}
           providerSpec:
             value:
               acceleratedNetworking: true
               apiVersion: machine.openshift.io/v1beta1
               credentialsSecret:
                 name: azure-cloud-credentials
                 namespace: openshift-machine-api
               diagnostics: {}
               image:
                 offer: ""
                 publisher: ""
                 resourceID: /resourceGroups/myclustername-rg/providers/Microsoft.Compute/galleries/gallery_myclustername_n6n4r/images/myclustername-gen2/versions/latest
                 sku: ""
                 version: ""
               kind: AzureMachineProviderSpec
               location: centralus
               managedIdentity: myclustername-identity
               metadata:
                 creationTimestamp: null
               networkResourceGroup: myclustername-rg
               osDisk:
                 diskSettings: {}
                 diskSizeGB: 128
                 managedDisk:
                   storageAccountType: Premium_LRS
                 osType: Linux
               publicIP: false
               publicLoadBalancer: myclustername
               resourceGroup: myclustername-rg
               spotVMOptions: {}
               subnet: myclustername-worker-subnet
               userDataSecret:
                 name: worker-user-data
               vmSize: Standard_NC4as_T4_v3
               vnet: myclustername-vnet
               zone: "1"
     status:
       availableReplicas: 1
       fullyLabeledReplicas: 1
       observedGeneration: 1
       readyReplicas: 1
       replicas: 1

6. To verify your changes, perform a diff of the original compute definition and the new GPU-enabled node definition by running the following command:

   $ diff machineset-azure.yaml machineset-azure-gpu.yaml

   Example output

   14c14
   <   name: myclustername-worker-centralus1
   ---
   >   name: myclustername-nc4ast4-gpu-worker-centralus1
   23c23
   <       machine.openshift.io/cluster-api-machineset: myclustername-worker-centralus1
   ---
   >       machine.openshift.io/cluster-api-machineset: myclustername-nc4ast4-gpu-worker-centralus1
   30c30
   <         machine.openshift.io/cluster-api-machineset: myclustername-worker-centralus1
   ---
   >         machine.openshift.io/cluster-api-machineset: myclustername-nc4ast4-gpu-worker-centralus1
   67c67
   <           vmSize: Standard_D4s_v3
   ---
   >           vmSize: Standard_NC4as_T4_v3

7. Create the GPU-enabled compute machine set from the definition file by running the following command:

   $ oc create -f machineset-azure-gpu.yaml

   Example output

   machineset.machine.openshift.io/myclustername-nc4ast4-gpu-worker-centralus1 created

8. View the machines and machine sets that exist in the openshift-machine-api namespace by running the following command. Each compute machine set is associated with a different availability zone within the Azure region. The installer automatically load balances compute machines across availability zones.

   $ oc get machineset -n openshift-machine-api

   Example output

   NAME                                               DESIRED   CURRENT   READY   AVAILABLE   AGE
   clustername-n6n4r-nc4ast4-gpu-worker-centralus1    1         1         1       1           122m
   clustername-n6n4r-worker-centralus1                1         1         1       1           8h
   clustername-n6n4r-worker-centralus2                1         1         1       1           8h
   clustername-n6n4r-worker-centralus3                1         1         1       1           8h

9. View the machines that exist in the openshift-machine-api namespace by running the following command. You can only configure one compute machine per set, although you can scale a compute machine set to add a node in a particular region and zone.

   $ oc get machines -n openshift-machine-api

   Example output

   NAME                                                PHASE     TYPE                   REGION      ZONE   AGE
   myclustername-master-0                              Running   Standard_D8s_v3        centralus   2      6h40m
   myclustername-master-1                              Running   Standard_D8s_v3        centralus   1      6h40m
   myclustername-master-2                              Running   Standard_D8s_v3        centralus   3      6h40m
   myclustername-nc4ast4-gpu-worker-centralus1-w9bqn   Running      centralus   1      21m
   myclustername-worker-centralus1-rbh6b               Running   Standard_D4s_v3        centralus   1      6h38m
   myclustername-worker-centralus2-dbz7w               Running   Standard_D4s_v3        centralus   2      6h38m
   myclustername-worker-centralus3-p9b8c               Running   Standard_D4s_v3        centralus   3      6h38m

10. View the existing nodes, machines, and machine sets by running the following command. Note that each node is an instance of a machine definition with a specific Azure region and OpenShift Container Platform role.

    $ oc get nodes

    Example output

    NAME                                                STATUS   ROLES                  AGE     VERSION
    myclustername-master-0                              Ready    control-plane,master   6h39m   v1.25.4+a34b9e9
    myclustername-master-1                              Ready    control-plane,master   6h41m   v1.25.4+a34b9e9
    myclustername-master-2                              Ready    control-plane,master   6h39m   v1.25.4+a34b9e9
    myclustername-nc4ast4-gpu-worker-centralus1-w9bqn   Ready    worker                 14m     v1.25.4+a34b9e9
    myclustername-worker-centralus1-rbh6b               Ready    worker                 6h29m   v1.25.4+a34b9e9
    myclustername-worker-centralus2-dbz7w               Ready    worker                 6h29m   v1.25.4+a34b9e9
    myclustername-worker-centralus3-p9b8c               Ready    worker                 6h31m   v1.25.4+a34b9e9

11. View the list of compute machine sets:

    $ oc get machineset -n openshift-machine-api

    Example output

    NAME                                   DESIRED   CURRENT   READY   AVAILABLE   AGE
    myclustername-worker-centralus1        1         1         1       1           8h
    myclustername-worker-centralus2        1         1         1       1           8h
    myclustername-worker-centralus3        1         1         1       1           8h

12. Create the GPU-enabled compute machine set from the definition file by running the following command:

    $ oc create -f machineset-azure-gpu.yaml

13. View the list of compute machine sets:

    oc get machineset -n openshift-machine-api

    Example output

    NAME                                          DESIRED   CURRENT   READY   AVAILABLE   AGE
    myclustername-nc4ast4-gpu-worker-centralus1   1         1         1       1           121m
    myclustername-worker-centralus1               1         1         1       1           8h
    myclustername-worker-centralus2               1         1         1       1           8h
    myclustername-worker-centralus3               1         1         1       1           8h

Verification

1. View the machine set you created by running the following command:

   $ oc get machineset -n openshift-machine-api | grep gpu

   The MachineSet replica count is set to 1 so a new Machine object is created automatically.

   Example output

   myclustername-nc4ast4-gpu-worker-centralus1   1         1         1       1           121m

2. View the Machine object that the machine set created by running the following command:

   $ oc -n openshift-machine-api get machines | grep gpu

   Example output

   myclustername-nc4ast4-gpu-worker-centralus1-w9bqn   Running   Standard_NC4as_T4_v3   centralus   1      21m

Procedure

1. Install the Node Feature Discovery Operator from OperatorHub in the OpenShift Container Platform console.
2. After installing the NFD Operator into OperatorHub, select Node Feature Discovery from the installed Operators list and select Create instance. This installs the nfd-master and nfd-worker pods, one nfd-worker pod for each compute node, in the openshift-nfd namespace.

3. Verify that the Operator is installed and running by running the following command:

   $ oc get pods -n openshift-nfd

   Example output

   NAME                                       READY    STATUS     RESTARTS   AGE
   
   nfd-controller-manager-8646fcbb65-x5qgk    2/2      Running 7  (8h ago)   1d

4. Browse to the installed Oerator in the console and select Create Node Feature Discovery.
5. Select Create to build a NFD custom resource. This creates NFD pods in the openshift-nfd namespace that poll the OpenShift Container Platform nodes for hardware resources and catalogue them.

Verification

1. After a successful build, verify that a NFD pod is running on each nodes by running the following command:

   $ oc get pods -n openshift-nfd

   Example output

   NAME                                       READY   STATUS      RESTARTS        AGE
   nfd-controller-manager-8646fcbb65-x5qgk    2/2     Running     7 (8h ago)      12d
   nfd-master-769656c4cb-w9vrv                1/1     Running     0               12d
   nfd-worker-qjxb2                           1/1     Running     3 (3d14h ago)   12d
   nfd-worker-xtz9b                           1/1     Running     5 (3d14h ago)   12d

   The NFD Operator uses vendor PCI IDs to identify hardware in a node. NVIDIA uses the PCI ID 10de.

2. View the NVIDIA GPU discovered by the NFD Operator by running the following command:

   $ oc describe node ip-10-0-132-138.us-east-2.compute.internal | egrep 'Roles|pci'

   Example output

   Roles: worker
   
   feature.node.kubernetes.io/pci-1013.present=true
   
   feature.node.kubernetes.io/pci-10de.present=true
   
   feature.node.kubernetes.io/pci-1d0f.present=true

   10de appears in the node feature list for the GPU-enabled node. This mean the NFD Operator correctly identified the node from the GPU-enabled MachineSet.

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <availabilitySet>, <clusterID>, and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. In a text editor, open the YAML file for an existing compute machine set or create a new one.

2. Edit the following line under the providerSpec field:

   providerSpec:
     value:
       disks:
         type: <pd-disk-type> 1

   1

   Specify the disk persistent type. Valid values are pd-ssd, pd-standard, and pd-balanced. The default value is pd-standard.

Procedure

1. Run the following command with your KMS key name, key ring name, and location to allow a specific service account to use your KMS key and to grant the service account the correct IAM role:

   gcloud kms keys add-iam-policy-binding <key_name> \
     --keyring <key_ring_name> \
     --location <key_ring_location> \
     --member "serviceAccount:service-<project_number>@compute-system.iam.gserviceaccount.com” \
     --role roles/cloudkms.cryptoKeyEncrypterDecrypter

2. Configure the encryption key under the providerSpec field in your compute machine set YAML file. For example:

   providerSpec:
     value:
       # ...
       disks:
       - type:
         # ...
         encryptionKey:
           kmsKey:
             name: machine-encryption-key 1
             keyRing: openshift-encrpytion-ring 2
             location: global 3
             projectID: openshift-gcp-project 4
           kmsKeyServiceAccount: openshift-service-account@openshift-gcp-project.iam.gserviceaccount.com 5

   1

   The name of the customer-managed encryption key that is used for the disk encryption.

   2

   The name of the KMS key ring that the KMS key belongs to.

   3

   The GCP location in which the KMS key ring exists.

   4

   Optional: The ID of the project in which the KMS key ring exists. If a project ID is not set, the compute machine set projectID in which the compute machine set was created is used.

   5

   Optional: The service account that is used for the encryption request for the given KMS key. If a service account is not set, the Compute Engine default service account is used.

   After a new machine is created by using the updated providerSpec object configuration, the disk encryption key is encrypted with the KMS key.

1. For more information about machine types, including specifications, compatibility, regional availability, and limitations, see the GCP Compute Engine documentation about N1 machine series, A2 machine series, and GPU regions and zones availability.

Procedure

1. In a text editor, open the YAML file for an existing compute machine set or create a new one.

2. Specify a GPU configuration under the providerSpec field in your compute machine set YAML file. See the following examples of valid configurations:

   Example configuration for the A2 machine series:

     providerSpec:
       value:
         machineType: a2-highgpu-1g 1
         onHostMaintenance: Terminate 2
         restartPolicy: Always 3

   1

   Specify the machine type. Ensure that the machine type is included in the A2 machine series.

   2

   When using GPU support, you must set onHostMaintenance to Terminate.

   3

   Specify the restart policy for machines deployed by the compute machine set. Allowed values are Always or Never.

   Example configuration for the N1 machine series:

   providerSpec:
     value:
       gpus:
       - count: 1 1
         type: nvidia-tesla-p100 2
       machineType: n1-standard-1 3
       onHostMaintenance: Terminate 4
       restartPolicy: Always 5

   1

   Specify the number of GPUs to attach to the machine.

   2

   Specify the type of GPUs to attach to the machine. Ensure that the machine type and GPU type are compatible.

   3

   Specify the machine type. Ensure that the machine type and GPU type are compatible.

   4

   When using GPU support, you must set onHostMaintenance to Terminate.

   5

   Specify the restart policy for machines deployed by the compute machine set. Allowed values are Always or Never.

Procedure

1. Make a copy of an existing MachineSet.
2. In the new copy, change the machine set name in metadata.name and in both instances of machine.openshift.io/cluster-api-machineset.

3. Change the instance type to add the following two lines to the newly copied MachineSet:

   machineType: a2-highgpu-1g
   onHostMaintenance: Terminate

   Example a2-highgpu-1g.json file

   {
       "apiVersion": "machine.openshift.io/v1beta1",
       "kind": "MachineSet",
       "metadata": {
           "annotations": {
               "machine.openshift.io/GPU": "0",
               "machine.openshift.io/memoryMb": "16384",
               "machine.openshift.io/vCPU": "4"
           },
           "creationTimestamp": "2023-01-13T17:11:02Z",
           "generation": 1,
           "labels": {
               "machine.openshift.io/cluster-api-cluster": "myclustername-2pt9p"
           },
           "name": "myclustername-2pt9p-worker-gpu-a",
           "namespace": "openshift-machine-api",
           "resourceVersion": "20185",
           "uid": "2daf4712-733e-4399-b4b4-d43cb1ed32bd"
       },
       "spec": {
           "replicas": 1,
           "selector": {
               "matchLabels": {
                   "machine.openshift.io/cluster-api-cluster": "myclustername-2pt9p",
                   "machine.openshift.io/cluster-api-machineset": "myclustername-2pt9p-worker-gpu-a"
               }
           },
           "template": {
               "metadata": {
                   "labels": {
                       "machine.openshift.io/cluster-api-cluster": "myclustername-2pt9p",
                       "machine.openshift.io/cluster-api-machine-role": "worker",
                       "machine.openshift.io/cluster-api-machine-type": "worker",
                       "machine.openshift.io/cluster-api-machineset": "myclustername-2pt9p-worker-gpu-a"
                   }
               },
               "spec": {
                   "lifecycleHooks": {},
                   "metadata": {},
                   "providerSpec": {
                       "value": {
                           "apiVersion": "machine.openshift.io/v1beta1",
                           "canIPForward": false,
                           "credentialsSecret": {
                               "name": "gcp-cloud-credentials"
                           },
                           "deletionProtection": false,
                           "disks": [
                               {
                                   "autoDelete": true,
                                   "boot": true,
                                   "image": "projects/rhcos-cloud/global/images/rhcos-412-86-202212081411-0-gcp-x86-64",
                                   "labels": null,
                                   "sizeGb": 128,
                                   "type": "pd-ssd"
                               }
                           ],
                           "kind": "GCPMachineProviderSpec",
                           "machineType": "a2-highgpu-1g",
                           "onHostMaintenance": "Terminate",
                           "metadata": {
                               "creationTimestamp": null
                           },
                           "networkInterfaces": [
                               {
                                   "network": "myclustername-2pt9p-network",
                                   "subnetwork": "myclustername-2pt9p-worker-subnet"
                               }
                           ],
                           "preemptible": true,
                           "projectID": "myteam",
                           "region": "us-central1",
                           "serviceAccounts": [
                               {
                                   "email": "myclustername-2pt9p-w@myteam.iam.gserviceaccount.com",
                                   "scopes": [
                                       "https://www.googleapis.com/auth/cloud-platform"
                                   ]
                               }
                           ],
                           "tags": [
                               "myclustername-2pt9p-worker"
                           ],
                           "userDataSecret": {
                               "name": "worker-user-data"
                           },
                           "zone": "us-central1-a"
                       }
                   }
               }
           }
       },
       "status": {
           "availableReplicas": 1,
           "fullyLabeledReplicas": 1,
           "observedGeneration": 1,
           "readyReplicas": 1,
           "replicas": 1
       }
   }

4. View the existing nodes, machines, and machine sets by running the following command. Note that each node is an instance of a machine definition with a specific GCP region and OpenShift Container Platform role.

   $ oc get nodes

   Example output

   NAME                                                             STATUS     ROLES                  AGE     VERSION
   myclustername-2pt9p-master-0.c.openshift-qe.internal             Ready      control-plane,master   8h      v1.25.4+77bec7a
   myclustername-2pt9p-master-1.c.openshift-qe.internal             Ready      control-plane,master   8h      v1.25.4+77bec7a
   myclustername-2pt9p-master-2.c.openshift-qe.internal             Ready      control-plane,master   8h      v1.25.4+77bec7a
   myclustername-2pt9p-worker-a-mxtnz.c.openshift-qe.internal       Ready      worker                 8h      v1.25.4+77bec7a
   myclustername-2pt9p-worker-b-9pzzn.c.openshift-qe.internal       Ready      worker                 8h      v1.25.4+77bec7a
   myclustername-2pt9p-worker-c-6pbg6.c.openshift-qe.internal       Ready      worker                 8h      v1.25.4+77bec7a
   myclustername-2pt9p-worker-gpu-a-wxcr6.c.openshift-qe.internal   Ready      worker                 4h35m   v1.25.4+77bec7a

5. View the machines and machine sets that exist in the openshift-machine-api namespace by running the following command. Each compute machine set is associated with a different availability zone within the GCP region. The installer automatically load balances compute machines across availability zones.

   $ oc get machinesets -n openshift-machine-api

   Example output

   NAME                               DESIRED   CURRENT   READY   AVAILABLE   AGE
   myclustername-2pt9p-worker-a       1         1         1       1           8h
   myclustername-2pt9p-worker-b       1         1         1       1           8h
   myclustername-2pt9p-worker-c       1         1                             8h
   myclustername-2pt9p-worker-f       0         0                             8h

6. View the machines that exist in the openshift-machine-api namespace by running the following command. You can only configure one compute machine per set, although you can scale a compute machine set to add a node in a particular region and zone.

   $ oc get machines -n openshift-machine-api | grep worker

   Example output

   myclustername-2pt9p-worker-a-mxtnz       Running   n2-standard-4   us-central1   us-central1-a   8h
   myclustername-2pt9p-worker-b-9pzzn       Running   n2-standard-4   us-central1   us-central1-b   8h
   myclustername-2pt9p-worker-c-6pbg6       Running   n2-standard-4   us-central1   us-central1-c   8h

7. Make a copy of one of the existing compute MachineSet definitions and output the result to a JSON file by running the following command. This will be the basis for the GPU-enabled compute machine set definition.

   $ oc get machineset myclustername-2pt9p-worker-a -n openshift-machine-api -o json  > <output_file.json>

8. Edit the JSON file to make the following changes to the new MachineSet definition:

   • Rename the machine set name by inserting the substring gpu in metadata.name and in both instances of machine.openshift.io/cluster-api-machineset.

   • Change the machineType of the new MachineSet definition to a2-highgpu-1g, which includes an NVIDIA A100 GPU.

     jq .spec.template.spec.providerSpec.value.machineType ocp_4.12_machineset-a2-highgpu-1g.json
     
     "a2-highgpu-1g"

     The <output_file.json> file is saved as ocp_4.12_machineset-a2-highgpu-1g.json.

9. Update the following fields in ocp_4.12_machineset-a2-highgpu-1g.json:

   • Change .metadata.name to a name containing gpu.
   • Change .spec.selector.matchLabels["machine.openshift.io/cluster-api-machineset"] to match the new .metadata.name.
   • Change .spec.template.metadata.labels["machine.openshift.io/cluster-api-machineset"] to match the new .metadata.name.
   • Change .spec.template.spec.providerSpec.value.MachineType to a2-highgpu-1g.

   • Add the following line under machineType: `"onHostMaintenance": "Terminate". For example:

     "machineType": "a2-highgpu-1g",
     "onHostMaintenance": "Terminate",

10. To verify your changes, perform a diff of the original compute definition and the new GPU-enabled node definition by running the following command:

    $ oc get machineset/myclustername-2pt9p-worker-a -n openshift-machine-api -o json | diff ocp_4.12_machineset-a2-highgpu-1g.json -

    Example output

    15c15
    <         "name": "myclustername-2pt9p-worker-gpu-a",
    ---
    >         "name": "myclustername-2pt9p-worker-a",
    25c25
    <                 "machine.openshift.io/cluster-api-machineset": "myclustername-2pt9p-worker-gpu-a"
    ---
    >                 "machine.openshift.io/cluster-api-machineset": "myclustername-2pt9p-worker-a"
    34c34
    <                     "machine.openshift.io/cluster-api-machineset": "myclustername-2pt9p-worker-gpu-a"
    ---
    >                     "machine.openshift.io/cluster-api-machineset": "myclustername-2pt9p-worker-a"
    59,60c59
    <                         "machineType": "a2-highgpu-1g",
    <                         "onHostMaintenance": "Terminate",
    ---
    >                         "machineType": "n2-standard-4",

11. Create the GPU-enabled compute machine set from the definition file by running the following command:

    $ oc create -f ocp_4.12_machineset-a2-highgpu-1g.json

    Example output

    machineset.machine.openshift.io/myclustername-2pt9p-worker-gpu-a created

Verification

1. View the machine set you created by running the following command:

   $ oc -n openshift-machine-api get machinesets | grep gpu

   The MachineSet replica count is set to 1 so a new Machine object is created automatically.

   Example output

   myclustername-2pt9p-worker-gpu-a   1         1         1       1           5h24m

2. View the Machine object that the machine set created by running the following command:

   $ oc -n openshift-machine-api get machines | grep gpu

   Example output

   myclustername-2pt9p-worker-gpu-a-wxcr6   Running   a2-highgpu-1g   us-central1   us-central1-a   5h25m

Procedure

1. Install the Node Feature Discovery Operator from OperatorHub in the OpenShift Container Platform console.
2. After installing the NFD Operator into OperatorHub, select Node Feature Discovery from the installed Operators list and select Create instance. This installs the nfd-master and nfd-worker pods, one nfd-worker pod for each compute node, in the openshift-nfd namespace.

3. Verify that the Operator is installed and running by running the following command:

   $ oc get pods -n openshift-nfd

   Example output

   NAME                                       READY    STATUS     RESTARTS   AGE
   
   nfd-controller-manager-8646fcbb65-x5qgk    2/2      Running 7  (8h ago)   1d

4. Browse to the installed Oerator in the console and select Create Node Feature Discovery.
5. Select Create to build a NFD custom resource. This creates NFD pods in the openshift-nfd namespace that poll the OpenShift Container Platform nodes for hardware resources and catalogue them.

Verification

1. After a successful build, verify that a NFD pod is running on each nodes by running the following command:

   $ oc get pods -n openshift-nfd

   Example output

   NAME                                       READY   STATUS      RESTARTS        AGE
   nfd-controller-manager-8646fcbb65-x5qgk    2/2     Running     7 (8h ago)      12d
   nfd-master-769656c4cb-w9vrv                1/1     Running     0               12d
   nfd-worker-qjxb2                           1/1     Running     3 (3d14h ago)   12d
   nfd-worker-xtz9b                           1/1     Running     5 (3d14h ago)   12d

   The NFD Operator uses vendor PCI IDs to identify hardware in a node. NVIDIA uses the PCI ID 10de.

2. View the NVIDIA GPU discovered by the NFD Operator by running the following command:

   $ oc describe node ip-10-0-132-138.us-east-2.compute.internal | egrep 'Roles|pci'

   Example output

   Roles: worker
   
   feature.node.kubernetes.io/pci-1013.present=true
   
   feature.node.kubernetes.io/pci-10de.present=true
   
   feature.node.kubernetes.io/pci-1d0f.present=true

   10de appears in the node feature list for the GPU-enabled node. This mean the NFD Operator correctly identified the node from the GPU-enabled MachineSet.

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. To determine whether the required credentials exist, run the following command:

   $ oc get secret \
     -n openshift-machine-api vsphere-cloud-credentials \
     -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

   Sample output

   <vcenter-server>.password=<openshift-user-password>
   <vcenter-server>.username=<openshift-user>

   where <vcenter-server> is the IP address or fully qualified domain name (FQDN) of the vCenter server and <openshift-user> and <openshift-user-password> are the OpenShift Container Platform administrator credentials to use.

2. If the secret does not exist, create it by running the following command:

   $ oc create secret generic vsphere-cloud-credentials \
     -n openshift-machine-api \
     --from-literal=<vcenter-server>.username=<openshift-user> --from-literal=<vcenter-server>.password=<openshift-user-password>

Procedure

1. To determine whether the required secret exists, run the following command:

   $ oc get secret \
     -n openshift-machine-api worker-user-data \
     -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

   Sample output

   disableTemplating: false
   userData: 1
     {
       "ignition": {
         ...
         },
       ...
     }

   1

   The full output is omitted here, but should have this format.

2. If the secret does not exist, create it by running the following command:

   $ oc create secret generic worker-user-data \
     -n openshift-machine-api \
     --from-file=<installation_directory>/worker.ign

   where <installation_directory> is the directory that was used to store your installation assets during cluster installation.

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

   3. If you are creating a compute machine set for a cluster that has user-provisioned infrastructure, note the following important values:

      Example vSphere providerSpec values

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      ...
      template:
        ...
        spec:
          providerSpec:
            value:
              apiVersion: machine.openshift.io/v1beta1
              credentialsSecret:
                name: vsphere-cloud-credentials 1
              diskGiB: 120
              kind: VSphereMachineProviderSpec
              memoryMiB: 16384
              network:
                devices:
                  - networkName: "<vm_network_name>"
              numCPUs: 4
              numCoresPerSocket: 4
              snapshot: ""
              template: <vm_template_name> 2
              userDataSecret:
                name: worker-user-data 3
              workspace:
                datacenter: <vcenter_datacenter_name>
                datastore: <vcenter_datastore_name>
                folder: <vcenter_vm_folder_path>
                resourcepool: <vsphere_resource_pool>
                server: <vcenter_server_address> 4

      1

      The name of the secret in the openshift-machine-api namespace that contains the required vCenter credentials.

      2

      The name of the RHCOS VM template for your cluster that was created during installation.

      3

      The name of the secret in the openshift-machine-api namespace that contains the required Ignition configuration credentials.

      4

      The IP address or fully qualified domain name (FQDN) of the vCenter server.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Identify the virtual machine template by running the following command:

   $ oc get -o jsonpath='{.items[0].spec.template.spec.providerSpec.value.template_name}{"\n"}' machineset -A

2. Create a new virtual machine in the Manager, based on the template you identified. Leave all other settings unchanged. For details, see Creating a Virtual Machine Based on a Template in the Red Hat Virtualization Virtual Machine Management Guide.

   Tip

   You do not need to start the new virtual machine.

3. Create a new template from the new virtual machine. Specify the target storage domain under Target. For details, see Creating a Template in the Red Hat Virtualization Virtual Machine Management Guide.

4. Add a new compute machine set to the OpenShift Container Platform cluster with the new template.

   1. Get the details of the current compute machine set by running the following command:

      $ oc get machineset -o yaml

   2. Use these details to create a compute machine set. For more information see Creating a compute machine set.

      Enter the new virtual machine template name in the template_name field. Use the same template name you used in the New template dialog in the Manager.

   3. Note the names of both the old and new compute machine sets. You need to refer to them in subsequent steps.

5. Migrate the workloads.

   1. Scale up the new compute machine set. For details on manually scaling compute machine sets, see Scaling a compute machine set manually.

      OpenShift Container Platform moves the pods to an available worker when the old machine is removed.

   2. Scale down the old compute machine set.

6. Remove the old compute machine set by running the following command:

   $ oc delete machineset <machineset-name>

Procedure

1. Select the virtual machine labeled master.
2. Shut down the virtual machine.
3. Click the Disks tab.
4. Click the virtual machine’s disk.
5. Click More Actions and select Move.
6. Select the target storage domain and wait for the migration process to complete.
7. Start the virtual machine.

8. Verify that the OpenShift Container Platform cluster is stable:

   $ oc get nodes

   The output should display the node with the status Ready.

9. Repeat this procedure for each control plane node.

Procedure

1. Log in to the web console as a user with the cluster-admin role.
2. Navigate to Compute → Machines.
3. On the Machines page, select the name of the machine that you want to find the phase of.
4. On the Machine details page, select the YAML tab.

5. In the YAML block, find the value of the status.phase field.

   Example YAML snippet

   apiVersion: machine.openshift.io/v1beta1
   kind: Machine
   metadata:
     name: mycluster-5kbsp-worker-us-west-1a-fmx8t
   # ...
   status:
     phase: Running 1

   1

   In this example, the phase is Running.

1. An existing machine is slated for deletion for one of the following reasons:

   • A user with cluster-admin permissions uses the oc delete machine command.
   • The machine gets a machine.openshift.io/delete-machine annotation.
   • The machine set that manages the machine marks it for deletion to reduce the replica count as part of reconciliation.
   • The cluster autoscaler identifies a node that is unnecessary to meet the deployment needs of the cluster.
   • A machine health check is configured to replace an unhealthy machine.
2. The machine enters the Deleting phase, in which it is marked for deletion but is still present in the API.

3. If a preDrain lifecycle hook exists, the hook-implementing controller that manages it does a specified action.

   Until all preDrain lifecycle hooks are satisfied, the machine status condition Drainable is set to False.

4. There are no unresolved preDrain lifecycle hooks and the machine status condition Drainable is set to True.

5. The machine controller attempts to drain the node that is backed by the machine.

   • If draining fails, Drained is set to False and the machine controller attempts to drain the node again.
   • If draining succeeds, Drained is set to True.
6. The machine status condition Drained is set to True.

7. If a preTerminate lifecycle hook exists, the hook-implementing controller that manages it does a specified action.

   Until all preTerminate lifecycle hooks are satisfied, the machine status condition Terminable is set to False.

8. There are no unresolved preTerminate lifecycle hooks and the machine status condition Terminable is set to True.
9. The machine controller removes the instance from the infrastructure provider.
10. The machine controller deletes the Node object.

Procedure

1. Create a YAML file for a ClusterAutoscaler resource that contains the custom resource definition.

2. Create the custom resource in the cluster by running the following command:

   $ oc create -f <filename>.yaml 1

   1

   <filename> is the name of the custom resource file.

Procedure

1. Create a YAML file for a MachineAutoscaler resource that contains the custom resource definition.

2. Create the custom resource in the cluster by running the following command:

   $ oc create -f <filename>.yaml 1

   1

   <filename> is the name of the custom resource file.

Procedure

1. List the MachineAutoscaler CRs for the cluster by running the following command:

   $ oc get MachineAutoscaler -n openshift-machine-api

   Example output

   NAME                 REF KIND     REF NAME             MIN   MAX   AGE
   compute-us-east-1a   MachineSet   compute-us-east-1a   1     12    39m
   compute-us-west-1a   MachineSet   compute-us-west-1a   2     4     37m

2. Optional: Create a YAML file backup of the MachineAutoscaler CR by running the following command:

   $ oc get MachineAutoscaler/<machine_autoscaler_name> \1
     -n openshift-machine-api \
     -o yaml> <machine_autoscaler_name_backup>.yaml 2

   1

   <machine_autoscaler_name> is the name of the CR that you want to delete.

   2

   <machine_autoscaler_name_backup> is the name for the backup of the CR.

3. Delete the MachineAutoscaler CR by running the following command:

   $ oc delete MachineAutoscaler/<machine_autoscaler_name> -n openshift-machine-api

   Example output

   machineautoscaler.autoscaling.openshift.io "compute-us-east-1a" deleted

Procedure

1. List the ClusterAutoscaler resource for the cluster by running the following command:

   $ oc get ClusterAutoscaler

   Example output

   NAME      AGE
   default   42m

2. Optional: Create a YAML file backup of the ClusterAutoscaler CR by running the following command:

   $ oc get ClusterAutoscaler/default \1
     -o yaml> <cluster_autoscaler_backup_name>.yaml 2

   1

   default is the name of the ClusterAutoscaler CR.

   2

   <cluster_autoscaler_backup_name> is the name for the backup of the CR.

3. Delete the ClusterAutoscaler CR by running the following command:

   $ oc delete ClusterAutoscaler/default

   Example output

   clusterautoscaler.autoscaling.openshift.io "default" deleted