Chapter 8. Creating infrastructure machine sets

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

      Copy to Clipboard Toggle word wrap

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Copy to Clipboard Toggle word wrap

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 

      1

      
        name: <infrastructure_id>-<role> 

      2

      
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 

      3

      
              ...

      Copy to Clipboard Toggle word wrap

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a label to the worker nodes that you want to act as infrastructure nodes:

   $ oc label node <node-name> node-role.kubernetes.io/infra=""

   Copy to Clipboard Toggle word wrap

2. Check to see if applicable nodes now have the infra role:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

3. Optional: Create a default cluster-wide node selector:

   1. Edit the Scheduler object:

      $ oc edit scheduler cluster

      Copy to Clipboard Toggle word wrap

   2. Add the defaultNodeSelector field with the appropriate node selector:

      apiVersion: config.openshift.io/v1
      kind: Scheduler
      metadata:
        name: cluster
      spec:
        defaultNodeSelector: node-role.kubernetes.io/infra="" 

      1

      
      # ...

      Copy to Clipboard Toggle word wrap

      1

      This example node selector deploys pods on infrastructure nodes by default.

   3. Save the file to apply the changes.

Procedure

1. Add a label to the node you want to assign as the infra node with a specific label:

   $ oc label node <node_name> <label>

   Copy to Clipboard Toggle word wrap

   $ oc label node ci-ln-n8mqwr2-f76d1-xscn2-worker-c-6fmtx node-role.kubernetes.io/infra=

   Copy to Clipboard Toggle word wrap

2. Create a machine config pool that contains both the worker role and your custom role as machine config selector:

   $ cat infra.mcp.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: infra
   spec:
     machineConfigSelector:
       matchExpressions:
         - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} 

   1

   
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/infra: "" 

   2

   Copy to Clipboard Toggle word wrap

   1

   Add the worker role and your custom role.

   2

   Add the label you added to the node as a nodeSelector.

   Note

   Custom machine config pools inherit machine configs from the worker pool. Custom pools use any machine config targeted for the worker pool, but add the ability to also deploy changes that are targeted at only the custom pool. Because a custom pool inherits resources from the worker pool, any change to the worker pool also affects the custom pool.

3. After you have the YAML file, you can create the machine config pool:

   $ oc create -f infra.mcp.yaml

   Copy to Clipboard Toggle word wrap

4. Check the machine configs to ensure that the infrastructure configuration rendered successfully:

   $ oc get machineconfig

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   00-worker                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-1ae2a1e0-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-ssh                                                                                          3.2.0             31d
   99-worker-1ae64748-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-worker-ssh                                                                                          3.2.0             31d
   rendered-infra-4e48906dca84ee702959c71a53ee80e7             365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             23m
   rendered-master-072d4b2da7f88162636902b074e9e28e            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-master-3e88ec72aed3886dec061df60d16d1af            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-master-419bee7de96134963a15fdf9dd473b25            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-master-53f5c91c7661708adce18739cc0f40fb            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d
   rendered-master-a6a357ec18e5bce7f5ac426fc7c5ffcd            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-master-dc7f874ec77fc4b969674204332da037            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-1a75960c52ad18ff5dfa6674eb7e533d            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-2640531be11ba43c61d72e82dc634ce6            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-4e48906dca84ee702959c71a53ee80e7            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-worker-4f110718fe88e5f349987854a1147755            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-worker-afc758e194d6188677eb837842d3b379            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-worker-daa08cc1e8f5fcdeba24de60cd955cc3            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d

   Copy to Clipboard Toggle word wrap

   You should see a new machine config, with the rendered-infra-* prefix.

5. Optional: To deploy changes to a custom pool, create a machine config that uses the custom pool name as the label, such as infra. Note that this is not required and only shown for instructional purposes. In this manner, you can apply any custom configurations specific to only your infra nodes.

   Note

   After you create the new machine config pool, the MCO generates a new rendered config for that pool, and associated nodes of that pool reboot to apply the new configuration.

   1. Create a machine config:

      $ cat infra.mc.yaml

      Copy to Clipboard Toggle word wrap

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        name: 51-infra
        labels:
          machineconfiguration.openshift.io/role: infra 

      1

      
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - path: /etc/infratest
              mode: 0644
              contents:
                source: data:,infra

      Copy to Clipboard Toggle word wrap

      1

      Add the label you added to the node as a nodeSelector.

   2. Apply the machine config to the infra-labeled nodes:

      $ oc create -f infra.mc.yaml

      Copy to Clipboard Toggle word wrap

6. Confirm that your new machine config pool is available:

   $ oc get mcp

   Copy to Clipboard Toggle word wrap

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   infra    rendered-infra-60e35c2e99f42d976e084fa94da4d0fc    True      False      False      1              1                   1                     0                      4m20s
   master   rendered-master-9360fdb895d4c131c7c4bebbae099c90   True      False      False      3              3                   3                     0                      91m
   worker   rendered-worker-60e35c2e99f42d976e084fa94da4d0fc   True      False      False      2              2                   2                     0                      91m

   Copy to Clipboard Toggle word wrap

   In this example, a worker node was changed to an infra node.

Procedure

1. Add a taint to the infrastructure node to prevent scheduling user workloads on it:

   1. Determine if the node has the taint:

      $ oc describe nodes <node_name>

      Copy to Clipboard Toggle word wrap

      Sample output

      oc describe node ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Name:               ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Roles:              worker
       ...
      Taints:             node-role.kubernetes.io/infra=reserved:NoSchedule
       ...

      Copy to Clipboard Toggle word wrap

      This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.

   2. If you have not configured a taint to prevent scheduling user workloads on it:

      $ oc adm taint nodes <node_name> <key>=<value>:<effect>

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc adm taint nodes node1 node-role.kubernetes.io/infra=reserved:NoSchedule

      Copy to Clipboard Toggle word wrap
      Tip

      You can alternatively edit the pod specification to add the taint:

      apiVersion: v1
      kind: Node
      metadata:
        name: node1
      # ...
      spec:
        taints:
          - key: node-role.kubernetes.io/infra
            value: reserved
            effect: NoSchedule
      # ...

      Copy to Clipboard Toggle word wrap

      These examples place a taint on node1 that has the node-role.kubernetes.io/infra key and the NoSchedule taint effect. Nodes with the NoSchedule effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.

      If you added a NoSchedule taint to the infrastructure node, any pods that are controlled by a daemon set on that node are marked as misscheduled. You must either delete the pods or add a toleration to the pods as shown in the Red Hat Knowledgebase solution add toleration on misscheduled DNS pods. Note that you cannot add a toleration to a daemon set object that is managed by an operator.

      Note

      If a descheduler is used, pods violating node taints could be evicted from the cluster.

2. Add tolerations to the pods that you want to schedule on the infrastructure node, such as the router, registry, and monitoring workloads. Referencing the previous examples, add the following tolerations to the Pod object specification:

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
   
   # ...
   spec:
   # ...
     tolerations:
       - key: node-role.kubernetes.io/infra 

   1

   
         value: reserved 

   2

   
         effect: NoSchedule 

   3

   
         operator: Equal 

   4

   Copy to Clipboard Toggle word wrap

   1

   Specify the key that you added to the node.

   2

   Specify the value of the key-value pair taint that you added to the node.

   3

   Specify the effect that you added to the node.

   4

   Specify the Equal Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node.

   This toleration matches the taint created by the oc adm taint command. A pod with this toleration can be scheduled onto the infrastructure node.

   Note

   Moving pods for an Operator installed via OLM to an infrastructure node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.

3. Schedule the pod to the infrastructure node by using a scheduler. See the documentation for "Controlling pod placement using the scheduler" for details.
4. Remove any workloads that you do not want, or that do not belong, on the new infrastructure node. See the list of workloads supported for use on infrastructure nodes in "OpenShift Container Platform infrastructure components".

Procedure

1. View the IngressController custom resource for the router Operator:

   $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

   Copy to Clipboard Toggle word wrap

   The command output resembles the following text:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: 2019-04-18T12:35:39Z
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
     namespace: openshift-ingress-operator
     resourceVersion: "11341"
     selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a
   spec: {}
   status:
     availableReplicas: 2
     conditions:
     - lastTransitionTime: 2019-04-18T12:36:15Z
       status: "True"
       type: Available
     domain: apps.<cluster>.example.com
     endpointPublishingStrategy:
       type: LoadBalancerService
     selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

   Copy to Clipboard Toggle word wrap

2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

   $ oc edit ingresscontroller default -n openshift-ingress-operator

   Copy to Clipboard Toggle word wrap

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: "2025-03-26T21:15:43Z"
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
   # ...
   spec:
     nodePlacement:
       nodeSelector: 

   1

   
         matchLabels:
           node-role.kubernetes.io/infra: ""
       tolerations:
       - effect: NoSchedule
         key: node-role.kubernetes.io/infra
         value: reserved
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector parameter in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.

3. Confirm that the router pod is running on the infra node.

   1. View the list of router pods and note the node name of the running pod:

      $ oc get pod -n openshift-ingress -o wide

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      Copy to Clipboard Toggle word wrap

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

   2. View the node status of the running pod:

      $ oc get node <node_name> 

      1

      Copy to Clipboard Toggle word wrap

      1

      Specify the <node_name> that you obtained from the pod list.

      Example output

      NAME                          STATUS  ROLES         AGE   VERSION
      ip-10-0-217-226.ec2.internal  Ready   infra,worker  17h   v1.28.5

      Copy to Clipboard Toggle word wrap

      Because the role list includes infra, the pod is running on the correct node.

Procedure

1. View the config/instance object:

   $ oc get configs.imageregistry.operator.openshift.io/cluster -o yaml

   Copy to Clipboard Toggle word wrap

   Example output

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     creationTimestamp: 2019-02-05T13:52:05Z
     finalizers:
     - imageregistry.operator.openshift.io/finalizer
     generation: 1
     name: cluster
     resourceVersion: "56174"
     selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
     uid: 36fd3724-294d-11e9-a524-12ffeee2931b
   spec:
     httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
     logging: 2
     managementState: Managed
     proxy: {}
     replicas: 1
     requests:
       read: {}
       write: {}
     storage:
       s3:
         bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
         region: us-east-1
   status:
   ...

   Copy to Clipboard Toggle word wrap

2. Edit the config/instance object:

   $ oc edit configs.imageregistry.operator.openshift.io/cluster

   Copy to Clipboard Toggle word wrap

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     name: cluster
   # ...
   spec:
     logLevel: Normal
     managementState: Managed
     nodeSelector: 

   1

   
       node-role.kubernetes.io/infra: ""
     tolerations:
     - effect: NoSchedule
       key: node-role.kubernetes.io/infra
       value: reserved

   Copy to Clipboard Toggle word wrap

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector parameter in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

3. Verify the registry pod has been moved to the infrastructure node.

   1. Run the following command to identify the node where the registry pod is located:

      $ oc get pods -o wide -n openshift-image-registry

      Copy to Clipboard Toggle word wrap

   2. Confirm the node has the label you specified:

      $ oc describe node <node_name>

      Copy to Clipboard Toggle word wrap

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Procedure

1. Edit the cluster-monitoring-config config map and change the nodeSelector to use the infra label:

   $ oc edit configmap cluster-monitoring-config -n openshift-monitoring

   Copy to Clipboard Toggle word wrap

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |+
       alertmanagerMain:
         nodeSelector: 

   1

   
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       prometheusK8s:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       prometheusOperator:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       k8sPrometheusAdapter:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       kubeStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       telemeterClient:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       openshiftStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       thanosQuerier:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
       monitoringPlugin:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule

   Copy to Clipboard Toggle word wrap

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector parameter in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.

2. Watch the monitoring pods move to the new machines:

   $ watch 'oc get pod -n openshift-monitoring -o wide'

   Copy to Clipboard Toggle word wrap

3. If a component has not moved to the infra node, delete the pod with this component:

   $ oc delete pod -n openshift-monitoring <pod>

   Copy to Clipboard Toggle word wrap

   The component from the deleted pod is re-created on the infra node.