Chapter 6. Creating infrastructure machine sets

Procedure

1. Create a new YAML file that contains the machine set custom resource (CR) sample, as shown, and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

   1. If you are not sure about which value to set for a specific field, you can check an existing machine set from your cluster.

      $ oc get machinesets -n openshift-machine-api
      
      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. Check values of a specific machine set:

      $ oc get machineset <machineset_name> -n \
           openshift-machine-api -o yaml
      
      ....
      
      template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1
              machine.openshift.io/cluster-api-machine-role: worker 2
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

      1

      The cluster ID.

      2

      A default node label.

2. Create the new MachineSet CR:

   $ oc create -f <file_name>.yaml

3. View the list of machine sets:

   $ oc get machineset -n openshift-machine-api
   
   
   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
   agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1d   0         0                             55m
   agl030519-vplxk-worker-us-east-1e   0         0                             55m
   agl030519-vplxk-worker-us-east-1f   0         0                             55m

   When the new machine set is available, the DESIRED and CURRENT values match. If the machine set is not available, wait a few minutes and run the command again.

4. After the new machine set is available, check status of the machine and the node that it references:

   $ oc describe machine <name> -n openshift-machine-api

   For example:

   $ oc describe machine agl030519-vplxk-infra-us-east-1a -n openshift-machine-api
   
   status:
     addresses:
     - address: 10.0.133.18
       type: InternalIP
     - address: ""
       type: ExternalDNS
     - address: ip-10-0-133-18.ec2.internal
       type: InternalDNS
     lastUpdated: "2019-05-03T10:38:17Z"
     nodeRef:
       kind: Node
       name: ip-10-0-133-18.ec2.internal
       uid: 71fb8d75-6d8f-11e9-9ff3-0e3f103c7cd8
     providerStatus:
       apiVersion: awsproviderconfig.openshift.io/v1beta1
       conditions:
       - lastProbeTime: "2019-05-03T10:34:31Z"
         lastTransitionTime: "2019-05-03T10:34:31Z"
         message: machine successfully created
         reason: MachineCreationSucceeded
         status: "True"
         type: MachineCreation
       instanceId: i-09ca0701454124294
       instanceState: running
       kind: AWSMachineProviderStatus

5. View the new node and confirm that the new node has the label that you specified:

   $ oc get node <node_name> --show-labels

   Review the command output and confirm that node-role.kubernetes.io/<your_label> is in the LABELS list.

Procedure

1. Use the following command to add a taint to the infra node to prevent scheduling user workloads on it:

   $ oc adm taint nodes <node_name> <key>:<effect>

   For example:

   $ oc adm taint nodes node1 node-role.kubernetes.io/infra:NoSchedule

   This example places a taint on node1 that has key node-role.kubernetes.io/infra and taint effect NoSchedule. Nodes with the NoSchedule effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.

   Note

   If a descheduler is used, pods violating node taints could be evicted from the cluster.

2. Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the Pod object specification:

   tolerations:
     - effect: NoSchedule 1
       key: node-role.kubernetes.io/infra 2
       operator: Exists 3

   1

   Specify the effect that you added to the node.

   2

   Specify the key that you added to the node.

   3

   Specify the Exists Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node.

   This toleration matches the taint created by the oc adm taint command. A pod with this toleration can be scheduled onto the infra node.

   Note

   Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.

3. Schedule the pod to the infra node using a scheduler. See the documentation for Controlling pod placement onto nodes for details.

Procedure

1. View the IngressController custom resource for the router Operator:

   $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

   The command output resembles the following text:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: 2019-04-18T12:35:39Z
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
     namespace: openshift-ingress-operator
     resourceVersion: "11341"
     selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a
   spec: {}
   status:
     availableReplicas: 2
     conditions:
     - lastTransitionTime: 2019-04-18T12:36:15Z
       status: "True"
       type: Available
     domain: apps.<cluster>.example.com
     endpointPublishingStrategy:
       type: LoadBalancerService
     selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

   $ oc edit ingresscontroller default -n openshift-ingress-operator

   Add the nodeSelector stanza that references the infra label to the spec section, as shown:

     spec:
       nodePlacement:
         nodeSelector:
           matchLabels:
             node-role.kubernetes.io/infra: ""

3. Confirm that the router pod is running on the infra node.

   1. View the list of router pods and note the node name of the running pod:

      $ oc get pod -n openshift-ingress -o wide
      
      NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

   2. View the node status of the running pod:

      $ oc get node <node_name> 1
      
      NAME                          STATUS  ROLES         AGE   VERSION
      ip-10-0-217-226.ec2.internal  Ready   infra,worker  17h   v1.17.1

      1

      Specify the <node_name> that you obtained from the pod list.

      Because the role list includes infra, the pod is running on the correct node.

Procedure

1. View the config/instance object:

   $ oc get config/cluster -o yaml

   The output resembles the following text:

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     creationTimestamp: 2019-02-05T13:52:05Z
     finalizers:
     - imageregistry.operator.openshift.io/finalizer
     generation: 1
     name: cluster
     resourceVersion: "56174"
     selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
     uid: 36fd3724-294d-11e9-a524-12ffeee2931b
   spec:
     httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
     logging: 2
     managementState: Managed
     proxy: {}
     replicas: 1
     requests:
       read: {}
       write: {}
     storage:
       s3:
         bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
         region: us-east-1
   status:
   ...

2. Edit the config/instance object:

   $ oc edit config/cluster

3. Add the following lines of text the spec section of the object:

     nodeSelector:
       node-role.kubernetes.io/infra: ""

4. Verify the registry pod has been moved to the infrastructure node.

   1. Run the following command to identify the node where the registry pod is located:

      $ oc get pods -o wide -n openshift-image-registry

   2. Confirm the node has the label you specified:

      $ oc describe node <node_name>

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Procedure

1. Save the following ConfigMap definition as the cluster-monitoring-configmap.yaml file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |+
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       prometheusK8s:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       prometheusOperator:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       grafana:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       k8sPrometheusAdapter:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       kubeStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       telemeterClient:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       openshiftStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       thanosQuerier:
         nodeSelector:
           node-role.kubernetes.io/infra: ""

   Running this config map forces the components of the monitoring stack to redeploy to infrastructure nodes.

2. Apply the new config map:

   $ oc create -f cluster-monitoring-configmap.yaml

3. Watch the monitoring pods move to the new machines:

   $ watch 'oc get pod -n openshift-monitoring -o wide'

4. If a component has not moved to the infra node, delete the pod with this component:

   $ oc delete pod -n openshift-monitoring <pod>

   The component from the deleted pod is re-created on the infra node.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     curation:
       curator:
         nodeSelector: 1
             node-role.kubernetes.io/infra: ''
         resources: null
         schedule: 30 3 * * *
       type: curator
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 2
             node-role.kubernetes.io/infra: ''
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 3
             node-role.kubernetes.io/infra: '' 4
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ....