Chapter 6. Creating infrastructure MachineSets

Procedure

1. Create a new YAML file that contains the MachineSet Custom Resource sample, as shown, and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

   1. If you are not sure about which value to set for a specific field, you can check an existing MachineSet from your cluster.

      $ oc get machinesets -n openshift-machine-api
      
      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

      Copy to Clipboard Toggle word wrap

   2. Check values of a specific MachineSet:

      $ oc get machineset <machineset_name> -n \
           openshift-machine-api -o yaml
      
      ....
      
      template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: agl030519-vplxk 

      1

      
              machine.openshift.io/cluster-api-machine-role: worker 

      2

      
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

      Copy to Clipboard Toggle word wrap

      1

      The cluster ID.

      2

      A default node label.

2. Create the new MachineSet:

   $ oc create -f <file_name>.yaml

   Copy to Clipboard Toggle word wrap

3. View the list of MachineSets:

   $ oc get machineset -n openshift-machine-api
   
   
   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
   agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1d   0         0                             55m
   agl030519-vplxk-worker-us-east-1e   0         0                             55m
   agl030519-vplxk-worker-us-east-1f   0         0                             55m

   Copy to Clipboard Toggle word wrap

   When the new MachineSet is available, the DESIRED and CURRENT values match. If the MachineSet is not available, wait a few minutes and run the command again.

4. After the new MachineSet is available, check status of the machine and the node that it references:

   $ oc describe machine <name> -n openshift-machine-api

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc describe machine agl030519-vplxk-infra-us-east-1a -n openshift-machine-api
   
   status:
     addresses:
     - address: 10.0.133.18
       type: InternalIP
     - address: ""
       type: ExternalDNS
     - address: ip-10-0-133-18.ec2.internal
       type: InternalDNS
     lastUpdated: "2019-05-03T10:38:17Z"
     nodeRef:
       kind: Node
       name: ip-10-0-133-18.ec2.internal
       uid: 71fb8d75-6d8f-11e9-9ff3-0e3f103c7cd8
     providerStatus:
       apiVersion: awsproviderconfig.openshift.io/v1beta1
       conditions:
       - lastProbeTime: "2019-05-03T10:34:31Z"
         lastTransitionTime: "2019-05-03T10:34:31Z"
         message: machine successfully created
         reason: MachineCreationSucceeded
         status: "True"
         type: MachineCreation
       instanceId: i-09ca0701454124294
       instanceState: running
       kind: AWSMachineProviderStatus

   Copy to Clipboard Toggle word wrap

5. View the new node and confirm that the new node has the label that you specified:

   $ oc get node <node_name> --show-labels

   Copy to Clipboard Toggle word wrap

   Review the command output and confirm that node-role.kubernetes.io/<your_label> is in the LABELS list.

Procedure

1. View the IngressController Custom Resource for the router Operator:

   $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

   Copy to Clipboard Toggle word wrap

   The command output resembles the following text:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: 2019-04-18T12:35:39Z
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
     namespace: openshift-ingress-operator
     resourceVersion: "11341"
     selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a
   spec: {}
   status:
     availableReplicas: 2
     conditions:
     - lastTransitionTime: 2019-04-18T12:36:15Z
       status: "True"
       type: Available
     domain: apps.<cluster>.example.com
     endpointPublishingStrategy:
       type: LoadBalancerService
     selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

   Copy to Clipboard Toggle word wrap

2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

   $ oc edit ingresscontroller default -n openshift-ingress-operator -o yaml

   Copy to Clipboard Toggle word wrap

   Add the nodeSelector stanza that references the infra label to the spec section, as shown:

     spec:
       nodePlacement:
         nodeSelector:
           matchLabels:
             node-role.kubernetes.io/infra: ""

   Copy to Clipboard Toggle word wrap

3. Confirm that the router pod is running on the infra node.

   1. View the list of router pods and note the node name of the running pod:

      $ oc get pod -n openshift-ingress -o wide
      
      NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      Copy to Clipboard Toggle word wrap

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

   2. View the node status of the running pod:

      $ oc get node <node_name> 

      1

      
      
      NAME                           STATUS    ROLES          AGE       VERSION
      ip-10-0-217-226.ec2.internal   Ready     infra,worker   17h       v1.14.6+c4799753c

      Copy to Clipboard Toggle word wrap

      1

      Specify the <node_name> that you obtained from the pod list.

      Because the role list includes infra, the pod is running on the correct node.

Procedure

1. View the config/instance object:

   $ oc get config/cluster -o yaml

   Copy to Clipboard Toggle word wrap

   The output resembles the following text:

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     creationTimestamp: 2019-02-05T13:52:05Z
     finalizers:
     - imageregistry.operator.openshift.io/finalizer
     generation: 1
     name: cluster
     resourceVersion: "56174"
     selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
     uid: 36fd3724-294d-11e9-a524-12ffeee2931b
   spec:
     httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
     logging: 2
     managementState: Managed
     proxy: {}
     replicas: 1
     requests:
       read: {}
       write: {}
     storage:
       s3:
         bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
         region: us-east-1
   status:
   ...

   Copy to Clipboard Toggle word wrap

2. Edit the config/instance object:

   $ oc edit config/cluster

   Copy to Clipboard Toggle word wrap

3. Add the following lines of text the spec section of the object:

     nodeSelector:
       node-role.kubernetes.io/infra: ""

   Copy to Clipboard Toggle word wrap

4. Verify the registry Pod has been moved to the infrastructure node.

   1. Run the following command to identify the node where the registry Pod is located:

      $ oc get pods -o wide -n openshift-image-registry

      Copy to Clipboard Toggle word wrap

   2. Confirm the node has the label you specified:

      $ oc describe node <node_name>

      Copy to Clipboard Toggle word wrap

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Procedure

1. Save the following ConfigMap definition as the cluster-monitoring-configmap.yaml file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |+
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       prometheusK8s:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       prometheusOperator:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       grafana:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       k8sPrometheusAdapter:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       kubeStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       telemeterClient:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       openshiftStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""

   Copy to Clipboard Toggle word wrap

   Running this ConfigMap forces the components of the monitoring stack to redeploy to infrastructure nodes.

2. Apply the new ConfigMap:

   $ oc create -f cluster-monitoring-configmap.yaml

   Copy to Clipboard Toggle word wrap

3. Watch the monitoring Pods move to the new machines:

   $ watch 'oc get pod -n openshift-monitoring -o wide'

   Copy to Clipboard Toggle word wrap

4. If a component has not moved to the infra node, delete the pod with this component:

   $ oc delete pod -n openshift-monitoring <pod>

   Copy to Clipboard Toggle word wrap

   The component from the deleted pod is re-created on the infra node.

Procedure

1. Edit the Cluster Logging Custom Resource in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     curation:
       curator:
         nodeSelector: 

   1

   
             node-role.kubernetes.io/infra: ''
         resources: null
         schedule: 30 3 * * *
       type: curator
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 

   2

   
             node-role.kubernetes.io/infra: ''
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 

   3

   
             node-role.kubernetes.io/infra: '' 

   4

   
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ....

   Copy to Clipboard Toggle word wrap