Chapter 6. Creating infrastructure MachineSets

Procedure

1. Create a new YAML file that contains the MachineSet Custom Resource sample, as shown, and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

   1. If you are not sure about which value to set for a specific field, you can check an existing MachineSet from your cluster.

      $ oc get machinesets -n openshift-machine-api
      
      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. Check values of a specific MachineSet:

      $ oc get machineset <machineset_name> -n \
           openshift-machine-api -o yaml
      
      ....
      
      template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1
              machine.openshift.io/cluster-api-machine-role: worker 2
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

      1

      The cluster ID.

      2

      A default node label.

2. Create the new MachineSet:

   $ oc create -f <file_name>.yaml

3. View the list of MachineSets:

   $ oc get machineset -n openshift-machine-api
   
   
   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
   agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1d   0         0                             55m
   agl030519-vplxk-worker-us-east-1e   0         0                             55m
   agl030519-vplxk-worker-us-east-1f   0         0                             55m

   When the new MachineSet is available, the DESIRED and CURRENT values match. If the MachineSet is not available, wait a few minutes and run the command again.

4. After the new MachineSet is available, check status of the machine and the node that it references:

   $ oc describe machine <name> -n openshift-machine-api

   For example:

   $ oc describe machine agl030519-vplxk-infra-us-east-1a -n openshift-machine-api
   
   status:
     addresses:
     - address: 10.0.133.18
       type: InternalIP
     - address: ""
       type: ExternalDNS
     - address: ip-10-0-133-18.ec2.internal
       type: InternalDNS
     lastUpdated: "2019-05-03T10:38:17Z"
     nodeRef:
       kind: Node
       name: ip-10-0-133-18.ec2.internal
       uid: 71fb8d75-6d8f-11e9-9ff3-0e3f103c7cd8
     providerStatus:
       apiVersion: awsproviderconfig.openshift.io/v1beta1
       conditions:
       - lastProbeTime: "2019-05-03T10:34:31Z"
         lastTransitionTime: "2019-05-03T10:34:31Z"
         message: machine successfully created
         reason: MachineCreationSucceeded
         status: "True"
         type: MachineCreation
       instanceId: i-09ca0701454124294
       instanceState: running
       kind: AWSMachineProviderStatus

5. View the new node and confirm that the new node has the label that you specified:

   $ oc get node <node_name> --show-labels

   Review the command output and confirm that node-role.kubernetes.io/<your_label> is in the LABELS list.

Procedure

1. View the IngressController Custom Resource for the router Operator:

   $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

   The command output resembles the following text:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: 2019-04-18T12:35:39Z
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
     namespace: openshift-ingress-operator
     resourceVersion: "11341"
     selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a
   spec: {}
   status:
     availableReplicas: 2
     conditions:
     - lastTransitionTime: 2019-04-18T12:36:15Z
       status: "True"
       type: Available
     domain: apps.<cluster>.example.com
     endpointPublishingStrategy:
       type: LoadBalancerService
     selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

   $ oc edit ingresscontroller default -n openshift-ingress-operator -o yaml

   Add the nodeSelector stanza that references the infra label to the spec section, as shown:

     spec:
       nodePlacement:
         nodeSelector:
           matchLabels:
             node-role.kubernetes.io/infra: ""

3. Confirm that the router pod is running on the infra node.

   1. View the list of router pods and note the node name of the running pod:

      $ oc get pod -n openshift-ingress -o wide
      
      NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

   2. View the node status of the running pod:

      $ oc get node <node_name> 1
      
      NAME                           STATUS    ROLES          AGE       VERSION
      ip-10-0-217-226.ec2.internal   Ready     infra,worker   17h       v1.14.6+c4799753c

      1

      Specify the <node_name> that you obtained from the pod list.

      Because the role list includes infra, the pod is running on the correct node.

Procedure

1. View the config/instance object:

   $ oc get config/cluster -o yaml

   The output resembles the following text:

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     creationTimestamp: 2019-02-05T13:52:05Z
     finalizers:
     - imageregistry.operator.openshift.io/finalizer
     generation: 1
     name: cluster
     resourceVersion: "56174"
     selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
     uid: 36fd3724-294d-11e9-a524-12ffeee2931b
   spec:
     httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
     logging: 2
     managementState: Managed
     proxy: {}
     replicas: 1
     requests:
       read: {}
       write: {}
     storage:
       s3:
         bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
         region: us-east-1
   status:
   ...

2. Edit the config/instance object:

   $ oc edit config/cluster

3. Add the following lines of text the spec section of the object:

     nodeSelector:
       node-role.kubernetes.io/infra: ""

4. Verify the registry Pod has been moved to the infrastructure node.

   1. Run the following command to identify the node where the registry Pod is located:

      $ oc get pods -o wide -n openshift-image-registry

   2. Confirm the node has the label you specified:

      $ oc describe node <node_name>

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Procedure

1. Save the following ConfigMap definition as the cluster-monitoring-configmap.yaml file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |+
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       prometheusK8s:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       prometheusOperator:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       grafana:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       k8sPrometheusAdapter:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       kubeStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       telemeterClient:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       openshiftStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""

   Running this ConfigMap forces the components of the monitoring stack to redeploy to infrastructure nodes.

2. Apply the new ConfigMap:

   $ oc create -f cluster-monitoring-configmap.yaml

3. Watch the monitoring Pods move to the new machines:

   $ watch 'oc get pod -n openshift-monitoring -o wide'

4. If a component has not moved to the infra node, delete the pod with this component:

   $ oc delete pod -n openshift-monitoring <pod>

   The component from the deleted pod is re-created on the infra node.

Procedure

1. Edit the Cluster Logging Custom Resource in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     curation:
       curator:
         nodeSelector: 1
             node-role.kubernetes.io/infra: ''
         resources: null
         schedule: 30 3 * * *
       type: curator
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 2
             node-role.kubernetes.io/infra: ''
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 3
             node-role.kubernetes.io/infra: '' 4
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ....