Chapter 4. Post-installation cluster tasks

Procedure

1. View the machine sets that are in the cluster:

   $ oc get machinesets -n openshift-machine-api

   The machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. View the machines that are in the cluster:

   $ oc get machine -n openshift-machine-api

3. Set the annotation on the machine that you want to delete:

   $ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/cluster-api-delete-machine="true"

4. Cordon and drain the node that you want to delete:

   $ oc adm cordon <node_name>
   $ oc adm drain <node_name>

5. Scale the machine set:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   Tip

   You can alternatively apply the following YAML to scale the machine set:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     name: <machineset>
     namespace: openshift-machine-api
   spec:
     replicas: 2

   You can scale the machine set up or down. It takes several minutes for the new machines to be available.

1. Edit the Scheduler Operator CR to add the default cluster-wide node selectors:

   $ oc edit scheduler cluster

   Example Scheduler Operator CR with a node selector

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     name: cluster
   ...
   
   spec:
     defaultNodeSelector: type=user-node,region=east 1
     mastersSchedulable: false
     policy:
       name: ""

   1

   Add a node selector with the appropriate <key>:<value> pairs.

   After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.

2. Add labels to a node by using a machine set or editing the node directly:

   • Use a machine set to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api 1

        1

        Add a <key>/<value> pair for each label.

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

        Tip

        You can alternatively apply the following YAML to add labels to a machine set:

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
          name: <machineset>
          namespace: openshift-machine-api
        spec:
          template:
            spec:
              metadata:
                labels:
                  region: "east"
                  type: "user-node"

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
          ...
        spec:
          ...
          template:
            metadata:
          ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
          ...

     3. Redeploy the nodes associated with that machine set by scaling down to 0 and scaling up the nodes:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. When the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east

        Tip

        You can alternatively apply the following YAML to add labels to a node:

        kind: Node
        apiVersion: v1
        metadata:
          name: <node_name>
          labels:
            type: "user-node"
            region: "east"

     2. Verify that the labels are added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f

Procedure

1. Create a new YAML file that contains the machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

   1. If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. Check values of a specific machine set:

      $ oc get machineset <machineset_name> -n \
           openshift-machine-api -o yaml

      Example output

      ...
      template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1
              machine.openshift.io/cluster-api-machine-role: worker 2
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

      1

      The cluster ID.

      2

      A default node label.

2. Create the new MachineSet CR:

   $ oc create -f <file_name>.yaml

3. View the list of machine sets:

   $ oc get machineset -n openshift-machine-api

   Example output

   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
   agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1d   0         0                             55m
   agl030519-vplxk-worker-us-east-1e   0         0                             55m
   agl030519-vplxk-worker-us-east-1f   0         0                             55m

   When the new machine set is available, the DESIRED and CURRENT values match. If the machine set is not available, wait a few minutes and run the command again.

Procedure

1. Add a label to the worker node that you want to act as application node:

   $ oc label node <node-name> node-role.kubernetes.io/app=""

2. Add a label to the worker nodes that you want to act as infrastructure nodes:

   $ oc label node <node-name> node-role.kubernetes.io/infra=""

3. Check to see if applicable nodes now have the infra role and app roles:

   $ oc get nodes

4. Create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces. This creates an intersection with any existing node selectors on a pod, which additionally constrains the pod’s selector.

   Important

   If the default node selector key conflicts with the key of a pod’s label, then the default node selector is not applied.

   However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as node-role.kubernetes.io/infra="", when a pod’s label is set to a different node role, such as node-role.kubernetes.io/master="", can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles.

   You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts.

   1. Edit the Scheduler object:

      $ oc edit scheduler cluster

   2. Add the defaultNodeSelector field with the appropriate node selector:

      apiVersion: config.openshift.io/v1
      kind: Scheduler
      metadata:
        name: cluster
      ...
      spec:
        defaultNodeSelector: topology.kubernetes.io/region=us-east-1 1
      ...

      1

      This example node selector deploys pods on nodes in the us-east-1 region by default.

   3. Save the file to apply the changes.

Procedure

1. Add a label to the node you want to assign as the infra node with a specific label:

   $ oc label node <node_name> <label>

   $ oc label node ci-ln-n8mqwr2-f76d1-xscn2-worker-c-6fmtx node-role.kubernetes.io/infra=

2. Create a machine config pool that contains both the worker role and your custom role as machine config selector:

   $ cat infra.mcp.yaml

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: infra
   spec:
     machineConfigSelector:
       matchExpressions:
         - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} 1
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/infra: "" 2

   1

   Add the worker role and your custom role.

   2

   Add the label you added to the node as a nodeSelector.

   Note

   Custom machine config pools inherit machine configs from the worker pool. Custom pools use any machine config targeted for the worker pool, but add the ability to also deploy changes that are targeted at only the custom pool. Because a custom pool inherits resources from the worker pool, any change to the worker pool also affects the custom pool.

3. After you have the YAML file, you can create the machine config pool:

   $ oc create -f infra.mcp.yaml

4. Check the machine configs to ensure that the infrastructure configuration rendered successfully:

   $ oc get machineconfig

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   00-worker                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-1ae2a1e0-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-ssh                                                                                          3.2.0             31d
   99-worker-1ae64748-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-worker-ssh                                                                                          3.2.0             31d
   rendered-infra-4e48906dca84ee702959c71a53ee80e7             365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             23m
   rendered-master-072d4b2da7f88162636902b074e9e28e            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-master-3e88ec72aed3886dec061df60d16d1af            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-master-419bee7de96134963a15fdf9dd473b25            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-master-53f5c91c7661708adce18739cc0f40fb            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d
   rendered-master-a6a357ec18e5bce7f5ac426fc7c5ffcd            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-master-dc7f874ec77fc4b969674204332da037            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-1a75960c52ad18ff5dfa6674eb7e533d            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-2640531be11ba43c61d72e82dc634ce6            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-4e48906dca84ee702959c71a53ee80e7            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-worker-4f110718fe88e5f349987854a1147755            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-worker-afc758e194d6188677eb837842d3b379            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-worker-daa08cc1e8f5fcdeba24de60cd955cc3            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d

   You should see a new machine config, with the rendered-infra-* prefix.

5. Optional: To deploy changes to a custom pool, create a machine config that uses the custom pool name as the label, such as infra. Note that this is not required and only shown for instructional purposes. In this manner, you can apply any custom configurations specific to only your infra nodes.

   Note

   After you create the new machine config pool, the MCO generates a new rendered config for that pool, and associated nodes of that pool reboot to apply the new configuration.

   1. Create a machine config:

      $ cat infra.mc.yaml

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        name: 51-infra
        labels:
          machineconfiguration.openshift.io/role: infra 1
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - path: /etc/infratest
              mode: 0644
              contents:
                source: data:,infra

      1

      Add the label you added to the node as a nodeSelector.

   2. Apply the machine config to the infra-labeled nodes:

      $ oc create -f infra.mc.yaml

6. Confirm that your new machine config pool is available:

   $ oc get mcp

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   infra    rendered-infra-60e35c2e99f42d976e084fa94da4d0fc    True      False      False      1              1                   1                     0                      4m20s
   master   rendered-master-9360fdb895d4c131c7c4bebbae099c90   True      False      False      3              3                   3                     0                      91m
   worker   rendered-worker-60e35c2e99f42d976e084fa94da4d0fc   True      False      False      2              2                   2                     0                      91m

   In this example, a worker node was changed to an infra node.

Procedure

1. Add a taint to the infra node to prevent scheduling user workloads on it:

   1. Determine if the node has the taint:

      $ oc describe nodes <node_name>

      Sample output

      oc describe node ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Name:               ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Roles:              worker
       ...
      Taints:             node-role.kubernetes.io/infra:NoSchedule
       ...

      This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.

   2. If you have not configured a taint to prevent scheduling user workloads on it:

      $ oc adm taint nodes <node_name> <key>:<effect>

      For example:

      $ oc adm taint nodes node1 node-role.kubernetes.io/infra:NoSchedule

      Tip

      You can alternatively apply the following YAML to add the taint:

      kind: Node
      apiVersion: v1
      metadata:
        name: <node_name>
        labels:
          ...
      spec:
        taints:
          - key: node-role.kubernetes.io/infra
            effect: NoSchedule
        ...

      This example places a taint on node1 that has key node-role.kubernetes.io/infra and taint effect NoSchedule. Nodes with the NoSchedule effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.

      Note

      If a descheduler is used, pods violating node taints could be evicted from the cluster.

2. Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the Pod object specification:

   tolerations:
     - effect: NoSchedule 1
       key: node-role.kubernetes.io/infra 2
       operator: Exists 3

   1

   Specify the effect that you added to the node.

   2

   Specify the key that you added to the node.

   3

   Specify the Exists Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node.

   This toleration matches the taint created by the oc adm taint command. A pod with this toleration can be scheduled onto the infra node.

   Note

   Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.

3. Schedule the pod to the infra node using a scheduler. See the documentation for Controlling pod placement onto nodes for details.

Procedure

1. View the IngressController custom resource for the router Operator:

   $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

   The command output resembles the following text:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: 2019-04-18T12:35:39Z
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
     namespace: openshift-ingress-operator
     resourceVersion: "11341"
     selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a
   spec: {}
   status:
     availableReplicas: 2
     conditions:
     - lastTransitionTime: 2019-04-18T12:36:15Z
       status: "True"
       type: Available
     domain: apps.<cluster>.example.com
     endpointPublishingStrategy:
       type: LoadBalancerService
     selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

   $ oc edit ingresscontroller default -n openshift-ingress-operator

     spec:
       nodePlacement:
         nodeSelector: 1
           matchLabels:
             node-role.kubernetes.io/infra: ""
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.

3. Confirm that the router pod is running on the infra node.

   1. View the list of router pods and note the node name of the running pod:

      $ oc get pod -n openshift-ingress -o wide

      Example output

      NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

   2. View the node status of the running pod:

      $ oc get node <node_name> 1

      1

      Specify the <node_name> that you obtained from the pod list.

      Example output

      NAME                          STATUS  ROLES         AGE   VERSION
      ip-10-0-217-226.ec2.internal  Ready   infra,worker  17h   v1.21.0

      Because the role list includes infra, the pod is running on the correct node.

Procedure

1. View the config/instance object:

   $ oc get configs.imageregistry.operator.openshift.io/cluster -o yaml

   Example output

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     creationTimestamp: 2019-02-05T13:52:05Z
     finalizers:
     - imageregistry.operator.openshift.io/finalizer
     generation: 1
     name: cluster
     resourceVersion: "56174"
     selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
     uid: 36fd3724-294d-11e9-a524-12ffeee2931b
   spec:
     httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
     logging: 2
     managementState: Managed
     proxy: {}
     replicas: 1
     requests:
       read: {}
       write: {}
     storage:
       s3:
         bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
         region: us-east-1
   status:
   ...

2. Edit the config/instance object:

   $ oc edit configs.imageregistry.operator.openshift.io/cluster

   spec:
     affinity:
       podAntiAffinity:
         preferredDuringSchedulingIgnoredDuringExecution:
         - podAffinityTerm:
             namespaces:
             - openshift-image-registry
             topologyKey: kubernetes.io/hostname
           weight: 100
     logLevel: Normal
     managementState: Managed
     nodeSelector: 1
       node-role.kubernetes.io/infra: ""
     tolerations:
     - effect: NoSchedule
       key: node-role.kubernetes.io/infra
       value: reserved
     - effect: NoExecute
       key: node-role.kubernetes.io/infra
       value: reserved

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

3. Verify the registry pod has been moved to the infrastructure node.

   1. Run the following command to identify the node where the registry pod is located:

      $ oc get pods -o wide -n openshift-image-registry

   2. Confirm the node has the label you specified:

      $ oc describe node <node_name>

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Procedure

1. Edit the cluster-monitoring-config config map and change the nodeSelector to use the infra label:

   $ oc edit configmap cluster-monitoring-config -n openshift-monitoring

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |+
       alertmanagerMain:
         nodeSelector: 1
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       prometheusK8s:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       prometheusOperator:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       grafana:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       k8sPrometheusAdapter:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       kubeStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       telemeterClient:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       openshiftStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       thanosQuerier:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

2. Watch the monitoring pods move to the new machines:

   $ watch 'oc get pod -n openshift-monitoring -o wide'

3. If a component has not moved to the infra node, delete the pod with this component:

   $ oc delete pod -n openshift-monitoring <pod>

   The component from the deleted pod is re-created on the infra node.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ...
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 1
           node-role.kubernetes.io/infra: ''
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 2
           node-role.kubernetes.io/infra: ''
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ...

   1 2

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

Procedure

1. Create a YAML file for the ClusterAutoscaler resource that contains the customized resource definition.

2. Create the resource in the cluster:

   $ oc create -f <filename>.yaml 1

   1

   <filename> is the name of the resource file that you customized.

Procedure

1. Create a YAML file for the MachineAutoscaler resource that contains the customized resource definition.

2. Create the resource in the cluster:

   $ oc create -f <filename>.yaml 1

   1

   <filename> is the name of the resource file that you customized.

1. In the OpenShift Container Platform web console, switch to the Administration Custom Resource Definitions page.
2. On the Custom Resource Definitions page, click FeatureGate.
3. On the Custom Resource Definition Details page, click the Instances tab.
4. Click the cluster feature gate, then click the YAML tab.

5. Edit the cluster instance to add specific feature sets:

   Warning

   Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.

   Sample Feature Gate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   ....
   
   spec:
     featureSet: TechPreviewNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the feature set that you want to enable:

   • TechPreviewNoUpgrade enables specific Technology Preview features.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

1. From the Administrator perspective in the web console, navigate to Compute Nodes.
2. Select a node.
3. In the Node details page, click Terminal.

4. In the terminal window, change your root directory to /host:

   sh-4.2# chroot /host

5. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

    ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
    ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.

1. Edit the FeatureGate CR named cluster:

   $ oc edit featuregate cluster

   Warning

   Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.

   Sample FeatureGate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   spec:
     featureSet: TechPreviewNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the feature set that you want to enable:

   • TechPreviewNoUpgrade enables specific Technology Preview features.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

1. From the Administrator perspective in the web console, navigate to Compute Nodes.
2. Select a node.
3. In the Node details page, click Terminal.

4. In the terminal window, change your root directory to /host:

   sh-4.2# chroot /host

5. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

    ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
    ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.

Procedure

1. Modify the APIServer object:

   $ oc edit apiserver

2. Set the encryption field type to aescbc:

   spec:
     encryption:
       type: aescbc 1

   1

   The aescbc type means that AES-CBC with PKCS#7 padding and a 32 byte key is used to perform the encryption.

3. Save the file to apply the changes.

   The encryption process starts. It can take 20 minutes or longer for this process to complete, depending on the size of your cluster.

4. Verify that etcd encryption was successful.

   1. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully encrypted:

      $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      EncryptionCompleted
      All resources encrypted: routes.route.openshift.io

      If the output shows EncryptionInProgress, encryption is still in progress. Wait a few minutes and try again.

   2. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully encrypted:

      $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      EncryptionCompleted
      All resources encrypted: secrets, configmaps

      If the output shows EncryptionInProgress, encryption is still in progress. Wait a few minutes and try again.

   3. Review the Encrypted status condition for the OpenShift OAuth API server to verify that its resources were successfully encrypted:

      $ oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      EncryptionCompleted
      All resources encrypted: oauthaccesstokens.oauth.openshift.io, oauthauthorizetokens.oauth.openshift.io

      If the output shows EncryptionInProgress, encryption is still in progress. Wait a few minutes and try again.

Procedure

1. Modify the APIServer object:

   $ oc edit apiserver

2. Set the encryption field type to identity:

   spec:
     encryption:
       type: identity 1

   1

   The identity type is the default value and means that no encryption is performed.

3. Save the file to apply the changes.

   The decryption process starts. It can take 20 minutes or longer for this process to complete, depending on the size of your cluster.

4. Verify that etcd decryption was successful.

   1. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully decrypted:

      $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      DecryptionCompleted
      Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, decryption is still in progress. Wait a few minutes and try again.

   2. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully decrypted:

      $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      DecryptionCompleted
      Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, decryption is still in progress. Wait a few minutes and try again.

   3. Review the Encrypted status condition for the OpenShift OAuth API server to verify that its resources were successfully decrypted:

      $ oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      DecryptionCompleted
      Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, decryption is still in progress. Wait a few minutes and try again.

Procedure

1. Start a debug session for a control plane node:

   $ oc debug node/<node_name>

2. Change your root directory to /host:

   sh-4.2# chroot /host

3. If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables.

4. Run the cluster-backup.sh script and pass in the location to save the backup to.

   Tip

   The cluster-backup.sh script is maintained as a component of the etcd Cluster Operator and is a wrapper around the etcdctl snapshot save command.

   sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

   Example script output

   found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6
   found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-7
   found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-6
   found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3
   ede95fe6b88b87ba86a03c15e669fb4aa5bf0991c180d3c6895ce72eaade54a1
   etcdctl version: 3.4.14
   API version: 3.4
   {"level":"info","ts":1624647639.0188997,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db.part"}
   {"level":"info","ts":"2021-06-25T19:00:39.030Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
   {"level":"info","ts":1624647639.0301006,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://10.0.0.5:2379"}
   {"level":"info","ts":"2021-06-25T19:00:40.215Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
   {"level":"info","ts":1624647640.6032252,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://10.0.0.5:2379","size":"114 MB","took":1.584090459}
   {"level":"info","ts":1624647640.6047094,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db"}
   Snapshot saved at /home/core/assets/backup/snapshot_2021-06-25_190035.db
   {"hash":3866667823,"revision":31407,"totalKey":12828,"totalSize":114446336}
   snapshot db and kube resources are successfully saved to /home/core/assets/backup

   In this example, two files are created in the /home/core/assets/backup/ directory on the control plane host:

   • snapshot_<datetimestamp>.db: This file is the etcd snapshot. The cluster-backup.sh script confirms its validity.

   • static_kuberesources_<datetimestamp>.tar.gz: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot.

     Note

     If etcd encryption is enabled, it is recommended to store this second file separately from the etcd snapshot for security reasons. However, this file is required to restore from the etcd snapshot.

     Keep in mind that etcd encryption only encrypts values, not keys. This means that resource types, namespaces, and object names are unencrypted.

Procedure

1. Determine which etcd member is the leader, because the leader should be defragmented last.

   1. Get the list of etcd pods:

      $ oc get pods -n openshift-etcd -o wide | grep -v quorum-guard | grep etcd

      Example output

      etcd-ip-10-0-159-225.example.redhat.com                3/3     Running     0          175m   10.0.159.225   ip-10-0-159-225.example.redhat.com   <none>           <none>
      etcd-ip-10-0-191-37.example.redhat.com                 3/3     Running     0          173m   10.0.191.37    ip-10-0-191-37.example.redhat.com    <none>           <none>
      etcd-ip-10-0-199-170.example.redhat.com                3/3     Running     0          176m   10.0.199.170   ip-10-0-199-170.example.redhat.com   <none>           <none>

   2. Choose a pod and run the following command to determine which etcd member is the leader:

      $ oc rsh -n openshift-etcd etcd-ip-10-0-159-225.example.redhat.com etcdctl endpoint status --cluster -w table

      Example output

      Defaulting container name to etcdctl.
      Use 'oc describe pod/etcd-ip-10-0-159-225.example.redhat.com -n openshift-etcd' to see all of the containers in this pod.
      +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      |         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
      +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      |  https://10.0.191.37:2379 | 251cd44483d811c3 |   3.4.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      | https://10.0.159.225:2379 | 264c7c58ecbdabee |   3.4.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      | https://10.0.199.170:2379 | 9ac311f93915cc79 |   3.4.9 |  104 MB |      true |      false |         7 |      91624 |              91624 |        |
      +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

      Based on the IS LEADER column of this output, the https://10.0.199.170:2379 endpoint is the leader. Matching this endpoint with the output of the previous step, the pod name of the leader is etcd-ip-10-0-199-170.example.redhat.com.

2. Defragment an etcd member.

   1. Connect to the running etcd container, passing in the name of a pod that is not the leader:

      $ oc rsh -n openshift-etcd etcd-ip-10-0-159-225.example.redhat.com

   2. Unset the ETCDCTL_ENDPOINTS environment variable:

      sh-4.4# unset ETCDCTL_ENDPOINTS

   3. Defragment the etcd member:

      sh-4.4# etcdctl --command-timeout=30s --endpoints=https://localhost:2379 defrag

      Example output

      Finished defragmenting etcd member[https://localhost:2379]

      If a timeout error occurs, increase the value for --command-timeout until the command succeeds.

   4. Verify that the database size was reduced:

      sh-4.4# etcdctl endpoint status -w table --cluster

      Example output

      +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      |         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
      +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      |  https://10.0.191.37:2379 | 251cd44483d811c3 |   3.4.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      | https://10.0.159.225:2379 | 264c7c58ecbdabee |   3.4.9 |   41 MB |     false |      false |         7 |      91624 |              91624 |        | 1
      | https://10.0.199.170:2379 | 9ac311f93915cc79 |   3.4.9 |  104 MB |      true |      false |         7 |      91624 |              91624 |        |
      +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

      This example shows that the database size for this etcd member is now 41 MB as opposed to the starting size of 104 MB.

   5. Repeat these steps to connect to each of the other etcd members and defragment them. Always defragment the leader last.

      Wait at least one minute between defragmentation actions to allow the etcd pod to recover. Until the etcd pod recovers, the etcd member will not respond.

3. If any NOSPACE alarms were triggered due to the space quota being exceeded, clear them.

   1. Check if there are any NOSPACE alarms:

      sh-4.4# etcdctl alarm list

      Example output

      memberID:12345678912345678912 alarm:NOSPACE

   2. Clear the alarms:

      sh-4.4# etcdctl alarm disarm

Procedure

1. Select a control plane host to use as the recovery host. This is the host that you will run the restore operation on.

2. Establish SSH connectivity to each of the control plane nodes, including the recovery host.

   The Kubernetes API server becomes inaccessible after the restore process starts, so you cannot access the control plane nodes. For this reason, it is recommended to establish SSH connectivity to each control plane host in a separate terminal.

   Important

   If you do not complete this step, you will not be able to access the control plane hosts to complete the restore procedure, and you will be unable to recover your cluster from this state.

3. Copy the etcd backup directory to the recovery control plane host.

   This procedure assumes that you copied the backup directory containing the etcd snapshot and the resources for the static pods to the /home/core/ directory of your recovery control plane host.

4. Stop the static pods on any other control plane nodes.

   Note

   It is not required to manually stop the pods on the recovery host. The recovery script will stop the pods on the recovery host.

   1. Access a control plane host that is not the recovery host.

   2. Move the existing etcd pod file out of the kubelet manifest directory:

      $ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp

   3. Verify that the etcd pods are stopped.

      $ sudo crictl ps | grep etcd | grep -v operator

      The output of this command should be empty. If it is not empty, wait a few minutes and check again.

   4. Move the existing Kubernetes API server pod file out of the kubelet manifest directory:

      $ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp

   5. Verify that the Kubernetes API server pods are stopped.

      $ sudo crictl ps | grep kube-apiserver | grep -v operator

      The output of this command should be empty. If it is not empty, wait a few minutes and check again.

   6. Move the etcd data directory to a different location:

      $ sudo mv /var/lib/etcd/ /tmp

   7. Repeat this step on each of the other control plane hosts that is not the recovery host.
5. Access the recovery control plane host.

6. If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables.

   Tip

   You can check whether the proxy is enabled by reviewing the output of oc get proxy cluster -o yaml. The proxy is enabled if the httpProxy, httpsProxy, and noProxy fields have values set.

7. Run the restore script on the recovery control plane host and pass in the path to the etcd backup directory:

   $ sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup

   Example script output

   ...stopping kube-scheduler-pod.yaml
   ...stopping kube-controller-manager-pod.yaml
   ...stopping etcd-pod.yaml
   ...stopping kube-apiserver-pod.yaml
   Waiting for container etcd to stop
   .complete
   Waiting for container etcdctl to stop
   .............................complete
   Waiting for container etcd-metrics to stop
   complete
   Waiting for container kube-controller-manager to stop
   complete
   Waiting for container kube-apiserver to stop
   ..........................................................................................complete
   Waiting for container kube-scheduler to stop
   complete
   Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup
   starting restore-etcd static pod
   starting kube-apiserver-pod.yaml
   static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
   starting kube-controller-manager-pod.yaml
   static-pod-resources/kube-controller-manager-pod-7/kube-controller-manager-pod.yaml
   starting kube-scheduler-pod.yaml
   static-pod-resources/kube-scheduler-pod-8/kube-scheduler-pod.yaml

   Note

   The restore process can cause nodes to enter the NotReady state if the node certificates were updated after the last etcd backup.

8. Check the nodes to ensure they are in the Ready state.

   1. Run the following command:

      $ oc get nodes -w

      Sample output

      NAME                STATUS  ROLES          AGE     VERSION
      host-172-25-75-28   Ready   master         3d20h   v1.23.3+e419edf
      host-172-25-75-38   Ready   infra,worker   3d20h   v1.23.3+e419edf
      host-172-25-75-40   Ready   master         3d20h   v1.23.3+e419edf
      host-172-25-75-65   Ready   master         3d20h   v1.23.3+e419edf
      host-172-25-75-74   Ready   infra,worker   3d20h   v1.23.3+e419edf
      host-172-25-75-79   Ready   worker         3d20h   v1.23.3+e419edf
      host-172-25-75-86   Ready   worker         3d20h   v1.23.3+e419edf
      host-172-25-75-98   Ready   infra,worker   3d20h   v1.23.3+e419edf

      It can take several minutes for all nodes to report their state.

   2. If any nodes are in the NotReady state, log in to the nodes and remove all of the PEM files from the /var/lib/kubelet/pki directory on each node. You can SSH into the nodes or use the terminal window in the web console.

      $  ssh -i <ssh-key-path> core@<master-hostname>

      Sample pki directory

      sh-4.4# pwd
      /var/lib/kubelet/pki
      sh-4.4# ls
      kubelet-client-2022-04-28-11-24-09.pem  kubelet-server-2022-04-28-11-24-15.pem
      kubelet-client-current.pem              kubelet-server-current.pem

9. Restart the kubelet service on all control plane hosts.

   1. From the recovery host, run the following command:

      $ sudo systemctl restart kubelet.service

   2. Repeat this step on all other control plane hosts.

10. Approve the pending CSRs:

    1. Get the list of current CSRs:

       $ oc get csr

       Example output

       NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
       csr-2s94x   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending 1
       csr-4bd6t   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending 2
       csr-4hl85   13m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 3
       csr-zhhhp   3m8s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 4
       ...

       1 1 2

       A pending kubelet service CSR (for user-provisioned installations).

       3 4

       A pending node-bootstrapper CSR.

    2. Review the details of a CSR to verify that it is valid:

       $ oc describe csr <csr_name> 1

       1

       <csr_name> is the name of a CSR from the list of current CSRs.

    3. Approve each valid node-bootstrapper CSR:

       $ oc adm certificate approve <csr_name>

    4. For user-provisioned installations, approve each valid kubelet service CSR:

       $ oc adm certificate approve <csr_name>

11. Verify that the single member control plane has started successfully.

    1. From the recovery host, verify that the etcd container is running.

       $ sudo crictl ps | grep etcd | grep -v operator

       Example output

       3ad41b7908e32       36f86e2eeaaffe662df0d21041eb22b8198e0e58abeeae8c743c3e6e977e8009                                                         About a minute ago   Running             etcd                                          0                   7c05f8af362f0

    2. From the recovery host, verify that the etcd pod is running.

       $ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd

       Note

       If you attempt to run oc login prior to running this command and receive the following error, wait a few moments for the authentication controllers to start and try again.

       Unable to connect to the server: EOF

       Example output

       NAME                                             READY   STATUS      RESTARTS   AGE
       etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1          2m47s

       If the status is Pending, or the output lists more than one running etcd pod, wait a few minutes and check again.

    3. Repeat this step for each lost control plane host that is not the recovery host.

12. Delete and recreate other non-recovery, control plane machines, one by one. After these machines are recreated, a new revision is forced and etcd scales up automatically.

    If you are running installer-provisioned infrastructure, or you used the Machine API to create your machines, follow these steps. Otherwise, you must create the new control plane node using the same method that was used to originally create it.

    Warning

    Do not delete and recreate the machine for the recovery host.

    1. Obtain the machine for one of the lost control plane hosts.

       In a terminal that has access to the cluster as a cluster-admin user, run the following command:

       $ oc get machines -n openshift-machine-api -o wide

       Example output:

       NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
       clustername-8qw5l-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   3h37m   ip-10-0-131-183.ec2.internal   aws:///us-east-1a/i-0ec2782f8287dfb7e   stopped 1
       clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-143-125.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
       clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-154-194.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba  running
       clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
       clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
       clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

       1

       This is the control plane machine for the lost control plane host, ip-10-0-131-183.ec2.internal.

    2. Save the machine configuration to a file on your file system:

       $ oc get machine clustername-8qw5l-master-0 \ 1
           -n openshift-machine-api \
           -o yaml \
           > new-master-machine.yaml

       1

       Specify the name of the control plane machine for the lost control plane host.

    3. Edit the new-master-machine.yaml file that was created in the previous step to assign a new name and remove unnecessary fields.

       1. Remove the entire status section:

          status:
            addresses:
            - address: 10.0.131.183
              type: InternalIP
            - address: ip-10-0-131-183.ec2.internal
              type: InternalDNS
            - address: ip-10-0-131-183.ec2.internal
              type: Hostname
            lastUpdated: "2020-04-20T17:44:29Z"
            nodeRef:
              kind: Node
              name: ip-10-0-131-183.ec2.internal
              uid: acca4411-af0d-4387-b73e-52b2484295ad
            phase: Running
            providerStatus:
              apiVersion: awsproviderconfig.openshift.io/v1beta1
              conditions:
              - lastProbeTime: "2020-04-20T16:53:50Z"
                lastTransitionTime: "2020-04-20T16:53:50Z"
                message: machine successfully created
                reason: MachineCreationSucceeded
                status: "True"
                type: MachineCreation
              instanceId: i-0fdb85790d76d0c3f
              instanceState: stopped
              kind: AWSMachineProviderStatus

       2. Change the metadata.name field to a new name.

          It is recommended to keep the same base name as the old machine and change the ending number to the next available number. In this example, clustername-8qw5l-master-0 is changed to clustername-8qw5l-master-3:

          apiVersion: machine.openshift.io/v1beta1
          kind: Machine
          metadata:
            ...
            name: clustername-8qw5l-master-3
            ...

       3. Remove the spec.providerID field:

          providerID: aws:///us-east-1a/i-0fdb85790d76d0c3f

       4. Remove the metadata.annotations and metadata.generation fields:

          annotations:
            machine.openshift.io/instance-state: running
          ...
          generation: 2

       5. Remove the metadata.resourceVersion and metadata.uid fields:

          resourceVersion: "13291"
          uid: a282eb70-40a2-4e89-8009-d05dd420d31a

    4. Delete the machine of the lost control plane host:

       $ oc delete machine -n openshift-machine-api clustername-8qw5l-master-0 1

       1

       Specify the name of the control plane machine for the lost control plane host.

    5. Verify that the machine was deleted:

       $ oc get machines -n openshift-machine-api -o wide

       Example output:

       NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
       clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-143-125.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
       clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-154-194.ec2.internal   aws:///us-east-1c/i-02626f1dba9ed5bba  running
       clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
       clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
       clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

    6. Create the new machine using the new-master-machine.yaml file:

       $ oc apply -f new-master-machine.yaml

    7. Verify that the new machine has been created:

       $ oc get machines -n openshift-machine-api -o wide

       Example output:

       NAME                                        PHASE          TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
       clustername-8qw5l-master-1                  Running        m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-143-125.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
       clustername-8qw5l-master-2                  Running        m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-154-194.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba  running
       clustername-8qw5l-master-3                  Provisioning   m4.xlarge   us-east-1   us-east-1a   85s     ip-10-0-173-171.ec2.internal    aws:///us-east-1a/i-015b0888fe17bc2c8  running 1
       clustername-8qw5l-worker-us-east-1a-wbtgd   Running        m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
       clustername-8qw5l-worker-us-east-1b-lrdxb   Running        m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
       clustername-8qw5l-worker-us-east-1c-pkg26   Running        m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

       1

       The new machine, clustername-8qw5l-master-3 is being created and is ready after the phase changes from Provisioning to Running.

       It might take a few minutes for the new machine to be created. The etcd cluster Operator will automatically sync when the machine or node returns to a healthy state.

    8. Repeat these steps for each lost control plane host that is not the recovery host.

13. In a separate terminal window, log in to the cluster as a user with the cluster-admin role by using the following command:

    $ oc login -u <cluster_admin> 1

    1

    For <cluster_admin>, specify a user name with the cluster-admin role.

14. Force etcd redeployment.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command:

    $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge 1

    1

    The forceRedeploymentReason value must be unique, which is why a timestamp is appended.

    When the etcd cluster Operator performs a redeployment, the existing nodes are started with new pods similar to the initial bootstrap scale up.

15. Verify all nodes are updated to the latest revision.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command:

    $ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

    Review the NodeInstallerProgressing status condition for etcd to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

    AllNodesAtLatestRevision
    3 nodes are at revision 7 1

    1

    In this example, the latest revision number is 7.

    If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

16. After etcd is redeployed, force new rollouts for the control plane. The Kubernetes API server will reinstall itself on the other nodes because the kubelet is connected to API servers using an internal load balancer.

    In a terminal that has access to the cluster as a cluster-admin user, run the following commands.

    1. Force a new rollout for the Kubernetes API server:

       $ oc patch kubeapiserver cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

       Verify all nodes are updated to the latest revision.

       $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

       Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

       AllNodesAtLatestRevision
       3 nodes are at revision 7 1

       1

       In this example, the latest revision number is 7.

       If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

    2. Force a new rollout for the Kubernetes controller manager:

       $ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

       Verify all nodes are updated to the latest revision.

       $ oc get kubecontrollermanager -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

       Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

       AllNodesAtLatestRevision
       3 nodes are at revision 7 1

       1

       In this example, the latest revision number is 7.

       If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

    3. Force a new rollout for the Kubernetes scheduler:

       $ oc patch kubescheduler cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

       Verify all nodes are updated to the latest revision.

       $ oc get kubescheduler -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

       Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

       AllNodesAtLatestRevision
       3 nodes are at revision 7 1

       1

       In this example, the latest revision number is 7.

       If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

17. Verify that all control plane hosts have started and joined the cluster.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command:

    $ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd

    Example output

    etcd-ip-10-0-143-125.ec2.internal                2/2     Running     0          9h
    etcd-ip-10-0-154-194.ec2.internal                2/2     Running     0          9h
    etcd-ip-10-0-173-171.ec2.internal                2/2     Running     0          9h

1. Create a YAML file with the an object definition similar to the following:

   apiVersion: policy/v1 1
   kind: PodDisruptionBudget
   metadata:
     name: my-pdb
   spec:
     minAvailable: 2  2
     selector:  3
       matchLabels:
         foo: bar

   1

   PodDisruptionBudget is part of the policy/v1 API group.

   2

   The minimum number of pods that must be available simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.

   3

   A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined. Leave this paramter blank, for example selector {}, to select all pods in the project.

   Or:

   apiVersion: policy/v1 1
   kind: PodDisruptionBudget
   metadata:
     name: my-pdb
   spec:
     maxUnavailable: 25% 2
     selector: 3
       matchLabels:
         foo: bar

   1

   PodDisruptionBudget is part of the policy/v1 API group.

   2

   The maximum number of pods that can be unavailable simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.

   3

   A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined. Leave this paramter blank, for example selector {}, to select all pods in the project.

2. Run the following command to add the object to project:

   $ oc create -f </path/to/file> -n <project_name>

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   Azure	azure-credentials
   GCP	gcp-credentials
   RHOSP	openstack-credentials
   RHV	ovirt-credentials
   VMware vSphere	vsphere-creds

3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.

   Note

   If the vSphere CSI Driver Operator is enabled, this step is not required.

   To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the cluster-admin role and run the following command:

   $ oc patch kubecontrollermanager cluster \
     -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \
     --type=merge

   While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports Progressing=true. To view the status, run the following command:

   $ oc get co kube-controller-manager

7. If the CCO for your cluster is configured to use mint mode, delete each component secret that is referenced by the individual CredentialsRequest objects.

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • GCP: GCPProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \1
        -n <secret_namespace> 2

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

3. Click the Options menu in the same row as the secret and select Delete Secret.

Procedure

1. Access the images of a specific image stream to mirror, for example:

   $ oc get is <imagestream> -n openshift -o json | jq .spec.tags[].from.name | grep registry.redhat.io

2. Mirror images from registry.redhat.io associated with any image streams you need in the restricted network environment into one of the defined mirrors, for example:

   $ oc image mirror registry.redhat.io/rhscl/ruby-25-rhel7:latest ${MIRROR_ADDR}/rhscl/ruby-25-rhel7:latest

3. Create the cluster’s image configuration object:

   $ oc create configmap registry-config --from-file=${MIRROR_ADDR_HOSTNAME}..5000=$path/ca.crt -n openshift-config

4. Add the required trusted CAs for the mirror in the cluster’s image configuration object:

   $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge

5. Update the samplesRegistry field in the Cluster Samples Operator configuration object to contain the hostname portion of the mirror location defined in the mirror configuration:

   $ oc edit configs.samples.operator.openshift.io -n openshift-cluster-samples-operator

   Note

   This is required because the image stream import process does not use the mirror or search mechanism at this time.

6. Add any image streams that are not mirrored into the skippedImagestreams field of the Cluster Samples Operator configuration object. Or if you do not want to support any of the sample image streams, set the Cluster Samples Operator to Removed in the Cluster Samples Operator configuration object.

   Note

   The Cluster Samples Operator issues alerts if image stream imports are failing but the Cluster Samples Operator is either periodically retrying or does not appear to be retrying them.

   Many of the templates in the openshift namespace reference the image streams. So using Removed to purge both the image streams and templates will eliminate the possibility of attempts to use them if they are not functional because of any missing image streams.

Procedure

1. If you have not added your mirror registry’s trusted CA to your cluster’s image configuration object as part of the Cluster Samples Operator configuration, perform the following steps:

   1. Create the cluster’s image configuration object:

      $ oc create configmap registry-config --from-file=${MIRROR_ADDR_HOSTNAME}..5000=$path/ca.crt -n openshift-config

   2. Add the required trusted CAs for the mirror in the cluster’s image configuration object:

      $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge

2. Import the default must-gather image from your installation payload:

   $ oc import-image is/must-gather -n openshift