Machine management

Procedure

1. Create a new YAML file that contains the MachineSet Custom Resource sample, as shown, and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

   1. If you are not sure about which value to set for an specific field, you can check an existing MachineSet from your cluster.

      $ oc get machinesets -n openshift-machine-api
      
      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. Check values of an specific MachineSet:

      $ oc get machineset <machineset_name> -n \
           openshift-machine-api -o yaml
      
      ....
      
      template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1
              machine.openshift.io/cluster-api-machine-role: worker 2
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a

      1

      The cluster ID.

      2

      A default node label.

2. Create the new MachineSet:

   $ oc create -f <file_name>.yaml

3. View the list of MachineSets:

   $ oc get machineset -n openshift-machine-api
   
   
   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
   agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
   agl030519-vplxk-worker-us-east-1d   0         0                             55m
   agl030519-vplxk-worker-us-east-1e   0         0                             55m
   agl030519-vplxk-worker-us-east-1f   0         0                             55m

   When the new MachineSet is available, the DESIRED and CURRENT values match. If the MachineSet is not available, wait a few minutes and run the command again.

4. After the new MachineSet is available, check status of the machine and the node that it references:

   $ oc describe machine <name> -n openshift-machine-api

   For example:

   $ oc describe machine agl030519-vplxk-infra-us-east-1a -n openshift-machine-api
   
   status:
     addresses:
     - address: 10.0.133.18
       type: InternalIP
     - address: ""
       type: ExternalDNS
     - address: ip-10-0-133-18.ec2.internal
       type: InternalDNS
     lastUpdated: "2019-05-03T10:38:17Z"
     nodeRef:
       kind: Node
       name: ip-10-0-133-18.ec2.internal
       uid: 71fb8d75-6d8f-11e9-9ff3-0e3f103c7cd8
     providerStatus:
       apiVersion: awsproviderconfig.openshift.io/v1beta1
       conditions:
       - lastProbeTime: "2019-05-03T10:34:31Z"
         lastTransitionTime: "2019-05-03T10:34:31Z"
         message: machine successfully created
         reason: MachineCreationSucceeded
         status: "True"
         type: MachineCreation
       instanceId: i-09ca0701454124294
       instanceState: running
       kind: AWSMachineProviderStatus

5. View the new node and confirm that the new node has the label that you specified:

   $ oc get node <node_name> --show-labels

   Review the command output and confirm that node-role.kubernetes.io/<your_label> is in the LABELS list.

Procedure

1. View the MachineSets that are in the cluster:

   $ oc get machinesets -n openshift-machine-api

   The MachineSets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. Scale the MachineSet:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   You can scale the MachineSet up or down. It takes several minutes for the new machines to be available.

   Important

   By default, the OpenShift Container Platform router pods are deployed on workers. Because the router is required to access some cluster resources, including the web console, do not scale the worker MachineSet to 0 unless you first relocate the router pods.

Procedure

1. Edit the MachineSet:

   $ oc edit machineset <machineset> -n openshift-machine-api

2. Scale down the MachineSet to 0:

   $ oc scale --replicas=0 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   Wait for the machines to be removed.

3. Scale up the MachineSet as needed:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   Wait for the machines to start. The new Machines contain changes you made to the Machineset.

Procedure

1. View the Machines that are in the cluster and identify the one to delete:

   $ oc get machine -n openshift-machine-api

   The command output contains a list of Machines in the <clusterid>-worker-<cloud_region> format.

2. Delete the Machine:

   $ oc delete machine <machine> -n openshift-machine-api

   Important

   By default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured Pod disruption budget, the drain operation might not be able to succeed in preventing the machine from being deleted. You can skip draining the node by annotating "machine.openshift.io/exclude-node-draining" in a specific machine. If the machine being deleted belongs to a MachineSet, a new machine is immediately created to satisfy the specified number of replicas.

Procedure

1. Ensure that the kubeconfig file for the cluster and the installation program that you used to install the cluster are on the machine. One way to accomplish this is to use the same machine that you used to install the cluster.
2. Configure the machine to access all of the RHEL hosts that you plan to use as compute machines. You can use any method that your company allows, including a bastion with an SSH proxy or a VPN.

3. Configure a user on the machine that you run the playbook on that has SSH access to all of the RHEL hosts.

   Important

   If you use SSH key-based authentication, you must manage the key with an SSH agent.

4. If you have not already done so, register the machine with RHSM and attach a pool with an OpenShift subscription to it:

   1. Register the machine with RHSM:

      # subscription-manager register --username=<user_name> --password=<password>

   2. Pull the latest subscription data from RHSM:

      # subscription-manager refresh

   3. List the available subscriptions:

      # subscription-manager list --available --matches '*OpenShift*'

   4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

      # subscription-manager attach --pool=<pool_id>

5. Enable the repositories required by OpenShift Container Platform 4.1:

   # subscription-manager repos \
       --enable="rhel-7-server-rpms" \
       --enable="rhel-7-server-extras-rpms" \
       --enable="rhel-7-server-ansible-2.7-rpms" \
       --enable="rhel-7-server-ose-4.1-rpms"

6. Install the required packages, including Openshift-Ansible:

   # yum install openshift-ansible openshift-clients jq

   The openshift-ansible package provides installation program utilities and pulls in other packages that you require to add a RHEL compute node to your cluster, such as Ansible, playbooks, and related configuration files. The openshift-clients provides the oc CLI, and the jq package improves the display of JSON output on your command line.

1. On each host, register with RHSM:

   # subscription-manager register --username=<user_name> --password=<password>

2. Pull the latest subscription data from RHSM:

   # subscription-manager refresh

3. List the available subscriptions:

   # subscription-manager list --available --matches '*OpenShift*'

4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

   # subscription-manager attach --pool=<pool_id>

5. Disable all yum repositories:

   1. Disable all the enabled RHSM repositories:

      # subscription-manager repos --disable="*"

   2. List the remaining yum repositories and note their names under repo id, if any:

      # yum repolist

   3. Use yum-config-manager to disable the remaining yum repositories:

      # yum-config-manager --disable <repo_id>

      Alternatively, disable all repositories:

       yum-config-manager --disable \*

      Note that this might take a few minutes if you have a large number of available repositories

6. Enable only the repositories required by OpenShift Container Platform 4.1:

   # subscription-manager repos \
       --enable="rhel-7-server-rpms" \
       --enable="rhel-7-server-extras-rpms" \
       --enable="rhel-7-server-ose-4.1-rpms"

7. Stop and disable firewalld on the host:

   # systemctl disable --now firewalld.service

   Note

   You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker.

1. Extract the pull secret for your cluster:

   $ oc -n openshift-config get -o jsonpath='{.data.\.dockerconfigjson}' secret pull-secret | base64 -d | jq .

2. Save the pull secret in a file that is named pull-secret.txt.

3. Create an Ansible inventory file that is named /<path>/inventory/hosts that defines your compute machine hosts and required variables:

   [all:vars]
   ansible_user=root 1
   #ansible_become=True 2
   
   openshift_kubeconfig_path="~/.kube/config" 3
   openshift_pull_secret_path="~/pull-secret.txt" 4
   
   [new_workers] 5
   mycluster-worker-0.example.com
   mycluster-worker-1.example.com

   1

   Specify the user name that runs the Ansible tasks on the remote compute machines.

   2

   If you do not specify root for the ansible_user, you must set ansible_become to True and assign the user sudo permissions.

   3

   Specify the path to the kubeconfig file for your cluster.

   4

   Specify the path to the file that contains the pull secret for the image registry for your cluster.

   5

   List each RHEL machine to add to your cluster. You must provide the fully-qualified domain name for each host. This name is the host name that the cluster uses to access the machine, so set the correct public or private name to access the machine.

4. Run the playbook:

   $ cd /usr/share/ansible/openshift-ansible
   $ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml 1

   1

   For <path>, specify the path to the Ansible inventory file that you created.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.13.4+b626c2fe1
   master-1  Ready     master  63m  v1.13.4+b626c2fe1
   master-2  Ready     master  64m  v1.13.4+b626c2fe1
   worker-0  NotReady  worker  76s  v1.13.4+b626c2fe1
   worker-1  NotReady  worker  70s  v1.13.4+b626c2fe1

   The output lists all of the machines that you created.

2. Review the pending certificate signing requests (CSRs) and ensure that the you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   1

   A client request CSR.

   2

   A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • If all the CSRs are valid, approve them all by running the following command:

     $ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

1. On each host, register with RHSM:

   # subscription-manager register --username=<user_name> --password=<password>

2. Pull the latest subscription data from RHSM:

   # subscription-manager refresh

3. List the available subscriptions:

   # subscription-manager list --available --matches '*OpenShift*'

4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

   # subscription-manager attach --pool=<pool_id>

5. Disable all yum repositories:

   1. Disable all the enabled RHSM repositories:

      # subscription-manager repos --disable="*"

   2. List the remaining yum repositories and note their names under repo id, if any:

      # yum repolist

   3. Use yum-config-manager to disable the remaining yum repositories:

      # yum-config-manager --disable <repo_id>

      Alternatively, disable all repositories:

       yum-config-manager --disable \*

      Note that this might take a few minutes if you have a large number of available repositories

6. Enable only the repositories required by OpenShift Container Platform 4.1:

   # subscription-manager repos \
       --enable="rhel-7-server-rpms" \
       --enable="rhel-7-server-extras-rpms" \
       --enable="rhel-7-server-ose-4.1-rpms"

7. Stop and disable firewalld on the host:

   # systemctl disable --now firewalld.service

   Note

   You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker.

Procedure

1. Open the Ansible inventory file at /<path>/inventory/hosts that defines your compute machine hosts and required variables.
2. Rename the [new_workers] section of the file to [workers].

3. Add a [new_workers] section to the file and define the fully-qualified domain names for each new host. The file resembles the following example:

   [all:vars]
   ansible_user=root
   #ansible_become=True
   
   openshift_kubeconfig_path="~/.kube/config"
   openshift_pull_secret_path="~/pull-secret.txt"
   
   [workers]
   mycluster-worker-0.example.com
   mycluster-worker-1.example.com
   
   [new_workers]
   mycluster-worker-2.example.com
   mycluster-worker-3.example.com

   In this example, the mycluster-worker-0.example.com and mycluster-worker-1.example.com machines are in the cluster and you add the mycluster-worker-2.example.com and mycluster-worker-3.example.com machines.

4. Run the scaleup playbook:

   $ cd /usr/share/ansible/openshift-ansible
   $ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml 1

   1

   For <path>, specify the path to the Ansible inventory file that you created.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.13.4+b626c2fe1
   master-1  Ready     master  63m  v1.13.4+b626c2fe1
   master-2  Ready     master  64m  v1.13.4+b626c2fe1
   worker-0  NotReady  worker  76s  v1.13.4+b626c2fe1
   worker-1  NotReady  worker  70s  v1.13.4+b626c2fe1

   The output lists all of the machines that you created.

2. Review the pending certificate signing requests (CSRs) and ensure that the you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   1

   A client request CSR.

   2

   A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • If all the CSRs are valid, approve them all by running the following command:

     $ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

Procedure

1. Create a healthcheck.yml file that contains the definition of your MachineHealthCheck.

2. Apply the healthcheck.yml file to your cluster:

   $ oc apply -f healthcheck.yml