Chapter 4. Working with nodes

1. View the MachineSets that are in the cluster:

   $ oc get machinesets -n openshift-machine-api

   The MachineSets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. Scale the MachineSet:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

Procedure

1. Obtain the label associated with the static CRD, Machine Config Pool, for the type of node you want to configure. Perform one of the following steps:

   1. Check current labels of the desired machineconfigpool.

      For example:

      $  oc get machineconfigpool  --show-labels
      NAME      CONFIG                                             UPDATED   UPDATING   DEGRADED   LABELS
      master    rendered-master-e05b81f5ca4db1d249a1bf32f9ec24fd   True      False      False      operator.machineconfiguration.openshift.io/required-for-upgrade=
      worker    rendered-worker-f50e78e1bc06d8e82327763145bfcf62   True      False      False

   2. Add a custom label to the desired machineconfigpool.

      For example:

      $ oc label machineconfigpool worker custom-kubelet=enabled

2. Create a KubeletConfig Custom Resource (CR) for your configuration change.

   For example:

   Sample configuration for a custom-config CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: custom-config 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: enabled 2
     kubeletConfig: 3
       podsPerCore: 10
       maxPods: 250
       systemReserved:
         cpu: 1000m
         memory: 500Mi
       kubeReserved:
         cpu: 1000m
         memory: 500Mi

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change, this is the label you added to the machineconfigpool.

   3

   Specify the new value(s) you want to change.

3. Create the CR object.

   $ oc create -f <file-name>

   For example:

   $ oc create -f master-kube-config.yaml

Prerequisite

1. Obtain the label associated with the static Machine Config Pool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the Machine Config Pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker
      
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a Custom Resource (CR) for your configuration change.

   Sample configuration for a max-pods CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       podsPerCore: 10 3
       maxPods: 250 4

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change.

   3

   Specify the number of pods the node can run based on the number of processor cores on the node.

   4

   Specify the number of pods the node can run to a fixed value, regardless of the properties of the node.

   Note

   Setting podsPerCore to 0 disables this limit.

   In the above example, the default value for podsPerCore is 10 and the default value for maxPods is 250. This means that unless the node has 25 cores or more, by default, podsPerCore will be the limiting factor.

2. List the Machine Config Pool CRDs to see if the change is applied. The UPDATING column reports True if the change is picked up by the Machine Config Controller:

   $ oc get machineconfigpools
   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     False      False
   worker   worker-8cecd1236b33ee3f8a5e   False     True       False

   Once the change is complete, the UPDATED column reports True.

   $ oc get machineconfigpools
   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     True       False
   worker   worker-8cecd1236b33ee3f8a5e   True      False      False

Procedure

1. Run:

   $ oc get Tuned/default -o yaml -n openshift-cluster-node-tuning-operator

1. Edit the node specification to configure pod anti-affinity:

   apiVersion: v1
   kind: Pod
   metadata:
     name: with-pod-antiaffinity
   spec:
     affinity:
       podAntiAffinity: 1
         preferredDuringSchedulingIgnoredDuringExecution: 2
         - weight: 100 3
           podAffinityTerm:
             labelSelector:
               matchExpressions:
               - key: registry 4
                 operator: In 5
                 values:
                 - default
             topologyKey: kubernetes.io/hostname

   1

   Stanza to configure pod anti-affinity.

   2

   Defines a preferred rule.

   3

   Specifies a weight for a preferred rule. The node with the highest weight is preferred.

   4

   Description of the pod label that determines when the anti-affinity rule applies. Specify a key and value for the label.

   5

   The operator represents the relationship between the label on the existing pod and the set of values in the matchExpression parameters in the specification for the new pod. Can be In, NotIn, Exists, or DoesNotExist.

   This example assumes the container image registry pod has a label of registry=default. Pod anti-affinity can use any Kubernetes match expression.

2. Enable the MatchInterPodAffinity scheduler predicate in the scheduling policy file.

1. A list of images currently running in at least one pod.
2. A list of images available on a host.

Prerequisites

1. Obtain the label associated with the static Machine Config Pool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the Machine Config Pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker
      
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a Custom Resource (CR) for your configuration change.

   Sample configuration for a container garbage collection CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: worker-kubeconfig 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       evictionSoft: 3
       evictionSoft:
         memory.available: "500Mi" 4
         nodefs.available: "10%"
         nodefs.inodesFree: "5%"
         imagefs.available: "15%"
         imagefs.inodesFree: "10%"
       evictionSoftGracePeriod:  5
         memory.available: "1m30s"
         nodefs.available: "1m30s"
         nodefs.inodesFree: "1m30s"
         imagefs.available: "1m30s"
         imagefs.inodesFree: "1m30s"
       evictionHard:
         memory.available: "200Mi"
         nodefs.available: "5%"
         nodefs.inodesFree: "4%"
         imagefs.available: "10%"
         imagefs.inodesFree: "5%"
       evictionPressureTransitionPeriod: 0s 6
       imageMinimumGCAge: 5m 7
       imageGCHighThresholdPercent: 80 8
       imageGCLowThresholdPercent: 75 9

   1

   Name for the object.

   2

   Selector label.

   3

   Type of eviction: EvictionSoft and EvictionHard.

   4

   Eviction thresholds based on a specific eviction trigger signal.

   5

   Grace periods for the soft eviction. This parameter does not apply to eviction-hard.

   6

   The duration to wait before transitioning out of an eviction pressure condition

   7

   The minimum age for an unused image before the image is removed by garbage collection.

   8

   The percent of disk usage (expressed as an integer) which triggers image garbage collection.

   9

   The percent of disk usage (expressed as an integer) to which image garbage collection attempts to free.

2. Create the object:

   $ oc create -f <file-name>.yaml

   For example:

   oc create -f gc-container.yaml
   
   kubeletconfig.machineconfiguration.openshift.io/gc-container created

3. Verify that garbage collection is active. The Machine Config Pool you specified in the custom resource appears with UPDATING as 'true` until the change is fully implemented:

   $ oc get machineconfigpool
   
   NAME     CONFIG                                   UPDATED   UPDATING
   master   rendered-master-546383f80705bd5aeaba93   True      False
   worker   rendered-worker-b4c51bb33ccaae6fc4a6a5   False     True

Prerequisites

1. To help you determine setting for --system-reserved and --kube-reserved you can introspect the corresponding node’s resource usage using the node summary API, which is accessible at <master>/api/v1/nodes/<node>/proxy/stats/summary. Run the following command for your node:

   $ curl <certificate details> https://<master>/api/v1/nodes/<node-name>/proxy/stats/summary

   The REST API Overview has details about certificate details.

   For example, to access the resources from cluster.node22 node, you can run:

   $ curl <certificate details> https://<master>/api/v1/nodes/cluster.node22/proxy/stats/summary
   {
       "node": {
           "nodeName": "cluster.node22",
           "systemContainers": [
               {
                   "cpu": {
                       "usageCoreNanoSeconds": 929684480915,
                       "usageNanoCores": 190998084
                   },
                   "memory": {
                       "rssBytes": 176726016,
                       "usageBytes": 1397895168,
                       "workingSetBytes": 1050509312
                   },
                   "name": "kubelet"
               },
               {
                   "cpu": {
                       "usageCoreNanoSeconds": 128521955903,
                       "usageNanoCores": 5928600
                   },
                   "memory": {
                       "rssBytes": 35958784,
                       "usageBytes": 129671168,
                       "workingSetBytes": 102416384
                   },
                   "name": "runtime"
               }
           ]
       }
   }

2. Obtain the label associated with the static Machine Config Pool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the Machine Config Pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker
      
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a Custom Resource (CR) for your configuration change.

   Sample configuration for a resource allocation CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-allocatable 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       systemReserved:
         cpu: 500m
         memory: 512Mi
       kubeReserved:
         cpu: 500m
         memory: 512Mi

1. The request line containing:

   1. A Unique ID allowing to match the response line (see #2)
   2. The source IP of the request
   3. The HTTP method being invoked
   4. The original user invoking the operation
   5. The impersonated user for the operation (self meaning himself)
   6. The impersonated group for the operation (lookup meaning user’s group)
   7. The namespace of the request or <none>
   8. The URI as requested

2. The response line containing:

   1. The unique ID from #1
   2. The response code

Procedure

1. Set the audit log level:

   1. Get the name of the API server Custom Resource (CR):

      $ oc get APIServer
      
      NAME      AGE
      cluster   18h

   2. Edit the API server CR:

      $ oc edit APIServer cluster

      apiVersion: config.openshift.io/v1
      kind: APIServer
      metadata:
        annotations:
          release.openshift.io/create-only: "true"
        creationTimestamp: 2019-05-09T18:56:37Z
        generation: 2
        name: cluster
        resourceVersion: "562502"
        selfLink: /apis/config.openshift.io/v1/apiservers/cluster
        uid: 2c4e980b-728c-11e9-967d-0a973adad40e
      spec:
        logLevel: "Normal" 1
        operatorloglevel: "TraceAll" 2

      1 1

      Specify the log level for an overall component.

      2 2

      Specify the log level for the API server Operator.

      You can set the log level for both settings to one of the following. The setting can be different for each setting:

      • Normal. Normal is the default. Normal working log information, including helpful notices for auditing or common operations. Similar to glog=2.
      • Debug. Debug is for troubleshooting problems. A greater quanitity of notices than Normal, but contain less information than Trace. Common operations might be logged. Similar to glog=4.
      • Trace. Trace is for troubleshooting problems when Debug is not verbose enough. Logs every function call as part of a common operation, including tracing execution of a query. Similar to glog=6.
      • TraceAll. TraceAll is troubleshoting at the level of API content/decoding. Contains complete body content. In production clusters, this setting causes performance degradation and results in a significant number of logs. Similar to ` glog=8`.

1. View the OpenShift Container Platform API server logs

   1. If necessary, get the node IP and name of the log you want to view:

      $ oc --insecure-skip-tls-verify adm node-logs --role=master --path=openshift-apiserver/
      
      ip-10-0-140-97.ec2.internal audit-2019-04-09T00-12-19.834.log
      ip-10-0-140-97.ec2.internal audit-2019-04-09T11-13-00.469.log
      ip-10-0-140-97.ec2.internal audit.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T00-11-49.835.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T11-08-30.469.log
      ip-10-0-153-35.ec2.internal audit.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T00-13-00.128.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T11-10-04.082.log
      ip-10-0-170-165.ec2.internal audit.log

   2. View the OpenShift Container Platform API server log for a specific master node and timestamp or view all the logs for that master:

      $ oc adm node-logs <node-ip> <log-name> --path=openshift-apiserver/<log-name>

      For example:

      $ oc adm node-logs ip-10-0-140-97.ec2.internal audit-2019-04-08T13-09-01.227.log --path=openshift-apiserver/audit-2019-04-08T13-09-01.227.log
      $ oc adm node-logs ip-10-0-140-97.ec2.internal audit.log --path=openshift-apiserver/audit.log

      The output appears similar to the following:

      ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T18:52:03Z"},"level":"Metadata","timestamp":"2019-04-09T18:52:03Z","auditID":"9708b50d-8956-4c87-b9eb-a53ba054c13d","stage":"ResponseComplete","requestURI":"/","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.128.0.1"],"userAgent":"Go-http-client/2.0","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-04-09T18:52:03.914638Z","stageTimestamp":"2019-04-09T18:52:03.915080Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-status-binding\" of ClusterRole \"cluster-status\" to Group \"system:unauthenticated\""}}

2. View the Kubernetes API server logs:

   1. If necessary, get the node IP and name of the log you want to view:

      $ oc --insecure-skip-tls-verify adm node-logs --role=master --path=kube-apiserver/
      
      ip-10-0-140-97.ec2.internal audit-2019-04-09T14-07-27.129.log
      ip-10-0-140-97.ec2.internal audit-2019-04-09T19-18-32.542.log
      ip-10-0-140-97.ec2.internal audit.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T19-24-22.620.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T19-51-30.905.log
      ip-10-0-153-35.ec2.internal audit.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T18-37-07.511.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T19-21-14.371.log
      ip-10-0-170-165.ec2.internal audit.log

   2. View the Kubernetes API server log for a specific master node and timestamp or view all the logs for that master:

      $ oc adm node-logs <node-ip> <log-name> --path=kube-apiserver/<log-name>

      For example:

      $ oc adm node-logs ip-10-0-140-97.ec2.internal audit-2019-04-09T14-07-27.129.log --path=kube-apiserver/audit-2019-04-09T14-07-27.129.log
      $ oc adm node-logs ip-10-0-170-165.ec2.internal audit.log --path=kube-apiserver/audit.log

      The output appears similar to the following:

      ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T19:56:58Z"},"level":"Metadata","timestamp":"2019-04-09T19:56:58Z","auditID":"6e96c88b-ab6f-44d2-b62e-d1413efd676b","stage":"ResponseComplete","requestURI":"/api/v1/nodes/audit-2019-04-09T14-07-27.129.log","verb":"get","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.0.57.93"],"userAgent":"oc/v1.13.4+b626c2fe1 (linux/amd64) kubernetes/ba88cb2","objectRef":{"resource":"nodes","name":"audit-2019-04-09T14-07-27.129.log","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2019-04-09T19:56:58.982157Z","stageTimestamp":"2019-04-09T19:56:58.985300Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}