Chapter 7. Troubleshooting

1. Ignition configuration files are created.
2. The bootstrap machine boots and starts hosting the remote resources required for the control plane machines to boot.
3. The control plane machines fetch the remote resources from the bootstrap machine and finish booting.
4. The control plane machines use the bootstrap machine to form an etcd cluster.
5. The bootstrap machine starts a temporary Kubernetes control plane using the new etcd cluster.
6. The temporary control plane schedules the production control plane to the control plane machines.
7. The temporary control plane shuts down and passes control to the production control plane.
8. The bootstrap machine adds OpenShift Container Platform components into the production control plane.
9. The installation program shuts down the bootstrap machine.
10. The control plane sets up the worker nodes.
11. The control plane installs additional services in the form of a set of Operators.
12. The cluster downloads and configures remaining components needed for the day-to-day operation, including the creation of worker machines in supported environments.

Procedure

1. Check that the haproxy systemd service is active:

   $ ssh <user_name>@<load_balancer> systemctl status haproxy

2. Verify that the load balancer is listening on the required ports. The following example references ports 80, 443, 6443, and 22623.

   • For HAProxy instances running on Red Hat Enterprise Linux (RHEL) 6, verify port status by using the netstat command:

     $ ssh <user_name>@<load_balancer> netstat -nltupe | grep -E ':80|:443|:6443|:22623'

   • For HAProxy instances running on Red Hat Enterprise Linux (RHEL) 7 or 8, verify port status by using the ss command:

     $ ssh <user_name>@<load_balancer> ss -nltupe | grep -E ':80|:443|:6443|:22623'

     Note

     Red Hat recommends the ss command instead of netstat in Red Hat Enterprise Linux (RHEL) 7 or later. ss is provided by the iproute package. For more information on the ss command, see the Red Hat Enterprise Linux (RHEL) 7 Performance Tuning Guide.

3. Check that the wildcard DNS record resolves to the load balancer:

   $ dig <wildcard_fqdn> @<dns_server>

Procedure

1. Watch the installation log as the installation progresses:

   $ tail -f ~/<installation_directory>/.openshift_install.log

2. Monitor the bootkube.service journald unit log on the bootstrap node, after it has booted. This provides visibility into the bootstrapping of the first control plane. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

   $ ssh core@<bootstrap_fqdn> journalctl -b -f -u bootkube.service

   Note

   The bootkube.service log on the bootstrap node outputs etcd connection refused errors, indicating that the bootstrap server is unable to connect to etcd on control plane nodes. After etcd has started on each control plane node and the nodes have joined the cluster, the errors should stop.

3. Monitor kubelet.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node agent activity.

   1. Monitor the logs using oc:

      $ oc adm node-logs --role=master -u kubelet

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

4. Monitor crio.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node CRI-O container runtime activity.

   1. Monitor the logs using oc:

      $ oc adm node-logs --role=master -u crio

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@master-N.cluster_name.sub_domain.domain journalctl -b -f -u crio.service

Procedure

1. If you have access to the bootstrap node’s console, monitor the console until the node reaches the login prompt.

2. Verify the Ignition file configuration.

   • If you are hosting Ignition configuration files by using an HTTP server.

     1. Verify the bootstrap node Ignition file URL. Replace <http_server_fqdn> with HTTP server’s fully qualified domain name:

        $ curl -I http://<http_server_fqdn>:<port>/bootstrap.ign  1

        1

        The -I option returns the header only. If the Ignition file is available on the specified URL, the command returns 200 OK status. If it is not available, the command returns 404 file not found.

     2. To verify that the Ignition file was received by the bootstrap node, query the HTTP server logs on the serving host. For example, if you are using an Apache web server to serve Ignition files, enter the following command:

        $ grep -is 'bootstrap.ign' /var/log/httpd/access_log

        If the bootstrap Ignition file is received, the associated HTTP GET log message will include a 200 OK success status, indicating that the request succeeded.

     3. If the Ignition file was not received, check that the Ignition files exist and that they have the appropriate file and web server permissions on the serving host directly.

   • If you are using a cloud provider mechanism to inject Ignition configuration files into hosts as part of their initial deployment.

     1. Review the bootstrap node’s console to determine if the mechanism is injecting the bootstrap node Ignition file correctly.
3. Verify the availability of the bootstrap node’s assigned storage device.
4. Verify that the bootstrap node has been assigned an IP address from the DHCP server.

5. Collect bootkube.service journald unit logs from the bootstrap node. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

   $ ssh core@<bootstrap_fqdn> journalctl -b -f -u bootkube.service

   Note

   The bootkube.service log on the bootstrap node outputs etcd connection refused errors, indicating that the bootstrap server is unable to connect to etcd on control plane nodes. After etcd has started on each control plane node and the nodes have joined the cluster, the errors should stop.

6. Collect logs from the bootstrap node containers.

   1. Collect the logs using podman on the bootstrap node. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

      $ ssh core@<bootstrap_fqdn> 'for pod in $(sudo podman ps -a -q); do sudo podman logs $pod; done'

7. If the bootstrap process fails, verify the following.

   • You can resolve api.<cluster_name>.<base_domain> from the installation host.
   • The load balancer proxies port 6443 connections to bootstrap and control plane nodes. Ensure that the proxy configuration meets OpenShift Container Platform installation requirements.

Procedure

1. If you have access to the console for the control plane node, monitor the console until the node reaches the login prompt. During the installation, Ignition log messages are output to the console.

2. Verify Ignition file configuration.

   • If you are hosting Ignition configuration files by using an HTTP server.

     1. Verify the control plane node Ignition file URL. Replace <http_server_fqdn> with HTTP server’s fully qualified domain name:

        $ curl -I http://<http_server_fqdn>:<port>/master.ign  1

        1

        The -I option returns the header only. If the Ignition file is available on the specified URL, the command returns 200 OK status. If it is not available, the command returns 404 file not found.

     2. To verify that the Ignition file was received by the control plane node query the HTTP server logs on the serving host. For example, if you are using an Apache web server to serve Ignition files:

        $ grep -is 'master.ign' /var/log/httpd/access_log

        If the master Ignition file is received, the associated HTTP GET log message will include a 200 OK success status, indicating that the request succeeded.

     3. If the Ignition file was not received, check that it exists on the serving host directly. Ensure that the appropriate file and web server permissions are in place.

   • If you are using a cloud provider mechanism to inject Ignition configuration files into hosts as part of their initial deployment.

     1. Review the console for the control plane node to determine if the mechanism is injecting the control plane node Ignition file correctly.
3. Check the availability of the storage device assigned to the control plane node.
4. Verify that the control plane node has been assigned an IP address from the DHCP server.

5. Determine control plane node status.

   1. Query control plane node status:

      $ oc get nodes

   2. If one of the control plane nodes does not reach a Ready status, retrieve a detailed node description:

      $ oc describe node <master_node>

      Note

      It is not possible to run oc commands if an installation issue prevents the OpenShift Container Platform API from running or if the kubelet is not running yet on each node:

6. Determine OpenShift Container Platform SDN status.

   1. Review sdn-controller, sdn, and ovs daemon set status, in the openshift-sdn namespace:

      $ oc get daemonsets -n openshift-sdn

   2. If those resources are listed as Not found, review pods in the openshift-sdn namespace:

      $ oc get pods -n openshift-sdn

   3. Review logs relating to failed OpenShift Container Platform SDN pods in the openshift-sdn namespace:

      $ oc logs <sdn_pod> -n openshift-sdn

7. Determine cluster network configuration status.

   1. Review whether the cluster’s network configuration exists:

      $ oc get network.config.openshift.io cluster -o yaml

   2. If the installer failed to create the network configuration, generate the Kubernetes manifests again and review message output:

      $ ./openshift-install create manifests

   3. Review the pod status in the openshift-network-operator namespace to determine whether the Cluster Network Operator (CNO) is running:

      $ oc get pods -n openshift-network-operator

   4. Gather network Operator pod logs from the openshift-network-operator namespace:

      $ oc logs pod/<network_operator_pod_name> -n openshift-network-operator

8. Monitor kubelet.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node agent activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=master -u kubelet

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

9. Retrieve crio.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node CRI-O container runtime activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=master -u crio

   2. If the API is not functional, review the logs using SSH instead:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u crio.service

10. Collect logs from specific subdirectories under /var/log/ on control plane nodes.

    1. Retrieve a list of logs contained within a /var/log/ subdirectory. The following example lists files in /var/log/openshift-apiserver/ on all control plane nodes:

       $ oc adm node-logs --role=master --path=openshift-apiserver

    2. Inspect a specific log within a /var/log/ subdirectory. The following example outputs /var/log/openshift-apiserver/audit.log contents from all control plane nodes:

       $ oc adm node-logs --role=master --path=openshift-apiserver/audit.log

    3. If the API is not functional, review the logs on each node using SSH instead. The following example tails /var/log/openshift-apiserver/audit.log:

       $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/openshift-apiserver/audit.log

11. Review control plane node container logs using SSH.

    1. List the containers:

       $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl ps -a

    2. Retrieve a container’s logs using crictl:

       $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

12. If you experience control plane node configuration issues, verify that the MCO, MCO endpoint, and DNS record are functioning. The Machine Config Operator (MCO) manages operating system configuration during the installation procedure. Also verify system clock accuracy and certificate validity.

    1. Test whether the MCO endpoint is available. Replace <cluster_name> with appropriate values:

       $ curl https://api-int.<cluster_name>:22623/config/master

    2. If the endpoint is unresponsive, verify load balancer configuration. Ensure that the endpoint is configured to run on port 22623.

    3. Verify that the MCO endpoint’s DNS record is configured and resolves to the load balancer.

       1. Run a DNS lookup for the defined MCO endpoint name:

          $ dig api-int.<cluster_name> @<dns_server>

       2. Run a reverse lookup to the assigned MCO IP address on the load balancer:

          $ dig -x <load_balancer_mco_ip_address> @<dns_server>

    4. Verify that the MCO is functioning from the bootstrap node directly. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

       $ ssh core@<bootstrap_fqdn> curl https://api-int.<cluster_name>:22623/config/master

    5. System clock time must be synchronized between bootstrap, master, and worker nodes. Check each node’s system clock reference time and time synchronization statistics:

       $ ssh core@<node>.<cluster_name>.<base_domain> chronyc tracking

    6. Review certificate validity:

       $ openssl s_client -connect api-int.<cluster_name>:22623 | openssl x509 -noout -text

Procedure

1. Check the status of etcd pods.

   1. Review the status of pods in the openshift-etcd namespace:

      $ oc get pods -n openshift-etcd

   2. Review the status of pods in the openshift-etcd-operator namespace:

      $ oc get pods -n openshift-etcd-operator

2. If any of the pods listed by the previous commands are not showing a Running or a Completed status, gather diagnostic information for the pod.

   1. Review events for the pod:

      $ oc describe pod/<pod_name> -n <namespace>

   2. Inspect the pod’s logs:

      $ oc logs pod/<pod_name> -n <namespace>

   3. If the pod has more than one container, the preceding command will create an error, and the container names will be provided in the error message. Inspect logs for each container:

      $ oc logs pod/<pod_name> -c <container_name> -n <namespace>

3. If the API is not functional, review etcd pod and container logs on each control plane node by using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values.

   1. List etcd pods on each control plane node:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl pods --name=etcd-

   2. For any pods not showing Ready status, inspect pod status in detail. Replace <pod_id> with the pod’s ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspectp <pod_id>

   3. List containers related to a pod:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl ps | grep '<pod_id>'

   4. For any containers not showing Ready status, inspect container status in detail. Replace <container_id> with container IDs listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspect <container_id>

   5. Review the logs for any containers not showing a Ready status. Replace <container_id> with the container IDs listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

4. Validate primary and secondary DNS server connectivity from control plane nodes.

Procedure

1. Verify that the API server’s DNS record directs the kubelet on control plane nodes to https://api-int.<cluster_name>.<base_domain>:6443. Ensure that the record references the load balancer.
2. Ensure that the load balancer’s port 6443 definition references each control plane node.
3. Check that unique control plane node hostnames have been provided by DHCP.

4. Inspect the kubelet.service journald unit logs on each control plane node.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=master -u kubelet

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

5. Check for certificate expiration messages in the control plane node kubelet logs.

   1. Retrieve the log using oc:

      $ oc adm node-logs --role=master -u kubelet | grep -is 'x509: certificate has expired'

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service  | grep -is 'x509: certificate has expired'

Procedure

1. If you have access to the worker node’s console, monitor the console until the node reaches the login prompt. During the installation, Ignition log messages are output to the console.

2. Verify Ignition file configuration.

   • If you are hosting Ignition configuration files by using an HTTP server.

     1. Verify the worker node Ignition file URL. Replace <http_server_fqdn> with HTTP server’s fully qualified domain name:

        $ curl -I http://<http_server_fqdn>:<port>/worker.ign  1

        1

        The -I option returns the header only. If the Ignition file is available on the specified URL, the command returns 200 OK status. If it is not available, the command returns 404 file not found.

     2. To verify that the Ignition file was received by the worker node, query the HTTP server logs on the HTTP host. For example, if you are using an Apache web server to serve Ignition files:

        $ grep -is 'worker.ign' /var/log/httpd/access_log

        If the worker Ignition file is received, the associated HTTP GET log message will include a 200 OK success status, indicating that the request succeeded.

     3. If the Ignition file was not received, check that it exists on the serving host directly. Ensure that the appropriate file and web server permissions are in place.

   • If you are using a cloud provider mechanism to inject Ignition configuration files into hosts as part of their initial deployment.

     1. Review the worker node’s console to determine if the mechanism is injecting the worker node Ignition file correctly.
3. Check the availability of the worker node’s assigned storage device.
4. Verify that the worker node has been assigned an IP address from the DHCP server.

5. Determine worker node status.

   1. Query node status:

      $ oc get nodes

   2. Retrieve a detailed node description for any worker nodes not showing a Ready status:

      $ oc describe node <worker_node>

      Note

      It is not possible to run oc commands if an installation issue prevents the OpenShift Container Platform API from running or if the kubelet is not running yet on each node.

6. Unlike control plane nodes, worker nodes are deployed and scaled using the Machine API Operator. Check the status of the Machine API Operator.

   1. Review Machine API Operator pod status:

      $ oc get pods -n openshift-machine-api

   2. If the Machine API Operator pod does not have a Ready status, detail the pod’s events:

      $ oc describe pod/<machine_api_operator_pod_name> -n openshift-machine-api

   3. Inspect machine-api-operator container logs. The container runs within the machine-api-operator pod:

      $ oc logs pod/<machine_api_operator_pod_name> -n openshift-machine-api -c machine-api-operator

   4. Also inspect kube-rbac-proxy container logs. The container also runs within the machine-api-operator pod:

      $ oc logs pod/<machine_api_operator_pod_name> -n openshift-machine-api -c kube-rbac-proxy

7. Monitor kubelet.service journald unit logs on worker nodes, after they have booted. This provides visibility into worker node agent activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=worker -u kubelet

   2. If the API is not functional, review the logs using SSH instead. Replace <worker-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<worker-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

8. Retrieve crio.service journald unit logs on worker nodes, after they have booted. This provides visibility into worker node CRI-O container runtime activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=worker -u crio

   2. If the API is not functional, review the logs using SSH instead:

      $ ssh core@<worker-node>.<cluster_name>.<base_domain> journalctl -b -f -u crio.service

9. Collect logs from specific subdirectories under /var/log/ on worker nodes.

   1. Retrieve a list of logs contained within a /var/log/ subdirectory. The following example lists files in /var/log/sssd/ on all worker nodes:

      $ oc adm node-logs --role=worker --path=sssd

   2. Inspect a specific log within a /var/log/ subdirectory. The following example outputs /var/log/sssd/audit.log contents from all worker nodes:

      $ oc adm node-logs --role=worker --path=sssd/sssd.log

   3. If the API is not functional, review the logs on each node using SSH instead. The following example tails /var/log/sssd/sssd.log:

      $ ssh core@<worker-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/sssd/sssd.log

10. Review worker node container logs using SSH.

    1. List the containers:

       $ ssh core@<worker-node>.<cluster_name>.<base_domain> sudo crictl ps -a

    2. Retrieve a container’s logs using crictl:

       $ ssh core@<worker-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

11. If you experience worker node configuration issues, verify that the MCO, MCO endpoint, and DNS record are functioning. The Machine Config Operator (MCO) manages operating system configuration during the installation procedure. Also verify system clock accuracy and certificate validity.

    1. Test whether the MCO endpoint is available. Replace <cluster_name> with appropriate values:

       $ curl https://api-int.<cluster_name>:22623/config/worker

    2. If the endpoint is unresponsive, verify load balancer configuration. Ensure that the endpoint is configured to run on port 22623.

    3. Verify that the MCO endpoint’s DNS record is configured and resolves to the load balancer.

       1. Run a DNS lookup for the defined MCO endpoint name:

          $ dig api-int.<cluster_name> @<dns_server>

       2. Run a reverse lookup to the assigned MCO IP address on the load balancer:

          $ dig -x <load_balancer_mco_ip_address> @<dns_server>

    4. Verify that the MCO is functioning from the bootstrap node directly. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

       $ ssh core@<bootstrap_fqdn> curl https://api-int.<cluster_name>:22623/config/worker

    5. System clock time must be synchronized between bootstrap, master, and worker nodes. Check each node’s system clock reference time and time synchronization statistics:

       $ ssh core@<node>.<cluster_name>.<base_domain> chronyc tracking

    6. Review certificate validity:

       $ openssl s_client -connect api-int.<cluster_name>:22623 | openssl x509 -noout -text

Procedure

1. Check that cluster Operators are all available at the end of an installation.

   $ oc get clusteroperators

2. Verify that all of the required certificate signing requests (CSRs) are approved. Some nodes might not move to a Ready status and some cluster Operators might not become available if there are pending CSRs.

   1. Check the status of the CSRs and ensure that you see a client and server request with the Pending or Approved status for each machine that you added to the cluster:

      $ oc get csr

      Example output

      NAME        AGE     REQUESTOR                                                                   CONDITION
      csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
      csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
      csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
      csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
      ...

      1

      A client request CSR.

      2

      A server request CSR.

      In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

   2. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

      Note

      Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager.

      Note

      For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

      • To approve them individually, run the following command for each valid CSR:

        $ oc adm certificate approve <csr_name> 1

        1

        <csr_name> is the name of a CSR from the list of current CSRs.

      • To approve all pending CSRs, run the following command:

        $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

3. View Operator events:

   $ oc describe clusteroperator <operator_name>

4. Review Operator pod status within the Operator’s namespace:

   $ oc get pods -n <operator_namespace>

5. Obtain a detailed description for pods that do not have Running status:

   $ oc describe pod/<operator_pod_name> -n <operator_namespace>

6. Inspect pod logs:

   $ oc logs pod/<operator_pod_name> -n <operator_namespace>

7. When experiencing pod base image related issues, review base image status.

   1. Obtain details of the base image used by a problematic pod:

      $ oc get pod -o "jsonpath={range .status.containerStatuses[*]}{.name}{'\t'}{.state}{'\t'}{.image}{'\n'}{end}" <operator_pod_name> -n <operator_namespace>

   2. List base image release information:

      $ oc adm release info <image_path>:<tag> --commits

Procedure

1. Generate the commands that are required to obtain the installation logs from the bootstrap and control plane machines:

   • If you used installer-provisioned infrastructure, change to the directory that contains the installation program and run the following command:

     $ ./openshift-install gather bootstrap --dir <installation_directory> 1

     1

     installation_directory is the directory you specified when you ran ./openshift-install create cluster. This directory contains the OpenShift Container Platform definition files that the installation program creates.

     For installer-provisioned infrastructure, the installation program stores information about the cluster, so you do not specify the hostnames or IP addresses.

   • If you used infrastructure that you provisioned yourself, change to the directory that contains the installation program and run the following command:

     $ ./openshift-install gather bootstrap --dir <installation_directory> \ 1
         --bootstrap <bootstrap_address> \ 2
         --master <master_1_address> \ 3
         --master <master_2_address> \ 4
         --master <master_3_address>" 5

     1

     For installation_directory, specify the same directory you specified when you ran ./openshift-install create cluster. This directory contains the OpenShift Container Platform definition files that the installation program creates.

     2

     <bootstrap_address> is the fully qualified domain name or IP address of the cluster’s bootstrap machine.

     3 4 5

     For each control plane, or master, machine in your cluster, replace <master_*_address> with its fully qualified domain name or IP address.

     Note

     A default cluster contains three control plane machines. List all of your control plane machines as shown, no matter how many your cluster uses.

   Example output

   INFO Pulling debug logs from the bootstrap machine
   INFO Bootstrap gather logs captured here "<installation_directory>/log-bundle-<timestamp>.tar.gz"

   If you open a Red Hat support case about your installation failure, include the compressed logs in the case.

Procedure

1. The kubelet is managed using a systemd service on each node. Review the kubelet’s status by querying the kubelet systemd service within a debug pod.

   1. Start a debug pod for a node:

      $ oc debug node/my-node

      Note

      If you are running oc debug on a control plane node, you can find administrative kubeconfig files in the /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs directory.

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Check whether the kubelet systemd service is active on the node:

      # systemctl is-active kubelet

   4. Output a more detailed kubelet.service status summary:

      # systemctl status kubelet

Procedure

1. Query kubelet journald unit logs from OpenShift Container Platform cluster nodes. The following example queries control plane nodes only:

   $ oc adm node-logs --role=master -u kubelet  1

   1

   Replace kubelet as appropriate to query other unit logs.

2. Collect logs from specific subdirectories under /var/log/ on cluster nodes.

   1. Retrieve a list of logs contained within a /var/log/ subdirectory. The following example lists files in /var/log/openshift-apiserver/ on all control plane nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver

   2. Inspect a specific log within a /var/log/ subdirectory. The following example outputs /var/log/openshift-apiserver/audit.log contents from all control plane nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver/audit.log

   3. If the API is not functional, review the logs on each node using SSH instead. The following example tails /var/log/openshift-apiserver/audit.log:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/openshift-apiserver/audit.log

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

Procedure

1. Review CRI-O status by querying the crio systemd service on a node, within a debug pod.

   1. Start a debug pod for a node:

      $ oc debug node/my-node

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Check whether the crio systemd service is active on the node:

      # systemctl is-active crio

   4. Output a more detailed crio.service status summary:

      # systemctl status crio.service

Procedure

1. Gather CRI-O journald unit logs. The following example collects logs from all control plane nodes (within the cluster:

   $ oc adm node-logs --role=master -u crio

2. Gather CRI-O journald unit logs from a specific node:

   $ oc adm node-logs <node_name> -u crio

3. If the API is not functional, review the logs using SSH instead. Replace <node>.<cluster_name>.<base_domain> with appropriate values:

   $ ssh core@<node>.<cluster_name>.<base_domain> journalctl -b -f -u crio.service

   Note

   OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

Procedure

1. Use cordon on the node. This is to avoid any workload getting scheduled if the node gets into the Ready status. You will know that scheduling is disabled when SchedulingDisabled is in your Status section:

   $ oc adm cordon <node_name>

2. Drain the node as the cluster-admin user:

   $ oc adm drain <node_name> --ignore-daemonsets --delete-emptydir-data

   Note

   The terminationGracePeriodSeconds attribute of a pod or pod template controls the graceful termination period. This attribute defaults at 30 seconds, but can be customized for each application as necessary. If set to more than 90 seconds, the pod might be marked as SIGKILLed and fail to terminate successfully.

3. When the node returns, connect back to the node via SSH or Console. Then connect to the root user:

   $ ssh core@node1.example.com
   $ sudo -i

4. Manually stop the kubelet:

   # systemctl stop kubelet

5. Stop the containers and pods:

   1. Use the following command to stop the pods that are not in the HostNetwork. They must be removed first because their removal relies on the networking plugin pods, which are in the HostNetwork.

      .. for pod in $(crictl pods -q); do if [[ "$(crictl inspectp $pod | jq -r .status.linux.namespaces.options.network)" != "NODE" ]]; then crictl rmp -f $pod; fi; done

   2. Stop all other pods:

      # crictl rmp -fa

6. Manually stop the crio services:

   # systemctl stop crio

7. After you run those commands, you can completely wipe the ephemeral storage:

   # crio wipe -f

8. Start the crio and kubelet service:

   # systemctl start crio
   # systemctl start kubelet

9. You will know if the clean up worked if the crio and kubelet services are started, and the node is in the Ready status:

   $ oc get nodes

   Example output

   NAME				    STATUS	                ROLES    AGE    VERSION
   ci-ln-tkbxyft-f76d1-nvwhr-master-1  Ready, SchedulingDisabled   master	 133m   v1.26.0

10. Mark the node schedulable. You will know that the scheduling is enabled when SchedulingDisabled is no longer in status:

    $ oc adm uncordon <node_name>

    Example output

    NAME				     STATUS	      ROLES    AGE    VERSION
    ci-ln-tkbxyft-f76d1-nvwhr-master-1   Ready            master   133m   v1.26.0

Procedure

1. Run the following command to show which service units failed:

   $ systemctl --failed

2. Optional: Run the following command on an individual service unit to find out more information:

   $ journalctl -u <unit>.service

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     55m
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-marketplace-57p8c                1/1     Running            0          36m
   redhat-operators-smxx8                  1/1     Running            0          36m

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. List Operators running in the cluster. The output includes Operator version, availability, and up-time information:

   $ oc get clusteroperators

2. List Operator pods running in the Operator’s namespace, plus pod status, restarts, and age:

   $ oc get pod -n <operator_namespace>

3. Output a detailed Operator pod summary:

   $ oc describe pod <operator_pod_name> -n <operator_namespace>

4. If an Operator issue is node-specific, query Operator container status on that node.

   1. Start a debug pod for the node:

      $ oc debug node/my-node

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. List details about the node’s containers, including state and associated pod IDs:

      # crictl ps

   4. List information about a specific Operator container on the node. The following example lists information about the network-operator container:

      # crictl ps --name network-operator

   5. Exit from the debug shell.

Procedure

1. List the Operator pods that are running in the Operator’s namespace, plus the pod status, restarts, and age:

   $ oc get pods -n <operator_namespace>

2. Review logs for an Operator pod:

   $ oc logs pod/<pod_name> -n <operator_namespace>

   If an Operator pod has multiple containers, the preceding command will produce an error that includes the name of each container. Query logs from an individual container:

   $ oc logs pod/<operator_pod_name> -c <container_name> -n <operator_namespace>

3. If the API is not functional, review Operator pod and container logs on each control plane node by using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values.

   1. List pods on each control plane node:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl pods

   2. For any Operator pods not showing a Ready status, inspect the pod’s status in detail. Replace <operator_pod_id> with the Operator pod’s ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspectp <operator_pod_id>

   3. List containers related to an Operator pod:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl ps --pod=<operator_pod_id>

   4. For any Operator container not showing a Ready status, inspect the container’s status in detail. Replace <container_id> with a container ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspect <container_id>

   5. Review the logs for any Operator containers not showing a Ready status. Replace <container_id> with a container ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

Procedure

1. Get the names of the Subscription and ClusterServiceVersion objects from the namespace where the Operator is installed:

   $ oc get sub,csv -n <namespace>

   Example output

   NAME                                                       PACKAGE                  SOURCE             CHANNEL
   subscription.operators.coreos.com/elasticsearch-operator   elasticsearch-operator   redhat-operators   5.0
   
   NAME                                                                         DISPLAY                            VERSION    REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/elasticsearch-operator.5.0.0-65   OpenShift Elasticsearch Operator   5.0.0-65              Succeeded

2. Delete the subscription:

   $ oc delete subscription <subscription_name> -n <namespace>

3. Delete the cluster service version:

   $ oc delete csv <csv_name> -n <namespace>

4. Get the names of any failing jobs and related config maps in the openshift-marketplace namespace:

   $ oc get job,configmap -n openshift-marketplace

   Example output

   NAME                                                                        COMPLETIONS   DURATION   AGE
   job.batch/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   1/1           26s        9m30s
   
   NAME                                                                        DATA   AGE
   configmap/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   3      9m30s

5. Delete the job:

   $ oc delete job <job_name> -n openshift-marketplace

   This ensures pods that try to pull the inaccessible image are not recreated.

6. Delete the config map:

   $ oc delete configmap <configmap_name> -n openshift-marketplace

7. Reinstall the Operator using OperatorHub in the web console.

Procedure

1. Check if there are any namespaces related to the Operator that are stuck in "Terminating" state:

   $ oc get namespaces

   Example output

   operator-ns-1                                       Terminating

2. Check if there are any CRDs related to the Operator that are still present after the failed uninstallation:

   $ oc get crds

   Note

   CRDs are global cluster definitions; the actual custom resource (CR) instances related to the CRDs could be in other namespaces or be global cluster instances.

3. If there are any CRDs that you know were provided or managed by the Operator and that should have been deleted after uninstallation, delete the CRD:

   $ oc delete crd <crd_name>

4. Check if there are any remaining CR instances related to the Operator that are still present after uninstallation, and if so, delete the CRs:

   1. The type of CRs to search for can be difficult to determine after uninstallation and can require knowing what CRDs the Operator manages. For example, if you are troubleshooting an uninstallation of the etcd Operator, which provides the EtcdCluster CRD, you can search for remaining EtcdCluster CRs in a namespace:

      $ oc get EtcdCluster -n <namespace_name>

      Alternatively, you can search across all namespaces:

      $ oc get EtcdCluster --all-namespaces

   2. If there are any remaining CRs that should be removed, delete the instances:

      $ oc delete <cr_name> <cr_instance_name> -n <namespace_name>

5. Check that the namespace deletion has successfully resolved:

   $ oc get namespace <namespace_name>

   Important

   If the namespace or other Operator resources are still not uninstalled cleanly, contact Red Hat Support.

6. Reinstall the Operator using OperatorHub in the web console.

Procedure

1. Switch into a project:

   $ oc project <project_name>

2. List pods running within the namespace, as well as pod status, error states, restarts, and age:

   $ oc get pods

3. Determine whether the namespace is managed by a deployment configuration:

   $ oc status

   If the namespace is managed by a deployment configuration, the output includes the deployment configuration name and a base image reference.

4. Inspect the base image referenced in the preceding command’s output:

   $ skopeo inspect docker://<image_reference>

5. If the base image reference is not correct, update the reference in the deployment configuration:

   $ oc edit deployment/my-deployment

6. When deployment configuration changes on exit, the configuration will automatically redeploy. Watch pod status as the deployment progresses, to determine whether the issue has been resolved:

   $ oc get pods -w

7. Review events within the namespace for diagnostic information relating to pod failures:

   $ oc get events

Procedure

1. Query logs for a specific pod:

   $ oc logs <pod_name>

2. Query logs for a specific container within a pod:

   $ oc logs <pod_name> -c <container_name>

   Logs retrieved using the preceding oc logs commands are composed of messages sent to stdout within pods or containers.

3. Inspect logs contained in /var/log/ within a pod.

   1. List log files and subdirectories contained in /var/log within a pod:

      $ oc exec <pod_name>  -- ls -alh /var/log

      Example output

      total 124K
      drwxr-xr-x. 1 root root   33 Aug 11 11:23 .
      drwxr-xr-x. 1 root root   28 Sep  6  2022 ..
      -rw-rw----. 1 root utmp    0 Jul 10 10:31 btmp
      -rw-r--r--. 1 root root  33K Jul 17 10:07 dnf.librepo.log
      -rw-r--r--. 1 root root  69K Jul 17 10:07 dnf.log
      -rw-r--r--. 1 root root 8.8K Jul 17 10:07 dnf.rpm.log
      -rw-r--r--. 1 root root  480 Jul 17 10:07 hawkey.log
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 lastlog
      drwx------. 2 root root   23 Aug 11 11:14 openshift-apiserver
      drwx------. 2 root root    6 Jul 10 10:31 private
      drwxr-xr-x. 1 root root   22 Mar  9 08:05 rhsm
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 wtmp

   2. Query a specific log file contained in /var/log within a pod:

      $ oc exec <pod_name> cat /var/log/<path_to_log>

      Example output

      2023-07-10T10:29:38+0000 INFO --- logging initialized ---
      2023-07-10T10:29:38+0000 DDEBUG timer: config: 13 ms
      2023-07-10T10:29:38+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
      2023-07-10T10:29:38+0000 INFO Updating Subscription Management repositories.
      2023-07-10T10:29:38+0000 INFO Unable to read consumer identity
      2023-07-10T10:29:38+0000 INFO Subscription Manager is operating in container mode.
      2023-07-10T10:29:38+0000 INFO

   3. List log files and subdirectories contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> ls /var/log

   4. Query a specific log file contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> cat /var/log/<path_to_log>

Procedure

1. Switch into the project that contains the pod you would like to access. This is necessary because the oc rsh command does not accept the -n namespace option:

   $ oc project <namespace>

2. Start a remote shell into a pod:

   $ oc rsh <pod_name>  1

   1

   If a pod has multiple containers, oc rsh defaults to the first container unless -c <container_name> is specified.

3. Start a remote shell into a specific container within a pod:

   $ oc rsh -c <container_name> pod/<pod_name>

4. Create a port forwarding session to a port on a pod:

   $ oc port-forward <pod_name> <host_port>:<pod_port>  1

   1

   Enter Ctrl+C to cancel the port forwarding session.

Procedure

1. Start a debug pod with root access, based on a deployment.

   1. Obtain a project’s deployment name:

      $ oc get deployment -n <project_name>

   2. Start a debug pod with root privileges, based on the deployment:

      $ oc debug deployment/my-deployment --as-root -n <project_name>

2. Start a debug pod with root access, based on a deployment configuration.

   1. Obtain a project’s deployment configuration name:

      $ oc get deploymentconfigs -n <project_name>

   2. Start a debug pod with root privileges, based on the deployment configuration:

      $ oc debug deploymentconfig/my-deployment-configuration --as-root -n <project_name>

Procedure

1. Copy a file to a pod:

   $ oc cp <local_path> <pod_name>:/<path> -c <container_name>  1

   1

   The first container in a pod is selected if the -c option is not specified.

2. Copy a file from a pod:

   $ oc cp <pod_name>:/<path>  -c <container_name> <local_path>  1

   1

   The first container in a pod is selected if the -c option is not specified.

   Note

   For oc cp to function, the tar binary must be available within the container.

1. During the build configuration stage, a build pod is used to create an application container image from a base image and application source code.
2. During the deployment configuration stage, a deployment pod is used to deploy application pods from the application container image that was built in the build configuration stage. The deployment pod also deploys other resources such as services and routes. The deployment configuration begins after the build configuration succeeds.
3. After the deployment pod has started the application pods, application failures can occur within the running application pods. For instance, an application might not behave as expected even though the application pods are in a Running state. In this scenario, you can access running application pods to investigate application failures within a pod.

1. Monitor build, deployment, and application pod status
2. Determine the stage of the S2I process where the problem occurred
3. Review logs corresponding to the failed stage

Procedure

1. Watch the pod status throughout the S2I process to determine at which stage a failure occurs:

   $ oc get pods -w  1

   1

   Use -w to monitor pods for changes until you quit the command using Ctrl+C.

2. Review a failed pod’s logs for errors.

   • If the build pod fails, review the build pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-build

     Note

     Alternatively, you can review the build configuration’s logs using oc logs -f bc/<application_name>. The build configuration’s logs include the logs from the build pod.

   • If the deployment pod fails, review the deployment pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-deploy

     Note

     Alternatively, you can review the deployment configuration’s logs using oc logs -f dc/<application_name>. This outputs logs from the deployment pod until the deployment pod completes successfully. The command outputs logs from the application pods if you run it after the deployment pod has completed. After a deployment pod completes, its logs can still be accessed by running oc logs -f pod/<application_name>-<build_number>-deploy.

   • If an application pod fails, or if an application is not behaving as expected within a running application pod, review the application pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-<random_string>

Procedure

1. List events relating to a specific application pod. The following example retrieves events for an application pod named my-app-1-akdlg:

   $ oc describe pod/my-app-1-akdlg

2. Review logs from an application pod:

   $ oc logs -f pod/my-app-1-akdlg

3. Query specific logs within a running application pod. Logs that are sent to stdout are collected by the OpenShift Logging framework and are included in the output of the preceding command. The following query is only required for logs that are not sent to stdout.

   1. If an application log can be accessed without root privileges within a pod, concatenate the log file as follows:

      $ oc exec my-app-1-akdlg -- cat /var/log/my-application.log

   2. If root access is required to view an application log, you can start a debug container with root privileges and then view the log file from within the container. Start the debug container from the project’s DeploymentConfig object. Pod users typically run with non-root privileges, but running troubleshooting pods with temporary root privileges can be useful during issue investigation:

      $ oc debug dc/my-deployment-configuration --as-root -- cat /var/log/my-application.log

      Note

      You can access an interactive shell with root access within the debug pod if you run oc debug dc/<deployment_configuration> --as-root without appending -- <command>.

4. Test application functionality interactively and run diagnostic tools, in an application container with an interactive shell.

   1. Start an interactive shell on the application container:

      $ oc exec -it my-app-1-akdlg /bin/bash

   2. Test application functionality interactively from within the shell. For example, you can run the container’s entry point command and observe the results. Then, test changes from the command line directly, before updating the source code and rebuilding the application container through the S2I process.

   3. Run diagnostic binaries available within the container.

      Note

      Root privileges are required to run some diagnostic binaries. In these situations you can start a debug pod with root access, based on a problematic pod’s DeploymentConfig object, by running oc debug dc/<deployment_configuration> --as-root. Then, you can run diagnostic binaries as root from within the debug pod.

5. If diagnostic binaries are not available within a container, you can run a host’s diagnostic binaries within a container’s namespace by using nsenter. The following example runs ip ad within a container’s namespace, using the host`s ip binary.

   1. Enter into a debug session on the target node. This step instantiates a debug pod called <node_name>-debug:

      $ oc debug node/my-cluster-node

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Note

      OpenShift Container Platform 4.13 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Determine the target container ID:

      # crictl ps

   4. Determine the container’s process ID. In this example, the target container ID is a7fe32346b120:

      # crictl inspect a7fe32346b120 --output yaml | grep 'pid:' | awk '{print $2}'

   5. Run ip ad within the container’s namespace, using the host’s ip binary. This example uses 31150 as the container’s process ID. The nsenter command enters the namespace of a target process and runs a command in its namespace. Because the target process in this example is a container’s process ID, the ip ad command is run in the container’s namespace from the host:

      # nsenter -n -t 31150 -- ip ad

      Note

      Running a host’s diagnostic binaries within a container’s namespace is only possible if you are using a privileged container such as a debug node.

Procedure

1. To view the logs under all directories in C:\var\logs, run the following command:

   $ oc adm node-logs -l kubernetes.io/os=windows --path= \
       /ip-10-0-138-252.us-east-2.compute.internal containers \
       /ip-10-0-138-252.us-east-2.compute.internal hybrid-overlay \
       /ip-10-0-138-252.us-east-2.compute.internal kube-proxy \
       /ip-10-0-138-252.us-east-2.compute.internal kubelet \
       /ip-10-0-138-252.us-east-2.compute.internal pods

2. You can now list files in the directories using the same command and view the individual log files. For example, to view the kubelet logs, run the following command:

   $ oc adm node-logs -l kubernetes.io/os=windows --path=/kubelet/kubelet.log

Procedure

1. SSH into the Windows node and enter PowerShell:

   C:\> powershell

2. View the Docker logs by running the following command:

   C:\> Get-EventLog -LogName Application -Source Docker

Procedure

1. Check that the corresponding labels match in the service and ServiceMonitor resource configurations.

   1. Obtain the label defined in the service. The following example queries the prometheus-example-app service in the ns1 project:

      $ oc -n ns1 get service prometheus-example-app -o yaml

      Example output

        labels:
          app: prometheus-example-app

   2. Check that the matchLabels definition in the ServiceMonitor resource configuration matches the label output in the preceding step. The following example queries the prometheus-example-monitor service monitor in the ns1 project:

      $ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml

      Example output

      apiVersion: v1
      kind: ServiceMonitor
      metadata:
        name: prometheus-example-monitor
        namespace: ns1
      spec:
        endpoints:
        - interval: 30s
          port: web
          scheme: http
        selector:
          matchLabels:
            app: prometheus-example-app

      Note

      You can check service and ServiceMonitor resource labels as a developer with view permissions for the project.

2. Inspect the logs for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. List the pods in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get pods

      Example output

      NAME                                   READY   STATUS    RESTARTS   AGE
      prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
      prometheus-user-workload-0             5/5     Running   1          132m
      prometheus-user-workload-1             5/5     Running   1          132m
      thanos-ruler-user-workload-0           3/3     Running   0          132m
      thanos-ruler-user-workload-1           3/3     Running   0          132m

   2. Obtain the logs from the prometheus-operator container in the prometheus-operator pod. In the following example, the pod is called prometheus-operator-776fcbbd56-2nbfm:

      $ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator

      If there is a issue with the service monitor, the logs might include an error similar to this example:

      level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload

3. Review the target status for your endpoint on the Metrics targets page in the OpenShift Container Platform web console UI.

   1. Log in to the OpenShift Container Platform web console and navigate to Observe Targets in the Administrator perspective.
   2. Locate the metrics endpoint in the list, and review the status of the target in the Status column.
   3. If the Status is Down, click the URL for the endpoint to view more information on the Target Details page for that metrics target.

4. Configure debug level logging for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   2. Add logLevel: debug for prometheusOperator under data/config.yaml to set the log level to debug:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheusOperator:
            logLevel: debug
      # ...

   3. Save the file to apply the changes. The affected prometheus-operator pod is automatically redeployed.

   4. Confirm that the debug log-level has been applied to the prometheus-operator deployment in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

      Example output

              - --log-level=debug

      Debug level logging will show all calls made by the Prometheus Operator.

   5. Check that the prometheus-operator pod is running:

      $ oc -n openshift-user-workload-monitoring get pods

      Note

      If an unrecognized Prometheus Operator loglevel value is included in the config map, the prometheus-operator pod might not restart successfully.

   6. Review the debug logs to see if the Prometheus Operator is using the ServiceMonitor resource. Review the logs for other related errors.

Procedure

1. In the Administrator perspective, navigate to Observe Metrics.

2. Enter a Prometheus Query Language (PromQL) query in the Expression field. The following example queries help to identify high cardinality metrics that might result in high disk space consumption:

   • By running the following query, you can identify the ten jobs that have the highest number of scrape samples:

     topk(10, max by(namespace, job) (topk by(namespace, job) (1, scrape_samples_post_metric_relabeling)))

   • By running the following query, you can pinpoint time series churn by identifying the ten jobs that have created the most time series data in the last hour:

     topk(10, sum by(namespace, job) (sum_over_time(scrape_series_added[1h])))

3. Investigate the number of unbound label values assigned to metrics with higher than expected scrape sample counts:

   • If the metrics relate to a user-defined project, review the metrics key-value pairs assigned to your workload. These are implemented through Prometheus client libraries at the application level. Try to limit the number of unbound attributes referenced in your labels.
   • If the metrics relate to a core OpenShift Container Platform project, create a Red Hat support case on the Red Hat Customer Portal.

4. Review the TSDB status using the Prometheus HTTP API by following these steps when logged in as a cluster administrator:

   1. Get the Prometheus API route URL by running the following command:

      $ HOST=$(oc -n openshift-monitoring get route prometheus-k8s -ojsonpath={.status.ingress[].host})

   2. Extract an authentication token by running the following command:

      $ TOKEN=$(oc whoami -t)

   3. Query the TSDB status for Prometheus by running the following command:

      $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/status/tsdb"

      Example output

      "status": "success","data":{"headStats":{"numSeries":507473,
      "numLabelPairs":19832,"chunkCount":946298,"minTime":1712253600010,
      "maxTime":1712257935346},"seriesCountByMetricName":
      [{"name":"etcd_request_duration_seconds_bucket","value":51840},
      {"name":"apiserver_request_sli_duration_seconds_bucket","value":47718},
      ...

Procedure

1. List the size of all TSDB blocks, sorted from oldest to newest, by running the following command:

   $ oc debug <prometheus_k8s_pod_name> -n openshift-monitoring \1
   -c prometheus --image=$(oc get po -n openshift-monitoring <prometheus_k8s_pod_name> \2
   -o jsonpath='{.spec.containers[?(@.name=="prometheus")].image}') \
   -- sh -c 'cd /prometheus/;du -hs $(ls -dt */ | grep -Eo "[0-9|A-Z]{26}")'

   1 2

   Replace <prometheus_k8s_pod_name> with the pod mentioned in the KubePersistentVolumeFillingUp alert description.

   Example output

   308M    01HVKMPKQWZYWS8WVDAYQHNMW6
   52M     01HVK64DTDA81799TBR9QDECEZ
   102M    01HVK64DS7TRZRWF2756KHST5X
   140M    01HVJS59K11FBVAPVY57K88Z11
   90M     01HVH2A5Z58SKT810EM6B9AT50
   152M    01HV8ZDVQMX41MKCN84S32RRZ1
   354M    01HV6Q2N26BK63G4RYTST71FBF
   156M    01HV664H9J9Z1FTZD73RD1563E
   216M    01HTHXB60A7F239HN7S2TENPNS
   104M    01HTHMGRXGS0WXA3WATRXHR36B

2. Identify which and how many blocks could be removed, then remove the blocks. The following example command removes the three oldest Prometheus TSDB blocks from the prometheus-k8s-0 pod:

   $ oc debug prometheus-k8s-0 -n openshift-monitoring \
   -c prometheus --image=$(oc get po -n openshift-monitoring prometheus-k8s-0 \
   -o jsonpath='{.spec.containers[?(@.name=="prometheus")].image}') \
   -- sh -c 'ls -latr /prometheus/ | egrep -o "[0-9|A-Z]{26}" | head -3 | \
   while read BLOCK; do rm -r /prometheus/$BLOCK; done'

3. Verify the usage of the mounted PV and ensure there is enough space available by running the following command:

   $ oc debug <prometheus_k8s_pod_name> -n openshift-monitoring \1
   --image=$(oc get po -n openshift-monitoring <prometheus_k8s_pod_name> \2
   -o jsonpath='{.spec.containers[?(@.name=="prometheus")].image}') -- df -h /prometheus/

   1 2

   Replace <prometheus_k8s_pod_name> with the pod mentioned in the KubePersistentVolumeFillingUp alert description.

   The following example output shows the mounted PV claimed by the prometheus-k8s-0 pod that has 63% of space remaining:

   Example output

   Starting pod/prometheus-k8s-0-debug-j82w4 ...
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme0n1p4  40G   15G  40G  37% /prometheus
   
   Removing debug pod ...