Validation and troubleshooting

Procedure

1. Obtain the cluster version and overall status:

   $ oc get clusterversion

   Example output

   NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   version   4.6.4     True        False         6m25s   Cluster version is 4.6.4

   The example output indicates that the cluster has been installed successfully.

2. If the cluster status indicates that the installation is still progressing, you can obtain more detailed progress information by checking the status of the Operators:

   $ oc get clusteroperators.config.openshift.io

3. View a detailed summary of cluster specifications, update availability, and update history:

   $ oc describe clusterversion

4. List the current update channel:

   $ oc get clusterversion -o jsonpath='{.items[0].spec}{"\n"}'

   Example output

   {"channel":"stable-4.6","clusterID":"245539c1-72a3-41aa-9cec-72ed8cf25c5c"}

5. Review the available cluster updates:

   $ oc adm upgrade

   Example output

   Cluster version is 4.6.4
   
   Updates:
   
   VERSION IMAGE
   4.6.6   quay.io/openshift-release-dev/ocp-release@sha256:c7e8f18e8116356701bd23ae3a23fb9892dd5ea66c8300662ef30563d7104f39

Procedure

1. List the status of the cluster nodes. Verify that the output lists all of the expected control plane and compute nodes and that each node has a Ready status:

   $ oc get nodes

   Example output

   NAME                          STATUS   ROLES    AGE   VERSION
   compute-1.example.com         Ready    worker   33m   v1.26.0
   control-plane-1.example.com   Ready    master   41m   v1.26.0
   control-plane-2.example.com   Ready    master   45m   v1.26.0
   compute-2.example.com         Ready    worker   38m   v1.26.0
   compute-3.example.com         Ready    worker   33m   v1.26.0
   control-plane-3.example.com   Ready    master   41m   v1.26.0

2. Review CPU and memory resource availability for each cluster node:

   $ oc adm top nodes

   Example output

   NAME                          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
   compute-1.example.com         128m         8%     1132Mi          16%
   control-plane-1.example.com   801m         22%    3471Mi          23%
   control-plane-2.example.com   1718m        49%    6085Mi          40%
   compute-2.example.com         935m         62%    5178Mi          75%
   compute-3.example.com         111m         7%     1131Mi          16%
   control-plane-3.example.com   942m         26%    4100Mi          27%

Procedure

1. Go to the Clusters list in OpenShift Cluster Manager Hybrid Cloud Console and locate your OpenShift Container Platform cluster.
2. Click the Overview tab for your cluster.

3. Review the following information about your cluster:

   • vCPU and memory availability and resource usage
   • The cluster ID, status, type, region, and the provider name
   • Node counts by node type
   • Cluster version details, the creation date of the cluster, and the name of the cluster owner
   • The life cycle support status of the cluster

   • Subscription information, including the service level agreement (SLA) status, the subscription unit type, the production status of the cluster, the subscription obligation, and the service level

     Tip

     To view the history for your cluster, click the Cluster history tab.

4. Navigate to the Monitoring page to review the following information:

   • A list of any issues that have been detected
   • A list of alerts that are firing
   • The cluster Operator status and version
   • The cluster’s resource usage

5. Optional: You can view information about your cluster that Red Hat Insights collects by navigating to the Overview menu. From this menu you can view the following information:

   • Potential issues that your cluster might be exposed to, categorized by risk level
   • Health-check status by category

Procedure

1. In the Administrator perspective in the OpenShift Container Platform web console, navigate to Observe → Dashboards.
2. Choose a dashboard in the Dashboard list. Some dashboards, such as the etcd dashboard, produce additional sub-menus when selected.

3. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by selecting Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
4. Optional: Select a Refresh Interval.
5. Hover over each of the graphs within a dashboard to display detailed information about specific items.

Procedure

1. In the Administrator perspective, navigate to the Observe → Alerting → Alerts page.
2. Review the alerts that are firing, including their Severity, State, and Source.
3. Select an alert to view more detailed information in the Alert Details page.

Procedure

1. Generate the commands that are required to obtain the installation logs from the bootstrap and control plane machines:

   • If you used installer-provisioned infrastructure, change to the directory that contains the installation program and run the following command:

     $ ./openshift-install gather bootstrap --dir <installation_directory> 1

     1

     installation_directory is the directory you specified when you ran ./openshift-install create cluster. This directory contains the OpenShift Container Platform definition files that the installation program creates.

     For installer-provisioned infrastructure, the installation program stores information about the cluster, so you do not specify the hostnames or IP addresses.

   • If you used infrastructure that you provisioned yourself, change to the directory that contains the installation program and run the following command:

     $ ./openshift-install gather bootstrap --dir <installation_directory> \ 1
         --bootstrap <bootstrap_address> \ 2
         --master <master_1_address> \ 3
         --master <master_2_address> \ 4
         --master <master_3_address>" 5

     1

     For installation_directory, specify the same directory you specified when you ran ./openshift-install create cluster. This directory contains the OpenShift Container Platform definition files that the installation program creates.

     2

     <bootstrap_address> is the fully qualified domain name or IP address of the cluster’s bootstrap machine.

     3 4 5

     For each control plane, or master, machine in your cluster, replace <master_*_address> with its fully qualified domain name or IP address.

     Note

     A default cluster contains three control plane machines. List all of your control plane machines as shown, no matter how many your cluster uses.

   Example output

   INFO Pulling debug logs from the bootstrap machine
   INFO Bootstrap gather logs captured here "<installation_directory>/log-bundle-<timestamp>.tar.gz"

   If you open a Red Hat support case about your installation failure, include the compressed logs in the case.

Procedure

1. Collect the bootkube.service service logs from the bootstrap host using the journalctl command by running:

   $ journalctl -b -f -u bootkube.service

2. Collect the bootstrap host’s container logs using the podman logs. This is shown as a loop to get all of the container logs from the host:

   $ for pod in $(sudo podman ps -a -q); do sudo podman logs $pod; done

3. Alternatively, collect the host’s container logs using the tail command by running:

   # tail -f /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log

4. Collect the kubelet.service and crio.service service logs from the master and worker hosts using the journalctl command by running:

   $ journalctl -b -f -u kubelet.service -u crio.service

5. Collect the master and worker host container logs using the tail command by running:

   $ sudo tail -f /var/log/containers/*

Procedure

1. Access journald unit logs under /var/log by running:

   $ oc adm node-logs --role=master -u kubelet

2. Access host file paths under /var/log by running:

   $ oc adm node-logs --role=master --path=openshift-apiserver

Procedure

1. Destroy the cluster and remove all the resources associated with the cluster, including the hidden installer state files in the installation directory:

   $ ./openshift-install destroy cluster --dir <installation_directory> 1

   1

   installation_directory is the directory you specified when you ran ./openshift-install create cluster. This directory contains the OpenShift Container Platform definition files that the installation program creates.

2. Before reinstalling the cluster, delete the installation directory:

   $ rm -rf <installation_directory>

3. Follow the procedure for installing a new OpenShift Container Platform cluster.