Support

Procedure

1. Find the URL for the Prometheus service that runs in the OpenShift Container Platform cluster:

   $ oc get route prometheus-k8s -n openshift-monitoring -o jsonpath="{.spec.host}"

   Copy to Clipboard Toggle word wrap
2. Navigate to the URL.

3. Enter this query in the Expression input box and press Execute:

   {__name__=~"cluster:usage:.*|count:up0|count:up1|cluster_version|cluster_version_available_updates|cluster_operator_up|cluster_operator_conditions|cluster_version_payload|cluster_installer|cluster_infrastructure_provider|cluster_feature_set|instance:etcd_object_counts:sum|ALERTS|code:apiserver_request_total:rate:sum|cluster:capacity_cpu_cores:sum|cluster:capacity_memory_bytes:sum|cluster:cpu_usage_cores:sum|cluster:memory_usage_bytes:sum|openshift:cpu_usage_cores:sum|openshift:memory_usage_bytes:sum|workload:cpu_usage_cores:sum|workload:memory_usage_bytes:sum|cluster:virt_platform_nodes:sum|cluster:node_instance_type_count:sum|cnv:vmi_status_running:count|node_role_os_version_machine:cpu_capacity_cores:sum|node_role_os_version_machine:cpu_capacity_sockets:sum|subscription_sync_total|csv_succeeded|csv_abnormal|ceph_cluster_total_bytes|ceph_cluster_total_used_raw_bytes|ceph_health_status|job:ceph_osd_metadata:count|job:kube_pv:count|job:ceph_pools_iops:total|job:ceph_pools_iops_bytes:total|job:ceph_versions_running:count|job:noobaa_total_unhealthy_buckets:sum|job:noobaa_bucket_count:sum|job:noobaa_total_object_count:sum|noobaa_accounts_num|noobaa_total_usage|console_url|cluster:network_attachment_definition_instances:max|cluster:network_attachment_definition_enabled_instance_up:max|insightsclient_request_send_total|cam_app_workload_migrations|cluster:apiserver_current_inflight_requests:sum:max_over_time:2m|cluster:telemetry_selected_series:count",alertstate=~"firing|"}

   Copy to Clipboard Toggle word wrap

   This query replicates the request that Telemetry makes against a running OpenShift Container Platform cluster’s Prometheus service and returns the full set of time series captured by Telemetry.

Procedure

1. Find the name of the currently running pod for the Insights Operator:

   $ INSIGHTS_OPERATOR_POD=$(oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running)

   Copy to Clipboard Toggle word wrap

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/$INSIGHTS_OPERATOR_POD:/var/lib/insights-operator ./insights-data

   Copy to Clipboard Toggle word wrap

Procedure

1. Download the global cluster pull secret to your local file system.

   $ oc extract secret/pull-secret -n openshift-config --to=.

   Copy to Clipboard Toggle word wrap
2. In a text editor, edit the .dockerconfigjson file that was downloaded.

3. Remove the cloud.openshift.com JSON entry, for example:

   "cloud.openshift.com":{"auth":"<hash>","email":"<email_address>"}

   Copy to Clipboard Toggle word wrap
4. Save the file.

Procedure

1. Optional: To append a new pull secret to the existing pull secret, complete the following steps:

   1. Enter the following command to download the pull secret:

      $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> 

      1

      Copy to Clipboard Toggle word wrap

      1

      Provide the path to the pull secret file.

   2. Enter the following command to add the new pull secret:

      $ oc registry login --registry="<registry>" \ 

      1

      
      --auth-basic="<username>:<password>" \ 

      2

      
      --to=<pull_secret_location> 

      3

      Copy to Clipboard Toggle word wrap

      1

      Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>".

      2

      Provide the credentials of the new registry.

      3

      Provide the path to the pull secret file.

      Alternatively, you can perform a manual update to the pull secret file.

2. Enter the following command to update the global pull secret for your cluster:

   $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Provide the path to the new pull secret file.

Procedure

1. Click the Clusters menu in the left pane.
2. Click the cluster’s name to display the details of the cluster.

3. Open the Insights tab of the cluster.

   Depending on the result, the tab displays one of the following:

   • Your cluster passed all health checks, if Insights did not identify any issues.
   • A list of issues Insights has detected, prioritized by risk (low, moderate, important, and critical).
   • No health checks to display, if Insights has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed and connected to the internet.

4. If any issues are displayed on the tab, click the > icon in front of the entry for further details.

   Depending on the issue, the details can also contain a link to an Red Hat Knowledge Base article. For details and information on how to solve the problem, click How to remediate this issue.

Procedure

1. Find the name of the running pod for the Insights Operator:

   $ oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running

   Copy to Clipboard Toggle word wrap

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/<insights_operator_pod_name>:/var/lib/insights-operator ./insights-data 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <insights_operator_pod_name> with the pod name output from the preceding command.

Procedure

1. From your archive, open /insights-operator/gathers.json.

   The file contains a list of Insights Operator gather operations:

       {
         "name": "clusterconfig/authentication",
         "duration_in_ms": 730, 

   1

   
         "records_count": 1,
         "errors": null,
         "panic": null
       }

   Copy to Clipboard Toggle word wrap

   1

   duration_in_ms is the amount of time in milliseconds for each gather operation.

2. Inspect each gather operation for abnormalities.

Procedure

1. Find the name of the Insights Operator pod that is currently running:

   $ INSIGHTS_OPERATOR_POD=$(oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running)

   Copy to Clipboard Toggle word wrap

2. Copy the recent data archives from the Insights Operator container:

   $ oc cp openshift-insights/$INSIGHTS_OPERATOR_POD:/var/lib/insights-operator ./insights-data

   Copy to Clipboard Toggle word wrap

Procedure

1. Download the dockerconfig.json file:

   $ oc extract secret/pull-secret -n openshift-config --to=.

   Copy to Clipboard Toggle word wrap

2. Copy your "cloud.openshift.com" "auth" token from the dockerconfig.json file:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "<your_token>",
         "email": "asd@redhat.com"
       }
   }

   Copy to Clipboard Toggle word wrap

3. Upload the archive to console.redhat.com:

   $ curl -v -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/<cluster_id>" -H "Authorization: Bearer <your_token>" -F "upload=@<path_to_archive>; type=application/vnd.redhat.openshift.periodic+tar" https://console.redhat.com/api/ingress/v1/upload

   Copy to Clipboard Toggle word wrap

   where <cluster_id> is your cluster ID, <your_token> is the token from your pull secret, and <path_to_archive> is the path to the Insights Operator archive.

   If the operation is successful, the command returns a "request_id" and "account_number":

   Example output

   * Connection #0 to host console.redhat.com left intact
   {"request_id":"393a7cf1093e434ea8dd4ab3eb28884c","upload":{"account_number":"6274079"}}%

   Copy to Clipboard Toggle word wrap

Verification steps

1. Log in to https://console.redhat.com/openshift.
2. Click the Clusters menu in the left pane.
3. To display the details of the cluster, click the cluster name.

4. Open the Insights Advisor tab of the cluster.

   If the upload was successful, the tab displays one of the following:

   • Your cluster passed all recommendations, if Insights Advisor did not identify any issues.
   • A list of issues that Insights Advisor has detected, prioritized by risk (low, moderate, important, and critical).

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command:

   $ oc adm must-gather

   Copy to Clipboard Toggle word wrap
   Note

   If this command fails, for example if you cannot schedule a pod on your cluster, then use the oc adm inspect command to gather information for particular resources. Contact Red Hat Support for the recommended resources to gather.

   Note

   If your cluster is using a restricted network, you must take additional steps. If your mirror registry has a trusted CA, you must first add the trusted CA to the cluster. For all clusters on restricted networks, you must import the default must-gather image as an image stream before you use the oc adm must-gather command.

   $ oc import-image is/must-gather -n openshift

   Copy to Clipboard Toggle word wrap

3. Create a compressed file from the must-gather directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 

   1

   Copy to Clipboard Toggle word wrap

   1

   Make sure to replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command with one or more --image or --image-stream arguments. For example, the following command gathers both the default cluster data and information specific to OpenShift Virtualization:

   $ oc adm must-gather \
    --image-stream=openshift/must-gather \ 

   1

   
    --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel8:v2.5.8 

   2

   Copy to Clipboard Toggle word wrap

   1

   The default OpenShift Container Platform must-gather image

   2

   The must-gather image for OpenShift Virtualization

   You can use the must-gather tool with additional arguments to gather data that is specifically related to cluster logging and the Cluster Logging Operator in your cluster. For cluster logging, run the following command:

   $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator \
    -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

   Copy to Clipboard Toggle word wrap

   Example 5.1. Example must-gather output for cluster logging

   ├── cluster-logging
   │  ├── clo
   │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
   │  │  ├── clusterlogforwarder_cr
   │  │  ├── cr
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── logforwarding_cr
   │  ├── collector
   │  │  ├── fluentd-2tr64
   │  ├── curator
   │  │  └── curator-1596028500-zkz4s
   │  ├── eo
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── elasticsearch-operator-7dc7d97b9d-jb4r4
   │  ├── es
   │  │  ├── cluster-elasticsearch
   │  │  │  ├── aliases
   │  │  │  ├── health
   │  │  │  ├── indices
   │  │  │  ├── latest_documents.json
   │  │  │  ├── nodes
   │  │  │  ├── nodes_stats.json
   │  │  │  └── thread_pool
   │  │  ├── cr
   │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  │  └── logs
   │  │     ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  ├── install
   │  │  ├── co_logs
   │  │  ├── install_plan
   │  │  ├── olmo_logs
   │  │  └── subscription
   │  └── kibana
   │     ├── cr
   │     ├── kibana-9d69668d4-2rkvz
   ├── cluster-scoped-resources
   │  └── core
   │     ├── nodes
   │     │  ├── ip-10-0-146-180.eu-west-1.compute.internal.yaml
   │     └── persistentvolumes
   │        ├── pvc-0a8d65d9-54aa-4c44-9ecc-33d9381e41c1.yaml
   ├── event-filter.html
   ├── gather-debug.log
   └── namespaces
      ├── openshift-logging
      │  ├── apps
      │  │  ├── daemonsets.yaml
      │  │  ├── deployments.yaml
      │  │  ├── replicasets.yaml
      │  │  └── statefulsets.yaml
      │  ├── batch
      │  │  ├── cronjobs.yaml
      │  │  └── jobs.yaml
      │  ├── core
      │  │  ├── configmaps.yaml
      │  │  ├── endpoints.yaml
      │  │  ├── events
      │  │  │  ├── curator-1596021300-wn2ks.162634ebf0055a94.yaml
      │  │  │  ├── curator.162638330681bee2.yaml
      │  │  │  ├── elasticsearch-im-app-1596020400-gm6nl.1626341a296c16a1.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-9l9n4.1626341a2af81bbd.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-v98tk.1626341a2d821069.yaml
      │  │  │  ├── elasticsearch-im-app-1596020400-cc5vc.1626341a3019b238.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-s8d5s.1626341a31f7b315.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-7mgv8.1626341a35ea59ed.yaml
      │  │  ├── events.yaml
      │  │  ├── persistentvolumeclaims.yaml
      │  │  ├── pods.yaml
      │  │  ├── replicationcontrollers.yaml
      │  │  ├── secrets.yaml
      │  │  └── services.yaml
      │  ├── openshift-logging.yaml
      │  ├── pods
      │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
      │  │  │  ├── cluster-logging-operator
      │  │  │  │  └── cluster-logging-operator
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  └── cluster-logging-operator-74dd5994f-6ttgt.yaml
      │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff
      │  │  │  ├── cluster-logging-operator-registry
      │  │  │  │  └── cluster-logging-operator-registry
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff.yaml
      │  │  │  └── mutate-csv-and-generate-sqlite-db
      │  │  │     └── mutate-csv-and-generate-sqlite-db
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── curator-1596028500-zkz4s
      │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
      │  │  ├── elasticsearch-im-app-1596030300-bpgcx
      │  │  │  ├── elasticsearch-im-app-1596030300-bpgcx.yaml
      │  │  │  └── indexmanagement
      │  │  │     └── indexmanagement
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── fluentd-2tr64
      │  │  │  ├── fluentd
      │  │  │  │  └── fluentd
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── fluentd-2tr64.yaml
      │  │  │  └── fluentd-init
      │  │  │     └── fluentd-init
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── kibana-9d69668d4-2rkvz
      │  │  │  ├── kibana
      │  │  │  │  └── kibana
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── kibana-9d69668d4-2rkvz.yaml
      │  │  │  └── kibana-proxy
      │  │  │     └── kibana-proxy
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  └── route.openshift.io
      │     └── routes.yaml
      └── openshift-operators-redhat
         ├── ...

   Copy to Clipboard Toggle word wrap

3. Create a compressed file from the must-gather directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 

   1

   Copy to Clipboard Toggle word wrap

   1

   Make sure to replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Procedure

1. Run the oc adm must-gather command with the -- /usr/bin/gather_audit_logs flag:

   $ oc adm must-gather -- /usr/bin/gather_audit_logs

   Copy to Clipboard Toggle word wrap

2. Create a compressed file from the must-gather directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.472290403699006248 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace must-gather-local.472290403699006248 with the actual directory name.

3. Attach the compressed file to your support case on the Red Hat Customer Portal.

1. Ignition configuration files are created.
2. The bootstrap machine boots and starts hosting the remote resources required for the control plane machines (also known as the master machines) to boot.
3. The control plane machines fetch the remote resources from the bootstrap machine and finish booting.
4. The control plane machines use the bootstrap machine to form an etcd cluster.
5. The bootstrap machine starts a temporary Kubernetes control plane using the new etcd cluster.
6. The temporary control plane schedules the production control plane to the control plane machines.
7. The temporary control plane shuts down and passes control to the production control plane.
8. The bootstrap machine adds OpenShift Container Platform components into the production control plane.
9. The installation program shuts down the bootstrap machine.
10. The control plane sets up the worker nodes.
11. The control plane installs additional services in the form of a set of Operators.
12. The cluster downloads and configures remaining components needed for the day-to-day operation, including the creation of worker machines in supported environments.

Procedure

1. Check that the haproxy systemd service is active:

   $ ssh <user_name>@<load_balancer> systemctl status haproxy

   Copy to Clipboard Toggle word wrap

2. Verify that the load balancer is listening on the required ports. The following example references ports 80, 443, 6443, and 22623.

   • For HAProxy instances running on Red Hat Enterprise Linux (RHEL) 6, verify port status by using the netstat command:

     $ ssh <user_name>@<load_balancer> netstat -nltupe | grep -E ':80|:443|:6443|:22623'

     Copy to Clipboard Toggle word wrap

   • For HAProxy instances running on Red Hat Enterprise Linux (RHEL) 7 or 8, verify port status by using the ss command:

     $ ssh <user_name>@<load_balancer> ss -nltupe | grep -E ':80|:443|:6443|:22623'

     Copy to Clipboard Toggle word wrap
     Note

     Red Hat recommends the ss command instead of netstat in Red Hat Enterprise Linux (RHEL) 7 or later. ss is provided by the iproute package. For more information on the ss command, see the Red Hat Enterprise Linux (RHEL) 7 Performance Tuning Guide.

3. Check that the wildcard DNS record resolves to the load balancer:

   $ dig <wildcard_fqdn> @<dns_server>

   Copy to Clipboard Toggle word wrap

Procedure

1. Watch the installation log as the installation progresses:

   $ tail -f ~/<installation_directory>/.openshift_install.log

   Copy to Clipboard Toggle word wrap

2. Monitor the bootkube.service journald unit log on the bootstrap node, after it has booted. This provides visibility into the bootstrapping of the first control plane. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

   $ ssh core@<bootstrap_fqdn> journalctl -b -f -u bootkube.service

   Copy to Clipboard Toggle word wrap
   Note

   The bootkube.service log on the bootstrap node outputs etcd connection refused errors, indicating that the bootstrap server is unable to connect to etcd on control plane nodes. After etcd has started on each control plane node and the nodes have joined the cluster, the errors should stop.

3. Monitor kubelet.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node agent activity.

   1. Monitor the logs using oc:

      $ oc adm node-logs --role=master -u kubelet

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

      Copy to Clipboard Toggle word wrap

4. Monitor crio.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node CRI-O container runtime activity.

   1. Monitor the logs using oc:

      $ oc adm node-logs --role=master -u crio

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@master-N.cluster_name.sub_domain.domain journalctl -b -f -u crio.service

      Copy to Clipboard Toggle word wrap

Procedure

1. If you have access to the bootstrap node’s console, monitor the console until the node reaches the login prompt.

2. Verify the Ignition file configuration.

   • If you are hosting Ignition configuration files by using an HTTP server.

     1. Verify the bootstrap node Ignition file URL. Replace <http_server_fqdn> with HTTP server’s fully qualified domain name:

        $ curl -I http://<http_server_fqdn>:<port>/bootstrap.ign  

        1

        Copy to Clipboard Toggle word wrap

        1

        The -I option returns the header only. If the Ignition file is available on the specified URL, the command returns 200 OK status. If it is not available, the command returns 404 file not found.

     2. To verify that the Ignition file was received by the bootstrap node, query the HTTP server logs on the serving host. For example, if you are using an Apache web server to serve Ignition files, enter the following command:

        $ grep -is 'bootstrap.ign' /var/log/httpd/access_log

        Copy to Clipboard Toggle word wrap

        If the bootstrap Ignition file is received, the associated HTTP GET log message will include a 200 OK success status, indicating that the request succeeded.

     3. If the Ignition file was not received, check that the Ignition files exist and that they have the appropriate file and web server permissions on the serving host directly.

   • If you are using a cloud provider mechanism to inject Ignition configuration files into hosts as part of their initial deployment.

     1. Review the bootstrap node’s console to determine if the mechanism is injecting the bootstrap node Ignition file correctly.
3. Verify the availability of the bootstrap node’s assigned storage device.
4. Verify that the bootstrap node has been assigned an IP address from the DHCP server.

5. Collect bootkube.service journald unit logs from the bootstrap node. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

   $ ssh core@<bootstrap_fqdn> journalctl -b -f -u bootkube.service

   Copy to Clipboard Toggle word wrap
   Note

   The bootkube.service log on the bootstrap node outputs etcd connection refused errors, indicating that the bootstrap server is unable to connect to etcd on control plane nodes (also known as the master nodes). After etcd has started on each control plane node and the nodes have joined the cluster, the errors should stop.

6. Collect logs from the bootstrap node containers.

   1. Collect the logs using podman on the bootstrap node. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

      $ ssh core@<bootstrap_fqdn> 'for pod in $(sudo podman ps -a -q); do sudo podman logs $pod; done'

      Copy to Clipboard Toggle word wrap

7. If the bootstrap process fails, verify the following.

   • You can resolve api.<cluster_name>.<base_domain> from the installation host.
   • The load balancer proxies port 6443 connections to bootstrap and control plane nodes. Ensure that the proxy configuration meets OpenShift Container Platform installation requirements.

Procedure

1. If you have access to the console for the control plane node, monitor the console until the node reaches the login prompt. During the installation, Ignition log messages are output to the console.

2. Verify Ignition file configuration.

   • If you are hosting Ignition configuration files by using an HTTP server.

     1. Verify the control plane node Ignition file URL. Replace <http_server_fqdn> with HTTP server’s fully qualified domain name:

        $ curl -I http://<http_server_fqdn>:<port>/master.ign  

        1

        Copy to Clipboard Toggle word wrap

        1

        The -I option returns the header only. If the Ignition file is available on the specified URL, the command returns 200 OK status. If it is not available, the command returns 404 file not found.

     2. To verify that the Ignition file was received by the control plane node query the HTTP server logs on the serving host. For example, if you are using an Apache web server to serve Ignition files:

        $ grep -is 'master.ign' /var/log/httpd/access_log

        Copy to Clipboard Toggle word wrap

        If the master Ignition file is received, the associated HTTP GET log message will include a 200 OK success status, indicating that the request succeeded.

     3. If the Ignition file was not received, check that it exists on the serving host directly. Ensure that the appropriate file and web server permissions are in place.

   • If you are using a cloud provider mechanism to inject Ignition configuration files into hosts as part of their initial deployment.

     1. Review the console for the control plane node to determine if the mechanism is injecting the control plane node Ignition file correctly.
3. Check the availability of the storage device assigned to the control plane node.
4. Verify that the control plane node has been assigned an IP address from the DHCP server.

5. Determine control plane node status.

   1. Query control plane node status:

      $ oc get nodes

      Copy to Clipboard Toggle word wrap

   2. If one of the control plane nodes does not reach a Ready status, retrieve a detailed node description:

      $ oc describe node <master_node>

      Copy to Clipboard Toggle word wrap
      Note

      It is not possible to run oc commands if an installation issue prevents the OpenShift Container Platform API from running or if the kubelet is not running yet on each node:

6. Determine OpenShift Container Platform SDN status.

   1. Review sdn-controller, sdn, and ovs daemon set status, in the openshift-sdn namespace:

      $ oc get daemonsets -n openshift-sdn

      Copy to Clipboard Toggle word wrap

   2. If those resources are listed as Not found, review pods in the openshift-sdn namespace:

      $ oc get pods -n openshift-sdn

      Copy to Clipboard Toggle word wrap

   3. Review logs relating to failed OpenShift Container Platform SDN pods in the openshift-sdn namespace:

      $ oc logs <sdn_pod> -n openshift-sdn

      Copy to Clipboard Toggle word wrap

7. Determine cluster network configuration status.

   1. Review whether the cluster’s network configuration exists:

      $ oc get network.config.openshift.io cluster -o yaml

      Copy to Clipboard Toggle word wrap

   2. If the installer failed to create the network configuration, generate the Kubernetes manifests again and review message output:

      $ ./openshift-install create manifests

      Copy to Clipboard Toggle word wrap

   3. Review the pod status in the openshift-network-operator namespace to determine whether the Cluster Network Operator (CNO) is running:

      $ oc get pods -n openshift-network-operator

      Copy to Clipboard Toggle word wrap

   4. Gather network Operator pod logs from the openshift-network-operator namespace:

      $ oc logs pod/<network_operator_pod_name> -n openshift-network-operator

      Copy to Clipboard Toggle word wrap

8. Monitor kubelet.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node agent activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=master -u kubelet

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

      Copy to Clipboard Toggle word wrap
      Note

      OpenShift Container Platform 4.6 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

9. Retrieve crio.service journald unit logs on control plane nodes, after they have booted. This provides visibility into control plane node CRI-O container runtime activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=master -u crio

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u crio.service

      Copy to Clipboard Toggle word wrap

10. Collect logs from specific subdirectories under /var/log/ on control plane nodes.

    1. Retrieve a list of logs contained within a /var/log/ subdirectory. The following example lists files in /var/log/openshift-apiserver/ on all control plane nodes:

       $ oc adm node-logs --role=master --path=openshift-apiserver

       Copy to Clipboard Toggle word wrap

    2. Inspect a specific log within a /var/log/ subdirectory. The following example outputs /var/log/openshift-apiserver/audit.log contents from all control plane nodes:

       $ oc adm node-logs --role=master --path=openshift-apiserver/audit.log

       Copy to Clipboard Toggle word wrap

    3. If the API is not functional, review the logs on each node using SSH instead. The following example tails /var/log/openshift-apiserver/audit.log:

       $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/openshift-apiserver/audit.log

       Copy to Clipboard Toggle word wrap

11. Review control plane node container logs using SSH.

    1. List the containers:

       $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl ps -a

       Copy to Clipboard Toggle word wrap

    2. Retrieve a container’s logs using crictl:

       $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

       Copy to Clipboard Toggle word wrap

12. If you experience control plane node configuration issues, verify that the MCO, MCO endpoint, and DNS record are functioning. The Machine Config Operator (MCO) manages operating system configuration during the installation procedure. Also verify system clock accuracy and certificate validity.

    1. Test whether the MCO endpoint is available. Replace <cluster_name> with appropriate values:

       $ curl https://api-int.<cluster_name>:22623/config/master

       Copy to Clipboard Toggle word wrap
    2. If the endpoint is unresponsive, verify load balancer configuration. Ensure that the endpoint is configured to run on port 22623.

    3. Verify that the MCO endpoint’s DNS record is configured and resolves to the load balancer.

       1. Run a DNS lookup for the defined MCO endpoint name:

          $ dig api-int.<cluster_name> @<dns_server>

          Copy to Clipboard Toggle word wrap

       2. Run a reverse lookup to the assigned MCO IP address on the load balancer:

          $ dig -x <load_balancer_mco_ip_address> @<dns_server>

          Copy to Clipboard Toggle word wrap

    4. Verify that the MCO is functioning from the bootstrap node directly. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

       $ ssh core@<bootstrap_fqdn> curl https://api-int.<cluster_name>:22623/config/master

       Copy to Clipboard Toggle word wrap

    5. System clock time must be synchronized between bootstrap, master, and worker nodes. Check each node’s system clock reference time and time synchronization statistics:

       $ ssh core@<node>.<cluster_name>.<base_domain> chronyc tracking

       Copy to Clipboard Toggle word wrap

    6. Review certificate validity:

       $ openssl s_client -connect api-int.<cluster_name>:22623 | openssl x509 -noout -text

       Copy to Clipboard Toggle word wrap

Procedure

1. Check the status of etcd pods.

   1. Review the status of pods in the openshift-etcd namespace:

      $ oc get pods -n openshift-etcd

      Copy to Clipboard Toggle word wrap

   2. Review the status of pods in the openshift-etcd-operator namespace:

      $ oc get pods -n openshift-etcd-operator

      Copy to Clipboard Toggle word wrap

2. If any of the pods listed by the previous commands are not showing a Running or a Completed status, gather diagnostic information for the pod.

   1. Review events for the pod:

      $ oc describe pod/<pod_name> -n <namespace>

      Copy to Clipboard Toggle word wrap

   2. Inspect the pod’s logs:

      $ oc logs pod/<pod_name> -n <namespace>

      Copy to Clipboard Toggle word wrap

   3. If the pod has more than one container, the preceding command will create an error, and the container names will be provided in the error message. Inspect logs for each container:

      $ oc logs pod/<pod_name> -c <container_name> -n <namespace>

      Copy to Clipboard Toggle word wrap

3. If the API is not functional, review etcd pod and container logs on each control plane node by using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values.

   1. List etcd pods on each control plane node:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl pods --name=etcd-

      Copy to Clipboard Toggle word wrap

   2. For any pods not showing Ready status, inspect pod status in detail. Replace <pod_id> with the pod’s ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspectp <pod_id>

      Copy to Clipboard Toggle word wrap

   3. List containers related to a pod:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl ps | grep '<pod_id>'

      Copy to Clipboard Toggle word wrap

   4. For any containers not showing Ready status, inspect container status in detail. Replace <container_id> with container IDs listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspect <container_id>

      Copy to Clipboard Toggle word wrap

   5. Review the logs for any containers not showing a Ready status. Replace <container_id> with the container IDs listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

      Copy to Clipboard Toggle word wrap
      Note

      OpenShift Container Platform 4.6 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

4. Validate primary and secondary DNS server connectivity from control plane nodes.

Procedure

1. Verify that the API server’s DNS record directs the kubelet on control plane nodes to https://api-int.<cluster_name>.<base_domain>:6443. Ensure that the record references the load balancer.
2. Ensure that the load balancer’s port 6443 definition references each control plane node.
3. Check that unique control plane node hostnames have been provided by DHCP.

4. Inspect the kubelet.service journald unit logs on each control plane node.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=master -u kubelet

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

      Copy to Clipboard Toggle word wrap
      Note

      OpenShift Container Platform 4.6 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

5. Check for certificate expiration messages in the control plane node kubelet logs.

   1. Retrieve the log using oc:

      $ oc adm node-logs --role=master -u kubelet | grep -is 'x509: certificate has expired'

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service  | grep -is 'x509: certificate has expired'

      Copy to Clipboard Toggle word wrap

Procedure

1. If you have access to the worker node’s console, monitor the console until the node reaches the login prompt. During the installation, Ignition log messages are output to the console.

2. Verify Ignition file configuration.

   • If you are hosting Ignition configuration files by using an HTTP server.

     1. Verify the worker node Ignition file URL. Replace <http_server_fqdn> with HTTP server’s fully qualified domain name:

        $ curl -I http://<http_server_fqdn>:<port>/worker.ign  

        1

        Copy to Clipboard Toggle word wrap

        1

        The -I option returns the header only. If the Ignition file is available on the specified URL, the command returns 200 OK status. If it is not available, the command returns 404 file not found.

     2. To verify that the Ignition file was received by the worker node, query the HTTP server logs on the HTTP host. For example, if you are using an Apache web server to serve Ignition files:

        $ grep -is 'worker.ign' /var/log/httpd/access_log

        Copy to Clipboard Toggle word wrap

        If the worker Ignition file is received, the associated HTTP GET log message will include a 200 OK success status, indicating that the request succeeded.

     3. If the Ignition file was not received, check that it exists on the serving host directly. Ensure that the appropriate file and web server permissions are in place.

   • If you are using a cloud provider mechanism to inject Ignition configuration files into hosts as part of their initial deployment.

     1. Review the worker node’s console to determine if the mechanism is injecting the worker node Ignition file correctly.
3. Check the availability of the worker node’s assigned storage device.
4. Verify that the worker node has been assigned an IP address from the DHCP server.

5. Determine worker node status.

   1. Query node status:

      $ oc get nodes

      Copy to Clipboard Toggle word wrap

   2. Retrieve a detailed node description for any worker nodes not showing a Ready status:

      $ oc describe node <worker_node>

      Copy to Clipboard Toggle word wrap
      Note

      It is not possible to run oc commands if an installation issue prevents the OpenShift Container Platform API from running or if the kubelet is not running yet on each node.

6. Unlike control plane nodes (also known as the master nodes), worker nodes are deployed and scaled using the Machine API Operator. Check the status of the Machine API Operator.

   1. Review Machine API Operator pod status:

      $ oc get pods -n openshift-machine-api

      Copy to Clipboard Toggle word wrap

   2. If the Machine API Operator pod does not have a Ready status, detail the pod’s events:

      $ oc describe pod/<machine_api_operator_pod_name> -n openshift-machine-api

      Copy to Clipboard Toggle word wrap

   3. Inspect machine-api-operator container logs. The container runs within the machine-api-operator pod:

      $ oc logs pod/<machine_api_operator_pod_name> -n openshift-machine-api -c machine-api-operator

      Copy to Clipboard Toggle word wrap

   4. Also inspect kube-rbac-proxy container logs. The container also runs within the machine-api-operator pod:

      $ oc logs pod/<machine_api_operator_pod_name> -n openshift-machine-api -c kube-rbac-proxy

      Copy to Clipboard Toggle word wrap

7. Monitor kubelet.service journald unit logs on worker nodes, after they have booted. This provides visibility into worker node agent activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=worker -u kubelet

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead. Replace <worker-node>.<cluster_name>.<base_domain> with appropriate values:

      $ ssh core@<worker-node>.<cluster_name>.<base_domain> journalctl -b -f -u kubelet.service

      Copy to Clipboard Toggle word wrap
      Note

      OpenShift Container Platform 4.6 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

8. Retrieve crio.service journald unit logs on worker nodes, after they have booted. This provides visibility into worker node CRI-O container runtime activity.

   1. Retrieve the logs using oc:

      $ oc adm node-logs --role=worker -u crio

      Copy to Clipboard Toggle word wrap

   2. If the API is not functional, review the logs using SSH instead:

      $ ssh core@<worker-node>.<cluster_name>.<base_domain> journalctl -b -f -u crio.service

      Copy to Clipboard Toggle word wrap

9. Collect logs from specific subdirectories under /var/log/ on worker nodes.

   1. Retrieve a list of logs contained within a /var/log/ subdirectory. The following example lists files in /var/log/sssd/ on all worker nodes:

      $ oc adm node-logs --role=worker --path=sssd

      Copy to Clipboard Toggle word wrap

   2. Inspect a specific log within a /var/log/ subdirectory. The following example outputs /var/log/sssd/audit.log contents from all worker nodes:

      $ oc adm node-logs --role=worker --path=sssd/sssd.log

      Copy to Clipboard Toggle word wrap

   3. If the API is not functional, review the logs on each node using SSH instead. The following example tails /var/log/sssd/sssd.log:

      $ ssh core@<worker-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/sssd/sssd.log

      Copy to Clipboard Toggle word wrap

10. Review worker node container logs using SSH.

    1. List the containers:

       $ ssh core@<worker-node>.<cluster_name>.<base_domain> sudo crictl ps -a

       Copy to Clipboard Toggle word wrap

    2. Retrieve a container’s logs using crictl:

       $ ssh core@<worker-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

       Copy to Clipboard Toggle word wrap

11. If you experience worker node configuration issues, verify that the MCO, MCO endpoint, and DNS record are functioning. The Machine Config Operator (MCO) manages operating system configuration during the installation procedure. Also verify system clock accuracy and certificate validity.

    1. Test whether the MCO endpoint is available. Replace <cluster_name> with appropriate values:

       $ curl https://api-int.<cluster_name>:22623/config/worker

       Copy to Clipboard Toggle word wrap
    2. If the endpoint is unresponsive, verify load balancer configuration. Ensure that the endpoint is configured to run on port 22623.

    3. Verify that the MCO endpoint’s DNS record is configured and resolves to the load balancer.

       1. Run a DNS lookup for the defined MCO endpoint name:

          $ dig api-int.<cluster_name> @<dns_server>

          Copy to Clipboard Toggle word wrap

       2. Run a reverse lookup to the assigned MCO IP address on the load balancer:

          $ dig -x <load_balancer_mco_ip_address> @<dns_server>

          Copy to Clipboard Toggle word wrap

    4. Verify that the MCO is functioning from the bootstrap node directly. Replace <bootstrap_fqdn> with the bootstrap node’s fully qualified domain name:

       $ ssh core@<bootstrap_fqdn> curl https://api-int.<cluster_name>:22623/config/worker

       Copy to Clipboard Toggle word wrap

    5. System clock time must be synchronized between bootstrap, master, and worker nodes. Check each node’s system clock reference time and time synchronization statistics:

       $ ssh core@<node>.<cluster_name>.<base_domain> chronyc tracking

       Copy to Clipboard Toggle word wrap

    6. Review certificate validity:

       $ openssl s_client -connect api-int.<cluster_name>:22623 | openssl x509 -noout -text

       Copy to Clipboard Toggle word wrap