Support

Procedure

1. Log in to a cluster.
2. Run the following command, which queries a cluster’s Prometheus service and returns the full set of time series data captured by Telemetry:

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

   Depending on the result, the Insights advisor service displays one of the following:

   • No matching recommendations found, if Insights did not identify any issues.
   • A list of issues Insights has detected, grouped by risk (low, moderate, important, and critical).
   • No clusters yet, if Insights has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed, registered, and connected to the internet.

2. If any issues are displayed, click the > icon in front of the entry for more details.

   Depending on the issue, the details can also contain a link to more information from Red Hat about the issue.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

2. Click the X icons next to the Clusters Impacted and Status filters.

   You can now browse through all of the potential recommendations for your cluster.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.
2. Optional: Use the Clusters Impacted and Status filters as needed.

3. Disable an alert by using one of the following methods:

   • To disable an alert:

     1. Click the Options menu for that alert, and then click Disable recommendation.
     2. Enter a justification note and click Save.

   • To view the clusters affected by this alert before disabling the alert:

     1. Click the name of the recommendation to disable. You are directed to the single recommendation page.
     2. Review the list of clusters in the Affected clusters section.
     3. Click Actions → Disable recommendation to disable the alert for all of your clusters.
     4. Enter a justification note and click Save.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

2. Filter the recommendations to display on the disabled recommendations:

   1. From the Status drop-down menu, select Status.
   2. From the Filter by status drop-down menu, select Disabled.
   3. Optional: Clear the Clusters impacted filter.
3. Locate the recommendation to enable.
4. Click the Options menu , and then click Enable recommendation.

Procedure

1. Navigate to Home → Overview in the Red Hat OpenShift Service on AWS web console.

2. Click Insights on the Status card.

   The pop-up window lists potential issues grouped by risk. Click the individual categories or View all recommendations in Insights Advisor to display more details.

Procedure

1. Go to Workloads → ConfigMaps and select Project: openshift-insights.
2. Click the insights-config ConfigMap object to open it.
3. Click Actions and select Edit ConfigMap.
4. Click the YAML view radio button.

5. In the file, set the obfuscation attribute with the workload_names value.

   apiVersion: v1
   kind: ConfigMap
   # ...
   data:
     config.yaml: |
       dataReporting:
         obfuscation:
           - workload_names
   # ...

   Copy to Clipboard Toggle word wrap
6. Click Save. The insights-config config-map details page opens.
7. Verify that the value of the config.yaml obfuscation attribute is set to - workload_names.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

   Note

   If your cluster is in a disconnected environment, you must take additional steps. If your mirror registry has a trusted CA, you must first add the trusted CA to the cluster. For all clusters in disconnected environments, you must import the default must-gather image as an image stream.

   $ oc import-image is/must-gather -n openshift

   Copy to Clipboard Toggle word wrap

2. Run the oc adm must-gather command:

   $ oc adm must-gather

   Copy to Clipboard Toggle word wrap
   Important

   If you are in a disconnected environment, use the --image flag as part of must-gather and point to the payload image.

   Note

   Because this command picks a random control plane node by default, the pod might be scheduled to a control plane node that is in the NotReady and SchedulingDisabled state.

   1. If this command fails, for example, if you cannot schedule a pod on your cluster, then use the oc adm inspect command to gather information for particular resources.

      Note

      Contact Red Hat Support for the recommended resources to gather.

3. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see How to find the cluster-id or name on OpenShift cluster. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather-`date +"%m-%d-%Y-%H-%M-%S"`-<cluster_id>.tar.gz <must_gather_local_dir> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <must_gather_local_dir> with the actual directory name.

4. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command with one or more --image or --image-stream arguments.

   Note
   • To collect the default must-gather data in addition to specific feature data, add the --image-stream=openshift/must-gather argument.
   • For information on gathering data about the Custom Metrics Autoscaler, see the Additional resources section that follows.

   For example, the following command gathers both the default cluster data and information specific to OpenShift Virtualization:

   $ oc adm must-gather \
     --image-stream=openshift/must-gather \ 

   1

   
     --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.19.6 

   2

   Copy to Clipboard Toggle word wrap

   1

   The default Red Hat OpenShift Service on AWS must-gather image

   2

   The must-gather image for OpenShift Virtualization

   You can use the must-gather tool with additional arguments to gather data that is specifically related to OpenShift Logging and the Red Hat OpenShift Logging Operator in your cluster. For OpenShift Logging, run the following command:

   $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator \
     -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

   Copy to Clipboard Toggle word wrap

   Example 6.1. Example must-gather output for OpenShift Logging

   ├── cluster-logging
   │  ├── clo
   │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
   │  │  ├── clusterlogforwarder_cr
   │  │  ├── cr
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── logforwarding_cr
   │  ├── collector
   │  │  ├── fluentd-2tr64
   │  ├── eo
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── elasticsearch-operator-7dc7d97b9d-jb4r4
   │  ├── es
   │  │  ├── cluster-elasticsearch
   │  │  │  ├── aliases
   │  │  │  ├── health
   │  │  │  ├── indices
   │  │  │  ├── latest_documents.json
   │  │  │  ├── nodes
   │  │  │  ├── nodes_stats.json
   │  │  │  └── thread_pool
   │  │  ├── cr
   │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  │  └── logs
   │  │     ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  ├── install
   │  │  ├── co_logs
   │  │  ├── install_plan
   │  │  ├── olmo_logs
   │  │  └── subscription
   │  └── kibana
   │     ├── cr
   │     ├── kibana-9d69668d4-2rkvz
   ├── cluster-scoped-resources
   │  └── core
   │     ├── nodes
   │     │  ├── ip-10-0-146-180.eu-west-1.compute.internal.yaml
   │     └── persistentvolumes
   │        ├── pvc-0a8d65d9-54aa-4c44-9ecc-33d9381e41c1.yaml
   ├── event-filter.html
   ├── gather-debug.log
   └── namespaces
      ├── openshift-logging
      │  ├── apps
      │  │  ├── daemonsets.yaml
      │  │  ├── deployments.yaml
      │  │  ├── replicasets.yaml
      │  │  └── statefulsets.yaml
      │  ├── batch
      │  │  ├── cronjobs.yaml
      │  │  └── jobs.yaml
      │  ├── core
      │  │  ├── configmaps.yaml
      │  │  ├── endpoints.yaml
      │  │  ├── events
      │  │  │  ├── elasticsearch-im-app-1596020400-gm6nl.1626341a296c16a1.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-9l9n4.1626341a2af81bbd.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-v98tk.1626341a2d821069.yaml
      │  │  │  ├── elasticsearch-im-app-1596020400-cc5vc.1626341a3019b238.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-s8d5s.1626341a31f7b315.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-7mgv8.1626341a35ea59ed.yaml
      │  │  ├── events.yaml
      │  │  ├── persistentvolumeclaims.yaml
      │  │  ├── pods.yaml
      │  │  ├── replicationcontrollers.yaml
      │  │  ├── secrets.yaml
      │  │  └── services.yaml
      │  ├── openshift-logging.yaml
      │  ├── pods
      │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
      │  │  │  ├── cluster-logging-operator
      │  │  │  │  └── cluster-logging-operator
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  └── cluster-logging-operator-74dd5994f-6ttgt.yaml
      │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff
      │  │  │  ├── cluster-logging-operator-registry
      │  │  │  │  └── cluster-logging-operator-registry
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff.yaml
      │  │  │  └── mutate-csv-and-generate-sqlite-db
      │  │  │     └── mutate-csv-and-generate-sqlite-db
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
      │  │  ├── elasticsearch-im-app-1596030300-bpgcx
      │  │  │  ├── elasticsearch-im-app-1596030300-bpgcx.yaml
      │  │  │  └── indexmanagement
      │  │  │     └── indexmanagement
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── fluentd-2tr64
      │  │  │  ├── fluentd
      │  │  │  │  └── fluentd
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── fluentd-2tr64.yaml
      │  │  │  └── fluentd-init
      │  │  │     └── fluentd-init
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── kibana-9d69668d4-2rkvz
      │  │  │  ├── kibana
      │  │  │  │  └── kibana
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── kibana-9d69668d4-2rkvz.yaml
      │  │  │  └── kibana-proxy
      │  │  │     └── kibana-proxy
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  └── route.openshift.io
      │     └── routes.yaml
      └── openshift-operators-redhat
         ├── ...

   Copy to Clipboard Toggle word wrap

3. Run the oc adm must-gather command with one or more --image or --image-stream arguments. For example, the following command gathers both the default cluster data and information specific to KubeVirt:

   $ oc adm must-gather \
    --image-stream=openshift/must-gather \ 

   1

   
    --image=quay.io/kubevirt/must-gather 

   2

   Copy to Clipboard Toggle word wrap

   1

   The default Red Hat OpenShift Service on AWS must-gather image

   2

   The must-gather image for KubeVirt

4. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see How to find the cluster-id or name on OpenShift cluster. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather-`date +"%m-%d-%Y-%H-%M-%S"`-<cluster_id>.tar.gz <must_gather_local_dir> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <must_gather_local_dir> with the actual directory name.

5. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Run the oc adm must-gather command with -- gather_network_logs:

   $ oc adm must-gather -- gather_network_logs

   Copy to Clipboard Toggle word wrap
   Note

   By default, the must-gather tool collects the OVN nbdb and sbdb databases from all of the nodes in the cluster. Adding the -- gather_network_logs option to include additional logs that contain OVN-Kubernetes transactions for OVN nbdb database.

2. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see How to find the cluster-id or name on OpenShift cluster. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather-`date +"%m-%d-%Y-%H-%M-%S"`-<cluster_id>.tar.gz <must_gather_local_dir>

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <must_gather_local_dir> with the actual directory name.

3. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

   # chroot /host

   Copy to Clipboard Toggle word wrap

2. Start the toolbox container:

   # toolbox

   Copy to Clipboard Toggle word wrap

3. Install the additional package, such as wget:

   # dnf install -y <package_name>

   Copy to Clipboard Toggle word wrap

Procedure

1. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

   # chroot /host

   Copy to Clipboard Toggle word wrap

2. Optional: If you need to use an alternative image instead of the default image, create a .toolboxrc file in the home directory for the root user ID, and specify the image metadata:

   REGISTRY=quay.io             

   1

   
   IMAGE=fedora/fedora:latest   

   2

   
   TOOLBOX_NAME=toolbox-fedora-latest  

   3

   Copy to Clipboard Toggle word wrap

   1

   Optional: Specify an alternative container registry.

   2

   Specify an alternative image to start.

   3

   Optional: Specify an alternative name for the toolbox container.

3. Start a toolbox container by entering the following command:

   # toolbox

   Copy to Clipboard Toggle word wrap
   Note

   If an existing toolbox pod is already running, the toolbox command outputs 'toolbox-' already exists. Trying to start…​. To avoid issues with sosreport plugins, remove the running toolbox container with podman rm toolbox- and then spawn a new toolbox container.

Procedure

1. Navigate to the Clusters page of the Hybrid Cloud Console.
2. Click the name of your cluster to go to the cluster details page.

3. Click the Cluster history tab.

   Cluster notifications appear under the Cluster history heading.

4. Optional: Filter for relevant cluster notifications

   Use the filter controls to hide cluster notifications that are not relevant to you, so that you can focus on your area of expertise or on resolving a critical issue. You can filter notifications based on text in the notification description, severity level, notification type, when the notification was received, and which system or person triggered the notification.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

   Copy to Clipboard Toggle word wrap

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

   Copy to Clipboard Toggle word wrap

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   Copy to Clipboard Toggle word wrap

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-operators-smxx8                  1/1     Running            0          36m

   Copy to Clipboard Toggle word wrap

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   Copy to Clipboard Toggle word wrap

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. List Operators running in the cluster. The output includes Operator version, availability, and up-time information:

   $ oc get clusteroperators

   Copy to Clipboard Toggle word wrap

2. List Operator pods running in the Operator’s namespace, plus pod status, restarts, and age:

   $ oc get pod -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

3. Output a detailed Operator pod summary:

   $ oc describe pod <operator_pod_name> -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

Procedure

1. Switch into a project:

   $ oc project <project_name>

   Copy to Clipboard Toggle word wrap

2. List pods running within the namespace, as well as pod status, error states, restarts, and age:

   $ oc get pods

   Copy to Clipboard Toggle word wrap

3. Determine whether the namespace is managed by a deployment configuration:

   $ oc status

   Copy to Clipboard Toggle word wrap

   If the namespace is managed by a deployment configuration, the output includes the deployment configuration name and a base image reference.

4. Inspect the base image referenced in the preceding command’s output:

   $ skopeo inspect docker://<image_reference>

   Copy to Clipboard Toggle word wrap

5. If the base image reference is not correct, update the reference in the deployment configuration:

   $ oc edit deployment/my-deployment

   Copy to Clipboard Toggle word wrap

6. When deployment configuration changes on exit, the configuration will automatically redeploy. Watch pod status as the deployment progresses, to determine whether the issue has been resolved:

   $ oc get pods -w

   Copy to Clipboard Toggle word wrap

7. Review events within the namespace for diagnostic information relating to pod failures:

   $ oc get events

   Copy to Clipboard Toggle word wrap

Procedure

1. Query logs for a specific pod:

   $ oc logs <pod_name>

   Copy to Clipboard Toggle word wrap

2. Query logs for a specific container within a pod:

   $ oc logs <pod_name> -c <container_name>

   Copy to Clipboard Toggle word wrap

   Logs retrieved using the preceding oc logs commands are composed of messages sent to stdout within pods or containers.

3. Inspect logs contained in /var/log/ within a pod.

   1. List log files and subdirectories contained in /var/log within a pod:

      $ oc exec <pod_name>  -- ls -alh /var/log

      Copy to Clipboard Toggle word wrap

      Example output

      total 124K
      drwxr-xr-x. 1 root root   33 Aug 11 11:23 .
      drwxr-xr-x. 1 root root   28 Sep  6  2022 ..
      -rw-rw----. 1 root utmp    0 Jul 10 10:31 btmp
      -rw-r--r--. 1 root root  33K Jul 17 10:07 dnf.librepo.log
      -rw-r--r--. 1 root root  69K Jul 17 10:07 dnf.log
      -rw-r--r--. 1 root root 8.8K Jul 17 10:07 dnf.rpm.log
      -rw-r--r--. 1 root root  480 Jul 17 10:07 hawkey.log
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 lastlog
      drwx------. 2 root root   23 Aug 11 11:14 openshift-apiserver
      drwx------. 2 root root    6 Jul 10 10:31 private
      drwxr-xr-x. 1 root root   22 Mar  9 08:05 rhsm
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 wtmp

      Copy to Clipboard Toggle word wrap

   2. Query a specific log file contained in /var/log within a pod:

      $ oc exec <pod_name> cat /var/log/<path_to_log>

      Copy to Clipboard Toggle word wrap

      Example output

      2023-07-10T10:29:38+0000 INFO --- logging initialized ---
      2023-07-10T10:29:38+0000 DDEBUG timer: config: 13 ms
      2023-07-10T10:29:38+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
      2023-07-10T10:29:38+0000 INFO Updating Subscription Management repositories.
      2023-07-10T10:29:38+0000 INFO Unable to read consumer identity
      2023-07-10T10:29:38+0000 INFO Subscription Manager is operating in container mode.
      2023-07-10T10:29:38+0000 INFO

      Copy to Clipboard Toggle word wrap

   3. List log files and subdirectories contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> ls /var/log

      Copy to Clipboard Toggle word wrap

   4. Query a specific log file contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> cat /var/log/<path_to_log>

      Copy to Clipboard Toggle word wrap

Procedure

1. Switch into the project that contains the pod you would like to access. This is necessary because the oc rsh command does not accept the -n namespace option:

   $ oc project <namespace>

   Copy to Clipboard Toggle word wrap

2. Start a remote shell into a pod:

   $ oc rsh <pod_name>  

   1

   Copy to Clipboard Toggle word wrap

   1

   If a pod has multiple containers, oc rsh defaults to the first container unless -c <container_name> is specified.

3. Start a remote shell into a specific container within a pod:

   $ oc rsh -c <container_name> pod/<pod_name>

   Copy to Clipboard Toggle word wrap

4. Create a port forwarding session to a port on a pod:

   $ oc port-forward <pod_name> <host_port>:<pod_port>  

   1

   Copy to Clipboard Toggle word wrap

   1

   Enter Ctrl+C to cancel the port forwarding session.

Procedure

1. Start a debug pod with root access, based on a deployment.

   1. Obtain a project’s deployment name:

      $ oc get deployment -n <project_name>

      Copy to Clipboard Toggle word wrap

   2. Start a debug pod with root privileges, based on the deployment:

      $ oc debug deployment/my-deployment --as-root -n <project_name>

      Copy to Clipboard Toggle word wrap

2. Start a debug pod with root access, based on a deployment configuration.

   1. Obtain a project’s deployment configuration name:

      $ oc get deploymentconfigs -n <project_name>

      Copy to Clipboard Toggle word wrap

   2. Start a debug pod with root privileges, based on the deployment configuration:

      $ oc debug deploymentconfig/my-deployment-configuration --as-root -n <project_name>

      Copy to Clipboard Toggle word wrap

Procedure

1. Copy a file to a pod:

   $ oc cp <local_path> <pod_name>:/<path> -c <container_name>  

   1

   Copy to Clipboard Toggle word wrap

   1

   The first container in a pod is selected if the -c option is not specified.

2. Copy a file from a pod:

   $ oc cp <pod_name>:/<path>  -c <container_name> <local_path>  

   1

   Copy to Clipboard Toggle word wrap

   1

   The first container in a pod is selected if the -c option is not specified.

   Note

   For oc cp to function, the tar binary must be available within the container.

1. During the build configuration stage, a build pod is used to create an application container image from a base image and application source code.
2. During the deployment configuration stage, a deployment pod is used to deploy application pods from the application container image that was built in the build configuration stage. The deployment pod also deploys other resources such as services and routes. The deployment configuration begins after the build configuration succeeds.
3. After the deployment pod has started the application pods, application failures can occur within the running application pods. For instance, an application might not behave as expected even though the application pods are in a Running state. In this scenario, you can access running application pods to investigate application failures within a pod.

1. Monitor build, deployment, and application pod status
2. Determine the stage of the S2I process where the problem occurred
3. Review logs corresponding to the failed stage

Procedure

1. Watch the pod status throughout the S2I process to determine at which stage a failure occurs:

   $ oc get pods -w  

   1

   Copy to Clipboard Toggle word wrap

   1

   Use -w to monitor pods for changes until you quit the command using Ctrl+C.

2. Review a failed pod’s logs for errors.

   • If the build pod fails, review the build pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-build

     Copy to Clipboard Toggle word wrap
     Note

     Alternatively, you can review the build configuration’s logs using oc logs -f bc/<application_name>. The build configuration’s logs include the logs from the build pod.

   • If the deployment pod fails, review the deployment pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-deploy

     Copy to Clipboard Toggle word wrap
     Note

     Alternatively, you can review the deployment configuration’s logs using oc logs -f dc/<application_name>. This outputs logs from the deployment pod until the deployment pod completes successfully. The command outputs logs from the application pods if you run it after the deployment pod has completed. After a deployment pod completes, its logs can still be accessed by running oc logs -f pod/<application_name>-<build_number>-deploy.

   • If an application pod fails, or if an application is not behaving as expected within a running application pod, review the application pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-<random_string>

     Copy to Clipboard Toggle word wrap

Procedure

1. List events relating to a specific application pod. The following example retrieves events for an application pod named my-app-1-akdlg:

   $ oc describe pod/my-app-1-akdlg

   Copy to Clipboard Toggle word wrap

2. Review logs from an application pod:

   $ oc logs -f pod/my-app-1-akdlg

   Copy to Clipboard Toggle word wrap

3. Query specific logs within a running application pod. Logs that are sent to stdout are collected by the OpenShift Logging framework and are included in the output of the preceding command. The following query is only required for logs that are not sent to stdout.

   1. If an application log can be accessed without root privileges within a pod, concatenate the log file as follows:

      $ oc exec my-app-1-akdlg -- cat /var/log/my-application.log

      Copy to Clipboard Toggle word wrap

   2. If root access is required to view an application log, you can start a debug container with root privileges and then view the log file from within the container. Start the debug container from the project’s DeploymentConfig object. Pod users typically run with non-root privileges, but running troubleshooting pods with temporary root privileges can be useful during issue investigation:

      $ oc debug dc/my-deployment-configuration --as-root -- cat /var/log/my-application.log

      Copy to Clipboard Toggle word wrap
      Note

      You can access an interactive shell with root access within the debug pod if you run oc debug dc/<deployment_configuration> --as-root without appending -- <command>.

4. Test application functionality interactively and run diagnostic tools, in an application container with an interactive shell.

   1. Start an interactive shell on the application container:

      $ oc exec -it my-app-1-akdlg /bin/bash

      Copy to Clipboard Toggle word wrap
   2. Test application functionality interactively from within the shell. For example, you can run the container’s entry point command and observe the results. Then, test changes from the command line directly, before updating the source code and rebuilding the application container through the S2I process.

   3. Run diagnostic binaries available within the container.

      Note

      Root privileges are required to run some diagnostic binaries. In these situations you can start a debug pod with root access, based on a problematic pod’s DeploymentConfig object, by running oc debug dc/<deployment_configuration> --as-root. Then, you can run diagnostic binaries as root from within the debug pod.

5. If diagnostic binaries are not available within a container, you can run a host’s diagnostic binaries within a container’s namespace by using nsenter. The following example runs ip ad within a container’s namespace, using the host`s ip binary.

   1. Enter into a debug session on the target node. This step instantiates a debug pod called <node_name>-debug:

      $ oc debug node/my-cluster-node

      Copy to Clipboard Toggle word wrap

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Copy to Clipboard Toggle word wrap
      Note

      Red Hat OpenShift Service on AWS 4 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the Red Hat OpenShift Service on AWS API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Determine the target container ID:

      # crictl ps

      Copy to Clipboard Toggle word wrap

   4. Determine the container’s process ID. In this example, the target container ID is a7fe32346b120:

      # crictl inspect a7fe32346b120 --output yaml | grep 'pid:' | awk '{print $2}'

      Copy to Clipboard Toggle word wrap

   5. Run ip ad within the container’s namespace, using the host’s ip binary. This example uses 31150 as the container’s process ID. The nsenter command enters the namespace of a target process and runs a command in its namespace. Because the target process in this example is a container’s process ID, the ip ad command is run in the container’s namespace from the host:

      # nsenter -n -t 31150 -- ip ad

      Copy to Clipboard Toggle word wrap
      Note

      Running a host’s diagnostic binaries within a container’s namespace is only possible if you are using a privileged container such as a debug node.