Chapter 8. Troubleshooting

Procedure

1. Navigate to the Clusters page of the Hybrid Cloud Console.
2. Click the name of your cluster to go to the cluster details page.

3. Click the Cluster history tab.

   Cluster notifications appear under the Cluster history heading.

4. Optional: Filter for relevant cluster notifications

   Use the filter controls to hide cluster notifications that are not relevant to you, so that you can focus on your area of expertise or on resolving a critical issue. You can filter notifications based on text in the notification description, severity level, notification type, when the notification was received, and which system or person triggered the notification.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

   Copy to Clipboard Toggle word wrap

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

   Copy to Clipboard Toggle word wrap

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   Copy to Clipboard Toggle word wrap

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-operators-smxx8                  1/1     Running            0          36m

   Copy to Clipboard Toggle word wrap

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   Copy to Clipboard Toggle word wrap

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. List Operators running in the cluster. The output includes Operator version, availability, and up-time information:

   $ oc get clusteroperators

   Copy to Clipboard Toggle word wrap

2. List Operator pods running in the Operator’s namespace, plus pod status, restarts, and age:

   $ oc get pod -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

3. Output a detailed Operator pod summary:

   $ oc describe pod <operator_pod_name> -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

Procedure

1. Switch into a project:

   $ oc project <project_name>

   Copy to Clipboard Toggle word wrap

2. List pods running within the namespace, as well as pod status, error states, restarts, and age:

   $ oc get pods

   Copy to Clipboard Toggle word wrap

3. Determine whether the namespace is managed by a deployment configuration:

   $ oc status

   Copy to Clipboard Toggle word wrap

   If the namespace is managed by a deployment configuration, the output includes the deployment configuration name and a base image reference.

4. Inspect the base image referenced in the preceding command’s output:

   $ skopeo inspect docker://<image_reference>

   Copy to Clipboard Toggle word wrap

5. If the base image reference is not correct, update the reference in the deployment configuration:

   $ oc edit deployment/my-deployment

   Copy to Clipboard Toggle word wrap

6. When deployment configuration changes on exit, the configuration will automatically redeploy. Watch pod status as the deployment progresses, to determine whether the issue has been resolved:

   $ oc get pods -w

   Copy to Clipboard Toggle word wrap

7. Review events within the namespace for diagnostic information relating to pod failures:

   $ oc get events

   Copy to Clipboard Toggle word wrap

Procedure

1. Query logs for a specific pod:

   $ oc logs <pod_name>

   Copy to Clipboard Toggle word wrap

2. Query logs for a specific container within a pod:

   $ oc logs <pod_name> -c <container_name>

   Copy to Clipboard Toggle word wrap

   Logs retrieved using the preceding oc logs commands are composed of messages sent to stdout within pods or containers.

3. Inspect logs contained in /var/log/ within a pod.

   1. List log files and subdirectories contained in /var/log within a pod:

      $ oc exec <pod_name>  -- ls -alh /var/log

      Copy to Clipboard Toggle word wrap

      Example output

      total 124K
      drwxr-xr-x. 1 root root   33 Aug 11 11:23 .
      drwxr-xr-x. 1 root root   28 Sep  6  2022 ..
      -rw-rw----. 1 root utmp    0 Jul 10 10:31 btmp
      -rw-r--r--. 1 root root  33K Jul 17 10:07 dnf.librepo.log
      -rw-r--r--. 1 root root  69K Jul 17 10:07 dnf.log
      -rw-r--r--. 1 root root 8.8K Jul 17 10:07 dnf.rpm.log
      -rw-r--r--. 1 root root  480 Jul 17 10:07 hawkey.log
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 lastlog
      drwx------. 2 root root   23 Aug 11 11:14 openshift-apiserver
      drwx------. 2 root root    6 Jul 10 10:31 private
      drwxr-xr-x. 1 root root   22 Mar  9 08:05 rhsm
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 wtmp

      Copy to Clipboard Toggle word wrap

   2. Query a specific log file contained in /var/log within a pod:

      $ oc exec <pod_name> cat /var/log/<path_to_log>

      Copy to Clipboard Toggle word wrap

      Example output

      2023-07-10T10:29:38+0000 INFO --- logging initialized ---
      2023-07-10T10:29:38+0000 DDEBUG timer: config: 13 ms
      2023-07-10T10:29:38+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
      2023-07-10T10:29:38+0000 INFO Updating Subscription Management repositories.
      2023-07-10T10:29:38+0000 INFO Unable to read consumer identity
      2023-07-10T10:29:38+0000 INFO Subscription Manager is operating in container mode.
      2023-07-10T10:29:38+0000 INFO

      Copy to Clipboard Toggle word wrap

   3. List log files and subdirectories contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> ls /var/log

      Copy to Clipboard Toggle word wrap

   4. Query a specific log file contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> cat /var/log/<path_to_log>

      Copy to Clipboard Toggle word wrap

Procedure

1. Switch into the project that contains the pod you would like to access. This is necessary because the oc rsh command does not accept the -n namespace option:

   $ oc project <namespace>

   Copy to Clipboard Toggle word wrap

2. Start a remote shell into a pod:

   $ oc rsh <pod_name>  

   1

   Copy to Clipboard Toggle word wrap

   1

   If a pod has multiple containers, oc rsh defaults to the first container unless -c <container_name> is specified.

3. Start a remote shell into a specific container within a pod:

   $ oc rsh -c <container_name> pod/<pod_name>

   Copy to Clipboard Toggle word wrap

4. Create a port forwarding session to a port on a pod:

   $ oc port-forward <pod_name> <host_port>:<pod_port>  

   1

   Copy to Clipboard Toggle word wrap

   1

   Enter Ctrl+C to cancel the port forwarding session.

Procedure

1. Start a debug pod with root access, based on a deployment.

   1. Obtain a project’s deployment name:

      $ oc get deployment -n <project_name>

      Copy to Clipboard Toggle word wrap

   2. Start a debug pod with root privileges, based on the deployment:

      $ oc debug deployment/my-deployment --as-root -n <project_name>

      Copy to Clipboard Toggle word wrap

2. Start a debug pod with root access, based on a deployment configuration.

   1. Obtain a project’s deployment configuration name:

      $ oc get deploymentconfigs -n <project_name>

      Copy to Clipboard Toggle word wrap

   2. Start a debug pod with root privileges, based on the deployment configuration:

      $ oc debug deploymentconfig/my-deployment-configuration --as-root -n <project_name>

      Copy to Clipboard Toggle word wrap

Procedure

1. Copy a file to a pod:

   $ oc cp <local_path> <pod_name>:/<path> -c <container_name>  

   1

   Copy to Clipboard Toggle word wrap

   1

   The first container in a pod is selected if the -c option is not specified.

2. Copy a file from a pod:

   $ oc cp <pod_name>:/<path>  -c <container_name> <local_path>  

   1

   Copy to Clipboard Toggle word wrap

   1

   The first container in a pod is selected if the -c option is not specified.

   Note

   For oc cp to function, the tar binary must be available within the container.

1. During the build configuration stage, a build pod is used to create an application container image from a base image and application source code.
2. During the deployment configuration stage, a deployment pod is used to deploy application pods from the application container image that was built in the build configuration stage. The deployment pod also deploys other resources such as services and routes. The deployment configuration begins after the build configuration succeeds.
3. After the deployment pod has started the application pods, application failures can occur within the running application pods. For instance, an application might not behave as expected even though the application pods are in a Running state. In this scenario, you can access running application pods to investigate application failures within a pod.

1. Monitor build, deployment, and application pod status
2. Determine the stage of the S2I process where the problem occurred
3. Review logs corresponding to the failed stage

Procedure

1. Watch the pod status throughout the S2I process to determine at which stage a failure occurs:

   $ oc get pods -w  

   1

   Copy to Clipboard Toggle word wrap

   1

   Use -w to monitor pods for changes until you quit the command using Ctrl+C.

2. Review a failed pod’s logs for errors.

   • If the build pod fails, review the build pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-build

     Copy to Clipboard Toggle word wrap
     Note

     Alternatively, you can review the build configuration’s logs using oc logs -f bc/<application_name>. The build configuration’s logs include the logs from the build pod.

   • If the deployment pod fails, review the deployment pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-deploy

     Copy to Clipboard Toggle word wrap
     Note

     Alternatively, you can review the deployment configuration’s logs using oc logs -f dc/<application_name>. This outputs logs from the deployment pod until the deployment pod completes successfully. The command outputs logs from the application pods if you run it after the deployment pod has completed. After a deployment pod completes, its logs can still be accessed by running oc logs -f pod/<application_name>-<build_number>-deploy.

   • If an application pod fails, or if an application is not behaving as expected within a running application pod, review the application pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-<random_string>

     Copy to Clipboard Toggle word wrap

Procedure

1. List events relating to a specific application pod. The following example retrieves events for an application pod named my-app-1-akdlg:

   $ oc describe pod/my-app-1-akdlg

   Copy to Clipboard Toggle word wrap

2. Review logs from an application pod:

   $ oc logs -f pod/my-app-1-akdlg

   Copy to Clipboard Toggle word wrap

3. Query specific logs within a running application pod. Logs that are sent to stdout are collected by the OpenShift Logging framework and are included in the output of the preceding command. The following query is only required for logs that are not sent to stdout.

   1. If an application log can be accessed without root privileges within a pod, concatenate the log file as follows:

      $ oc exec my-app-1-akdlg -- cat /var/log/my-application.log

      Copy to Clipboard Toggle word wrap

   2. If root access is required to view an application log, you can start a debug container with root privileges and then view the log file from within the container. Start the debug container from the project’s DeploymentConfig object. Pod users typically run with non-root privileges, but running troubleshooting pods with temporary root privileges can be useful during issue investigation:

      $ oc debug dc/my-deployment-configuration --as-root -- cat /var/log/my-application.log

      Copy to Clipboard Toggle word wrap
      Note

      You can access an interactive shell with root access within the debug pod if you run oc debug dc/<deployment_configuration> --as-root without appending -- <command>.

4. Test application functionality interactively and run diagnostic tools, in an application container with an interactive shell.

   1. Start an interactive shell on the application container:

      $ oc exec -it my-app-1-akdlg /bin/bash

      Copy to Clipboard Toggle word wrap
   2. Test application functionality interactively from within the shell. For example, you can run the container’s entry point command and observe the results. Then, test changes from the command line directly, before updating the source code and rebuilding the application container through the S2I process.

   3. Run diagnostic binaries available within the container.

      Note

      Root privileges are required to run some diagnostic binaries. In these situations you can start a debug pod with root access, based on a problematic pod’s DeploymentConfig object, by running oc debug dc/<deployment_configuration> --as-root. Then, you can run diagnostic binaries as root from within the debug pod.

5. If diagnostic binaries are not available within a container, you can run a host’s diagnostic binaries within a container’s namespace by using nsenter. The following example runs ip ad within a container’s namespace, using the host`s ip binary.

   1. Enter into a debug session on the target node. This step instantiates a debug pod called <node_name>-debug:

      $ oc debug node/my-cluster-node

      Copy to Clipboard Toggle word wrap

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Copy to Clipboard Toggle word wrap
      Note

      Red Hat OpenShift Service on AWS 4 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the Red Hat OpenShift Service on AWS API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Determine the target container ID:

      # crictl ps

      Copy to Clipboard Toggle word wrap

   4. Determine the container’s process ID. In this example, the target container ID is a7fe32346b120:

      # crictl inspect a7fe32346b120 --output yaml | grep 'pid:' | awk '{print $2}'

      Copy to Clipboard Toggle word wrap

   5. Run ip ad within the container’s namespace, using the host’s ip binary. This example uses 31150 as the container’s process ID. The nsenter command enters the namespace of a target process and runs a command in its namespace. Because the target process in this example is a container’s process ID, the ip ad command is run in the container’s namespace from the host:

      # nsenter -n -t 31150 -- ip ad

      Copy to Clipboard Toggle word wrap
      Note

      Running a host’s diagnostic binaries within a container’s namespace is only possible if you are using a privileged container such as a debug node.

Procedure

1. Ensure that your project and resources are not excluded from user workload monitoring. The following examples use the ns1 project.

   1. Verify that the project does not have the openshift.io/user-monitoring=false label attached:

      $ oc get namespace ns1 --show-labels | grep 'openshift.io/user-monitoring=false'

      Copy to Clipboard Toggle word wrap
      Note

      The default label set for user workload projects is openshift.io/user-monitoring=true. However, the label is not visible unless you manually apply it.

   2. Verify that the ServiceMonitor and PodMonitor resources do not have the openshift.io/user-monitoring=false label attached. The following example checks the prometheus-example-monitor service monitor.

      $ oc -n ns1 get servicemonitor prometheus-example-monitor --show-labels | grep 'openshift.io/user-monitoring=false'

      Copy to Clipboard Toggle word wrap

   3. If the label is attached, remove the label:

      Example of removing the label from the project

      $ oc label namespace ns1 'openshift.io/user-monitoring-'

      Copy to Clipboard Toggle word wrap

      Example of removing the label from the resource

      $ oc -n ns1 label servicemonitor prometheus-example-monitor 'openshift.io/user-monitoring-'

      Copy to Clipboard Toggle word wrap

      Example output

      namespace/ns1 unlabeled

      Copy to Clipboard Toggle word wrap

2. Check that the corresponding labels match in the service and ServiceMonitor resource configurations. The following examples use the prometheus-example-app service, the prometheus-example-monitor service monitor, and the ns1 project.

   1. Obtain the label defined in the service.

      $ oc -n ns1 get service prometheus-example-app -o yaml

      Copy to Clipboard Toggle word wrap

      Example output

        labels:
          app: prometheus-example-app

      Copy to Clipboard Toggle word wrap

   2. Check that the matchLabels definition in the ServiceMonitor resource configuration matches the label output in the previous step.

      $ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml

      Copy to Clipboard Toggle word wrap

      Example output

      apiVersion: v1
      kind: ServiceMonitor
      metadata:
        name: prometheus-example-monitor
        namespace: ns1
      spec:
        endpoints:
        - interval: 30s
          port: web
          scheme: http
        selector:
          matchLabels:
            app: prometheus-example-app

      Copy to Clipboard Toggle word wrap

      Note

      You can check service and ServiceMonitor resource labels as a developer with view permissions for the project.

3. Inspect the logs for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. List the pods in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get pods

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                   READY   STATUS    RESTARTS   AGE
      prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
      prometheus-user-workload-0             5/5     Running   1          132m
      prometheus-user-workload-1             5/5     Running   1          132m
      thanos-ruler-user-workload-0           3/3     Running   0          132m
      thanos-ruler-user-workload-1           3/3     Running   0          132m

      Copy to Clipboard Toggle word wrap

   2. Obtain the logs from the prometheus-operator container in the prometheus-operator pod. In the following example, the pod is called prometheus-operator-776fcbbd56-2nbfm:

      $ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator

      Copy to Clipboard Toggle word wrap

      If there is a issue with the service monitor, the logs might include an error similar to this example:

      level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload

      Copy to Clipboard Toggle word wrap

4. Review the target status for your endpoint on the Metrics targets page in the Red Hat OpenShift Service on AWS web console UI.

   1. Log in to the Red Hat OpenShift Service on AWS web console and go to Observe Targets.
   2. Locate the metrics endpoint in the list, and review the status of the target in the Status column.
   3. If the Status is Down, click the URL for the endpoint to view more information on the Target Details page for that metrics target.

5. Configure debug level logging for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      Copy to Clipboard Toggle word wrap

   2. Add logLevel: debug for prometheusOperator under data/config.yaml to set the log level to debug:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheusOperator:
            logLevel: debug
      # ...

      Copy to Clipboard Toggle word wrap
   3. Save the file to apply the changes. The affected prometheus-operator pod is automatically redeployed.

   4. Confirm that the debug log-level has been applied to the prometheus-operator deployment in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

      Copy to Clipboard Toggle word wrap

      Example output

              - --log-level=debug

      Copy to Clipboard Toggle word wrap

      Debug level logging will show all calls made by the Prometheus Operator.

   5. Check that the prometheus-operator pod is running:

      $ oc -n openshift-user-workload-monitoring get pods

      Copy to Clipboard Toggle word wrap
      Note

      If an unrecognized Prometheus Operator loglevel value is included in the config map, the prometheus-operator pod might not restart successfully.

   6. Review the debug logs to see if the Prometheus Operator is using the ServiceMonitor resource. Review the logs for other related errors.

Procedure

1. In the Red Hat OpenShift Service on AWS web console, go to Observe Metrics.

2. Enter a Prometheus Query Language (PromQL) query in the Expression field. The following example queries help to identify high cardinality metrics that might result in high disk space consumption:

   • By running the following query, you can identify the ten jobs that have the highest number of scrape samples:

     topk(10, max by(namespace, job) (topk by(namespace, job) (1, scrape_samples_post_metric_relabeling)))

     Copy to Clipboard Toggle word wrap

   • By running the following query, you can pinpoint time series churn by identifying the ten jobs that have created the most time series data in the last hour:

     topk(10, sum by(namespace, job) (sum_over_time(scrape_series_added[1h])))

     Copy to Clipboard Toggle word wrap

3. Investigate the number of unbound label values assigned to metrics with higher than expected scrape sample counts:

   • If the metrics relate to a user-defined project, review the metrics key-value pairs assigned to your workload. These are implemented through Prometheus client libraries at the application level. Try to limit the number of unbound attributes referenced in your labels.
   • If the metrics relate to a core Red Hat OpenShift Service on AWS project, create a Red Hat support case on the Red Hat Customer Portal.

4. Review the TSDB status using the Prometheus HTTP API by following these steps when logged in as a dedicated-admin:

   1. Get the Prometheus API route URL by running the following command:

      $ HOST=$(oc -n openshift-monitoring get route prometheus-k8s -ojsonpath='{.status.ingress[].host}')

      Copy to Clipboard Toggle word wrap

   2. Extract an authentication token by running the following command:

      $ TOKEN=$(oc whoami -t)

      Copy to Clipboard Toggle word wrap

   3. Query the TSDB status for Prometheus by running the following command:

      $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/status/tsdb"

      Copy to Clipboard Toggle word wrap

      Example output

      "status": "success","data":{"headStats":{"numSeries":507473,
      "numLabelPairs":19832,"chunkCount":946298,"minTime":1712253600010,
      "maxTime":1712257935346},"seriesCountByMetricName":
      [{"name":"etcd_request_duration_seconds_bucket","value":51840},
      {"name":"apiserver_request_sli_duration_seconds_bucket","value":47718},
      ...

      Copy to Clipboard Toggle word wrap

1. Delete the stack:

   $ rosa init --delete

   Copy to Clipboard Toggle word wrap

2. Reinitialize your account:

   $ rosa init

   Copy to Clipboard Toggle word wrap

Procedure

1. Run the following command to create the user-role IAM resource:

   $ rosa create user-role

   Copy to Clipboard Toggle word wrap

2. After you see that the role has been created, you can delete the cluster. The following confirms that the role was created and linked:

   I: Successfully linked role ARN <user role ARN> with account <account ID>

   Copy to Clipboard Toggle word wrap