Support

Procedure

1. Log in to a cluster.

2. Run the following command, which queries a cluster’s Prometheus service and returns the full set of time series data captured by Telemetry:

   Note

   The following example contains some values that are specific to Red Hat OpenShift Service on AWS on AWS.

   $ curl -G -k -H "Authorization: Bearer $(oc whoami -t)" \
   https://$(oc get route prometheus-k8s-federate -n \
   openshift-monitoring -o jsonpath="{.spec.host}")/federate \
   --data-urlencode 'match[]={__name__=~"cluster:usage:.*"}' \
   --data-urlencode 'match[]={__name__="count:up0"}' \
   --data-urlencode 'match[]={__name__="count:up1"}' \
   --data-urlencode 'match[]={__name__="cluster_version"}' \
   --data-urlencode 'match[]={__name__="cluster_version_available_updates"}' \
   --data-urlencode 'match[]={__name__="cluster_version_capability"}' \
   --data-urlencode 'match[]={__name__="cluster_operator_up"}' \
   --data-urlencode 'match[]={__name__="cluster_operator_conditions"}' \
   --data-urlencode 'match[]={__name__="cluster_version_payload"}' \
   --data-urlencode 'match[]={__name__="cluster_installer"}' \
   --data-urlencode 'match[]={__name__="cluster_infrastructure_provider"}' \
   --data-urlencode 'match[]={__name__="cluster_feature_set"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_object_counts:sum"}' \
   --data-urlencode 'match[]={__name__="ALERTS",alertstate="firing"}' \
   --data-urlencode 'match[]={__name__="code:apiserver_request_total:rate:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:capacity_cpu_cores:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:capacity_memory_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="workload:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="workload:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:virt_platform_nodes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:node_instance_type_count:sum"}' \
   --data-urlencode 'match[]={__name__="cnv:vmi_status_running:count"}' \
   --data-urlencode 'match[]={__name__="cluster:vmi_request_cpu_cores:sum"}' \
   --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_cores:sum"}' \
   --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_sockets:sum"}' \
   --data-urlencode 'match[]={__name__="subscription_sync_total"}' \
   --data-urlencode 'match[]={__name__="olm_resolution_duration_seconds"}' \
   --data-urlencode 'match[]={__name__="csv_succeeded"}' \
   --data-urlencode 'match[]={__name__="csv_abnormal"}' \
   --data-urlencode 'match[]={__name__="cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:kubelet_volume_stats_used_bytes:provisioner:sum"}' \
   --data-urlencode 'match[]={__name__="ceph_cluster_total_bytes"}' \
   --data-urlencode 'match[]={__name__="ceph_cluster_total_used_raw_bytes"}' \
   --data-urlencode 'match[]={__name__="ceph_health_status"}' \
   --data-urlencode 'match[]={__name__="odf_system_raw_capacity_total_bytes"}' \
   --data-urlencode 'match[]={__name__="odf_system_raw_capacity_used_bytes"}' \
   --data-urlencode 'match[]={__name__="odf_system_health_status"}' \
   --data-urlencode 'match[]={__name__="job:ceph_osd_metadata:count"}' \
   --data-urlencode 'match[]={__name__="job:kube_pv:count"}' \
   --data-urlencode 'match[]={__name__="job:odf_system_pvs:count"}' \
   --data-urlencode 'match[]={__name__="job:ceph_pools_iops:total"}' \
   --data-urlencode 'match[]={__name__="job:ceph_pools_iops_bytes:total"}' \
   --data-urlencode 'match[]={__name__="job:ceph_versions_running:count"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_total_unhealthy_buckets:sum"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_bucket_count:sum"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_total_object_count:sum"}' \
   --data-urlencode 'match[]={__name__="odf_system_bucket_count", system_type="OCS", system_vendor="Red Hat"}' \
   --data-urlencode 'match[]={__name__="odf_system_objects_total", system_type="OCS", system_vendor="Red Hat"}' \
   --data-urlencode 'match[]={__name__="noobaa_accounts_num"}' \
   --data-urlencode 'match[]={__name__="noobaa_total_usage"}' \
   --data-urlencode 'match[]={__name__="console_url"}' \
   --data-urlencode 'match[]={__name__="cluster:ovnkube_master_egress_routing_via_host:max"}' \
   --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_instances:max"}' \
   --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_enabled_instance_up:max"}' \
   --data-urlencode 'match[]={__name__="cluster:ingress_controller_aws_nlb_active:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:min"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:max"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:avg"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:median"}' \
   --data-urlencode 'match[]={__name__="cluster:openshift_route_info:tls_termination:sum"}' \
   --data-urlencode 'match[]={__name__="insightsclient_request_send_total"}' \
   --data-urlencode 'match[]={__name__="cam_app_workload_migrations"}' \
   --data-urlencode 'match[]={__name__="cluster:apiserver_current_inflight_requests:sum:max_over_time:2m"}' \
   --data-urlencode 'match[]={__name__="cluster:alertmanager_integrations:max"}' \
   --data-urlencode 'match[]={__name__="cluster:telemetry_selected_series:count"}' \
   --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_series:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_samples_appended_total:sum"}' \
   --data-urlencode 'match[]={__name__="monitoring:container_memory_working_set_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="namespace_job:scrape_series_added:topk3_sum1h"}' \
   --data-urlencode 'match[]={__name__="namespace_job:scrape_samples_post_metric_relabeling:topk3"}' \
   --data-urlencode 'match[]={__name__="monitoring:haproxy_server_http_responses_total:sum"}' \
   --data-urlencode 'match[]={__name__="rhmi_status"}' \
   --data-urlencode 'match[]={__name__="status:upgrading:version:rhoam_state:max"}' \
   --data-urlencode 'match[]={__name__="state:rhoam_critical_alerts:max"}' \
   --data-urlencode 'match[]={__name__="state:rhoam_warning_alerts:max"}' \
   --data-urlencode 'match[]={__name__="rhoam_7d_slo_percentile:max"}' \
   --data-urlencode 'match[]={__name__="rhoam_7d_slo_remaining_error_budget:max"}' \
   --data-urlencode 'match[]={__name__="cluster_legacy_scheduler_policy"}' \
   --data-urlencode 'match[]={__name__="cluster_master_schedulable"}' \
   --data-urlencode 'match[]={__name__="che_workspace_status"}' \
   --data-urlencode 'match[]={__name__="che_workspace_started_total"}' \
   --data-urlencode 'match[]={__name__="che_workspace_failure_total"}' \
   --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_sum"}' \
   --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_count"}' \
   --data-urlencode 'match[]={__name__="cco_credentials_mode"}' \
   --data-urlencode 'match[]={__name__="cluster:kube_persistentvolume_plugin_type_counts:sum"}' \
   --data-urlencode 'match[]={__name__="visual_web_terminal_sessions_total"}' \
   --data-urlencode 'match[]={__name__="acm_managed_cluster_info"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_vcenter_info:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_esxi_version_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_node_hw_version_total:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:build_by_strategy:sum"}' \
   --data-urlencode 'match[]={__name__="rhods_aggregate_availability"}' \
   --data-urlencode 'match[]={__name__="rhods_total_users"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="appsvcs:cores_by_product:sum"}' \
   --data-urlencode 'match[]={__name__="nto_custom_profiles:count"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_configmap"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_secret"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_mount_failures_total"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_mount_requests_total"}' \
   --data-urlencode 'match[]={__name__="cluster:velero_backup_total:max"}' \
   --data-urlencode 'match[]={__name__="cluster:velero_restore_total:max"}' \
   --data-urlencode 'match[]={__name__="eo_es_storage_info"}' \
   --data-urlencode 'match[]={__name__="eo_es_redundancy_policy_info"}' \
   --data-urlencode 'match[]={__name__="eo_es_defined_delete_namespaces_total"}' \
   --data-urlencode 'match[]={__name__="eo_es_misconfigured_memory_resources_info"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_data_nodes_total:max"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_documents_created_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_documents_deleted_total:sum"}' \
   --data-urlencode 'match[]={__name__="pod:eo_es_shards_total:max"}' \
   --data-urlencode 'match[]={__name__="eo_es_cluster_management_state_info"}' \
   --data-urlencode 'match[]={__name__="imageregistry:imagestreamtags_count:sum"}' \
   --data-urlencode 'match[]={__name__="imageregistry:operations_count:sum"}' \
   --data-urlencode 'match[]={__name__="log_logging_info"}' \
   --data-urlencode 'match[]={__name__="log_collector_error_count_total"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_pipeline_info"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_input_info"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_output_info"}' \
   --data-urlencode 'match[]={__name__="cluster:log_collected_bytes_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:log_logged_bytes_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:kata_monitor_running_shim_count:sum"}' \
   --data-urlencode 'match[]={__name__="platform:hypershift_hostedclusters:max"}' \
   --data-urlencode 'match[]={__name__="platform:hypershift_nodepools:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_bucket_claims:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_buckets_claims:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_resources:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_resources:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_buckets:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_buckets:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_accounts:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_usage:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_system_health_status:max"}' \
   --data-urlencode 'match[]={__name__="ocs_advanced_feature_usage"}' \
   --data-urlencode 'match[]={__name__="os_image_url_override:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:openshift_network_operator_ipsec_state:info"}'

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

   Depending on the result, the Red Hat Lightspeed advisor service displays one of the following:

   • No matching recommendations found, if Red Hat Lightspeed did not identify any issues.
   • A list of issues Red Hat Lightspeed has detected, grouped by risk (low, moderate, important, and critical).
   • No clusters yet, if Red Hat Lightspeed has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed, registered, and connected to the internet.

2. If any issues are displayed, click the > icon in front of the entry for more details.

   Depending on the issue, the details can also contain a link to more information from Red Hat about the issue.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

2. Click the X icons next to the Clusters Impacted and Status filters.

   You can now browse through all of the potential recommendations for your cluster.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.
2. Optional: Use the Clusters Impacted and Status filters as needed.

3. Disable an alert by using one of the following methods:

   • To disable an alert:

     1. Click the Options menu for that alert, and then click Disable recommendation.
     2. Enter a justification note and click Save.

   • To view the clusters affected by this alert before disabling the alert:

     1. Click the name of the recommendation to disable. You are directed to the single recommendation page.
     2. Review the list of clusters in the Affected clusters section.
     3. Click Actions → Disable recommendation to disable the alert for all of your clusters.
     4. Enter a justification note and click Save.

Procedure

1. Navigate to Advisor → Recommendations on OpenShift Cluster Manager.

2. Filter the recommendations to display on the disabled recommendations:

   1. From the Status drop-down menu, select Status.
   2. From the Filter by status drop-down menu, select Disabled.
   3. Optional: Clear the Clusters impacted filter.
3. Locate the recommendation to enable.
4. Click the Options menu , and then click Enable recommendation.

Procedure

1. Navigate to Home → Overview in the Red Hat OpenShift Service on AWS web console.

2. Click Red Hat Lightspeed on the Status card.

   The pop-up window lists potential issues grouped by risk. Click the individual categories or View all recommendations in Red Hat Lightspeed Advisor to display more details.

Procedure

1. Go to Workloads → ConfigMaps and select Project: openshift-insights.
2. Click the insights-config ConfigMap object to open it.
3. Click Actions and select Edit ConfigMap.
4. Click the YAML view radio button.

5. In the file, set the obfuscation attribute with the workload_names value.

   apiVersion: v1
   kind: ConfigMap
   # ...
   data:
     config.yaml: |
       dataReporting:
         obfuscation:
           - workload_names
   # ...

   Copy to Clipboard Toggle word wrap
6. Click Save. The insights-config config-map details page opens.
7. Verify that the value of the config.yaml obfuscation attribute is set to - workload_names.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

   Note

   If your cluster is in a disconnected environment, you must take additional steps. If your mirror registry has a trusted CA, you must first add the trusted CA to the cluster. For all clusters in disconnected environments, you must import the default must-gather image as an image stream.

   $ oc import-image is/must-gather -n openshift

   Copy to Clipboard Toggle word wrap

2. Run the oc adm must-gather command:

   $ oc adm must-gather

   Copy to Clipboard Toggle word wrap
   Important

   If you are in a disconnected environment, use the --image flag as part of must-gather and point to the payload image.

   Note

   Because this command picks a random control plane node by default, the pod might be scheduled to a control plane node that is in the NotReady and SchedulingDisabled state.

   1. If this command fails, for example, if you cannot schedule a pod on your cluster, then use the oc adm inspect command to gather information for particular resources.

      Note

      Contact Red Hat Support for the recommended resources to gather.

3. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see How to find the cluster-id or name on OpenShift cluster. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather-`date +"%m-%d-%Y-%H-%M-%S"`-<cluster_id>.tar.gz <must_gather_local_dir>

   Copy to Clipboard Toggle word wrap

   where:

   <must_gather_local_dir>

   Replace with the actual directory name.

4. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command with one or more --image or --image-stream arguments.

   Note
   • To collect the default must-gather data in addition to specific feature data, add the --image-stream=openshift/must-gather argument.
   • For information on gathering data about the Custom Metrics Autoscaler, see the Additional resources section that follows.

   For example, the following command gathers both the default cluster data and information specific to OpenShift Virtualization:

   $ oc adm must-gather \
     --image-stream=openshift/must-gather \
     --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.21.0

   Copy to Clipboard Toggle word wrap

   You can use the must-gather tool with additional arguments to gather data that is specifically related to OpenShift Logging and the Red Hat OpenShift Logging Operator in your cluster. For OpenShift Logging, run the following command:

   $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator \
     -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

   Copy to Clipboard Toggle word wrap

   Example must-gather output for OpenShift Logging

   ├── cluster-logging
   │  ├── clo
   │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
   │  │  ├── clusterlogforwarder_cr
   │  │  ├── cr
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── logforwarding_cr
   │  ├── collector
   │  │  ├── fluentd-2tr64
   │  ├── eo
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── elasticsearch-operator-7dc7d97b9d-jb4r4
   │  ├── es
   │  │  ├── cluster-elasticsearch
   │  │  │  ├── aliases
   │  │  │  ├── health
   │  │  │  ├── indices
   │  │  │  ├── latest_documents.json
   │  │  │  ├── nodes
   │  │  │  ├── nodes_stats.json
   │  │  │  └── thread_pool
   │  │  ├── cr
   │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  │  └── logs
   │  │     ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  ├── install
   │  │  ├── co_logs
   │  │  ├── install_plan
   │  │  ├── olmo_logs
   │  │  └── subscription
   │  └── kibana
   │     ├── cr
   │     ├── kibana-9d69668d4-2rkvz
   ├── cluster-scoped-resources
   │  └── core
   │     ├── nodes
   │     │  ├── ip-10-0-146-180.eu-west-1.compute.internal.yaml
   │     └── persistentvolumes
   │        ├── pvc-0a8d65d9-54aa-4c44-9ecc-33d9381e41c1.yaml
   ├── event-filter.html
   ├── gather-debug.log
   └── namespaces
      ├── openshift-logging
      │  ├── apps
      │  │  ├── daemonsets.yaml
      │  │  ├── deployments.yaml
      │  │  ├── replicasets.yaml
      │  │  └── statefulsets.yaml
      │  ├── batch
      │  │  ├── cronjobs.yaml
      │  │  └── jobs.yaml
      │  ├── core
      │  │  ├── configmaps.yaml
      │  │  ├── endpoints.yaml
      │  │  ├── events
      │  │  │  ├── elasticsearch-im-app-1596020400-gm6nl.1626341a296c16a1.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-9l9n4.1626341a2af81bbd.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-v98tk.1626341a2d821069.yaml
      │  │  │  ├── elasticsearch-im-app-1596020400-cc5vc.1626341a3019b238.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-s8d5s.1626341a31f7b315.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-7mgv8.1626341a35ea59ed.yaml
      │  │  ├── events.yaml
      │  │  ├── persistentvolumeclaims.yaml
      │  │  ├── pods.yaml
      │  │  ├── replicationcontrollers.yaml
      │  │  ├── secrets.yaml
      │  │  └── services.yaml
      │  ├── openshift-logging.yaml
      │  ├── pods
      │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
      │  │  │  ├── cluster-logging-operator
      │  │  │  │  └── cluster-logging-operator
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  └── cluster-logging-operator-74dd5994f-6ttgt.yaml
      │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff
      │  │  │  ├── cluster-logging-operator-registry
      │  │  │  │  └── cluster-logging-operator-registry
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff.yaml
      │  │  │  └── mutate-csv-and-generate-sqlite-db
      │  │  │     └── mutate-csv-and-generate-sqlite-db
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
      │  │  ├── elasticsearch-im-app-1596030300-bpgcx
      │  │  │  ├── elasticsearch-im-app-1596030300-bpgcx.yaml
      │  │  │  └── indexmanagement
      │  │  │     └── indexmanagement
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── fluentd-2tr64
      │  │  │  ├── fluentd
      │  │  │  │  └── fluentd
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── fluentd-2tr64.yaml
      │  │  │  └── fluentd-init
      │  │  │     └── fluentd-init
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── kibana-9d69668d4-2rkvz
      │  │  │  ├── kibana
      │  │  │  │  └── kibana
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── kibana-9d69668d4-2rkvz.yaml
      │  │  │  └── kibana-proxy
      │  │  │     └── kibana-proxy
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  └── route.openshift.io
      │     └── routes.yaml
      └── openshift-operators-redhat
         ├── ...

   Copy to Clipboard Toggle word wrap

3. Run the oc adm must-gather command with one or more --image or --image-stream arguments. For example, the following command gathers both the default cluster data and information specific to KubeVirt:

   $ oc adm must-gather \
    --image-stream=openshift/must-gather \
    --image=quay.io/kubevirt/must-gather

   Copy to Clipboard Toggle word wrap

4. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see How to find the cluster-id or name on OpenShift cluster. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather-`date +"%m-%d-%Y-%H-%M-%S"`-<cluster_id>.tar.gz <must_gather_local_dir>

   Copy to Clipboard Toggle word wrap

   where:

   <must_gather_local_dir>

   Replace with the actual directory name.

5. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Log in to the Red Hat OpenShift Service on AWS web console.
2. Navigate to Ecosystem → Software Catalog.
3. In the filter box, enter Support Log Gather.
4. Select Support Log Gather.
5. From Version list, select the Support Log Gather version, and click Install.

6. On the Install Operator page, configure the installation settings:

   1. Choose the Installed Namespace for the Operator.

      The default Operator namespace is must-gather-operator. The must-gather-operator namespace is created automatically if it does not exist.

   2. Select an Update approval strategy:

      • Select Automatic to have the Operator Lifecycle Manager (OLM) update the Operator automatically when a newer version is available.
      • Select Manual if Operator updates must be approved by a user with appropriate credentials.
   3. Click Install.

Verification

1. Verify that the Operator is installed successfully:

   1. Navigate to Ecosystem → Software Catalog.
   2. Verify that Support Log Gather is listed with a Status of Succeeded in the must-gather-operator namespace.

2. Verify that Support Log Gather pods are running:

   1. Navigate to Workloads → Pods

   2. Verify that the status of the Support Log Gather pods is Running.

      You can use the Support Log Gather only after the pods are up and running.

Procedure

1. Create a new project named must-gather-operator by running the following command:

   $ oc new-project must-gather-operator

   Copy to Clipboard Toggle word wrap

2. Create an OperatorGroup object:

   1. Create a YAML file, for example, operatorGroup.yaml, that defines the OperatorGroup object:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: must-gather-operator
        namespace: must-gather-operator
      spec: {}

      Copy to Clipboard Toggle word wrap

   2. Create the OperatorGroup object by running the following command:

      $ oc create -f operatorGroup.yaml

      Copy to Clipboard Toggle word wrap

3. Create a Subscription object:

   1. Create a YAML file, for example, subscription.yaml, that defines the Subscription object:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: support-log-gather-operator
        namespace: must-gather-operator
      spec:
        channel: tech-preview
        name: support-log-gather-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        installPlanApproval: Automatic

      Copy to Clipboard Toggle word wrap

   2. Create the Subscription object by running the following command:

      $ oc create -f subscription.yaml

      Copy to Clipboard Toggle word wrap

Verification

1. Verify the status of the pods in the Operator namespace by running the following command.

   $ oc get pods

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                                              READY   STATUS      RESTARTS   AGE
   must-gather-operator-657fc74d64-2gg2w                             1/1     Running     0          13m

   Copy to Clipboard Toggle word wrap

   The status of all the pods must be Running.

2. Verify that the subscription is created by running the following command:

   $ oc get subscription -n must-gather-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                          PACKAGE                       SOURCE            CHANNEL
   support-log-gather-operator   support-log-gather-operator   redhat-operators  tech-preview

   Copy to Clipboard Toggle word wrap

3. Verify that the Operator is installed by running the following command:

   $ oc get csv -n must-gather-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                  DISPLAY                VERSION   REPLACES   PHASE
   support-log-gather-operator.v4.21.0   support log gather     4.21.0               Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file for the MustGather CR, such as support-log-gather.yaml, that contains the following basic configuration::

   Example support-log-gather.yaml

   apiVersion: operator.openshift.io/v1alpha1
   kind: MustGather
   metadata:
     name: example-mg
     namespace: must-gather-operator
   spec:
     serviceAccountName: must-gather-operator
     audit: true
     proxyConfig:
       httpProxy: "http://proxy.example.com:8080"
       httpsProxy: "https://proxy.example.com:8443"
       noProxy: ".example.com,localhost"
     mustGatherTimeout: "1h30m9s"
     uploadTarget:
       type: SFTP
       sftp:
         caseID: "04230315"
         caseManagementAccountSecretRef:
           name: mustgather-creds
         host: "sftp.access.redhat.com"
     retainResourcesOnCompletion: true
     storage:
       type: PersistentVolume
       persistentVolume:
         claim:
           name: mustgather-pvc
         subPath: must-gather-bundles/case-04230315

   Copy to Clipboard Toggle word wrap

   For more information on the configuration parameters, see "Configuration parameters for MustGather custom resource".

2. Create the MustGather object by running the following command:

   $ oc create -f support-log-gather.yaml

   Copy to Clipboard Toggle word wrap

Verification

1. Verify that the MustGather CR was created by running the following command:

   $ oc get mustgather

   Copy to Clipboard Toggle word wrap

   Example output

   NAME          AGE
   example-mg    7s

   Copy to Clipboard Toggle word wrap

2. Verify the status of the pods in the Operator namespace by running the following command.

   $ oc get pods

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                                              READY   STATUS      RESTARTS   AGE
   must-gather-operator-657fc74d64-2gg2w                             1/1     Running     0          13m
   example-mg-gk8m8                                                  2/2     Running     0          13s

   Copy to Clipboard Toggle word wrap

   A new pod with a name based on the MustGather CR must be created. The status of all the pods must be Running.

3. To monitor the progress of the file upload, view the logs of the upload container in the job pod by running the following command:

   oc logs -f pod/example-mg-gk8m8 -c upload

   Copy to Clipboard Toggle word wrap

   When successful, the process must create an archive and upload it to the Red Hat Secure File Transfer Protocol (SFTP) server for the specified case.

Procedure

1. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

   # chroot /host

   Copy to Clipboard Toggle word wrap

2. Start the toolbox container:

   # toolbox

   Copy to Clipboard Toggle word wrap

3. Install the additional package, such as wget:

   # dnf install -y <package_name>

   Copy to Clipboard Toggle word wrap

Procedure

1. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

   # chroot /host

   Copy to Clipboard Toggle word wrap

2. Optional: If you need to use an alternative image instead of the default image, create a .toolboxrc file in the home directory for the root user ID, and specify the image metadata:

   REGISTRY=quay.io
   IMAGE=fedora/fedora:latest
   TOOLBOX_NAME=toolbox-fedora-latest

   Copy to Clipboard Toggle word wrap

   where:

   REGISTRY=quay.io

   Optional: Specify an alternative container registry.

   IMAGE=fedora/fedora:latest

   Specify an alternative image to start.

   TOOLBOX_NAME=toolbox-fedora-latest

   Optional: Specify an alternative name for the toolbox container.

3. Start a toolbox container by entering the following command:

   # toolbox

   Copy to Clipboard Toggle word wrap
   Note

   If an existing toolbox pod is already running, the toolbox command outputs 'toolbox-' already exists. Trying to start…​. To avoid issues with sosreport plugins, remove the running toolbox container with podman rm toolbox- and then spawn a new toolbox container.

Procedure

1. Navigate to the Clusters page of the Hybrid Cloud Console.
2. Click the name of your cluster to go to the cluster details page.

3. Click the Cluster history tab.

   Cluster notifications appear under the Cluster history heading.

4. Optional: Filter for relevant cluster notifications

   Use the filter controls to hide cluster notifications that are not relevant to you, so that you can focus on your area of expertise or on resolving a critical issue. You can filter notifications based on text in the notification description, severity level, notification type, when the notification was received, and which system or person triggered the notification.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

   Copy to Clipboard Toggle word wrap

   Note

   Default Red Hat OpenShift Service on AWS cluster Operators are managed by the Cluster Version Operator (CVO) and they do not have a Subscription object. Application Operators are managed by Operator Lifecycle Manager (OLM) and they have a Subscription object.

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

   Copy to Clipboard Toggle word wrap

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   Copy to Clipboard Toggle word wrap

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-operators-smxx8                  1/1     Running            0          36m

   Copy to Clipboard Toggle word wrap

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   Copy to Clipboard Toggle word wrap

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. List Operators running in the cluster. The output includes Operator version, availability, and up-time information:

   $ oc get clusteroperators

   Copy to Clipboard Toggle word wrap

2. List Operator pods running in the Operator’s namespace, plus pod status, restarts, and age:

   $ oc get pod -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

3. Output a detailed Operator pod summary:

   $ oc describe pod <operator_pod_name> -n <operator_namespace>

   Copy to Clipboard Toggle word wrap

Procedure

1. Switch into a project:

   $ oc project <project_name>

   Copy to Clipboard Toggle word wrap

2. List pods running within the namespace, as well as pod status, error states, restarts, and age:

   $ oc get pods

   Copy to Clipboard Toggle word wrap

3. Determine whether the namespace is managed by a deployment configuration:

   $ oc status

   Copy to Clipboard Toggle word wrap

   If the namespace is managed by a deployment configuration, the output includes the deployment configuration name and a base image reference.

4. Inspect the base image referenced in the preceding command’s output:

   $ skopeo inspect docker://<image_reference>

   Copy to Clipboard Toggle word wrap

5. If the base image reference is not correct, update the reference in the deployment configuration:

   $ oc edit deployment/my-deployment

   Copy to Clipboard Toggle word wrap

6. When deployment configuration changes on exit, the configuration will automatically redeploy. Watch pod status as the deployment progresses, to determine whether the issue has been resolved:

   $ oc get pods -w

   Copy to Clipboard Toggle word wrap

7. Review events within the namespace for diagnostic information relating to pod failures:

   $ oc get events

   Copy to Clipboard Toggle word wrap

Procedure

1. Query logs for a specific pod:

   $ oc logs <pod_name>

   Copy to Clipboard Toggle word wrap

2. Query logs for a specific container within a pod:

   $ oc logs <pod_name> -c <container_name>

   Copy to Clipboard Toggle word wrap

   Logs retrieved using the preceding oc logs commands are composed of messages sent to stdout within pods or containers.

3. Inspect logs contained in /var/log/ within a pod.

   1. List log files and subdirectories contained in /var/log within a pod:

      $ oc exec <pod_name>  -- ls -alh /var/log

      Copy to Clipboard Toggle word wrap

      Example output

      total 124K
      drwxr-xr-x. 1 root root   33 Aug 11 11:23 .
      drwxr-xr-x. 1 root root   28 Sep  6  2022 ..
      -rw-rw----. 1 root utmp    0 Jul 10 10:31 btmp
      -rw-r--r--. 1 root root  33K Jul 17 10:07 dnf.librepo.log
      -rw-r--r--. 1 root root  69K Jul 17 10:07 dnf.log
      -rw-r--r--. 1 root root 8.8K Jul 17 10:07 dnf.rpm.log
      -rw-r--r--. 1 root root  480 Jul 17 10:07 hawkey.log
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 lastlog
      drwx------. 2 root root   23 Aug 11 11:14 openshift-apiserver
      drwx------. 2 root root    6 Jul 10 10:31 private
      drwxr-xr-x. 1 root root   22 Mar  9 08:05 rhsm
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 wtmp

      Copy to Clipboard Toggle word wrap

   2. Query a specific log file contained in /var/log within a pod:

      $ oc exec <pod_name> cat /var/log/<path_to_log>

      Copy to Clipboard Toggle word wrap

      Example output

      2023-07-10T10:29:38+0000 INFO --- logging initialized ---
      2023-07-10T10:29:38+0000 DDEBUG timer: config: 13 ms
      2023-07-10T10:29:38+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
      2023-07-10T10:29:38+0000 INFO Updating Subscription Management repositories.
      2023-07-10T10:29:38+0000 INFO Unable to read consumer identity
      2023-07-10T10:29:38+0000 INFO Subscription Manager is operating in container mode.
      2023-07-10T10:29:38+0000 INFO

      Copy to Clipboard Toggle word wrap

   3. List log files and subdirectories contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> ls /var/log

      Copy to Clipboard Toggle word wrap

   4. Query a specific log file contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> cat /var/log/<path_to_log>

      Copy to Clipboard Toggle word wrap

Procedure

1. Switch into the project that contains the pod you would like to access. This is necessary because the oc rsh command does not accept the -n namespace option:

   $ oc project <namespace>

   Copy to Clipboard Toggle word wrap

2. Start a remote shell into a pod:

   $ oc rsh <pod_name>

   Copy to Clipboard Toggle word wrap

   where:

   <pod_name>

   If a pod has multiple containers, oc rsh defaults to the first container unless -c <container_name> is specified.

3. Start a remote shell into a specific container within a pod:

   $ oc rsh -c <container_name> pod/<pod_name>

   Copy to Clipboard Toggle word wrap

4. Create a port forwarding session to a port on a pod:

   $ oc port-forward <pod_name> <host_port>:<pod_port>

   Copy to Clipboard Toggle word wrap

   where:

   <pod_name> <host_port>:<pod_port>

   Enter Ctrl+C to cancel the port forwarding session.

Procedure

1. Start a debug pod with root access, based on a deployment.

   1. Obtain a project’s deployment name:

      $ oc get deployment -n <project_name>

      Copy to Clipboard Toggle word wrap

   2. Start a debug pod with root privileges, based on the deployment:

      $ oc debug deployment/my-deployment --as-root -n <project_name>

      Copy to Clipboard Toggle word wrap

2. Start a debug pod with root access, based on a deployment configuration.

   1. Obtain a project’s deployment configuration name:

      $ oc get deploymentconfigs -n <project_name>

      Copy to Clipboard Toggle word wrap

   2. Start a debug pod with root privileges, based on the deployment configuration:

      $ oc debug deploymentconfig/my-deployment-configuration --as-root -n <project_name>

      Copy to Clipboard Toggle word wrap
      Note

      You can append -- <command> to the preceding oc debug commands to run individual commands within a debug pod, instead of running an interactive shell.