Chapter 4. Remote health monitoring with connected clusters

Procedure

1. Find the URL for the Prometheus service that runs in the OpenShift Container Platform cluster:

   $ oc get route prometheus-k8s -n openshift-monitoring -o jsonpath="{.spec.host}"

2. Navigate to the URL.

3. Enter this query in the Expression input box and press Execute:

   {__name__=~"cluster:usage:.*|count:up0|count:up1|cluster_version|cluster_version_available_updates|cluster_operator_up|cluster_operator_conditions|cluster_version_payload|cluster_installer|cluster_infrastructure_provider|cluster_feature_set|instance:etcd_object_counts:sum|ALERTS|code:apiserver_request_total:rate:sum|cluster:capacity_cpu_cores:sum|cluster:capacity_memory_bytes:sum|cluster:cpu_usage_cores:sum|cluster:memory_usage_bytes:sum|openshift:cpu_usage_cores:sum|openshift:memory_usage_bytes:sum|workload:cpu_usage_cores:sum|workload:memory_usage_bytes:sum|cluster:virt_platform_nodes:sum|cluster:node_instance_type_count:sum|cnv:vmi_status_running:count|node_role_os_version_machine:cpu_capacity_cores:sum|node_role_os_version_machine:cpu_capacity_sockets:sum|subscription_sync_total|csv_succeeded|csv_abnormal|ceph_cluster_total_bytes|ceph_cluster_total_used_raw_bytes|ceph_health_status|job:ceph_osd_metadata:count|job:kube_pv:count|job:ceph_pools_iops:total|job:ceph_pools_iops_bytes:total|job:ceph_versions_running:count|job:noobaa_total_unhealthy_buckets:sum|job:noobaa_bucket_count:sum|job:noobaa_total_object_count:sum|noobaa_accounts_num|noobaa_total_usage|console_url|cluster:network_attachment_definition_instances:max|cluster:network_attachment_definition_enabled_instance_up:max|insightsclient_request_send_total|cam_app_workload_migrations|cluster:apiserver_current_inflight_requests:sum:max_over_time:2m|cluster:telemetry_selected_series:count",alertstate=~"firing|"}

   This query replicates the request that Telemetry makes against a running OpenShift Container Platform cluster’s Prometheus service and returns the full set of time series captured by Telemetry.

Procedure

1. Find the name of the currently running pod for the Insights Operator:

   $ INSIGHTS_OPERATOR_POD=$(oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running)

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/$INSIGHTS_OPERATOR_POD:/var/lib/insights-operator ./insights-data

Procedure

1. Download the global cluster pull secret to your local file system.

   $ oc extract secret/pull-secret -n openshift-config --to=.

2. In a text editor, edit the .dockerconfigjson file that was downloaded.

3. Remove the cloud.openshift.com JSON entry, for example:

   "cloud.openshift.com":{"auth":"<hash>","email":"<email_address>"}

4. Save the file.

Procedure

1. Optional: To append a new pull secret to the existing pull secret, complete the following steps:

   1. Enter the following command to download the pull secret:

      $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> 1

      1

      Provide the path to the pull secret file.

   2. Enter the following command to add the new pull secret:

      $ oc registry login --registry="<registry>" \ 1
      --auth-basic="<username>:<password>" \ 2
      --to=<pull_secret_location> 3

      1

      Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>".

      2

      Provide the credentials of the new registry.

      3

      Provide the path to the pull secret file.

      Alternatively, you can perform a manual update to the pull secret file.

2. Enter the following command to update the global pull secret for your cluster:

   $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> 1

   1

   Provide the path to the new pull secret file.

   This update is rolled out to all nodes, which can take some time depending on the size of your cluster.

   Note

   As of OpenShift Container Platform 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot.

Procedure

1. Navigate to Advisor Recommendations on Red Hat Hybrid Cloud Console.

   Depending on the result, Insights Advisor displays one of the following:

   • No matching recommendations found, if Insights did not identify any issues.
   • A list of issues Insights has detected, grouped by risk (low, moderate, important, and critical).
   • No clusters yet, if Insights has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed, registered, and connected to the internet.

2. If any issues are displayed, click the > icon in front of the entry for more details.

   Depending on the issue, the details can also contain a link to more information from Red Hat about the issue.

Procedure

1. Navigate to Advisor Recommendations on Red Hat Hybrid Cloud Console.

2. Click the X icons next to the Clusters Impacted and Status filters.

   You can now browse through all of the potential recommendations for your cluster.

Procedure

1. Navigate to Advisor Recommendations on Red Hat Hybrid Cloud Console.
2. Click the name of the recommendation to disable. You are directed to the single recommendation page.

3. To disable the recommendation for a single cluster:

   1. Click the Options menu for that cluster, and then click Disable recommendation for cluster.
   2. Enter a justification note and click Save.

4. To disable the recommendation for all of your clusters:

   1. Click Actions Disable recommendation.
   2. Enter a justification note and click Save.

Procedure

1. Navigate to Advisor Recommendations on Red Hat Hybrid Cloud Console.
2. Filter the recommendations by Status Disabled.
3. Locate the recommendation to enable.
4. Click the Options menu , and then click Enable recommendation.

Procedure

1. Navigate to Home Overview in the OpenShift Container Platform web console.

2. Click Insights on the Status card.

   The pop-up window lists potential issues grouped by risk. Click the individual categories or View all recommendations in Insights Advisor to display more details.

Procedure

1. Find the name of the running pod for the Insights Operator:

   $ oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/<insights_operator_pod_name>:/var/lib/insights-operator ./insights-data 1

   1

   Replace <insights_operator_pod_name> with the pod name output from the preceding command.

Procedure

1. From your archive, open /insights-operator/gathers.json.

   The file contains a list of Insights Operator gather operations:

       {
         "name": "clusterconfig/authentication",
         "duration_in_ms": 730, 1
         "records_count": 1,
         "errors": null,
         "panic": null
       }

   1

   duration_in_ms is the amount of time in milliseconds for each gather operation.

2. Inspect each gather operation for abnormalities.

Procedure

1. Find the name of the Insights Operator pod that is currently running:

   $ INSIGHTS_OPERATOR_POD=$(oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running)

2. Copy the recent data archives from the Insights Operator container:

   $ oc cp openshift-insights/$INSIGHTS_OPERATOR_POD:/var/lib/insights-operator ./insights-data

Procedure

1. Download the dockerconfig.json file:

   $ oc extract secret/pull-secret -n openshift-config --to=.

2. Copy your "cloud.openshift.com" "auth" token from the dockerconfig.json file:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "<your_token>",
         "email": "asd@redhat.com"
       }
   }

3. Upload the archive to console.redhat.com:

   $ curl -v -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/<cluster_id>" -H "Authorization: Bearer <your_token>" -F "upload=@<path_to_archive>; type=application/vnd.redhat.openshift.periodic+tar" https://console.redhat.com/api/ingress/v1/upload

   where <cluster_id> is your cluster ID, <your_token> is the token from your pull secret, and <path_to_archive> is the path to the Insights Operator archive.

   If the operation is successful, the command returns a "request_id" and "account_number":

   Example output

   * Connection #0 to host console.redhat.com left intact
   {"request_id":"393a7cf1093e434ea8dd4ab3eb28884c","upload":{"account_number":"6274079"}}%

Verification steps

1. Log in to https://console.redhat.com/openshift.
2. Click the Clusters menu in the left pane.
3. To display the details of the cluster, click the cluster name.

4. Open the Insights Advisor tab of the cluster.

   If the upload was successful, the tab displays one of the following:

   • Your cluster passed all recommendations, if Insights Advisor did not identify any issues.
   • A list of issues that Insights Advisor has detected, prioritized by risk (low, moderate, important, and critical).