Chapter 4. Remote health monitoring with connected clusters

Procedure

1. Log in to a cluster.

2. Run the following command, which queries a cluster’s Prometheus service and returns the full set of time series data captured by Telemetry:

   $ curl -G -k -H "Authorization: Bearer $(oc whoami -t)" \
   https://$(oc get route prometheus-k8s-federate -n \
   openshift-monitoring -o jsonpath="{.spec.host}")/federate \
   --data-urlencode 'match[]={__name__=~"cluster:usage:.*"}' \
   --data-urlencode 'match[]={__name__="count:up0"}' \
   --data-urlencode 'match[]={__name__="count:up1"}' \
   --data-urlencode 'match[]={__name__="cluster_version"}' \
   --data-urlencode 'match[]={__name__="cluster_version_available_updates"}' \
   --data-urlencode 'match[]={__name__="cluster_version_capability"}' \
   --data-urlencode 'match[]={__name__="cluster_operator_up"}' \
   --data-urlencode 'match[]={__name__="cluster_operator_conditions"}' \
   --data-urlencode 'match[]={__name__="cluster_version_payload"}' \
   --data-urlencode 'match[]={__name__="cluster_installer"}' \
   --data-urlencode 'match[]={__name__="cluster_infrastructure_provider"}' \
   --data-urlencode 'match[]={__name__="cluster_feature_set"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_object_counts:sum"}' \
   --data-urlencode 'match[]={__name__="ALERTS",alertstate="firing"}' \
   --data-urlencode 'match[]={__name__="code:apiserver_request_total:rate:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:capacity_cpu_cores:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:capacity_memory_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="workload:cpu_usage_cores:sum"}' \
   --data-urlencode 'match[]={__name__="workload:memory_usage_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:virt_platform_nodes:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:node_instance_type_count:sum"}' \
   --data-urlencode 'match[]={__name__="cnv:vmi_status_running:count"}' \
   --data-urlencode 'match[]={__name__="cluster:vmi_request_cpu_cores:sum"}' \
   --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_cores:sum"}' \
   --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_sockets:sum"}' \
   --data-urlencode 'match[]={__name__="subscription_sync_total"}' \
   --data-urlencode 'match[]={__name__="olm_resolution_duration_seconds"}' \
   --data-urlencode 'match[]={__name__="csv_succeeded"}' \
   --data-urlencode 'match[]={__name__="csv_abnormal"}' \
   --data-urlencode 'match[]={__name__="cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:kubelet_volume_stats_used_bytes:provisioner:sum"}' \
   --data-urlencode 'match[]={__name__="ceph_cluster_total_bytes"}' \
   --data-urlencode 'match[]={__name__="ceph_cluster_total_used_raw_bytes"}' \
   --data-urlencode 'match[]={__name__="ceph_health_status"}' \
   --data-urlencode 'match[]={__name__="odf_system_raw_capacity_total_bytes"}' \
   --data-urlencode 'match[]={__name__="odf_system_raw_capacity_used_bytes"}' \
   --data-urlencode 'match[]={__name__="odf_system_health_status"}' \
   --data-urlencode 'match[]={__name__="job:ceph_osd_metadata:count"}' \
   --data-urlencode 'match[]={__name__="job:kube_pv:count"}' \
   --data-urlencode 'match[]={__name__="job:odf_system_pvs:count"}' \
   --data-urlencode 'match[]={__name__="job:ceph_pools_iops:total"}' \
   --data-urlencode 'match[]={__name__="job:ceph_pools_iops_bytes:total"}' \
   --data-urlencode 'match[]={__name__="job:ceph_versions_running:count"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_total_unhealthy_buckets:sum"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_bucket_count:sum"}' \
   --data-urlencode 'match[]={__name__="job:noobaa_total_object_count:sum"}' \
   --data-urlencode 'match[]={__name__="odf_system_bucket_count", system_type="OCS", system_vendor="Red Hat"}' \
   --data-urlencode 'match[]={__name__="odf_system_objects_total", system_type="OCS", system_vendor="Red Hat"}' \
   --data-urlencode 'match[]={__name__="noobaa_accounts_num"}' \
   --data-urlencode 'match[]={__name__="noobaa_total_usage"}' \
   --data-urlencode 'match[]={__name__="console_url"}' \
   --data-urlencode 'match[]={__name__="cluster:ovnkube_master_egress_routing_via_host:max"}' \
   --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_instances:max"}' \
   --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_enabled_instance_up:max"}' \
   --data-urlencode 'match[]={__name__="cluster:ingress_controller_aws_nlb_active:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:min"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:max"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:avg"}' \
   --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:median"}' \
   --data-urlencode 'match[]={__name__="cluster:openshift_route_info:tls_termination:sum"}' \
   --data-urlencode 'match[]={__name__="insightsclient_request_send_total"}' \
   --data-urlencode 'match[]={__name__="cam_app_workload_migrations"}' \
   --data-urlencode 'match[]={__name__="cluster:apiserver_current_inflight_requests:sum:max_over_time:2m"}' \
   --data-urlencode 'match[]={__name__="cluster:alertmanager_integrations:max"}' \
   --data-urlencode 'match[]={__name__="cluster:telemetry_selected_series:count"}' \
   --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_series:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_samples_appended_total:sum"}' \
   --data-urlencode 'match[]={__name__="monitoring:container_memory_working_set_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="namespace_job:scrape_series_added:topk3_sum1h"}' \
   --data-urlencode 'match[]={__name__="namespace_job:scrape_samples_post_metric_relabeling:topk3"}' \
   --data-urlencode 'match[]={__name__="monitoring:haproxy_server_http_responses_total:sum"}' \
   --data-urlencode 'match[]={__name__="rhmi_status"}' \
   --data-urlencode 'match[]={__name__="status:upgrading:version:rhoam_state:max"}' \
   --data-urlencode 'match[]={__name__="state:rhoam_critical_alerts:max"}' \
   --data-urlencode 'match[]={__name__="state:rhoam_warning_alerts:max"}' \
   --data-urlencode 'match[]={__name__="rhoam_7d_slo_percentile:max"}' \
   --data-urlencode 'match[]={__name__="rhoam_7d_slo_remaining_error_budget:max"}' \
   --data-urlencode 'match[]={__name__="cluster_legacy_scheduler_policy"}' \
   --data-urlencode 'match[]={__name__="cluster_master_schedulable"}' \
   --data-urlencode 'match[]={__name__="che_workspace_status"}' \
   --data-urlencode 'match[]={__name__="che_workspace_started_total"}' \
   --data-urlencode 'match[]={__name__="che_workspace_failure_total"}' \
   --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_sum"}' \
   --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_count"}' \
   --data-urlencode 'match[]={__name__="cco_credentials_mode"}' \
   --data-urlencode 'match[]={__name__="cluster:kube_persistentvolume_plugin_type_counts:sum"}' \
   --data-urlencode 'match[]={__name__="visual_web_terminal_sessions_total"}' \
   --data-urlencode 'match[]={__name__="acm_managed_cluster_info"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_vcenter_info:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_esxi_version_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:vsphere_node_hw_version_total:sum"}' \
   --data-urlencode 'match[]={__name__="openshift:build_by_strategy:sum"}' \
   --data-urlencode 'match[]={__name__="rhods_aggregate_availability"}' \
   --data-urlencode 'match[]={__name__="rhods_total_users"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum"}' \
   --data-urlencode 'match[]={__name__="instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile",quantile="0.99"}' \
   --data-urlencode 'match[]={__name__="jaeger_operator_instances_storage_types"}' \
   --data-urlencode 'match[]={__name__="jaeger_operator_instances_strategies"}' \
   --data-urlencode 'match[]={__name__="jaeger_operator_instances_agent_strategies"}' \
   --data-urlencode 'match[]={__name__="appsvcs:cores_by_product:sum"}' \
   --data-urlencode 'match[]={__name__="nto_custom_profiles:count"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_configmap"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_secret"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_mount_failures_total"}' \
   --data-urlencode 'match[]={__name__="openshift_csi_share_mount_requests_total"}' \
   --data-urlencode 'match[]={__name__="cluster:velero_backup_total:max"}' \
   --data-urlencode 'match[]={__name__="cluster:velero_restore_total:max"}' \
   --data-urlencode 'match[]={__name__="eo_es_storage_info"}' \
   --data-urlencode 'match[]={__name__="eo_es_redundancy_policy_info"}' \
   --data-urlencode 'match[]={__name__="eo_es_defined_delete_namespaces_total"}' \
   --data-urlencode 'match[]={__name__="eo_es_misconfigured_memory_resources_info"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_data_nodes_total:max"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_documents_created_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:eo_es_documents_deleted_total:sum"}' \
   --data-urlencode 'match[]={__name__="pod:eo_es_shards_total:max"}' \
   --data-urlencode 'match[]={__name__="eo_es_cluster_management_state_info"}' \
   --data-urlencode 'match[]={__name__="imageregistry:imagestreamtags_count:sum"}' \
   --data-urlencode 'match[]={__name__="imageregistry:operations_count:sum"}' \
   --data-urlencode 'match[]={__name__="log_logging_info"}' \
   --data-urlencode 'match[]={__name__="log_collector_error_count_total"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_pipeline_info"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_input_info"}' \
   --data-urlencode 'match[]={__name__="log_forwarder_output_info"}' \
   --data-urlencode 'match[]={__name__="cluster:log_collected_bytes_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:log_logged_bytes_total:sum"}' \
   --data-urlencode 'match[]={__name__="cluster:kata_monitor_running_shim_count:sum"}' \
   --data-urlencode 'match[]={__name__="platform:hypershift_hostedclusters:max"}' \
   --data-urlencode 'match[]={__name__="platform:hypershift_nodepools:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_bucket_claims:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_buckets_claims:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_resources:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_resources:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_buckets:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_buckets:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_accounts:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_usage:max"}' \
   --data-urlencode 'match[]={__name__="namespace:noobaa_system_health_status:max"}' \
   --data-urlencode 'match[]={__name__="ocs_advanced_feature_usage"}' \
   --data-urlencode 'match[]={__name__="os_image_url_override:sum"}'

Procedure

1. Find the name of the currently running pod for the Insights Operator:

   $ INSIGHTS_OPERATOR_POD=$(oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running)

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/$INSIGHTS_OPERATOR_POD:/var/lib/insights-operator ./insights-data

Procedure

1. Download the global cluster pull secret to your local file system.

   $ oc extract secret/pull-secret -n openshift-config --to=.

2. In a text editor, edit the .dockerconfigjson file that was downloaded.

3. Remove the cloud.openshift.com JSON entry, for example:

   "cloud.openshift.com":{"auth":"<hash>","email":"<email_address>"}

4. Save the file.

Procedure

1. Go to the Register disconnected cluster web page on the Red Hat Hybrid Cloud Console.
2. Optional: To access the Register disconnected cluster web page from the home page of the Red Hat Hybrid Cloud Console, go to the Clusters navigation menu item and then select the Register cluster button.
3. Enter your cluster’s details in the provided fields on the Register disconnected cluster page.
4. From the Subscription settings section of the page, select the subcription settings that apply to your Red Hat subscription offering.
5. To register your disconnected cluster, select the Register cluster button.

Procedure

1. Optional: To append a new pull secret to the existing pull secret, complete the following steps:

   1. Enter the following command to download the pull secret:

      $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> 1

      1

      Provide the path to the pull secret file.

   2. Enter the following command to add the new pull secret:

      $ oc registry login --registry="<registry>" \ 1
      --auth-basic="<username>:<password>" \ 2
      --to=<pull_secret_location> 3

      1

      Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>".

      2

      Provide the credentials of the new registry.

      3

      Provide the path to the pull secret file.

      Alternatively, you can perform a manual update to the pull secret file.

2. Enter the following command to update the global pull secret for your cluster:

   $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> 1

   1

   Provide the path to the new pull secret file.

   This update is rolled out to all nodes, which can take some time depending on the size of your cluster.

   Note

   As of OpenShift Container Platform 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot.

Procedure

1. Navigate to https://console.redhat.com/openshift/downloads.

2. From Tokens Pull Secret, click Download.

   The file pull-secret.txt containing your cloud.openshift.com access token in JSON format downloads:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "<your_token>",
         "email": "<email_address>"
       }
     }
   }

3. Download the global cluster pull secret to your local file system.

   $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' > pull-secret

4. Make a backup copy of your pull secret.

   $ cp pull-secret pull-secret-backup

5. Open the pull-secret file in a text editor.
6. Append the cloud.openshift.com JSON entry from pull-secret.txt into auths.
7. Save the file.

8. Update the secret in your cluster.

   oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=pull-secret

Verification

1. Navigate to the OpenShift Container Platform Web Console Overview page.
2. Insights in the Status tile reports the number of issues found.

Procedure

1. Navigate to Advisor Recommendations on OpenShift Cluster Manager Hybrid Cloud Console.

   Depending on the result, Insights Advisor displays one of the following:

   • No matching recommendations found, if Insights did not identify any issues.
   • A list of issues Insights has detected, grouped by risk (low, moderate, important, and critical).
   • No clusters yet, if Insights has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed, registered, and connected to the internet.

2. If any issues are displayed, click the > icon in front of the entry for more details.

   Depending on the issue, the details can also contain a link to more information from Red Hat about the issue.

Procedure

1. Navigate to Advisor Recommendations on OpenShift Cluster Manager Hybrid Cloud Console.

2. Click the X icons next to the Clusters Impacted and Status filters.

   You can now browse through all of the potential recommendations for your cluster.

Procedure

1. Navigate to Advisor Recommendations on OpenShift Cluster Manager Hybrid Cloud Console.
2. Optional: Use the Clusters Impacted and Status filters as needed.

3. Disable an alert by using one of the following methods:

   • To disable an alert:

     1. Click the Options menu for that alert, and then click Disable recommendation.
     2. Enter a justification note and click Save.

   • To view the clusters affected by this alert before disabling the alert:

     1. Click the name of the recommendation to disable. You are directed to the single recommendation page.
     2. Review the list of clusters in the Affected clusters section.
     3. Click Actions Disable recommendation to disable the alert for all of your clusters.
     4. Enter a justification note and click Save.

Procedure

1. Navigate to Advisor Recommendations on OpenShift Cluster Manager Hybrid Cloud Console.

2. Filter the recommendations to display on the disabled recommendations:

   1. From the Status drop-down menu, select Status.
   2. From the Filter by status drop-down menu, select Disabled.
   3. Optional: Clear the Clusters impacted filter.
3. Locate the recommendation to enable.
4. Click the Options menu , and then click Enable recommendation.

Procedure

1. Navigate to Home Overview in the OpenShift Container Platform web console.

2. Click Insights on the Status card.

   The pop-up window lists potential issues grouped by risk. Click the individual categories or View all recommendations in Insights Advisor to display more details.

Procedure

1. Navigate to Workloads Secrets.
2. On the Secrets page, select All Projects from the Project list, and then set Show default projects to on.
3. Select the openshift-config project from the Projects list.

4. Search for the support secret by using the Search by name field.

   • If the secret exists:
5. Click the Options menu , and then click Edit Secret.

6. Click Add Key/Value.

   1. In the Key field, enter disableInsightsAlerts.

   2. In the Value field, enter True.

      • If the secret does not exist:

   3. Click Create Key/value secret.

      1. In the Secret name field, enter support.
      2. In the Key field, enter disableInsightsAlerts.
      3. In the Value field, enter True.
   4. Click Create.

Procedure

1. Navigate to Workloads Secrets.
2. On the Secrets page, select All Projects from the Project list, and then set Show default projects to ON.
3. Select the openshift-config project from the Projects list.
4. Search for the support secret by using the Search by name field.
5. Click the Options menu , and then click Edit Secret.
6. For the disableInsightsAlerts key, set the Value field to false.

Procedure

1. Find the name of the running pod for the Insights Operator:

   $ oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running

2. Copy the recent data archives collected by the Insights Operator:

   $ oc cp openshift-insights/<insights_operator_pod_name>:/var/lib/insights-operator ./insights-data 1

   1

   Replace <insights_operator_pod_name> with the pod name output from the preceding command.

Procedure

1. From your archive, open /insights-operator/gathers.json.

   The file contains a list of Insights Operator gather operations:

       {
         "name": "clusterconfig/authentication",
         "duration_in_ms": 730, 1
         "records_count": 1,
         "errors": null,
         "panic": null
       }

   1

   duration_in_ms is the amount of time in milliseconds for each gather operation.

2. Inspect each gather operation for abnormalities.

Procedure

1. Navigate to Administration CustomResourceDefinitions.
2. On the CustomResourceDefinitions page, use the Search by name field to find the InsightsDataGather resource definition and click it.
3. On the CustomResourceDefinition details page, click the Instances tab.
4. Click cluster, and then click the YAML tab.

5. Disable the gather operations by performing one of the following edits to the InsightsDataGather configuration file:

   1. To disable all the gather operations, enter all under the disabledGatherers key:

      apiVersion: config.openshift.io/v1alpha1
      kind: InsightsDataGather
      metadata:
      ....
      
      spec: 1
        gatherConfig:
          disabledGatherers:
            - all 2

      1

      The spec parameter specifies gather configurations.

      2

      The all value disables all gather operations.

   2. To disable individual gather operations, enter their values under the disabledGatherers key:

      spec:
        gatherConfig:
          disabledGatherers:
            - clusterconfig/container_images 1
            - clusterconfig/host_subnets
            - workloads/workload_info

      1

      Example individual gather operation

6. Click Save.

   After you save the changes, the Insights Operator gather configurations are updated and the operations will no longer occur.

Procedure

1. Navigate to Administration CustomResourceDefinitions.
2. On the CustomResourceDefinitions page, use the Search by name field to find the InsightsDataGather resource definition and click it.
3. On the CustomResourceDefinition details page, click the Instances tab.
4. Click cluster, and then click the YAML tab.

5. Enable the gather operations by performing one of the following edits:

   • To enable all disabled gather operations, remove the gatherConfig stanza:

     apiVersion: config.openshift.io/v1alpha1
     kind: InsightsDataGather
     metadata:
     ....
     
     spec:
       gatherConfig: 1
         disabledGatherers: all

     1

     Remove the gatherConfig stanza to enable all gather operations.

   • To enable individual gather operations, remove their values under the disabledGatherers key:

     spec:
       gatherConfig:
         disabledGatherers:
           - clusterconfig/container_images 1
           - clusterconfig/host_subnets
           - workloads/workload_info

     1

     Remove one or more gather operations.

6. Click Save.

   After you save the changes, the Insights Operator gather configurations are updated and the affected gather operations start.

Procedure

1. Navigate to Workloads Secrets.
2. On the Secrets page, select All Projects from the Project list, and then set Show default projects to on.
3. Select the openshift-config project from the Project list.
4. Search for the support secret using the Search by name field. If it does not exist, click Create Key/value secret to create it.
5. Click the Options menu for the secret, and then click Edit Secret.
6. Click Add Key/Value.
7. Enter an attribute name with an appropriate value (see table above), and click Save.
8. Repeat the above steps for any additional configurations.

Procedure

1. Create a file named gather-job.yaml using this template:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: insights-operator-job
     annotations:
       config.openshift.io/inject-proxy: insights-operator
   spec:
     backoffLimit: 6
     ttlSecondsAfterFinished: 600
     template:
       spec:
         restartPolicy: OnFailure
         serviceAccountName: operator
         nodeSelector:
           beta.kubernetes.io/os: linux
           node-role.kubernetes.io/master: ""
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/master
           operator: Exists
         - effect: NoExecute
           key: node.kubernetes.io/unreachable
           operator: Exists
           tolerationSeconds: 900
         - effect: NoExecute
           key: node.kubernetes.io/not-ready
           operator: Exists
           tolerationSeconds: 900
         volumes:
         - name: snapshots
           emptyDir: {}
         - name: service-ca-bundle
           configMap:
             name: service-ca-bundle
             optional: true
         initContainers:
         - name: insights-operator
           image: quay.io/openshift/origin-insights-operator:latest
           terminationMessagePolicy: FallbackToLogsOnError
           volumeMounts:
           - name: snapshots
             mountPath: /var/lib/insights-operator
           - name: service-ca-bundle
             mountPath: /var/run/configmaps/service-ca-bundle
             readOnly: true
           ports:
           - containerPort: 8443
             name: https
           resources:
             requests:
               cpu: 10m
               memory: 70Mi
           args:
           - gather
           - -v=4
           - --config=/etc/insights-operator/server.yaml
         containers:
           - name: sleepy
             image: quay.io/openshift/origin-base:latest
             args:
               - /bin/sh
               - -c
               - sleep 10m
             volumeMounts: [{name: snapshots, mountPath: /var/lib/insights-operator}]

2. Copy your insights-operator image version:

   $ oc get -n openshift-insights deployment insights-operator -o yaml

   Example output

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: insights-operator
     namespace: openshift-insights
   # ...
   spec:
     template:
   # ...
       spec:
         containers:
         - args:
   # ...
           image: registry.ci.openshift.org/ocp/4.15-2023-10-12-212500@sha256:a0aa581400805ad0... 1
   # ...

   1

   Specifies your insights-operator image version.

3. Paste your image version in gather-job.yaml:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: insights-operator-job
   # ...
   spec:
   # ...
     template:
       spec:
       initContainers:
       - name: insights-operator
         image: image: registry.ci.openshift.org/ocp/4.15-2023-10-12-212500@sha256:a0aa581400805ad0... 1
         terminationMessagePolicy: FallbackToLogsOnError
         volumeMounts:

   1

   Replace any existing value with your insights-operator image version.

4. Create the gather job:

   $ oc apply -n openshift-insights -f gather-job.yaml

5. Find the name of the job pod:

   $ oc describe -n openshift-insights job/insights-operator-job

   Example output

   Name:             insights-operator-job
   Namespace:        openshift-insights
   # ...
   Events:
     Type    Reason            Age    From            Message
     ----    ------            ----   ----            -------
     Normal  SuccessfulCreate  7m18s  job-controller  Created pod: insights-operator-job-<your_job>

   where

   insights-operator-job-<your_job> is the name of the pod.

6. Verify that the operation has finished:

   $ oc logs -n openshift-insights insights-operator-job-<your_job> insights-operator

   Example output

   I0407 11:55:38.192084       1 diskrecorder.go:34] Wrote 108 records to disk in 33ms

7. Save the created archive:

   $ oc cp openshift-insights/insights-operator-job-<your_job>:/var/lib/insights-operator ./insights-data

8. Clean up the job:

   $ oc delete -n openshift-insights job insights-operator-job

Procedure

1. Download the dockerconfig.json file:

   $ oc extract secret/pull-secret -n openshift-config --to=.

2. Copy your "cloud.openshift.com" "auth" token from the dockerconfig.json file:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "<your_token>",
         "email": "asd@redhat.com"
       }
   }

3. Upload the archive to console.redhat.com:

   $ curl -v -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/<cluster_id>" -H "Authorization: Bearer <your_token>" -F "upload=@<path_to_archive>; type=application/vnd.redhat.openshift.periodic+tar" https://console.redhat.com/api/ingress/v1/upload

   where <cluster_id> is your cluster ID, <your_token> is the token from your pull secret, and <path_to_archive> is the path to the Insights Operator archive.

   If the operation is successful, the command returns a "request_id" and "account_number":

   Example output

   * Connection #0 to host console.redhat.com left intact
   {"request_id":"393a7cf1093e434ea8dd4ab3eb28884c","upload":{"account_number":"6274079"}}%

Verification steps

1. Log in to https://console.redhat.com/openshift.
2. Click the Clusters menu in the left pane.
3. To display the details of the cluster, click the cluster name.

4. Open the Insights Advisor tab of the cluster.

   If the upload was successful, the tab displays one of the following:

   • Your cluster passed all recommendations, if Insights Advisor did not identify any issues.
   • A list of issues that Insights Advisor has detected, prioritized by risk (low, moderate, important, and critical).

Procedure

1. Navigate to Workloads Secrets.
2. Select the openshift-config project.
3. Search for the support secret using the Search by name field. If it does not exist, click Create Key/value secret to create it.
4. Click the Options menu , and then click Edit Secret.
5. Click Add Key/Value.
6. Create a key named enableGlobalObfuscation with a value of true, and click Save.
7. Navigate to Workloads Pods
8. Select the openshift-insights project.
9. Find the insights-operator pod.
10. To restart the insights-operator pod, click the Options menu , and then click Delete Pod.

Verification

1. Navigate to Workloads Secrets.
2. Select the openshift-insights project.
3. Search for the obfuscation-translation-table secret using the Search by name field.

Procedure

1. Navigate to Workloads Secrets.
2. Select the openshift-config project.

3. Search for the support secret by using the Search by name field. If it does not exist, click Create Key/value secret to create it.

   • If the secret exists:

     1. Click the Options menu , and then click Edit Secret.

     2. Click Add Key/Value.

        1. In the Key field, enter scaInterval.
        2. In the Value field, enter 1h.

   • If the secret does not exist:

     1. Click Create Key/value secret.

        1. In the Secret name field, enter support.
        2. In the Key field, enter scaInterval.
        3. In the Value field, enter 1h.
     2. Click Create.
   Note

   The interval 1h can also be entered as 60m for 60 minutes.

Procedure

1. Navigate to Workloads Secrets.
2. Select the openshift-config project.

3. Search for the support secret using the Search by name field.

   • If the secret exists:

     1. Click the Options menu , and then click Edit Secret.

     2. Click Add Key/Value.

        1. In the Key field, enter scaPullDisabled.
        2. In the Value field, enter true.

   • If the secret does not exist:

     1. Click Create Key/value secret.

        1. In the Secret name field, enter support.
        2. In the Key field, enter scaPullDisabled.
        3. In the Value field, enter true.
     2. Click Create.

   The simple content access entitlement import is now disabled.

   Note

   To enable the simple content access import again, edit the support secret and delete the scaPullDisabled key.

Procedure

1. Navigate to Workloads Secrets.
2. Select the openshift-config project.
3. Search for the support secret by using the Search by name field.
4. Click the Options menu , and then click Edit Secret.

5. For the scaPullDisabled key, set the Value field to false.

   The simple content access entitlement import is now disabled.