Search

Chapter 4. Remote health monitoring with connected clusters

download PDF

4.1. About remote health monitoring

OpenShift Container Platform collects telemetry and configuration data about your cluster and reports it to Red Hat by using the Telemeter Client and the Insights Operator. The data that is provided to Red Hat enables the benefits outlined in this document.

A cluster that reports data to Red Hat through Telemetry and the Insights Operator is considered a connected cluster.

Telemetry is the term that Red Hat uses to describe the information being sent to Red Hat by the OpenShift Container Platform Telemeter Client. Lightweight attributes are sent from connected clusters to Red Hat to enable subscription management automation, monitor the health of clusters, assist with support, and improve customer experience.

The Insights Operator gathers OpenShift Container Platform configuration data and sends it to Red Hat. The data is used to produce insights about potential issues that a cluster might be exposed to. These insights are communicated to cluster administrators on OpenShift Cluster Manager.

More information is provided in this document about these two processes.

Telemetry and Insights Operator benefits

Telemetry and the Insights Operator enable the following benefits for end-users:

  • Enhanced identification and resolution of issues. Events that might seem normal to an end-user can be observed by Red Hat from a broader perspective across a fleet of clusters. Some issues can be more rapidly identified from this point of view and resolved without an end-user needing to open a support case or file a Jira issue.
  • Advanced release management. OpenShift Container Platform offers the candidate, fast, and stable release channels, which enable you to choose an update strategy. The graduation of a release from fast to stable is dependent on the success rate of updates and on the events seen during upgrades. With the information provided by connected clusters, Red Hat can improve the quality of releases to stable channels and react more rapidly to issues found in the fast channels.
  • Targeted prioritization of new features and functionality. The data collected provides insights about which areas of OpenShift Container Platform are used most. With this information, Red Hat can focus on developing the new features and functionality that have the greatest impact for our customers.
  • A streamlined support experience. You can provide a cluster ID for a connected cluster when creating a support ticket on the Red Hat Customer Portal. This enables Red Hat to deliver a streamlined support experience that is specific to your cluster, by using the connected information. This document provides more information about that enhanced support experience.
  • Predictive analytics. The insights displayed for your cluster on OpenShift Cluster Manager are enabled by the information collected from connected clusters. Red Hat is investing in applying deep learning, machine learning, and artificial intelligence automation to help identify issues that OpenShift Container Platform clusters are exposed to.

4.1.1. About Telemetry

Telemetry sends a carefully chosen subset of the cluster monitoring metrics to Red Hat. The Telemeter Client fetches the metrics values every four minutes and thirty seconds and uploads the data to Red Hat. These metrics are described in this document.

This stream of data is used by Red Hat to monitor the clusters in real-time and to react as necessary to problems that impact our customers. It also allows Red Hat to roll out OpenShift Container Platform upgrades to customers to minimize service impact and continuously improve the upgrade experience.

This debugging information is available to Red Hat Support and Engineering teams with the same restrictions as accessing data reported through support cases. All connected cluster information is used by Red Hat to help make OpenShift Container Platform better and more intuitive to use.

Additional resources

4.1.1.1. Information collected by Telemetry

The following information is collected by Telemetry:

4.1.1.1.1. System information
  • Version information, including the OpenShift Container Platform cluster version and installed update details that are used to determine update version availability
  • Update information, including the number of updates available per cluster, the channel and image repository used for an update, update progress information, and the number of errors that occur in an update
  • The unique random identifier that is generated during an installation
  • Configuration details that help Red Hat Support to provide beneficial support for customers, including node configuration at the cloud infrastructure level, hostnames, IP addresses, Kubernetes pod names, namespaces, and services
  • The OpenShift Container Platform framework components installed in a cluster and their condition and status
  • Events for all namespaces listed as "related objects" for a degraded Operator
  • Information about degraded software
  • Information about the validity of certificates
  • The name of the provider platform that OpenShift Container Platform is deployed on and the data center location
4.1.1.1.2. Sizing Information
  • Sizing information about clusters, machine types, and machines, including the number of CPU cores and the amount of RAM used for each
  • The number of etcd members and the number of objects stored in the etcd cluster
  • Number of application builds by build strategy type
4.1.1.1.3. Usage information
  • Usage information about components, features, and extensions
  • Usage details about Technology Previews and unsupported configurations

Telemetry does not collect identifying information such as usernames or passwords. Red Hat does not intend to collect personal information. If Red Hat discovers that personal information has been inadvertently received, Red Hat will delete such information. To the extent that any telemetry data constitutes personal data, please refer to the Red Hat Privacy Statement for more information about Red Hat’s privacy practices.

Additional resources

4.1.2. About the Insights Operator

The Insights Operator periodically gathers configuration and component failure status and, by default, reports that data every two hours to Red Hat. This information enables Red Hat to assess configuration and deeper failure data than is reported through Telemetry.

Users of OpenShift Container Platform can display the report of each cluster in the Insights Advisor service on Red Hat Hybrid Cloud Console. If any issues have been identified, Insights provides further details and, if available, steps on how to solve a problem.

The Insights Operator does not collect identifying information, such as user names, passwords, or certificates. See Red Hat Insights Data & Application Security for information about Red Hat Insights data collection and controls.

Red Hat uses all connected cluster information to:

  • Identify potential cluster issues and provide a solution and preventive actions in the Insights Advisor service on Red Hat Hybrid Cloud Console
  • Improve OpenShift Container Platform by providing aggregated and critical information to product and support teams
  • Make OpenShift Container Platform more intuitive

Additional resources

4.1.2.1. Information collected by the Insights Operator

The following information is collected by the Insights Operator:

  • General information about your cluster and its components to identify issues that are specific to your OpenShift Container Platform version and environment
  • Configuration files, such as the image registry configuration, of your cluster to determine incorrect settings and issues that are specific to parameters you set
  • Errors that occur in the cluster components
  • Progress information of running updates, and the status of any component upgrades
  • Details of the platform that OpenShift Container Platform is deployed on, such as Amazon Web Services, and the region that the cluster is located in
  • Cluster workload information transformed into discreet Secure Hash Algorithm (SHA) values, which allows Red Hat to assess workloads for security and version vulnerabilities without disclosing sensitive details
  • If an Operator reports an issue, information is collected about core OpenShift Container Platform pods in the openshift-* and kube-* projects. This includes state, resource, security context, volume information, and more.

Additional resources

4.1.3. Understanding Telemetry and Insights Operator data flow

The Telemeter Client collects selected time series data from the Prometheus API. The time series data is uploaded to api.openshift.com every four minutes and thirty seconds for processing.

The Insights Operator gathers selected data from the Kubernetes API and the Prometheus API into an archive. The archive is uploaded to OpenShift Cluster Manager every two hours for processing. The Insights Operator also downloads the latest Insights analysis from OpenShift Cluster Manager. This is used to populate the Insights status pop-up that is included in the Overview page in the OpenShift Container Platform web console.

All of the communication with Red Hat occurs over encrypted channels by using Transport Layer Security (TLS) and mutual certificate authentication. All of the data is encrypted in transit and at rest.

Access to the systems that handle customer data is controlled through multi-factor authentication and strict authorization controls. Access is granted on a need-to-know basis and is limited to required operations.

Telemetry and Insights Operator data flow

Telemetry and Insights Operator data flow

Additional resources

  • See Monitoring overview for more information about the OpenShift Container Platform monitoring stack.
  • See Configuring your firewall for details about configuring a firewall and enabling endpoints for Telemetry and Insights

4.1.4. Additional details about how remote health monitoring data is used

The information collected to enable remote health monitoring is detailed in Information collected by Telemetry and Information collected by the Insights Operator.

As further described in the preceding sections of this document, Red Hat collects data about your use of the Red Hat Product(s) for purposes such as providing support and upgrades, optimizing performance or configuration, minimizing service impacts, identifying and remediating threats, troubleshooting, improving the offerings and user experience, responding to issues, and for billing purposes if applicable.

Collection safeguards

Red Hat employs technical and organizational measures designed to protect the telemetry and configuration data.

Sharing

Red Hat may share the data collected through Telemetry and the Insights Operator internally within Red Hat to improve your user experience. Red Hat may share telemetry and configuration data with its business partners in an aggregated form that does not identify customers to help the partners better understand their markets and their customers’ use of Red Hat offerings or to ensure the successful integration of products jointly supported by those partners.

Third parties

Red Hat may engage certain third parties to assist in the collection, analysis, and storage of the Telemetry and configuration data.

User control / enabling and disabling telemetry and configuration data collection

You may disable OpenShift Container Platform Telemetry and the Insights Operator by following the instructions in Opting out of remote health reporting.

4.2. Showing data collected by remote health monitoring

As an administrator, you can review the metrics collected by Telemetry and the Insights Operator.

4.2.1. Showing data collected by Telemetry

You can view the cluster and components time series data captured by Telemetry.

Prerequisites

  • You have installed the OpenShift Container Platform CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role or the cluster-monitoring-view role.

Procedure

  1. Log in to a cluster.
  2. Run the following command, which queries a cluster’s Prometheus service and returns the full set of time series data captured by Telemetry:

    $ curl -G -k -H "Authorization: Bearer $(oc whoami -t)" \
    https://$(oc get route prometheus-k8s-federate -n \
    openshift-monitoring -o jsonpath="{.spec.host}")/federate \
    --data-urlencode 'match[]={__name__=~"cluster:usage:.*"}' \
    --data-urlencode 'match[]={__name__="count:up0"}' \
    --data-urlencode 'match[]={__name__="count:up1"}' \
    --data-urlencode 'match[]={__name__="cluster_version"}' \
    --data-urlencode 'match[]={__name__="cluster_version_available_updates"}' \
    --data-urlencode 'match[]={__name__="cluster_version_capability"}' \
    --data-urlencode 'match[]={__name__="cluster_operator_up"}' \
    --data-urlencode 'match[]={__name__="cluster_operator_conditions"}' \
    --data-urlencode 'match[]={__name__="cluster_version_payload"}' \
    --data-urlencode 'match[]={__name__="cluster_installer"}' \
    --data-urlencode 'match[]={__name__="cluster_infrastructure_provider"}' \
    --data-urlencode 'match[]={__name__="cluster_feature_set"}' \
    --data-urlencode 'match[]={__name__="instance:etcd_object_counts:sum"}' \
    --data-urlencode 'match[]={__name__="ALERTS",alertstate="firing"}' \
    --data-urlencode 'match[]={__name__="code:apiserver_request_total:rate:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:capacity_cpu_cores:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:capacity_memory_bytes:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:cpu_usage_cores:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:memory_usage_bytes:sum"}' \
    --data-urlencode 'match[]={__name__="openshift:cpu_usage_cores:sum"}' \
    --data-urlencode 'match[]={__name__="openshift:memory_usage_bytes:sum"}' \
    --data-urlencode 'match[]={__name__="workload:cpu_usage_cores:sum"}' \
    --data-urlencode 'match[]={__name__="workload:memory_usage_bytes:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:virt_platform_nodes:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:node_instance_type_count:sum"}' \
    --data-urlencode 'match[]={__name__="cnv:vmi_status_running:count"}' \
    --data-urlencode 'match[]={__name__="cluster:vmi_request_cpu_cores:sum"}' \
    --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_cores:sum"}' \
    --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_sockets:sum"}' \
    --data-urlencode 'match[]={__name__="subscription_sync_total"}' \
    --data-urlencode 'match[]={__name__="olm_resolution_duration_seconds"}' \
    --data-urlencode 'match[]={__name__="csv_succeeded"}' \
    --data-urlencode 'match[]={__name__="csv_abnormal"}' \
    --data-urlencode 'match[]={__name__="cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:kubelet_volume_stats_used_bytes:provisioner:sum"}' \
    --data-urlencode 'match[]={__name__="ceph_cluster_total_bytes"}' \
    --data-urlencode 'match[]={__name__="ceph_cluster_total_used_raw_bytes"}' \
    --data-urlencode 'match[]={__name__="ceph_health_status"}' \
    --data-urlencode 'match[]={__name__="odf_system_raw_capacity_total_bytes"}' \
    --data-urlencode 'match[]={__name__="odf_system_raw_capacity_used_bytes"}' \
    --data-urlencode 'match[]={__name__="odf_system_health_status"}' \
    --data-urlencode 'match[]={__name__="job:ceph_osd_metadata:count"}' \
    --data-urlencode 'match[]={__name__="job:kube_pv:count"}' \
    --data-urlencode 'match[]={__name__="job:odf_system_pvs:count"}' \
    --data-urlencode 'match[]={__name__="job:ceph_pools_iops:total"}' \
    --data-urlencode 'match[]={__name__="job:ceph_pools_iops_bytes:total"}' \
    --data-urlencode 'match[]={__name__="job:ceph_versions_running:count"}' \
    --data-urlencode 'match[]={__name__="job:noobaa_total_unhealthy_buckets:sum"}' \
    --data-urlencode 'match[]={__name__="job:noobaa_bucket_count:sum"}' \
    --data-urlencode 'match[]={__name__="job:noobaa_total_object_count:sum"}' \
    --data-urlencode 'match[]={__name__="odf_system_bucket_count", system_type="OCS", system_vendor="Red Hat"}' \
    --data-urlencode 'match[]={__name__="odf_system_objects_total", system_type="OCS", system_vendor="Red Hat"}' \
    --data-urlencode 'match[]={__name__="noobaa_accounts_num"}' \
    --data-urlencode 'match[]={__name__="noobaa_total_usage"}' \
    --data-urlencode 'match[]={__name__="console_url"}' \
    --data-urlencode 'match[]={__name__="cluster:ovnkube_master_egress_routing_via_host:max"}' \
    --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_instances:max"}' \
    --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_enabled_instance_up:max"}' \
    --data-urlencode 'match[]={__name__="cluster:ingress_controller_aws_nlb_active:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:min"}' \
    --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:max"}' \
    --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:avg"}' \
    --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:median"}' \
    --data-urlencode 'match[]={__name__="cluster:openshift_route_info:tls_termination:sum"}' \
    --data-urlencode 'match[]={__name__="insightsclient_request_send_total"}' \
    --data-urlencode 'match[]={__name__="cam_app_workload_migrations"}' \
    --data-urlencode 'match[]={__name__="cluster:apiserver_current_inflight_requests:sum:max_over_time:2m"}' \
    --data-urlencode 'match[]={__name__="cluster:alertmanager_integrations:max"}' \
    --data-urlencode 'match[]={__name__="cluster:telemetry_selected_series:count"}' \
    --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_series:sum"}' \
    --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_samples_appended_total:sum"}' \
    --data-urlencode 'match[]={__name__="monitoring:container_memory_working_set_bytes:sum"}' \
    --data-urlencode 'match[]={__name__="namespace_job:scrape_series_added:topk3_sum1h"}' \
    --data-urlencode 'match[]={__name__="namespace_job:scrape_samples_post_metric_relabeling:topk3"}' \
    --data-urlencode 'match[]={__name__="monitoring:haproxy_server_http_responses_total:sum"}' \
    --data-urlencode 'match[]={__name__="rhmi_status"}' \
    --data-urlencode 'match[]={__name__="status:upgrading:version:rhoam_state:max"}' \
    --data-urlencode 'match[]={__name__="state:rhoam_critical_alerts:max"}' \
    --data-urlencode 'match[]={__name__="state:rhoam_warning_alerts:max"}' \
    --data-urlencode 'match[]={__name__="rhoam_7d_slo_percentile:max"}' \
    --data-urlencode 'match[]={__name__="rhoam_7d_slo_remaining_error_budget:max"}' \
    --data-urlencode 'match[]={__name__="cluster_legacy_scheduler_policy"}' \
    --data-urlencode 'match[]={__name__="cluster_master_schedulable"}' \
    --data-urlencode 'match[]={__name__="che_workspace_status"}' \
    --data-urlencode 'match[]={__name__="che_workspace_started_total"}' \
    --data-urlencode 'match[]={__name__="che_workspace_failure_total"}' \
    --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_sum"}' \
    --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_count"}' \
    --data-urlencode 'match[]={__name__="cco_credentials_mode"}' \
    --data-urlencode 'match[]={__name__="cluster:kube_persistentvolume_plugin_type_counts:sum"}' \
    --data-urlencode 'match[]={__name__="visual_web_terminal_sessions_total"}' \
    --data-urlencode 'match[]={__name__="acm_managed_cluster_info"}' \
    --data-urlencode 'match[]={__name__="cluster:vsphere_vcenter_info:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:vsphere_esxi_version_total:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:vsphere_node_hw_version_total:sum"}' \
    --data-urlencode 'match[]={__name__="openshift:build_by_strategy:sum"}' \
    --data-urlencode 'match[]={__name__="rhods_aggregate_availability"}' \
    --data-urlencode 'match[]={__name__="rhods_total_users"}' \
    --data-urlencode 'match[]={__name__="instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile",quantile="0.99"}' \
    --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_bytes:sum"}' \
    --data-urlencode 'match[]={__name__="instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile",quantile="0.99"}' \
    --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum"}' \
    --data-urlencode 'match[]={__name__="instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile",quantile="0.99"}' \
    --data-urlencode 'match[]={__name__="jaeger_operator_instances_storage_types"}' \
    --data-urlencode 'match[]={__name__="jaeger_operator_instances_strategies"}' \
    --data-urlencode 'match[]={__name__="jaeger_operator_instances_agent_strategies"}' \
    --data-urlencode 'match[]={__name__="appsvcs:cores_by_product:sum"}' \
    --data-urlencode 'match[]={__name__="nto_custom_profiles:count"}' \
    --data-urlencode 'match[]={__name__="openshift_csi_share_configmap"}' \
    --data-urlencode 'match[]={__name__="openshift_csi_share_secret"}' \
    --data-urlencode 'match[]={__name__="openshift_csi_share_mount_failures_total"}' \
    --data-urlencode 'match[]={__name__="openshift_csi_share_mount_requests_total"}' \
    --data-urlencode 'match[]={__name__="cluster:velero_backup_total:max"}' \
    --data-urlencode 'match[]={__name__="cluster:velero_restore_total:max"}' \
    --data-urlencode 'match[]={__name__="eo_es_storage_info"}' \
    --data-urlencode 'match[]={__name__="eo_es_redundancy_policy_info"}' \
    --data-urlencode 'match[]={__name__="eo_es_defined_delete_namespaces_total"}' \
    --data-urlencode 'match[]={__name__="eo_es_misconfigured_memory_resources_info"}' \
    --data-urlencode 'match[]={__name__="cluster:eo_es_data_nodes_total:max"}' \
    --data-urlencode 'match[]={__name__="cluster:eo_es_documents_created_total:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:eo_es_documents_deleted_total:sum"}' \
    --data-urlencode 'match[]={__name__="pod:eo_es_shards_total:max"}' \
    --data-urlencode 'match[]={__name__="eo_es_cluster_management_state_info"}' \
    --data-urlencode 'match[]={__name__="imageregistry:imagestreamtags_count:sum"}' \
    --data-urlencode 'match[]={__name__="imageregistry:operations_count:sum"}' \
    --data-urlencode 'match[]={__name__="log_logging_info"}' \
    --data-urlencode 'match[]={__name__="log_collector_error_count_total"}' \
    --data-urlencode 'match[]={__name__="log_forwarder_pipeline_info"}' \
    --data-urlencode 'match[]={__name__="log_forwarder_input_info"}' \
    --data-urlencode 'match[]={__name__="log_forwarder_output_info"}' \
    --data-urlencode 'match[]={__name__="cluster:log_collected_bytes_total:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:log_logged_bytes_total:sum"}' \
    --data-urlencode 'match[]={__name__="cluster:kata_monitor_running_shim_count:sum"}' \
    --data-urlencode 'match[]={__name__="platform:hypershift_hostedclusters:max"}' \
    --data-urlencode 'match[]={__name__="platform:hypershift_nodepools:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_bucket_claims:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_buckets_claims:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_resources:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_resources:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_buckets:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_buckets:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_accounts:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_usage:max"}' \
    --data-urlencode 'match[]={__name__="namespace:noobaa_system_health_status:max"}' \
    --data-urlencode 'match[]={__name__="ocs_advanced_feature_usage"}' \
    --data-urlencode 'match[]={__name__="os_image_url_override:sum"}'

4.2.2. Showing data collected by the Insights Operator

You can review the data that is collected by the Insights Operator.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

Procedure

  1. Find the name of the currently running pod for the Insights Operator:

    $ INSIGHTS_OPERATOR_POD=$(oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running)
  2. Copy the recent data archives collected by the Insights Operator:

    $ oc cp openshift-insights/$INSIGHTS_OPERATOR_POD:/var/lib/insights-operator ./insights-data

The recent Insights Operator archives are now available in the insights-data directory.

4.3. Opting out of remote health reporting

You may choose to opt out of reporting health and usage data for your cluster.

To opt out of remote health reporting, you must:

  1. Modify the global cluster pull secret to disable remote health reporting.
  2. Update the cluster to use this modified pull secret.

4.3.1. Consequences of disabling remote health reporting

In OpenShift Container Platform, customers can opt out of reporting usage information. However, connected clusters allow Red Hat to react more quickly to problems and better support our customers, as well as better understand how product upgrades impact clusters. Connected clusters also help to simplify the subscription and entitlement process and enable the OpenShift Cluster Manager service to provide an overview of your clusters and their subscription status.

Red Hat strongly recommends leaving health and usage reporting enabled for pre-production and test clusters even if it is necessary to opt out for production clusters. This allows Red Hat to be a participant in qualifying OpenShift Container Platform in your environments and react more rapidly to product issues.

Some of the consequences of opting out of having a connected cluster are:

  • Red Hat will not be able to monitor the success of product upgrades or the health of your clusters without a support case being opened.
  • Red Hat will not be able to use configuration data to better triage customer support cases and identify which configurations our customers find important.
  • The OpenShift Cluster Manager will not show data about your clusters including health and usage information.
  • Your subscription entitlement information must be manually entered via console.redhat.com without the benefit of automatic usage reporting.

In restricted networks, Telemetry and Insights data can still be reported through appropriate configuration of your proxy.

4.3.2. Modifying the global cluster pull secret to disable remote health reporting

You can modify your existing global cluster pull secret to disable remote health reporting. This disables both Telemetry and the Insights Operator.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Download the global cluster pull secret to your local file system.

    $ oc extract secret/pull-secret -n openshift-config --to=.
  2. In a text editor, edit the .dockerconfigjson file that was downloaded.
  3. Remove the cloud.openshift.com JSON entry, for example:

    "cloud.openshift.com":{"auth":"<hash>","email":"<email_address>"}
  4. Save the file.

You can now update your cluster to use this modified pull secret.

4.3.3. Registering your disconnected cluster

Register your disconnected OpenShift Container Platform cluster on the Red Hat Hybrid Cloud Console so that your cluster is not impacted by the consequences listed in the section named "Consequences of disabling remote health reporting".

Important

By registering your disconnected cluster, you can continue to report your subscription usage to Red Hat. In turn, Red Hat can return accurate usage and capacity trends associated with your subscription, so that you can use the returned information to better organize subscription allocations across all of your resources.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as cluster-admin.
  • You can log in to the Red Hat Hybrid Cloud Console.

Procedure

  1. Go to the Register disconnected cluster web page on the Red Hat Hybrid Cloud Console.
  2. Optional: To access the Register disconnected cluster web page from the home page of the Red Hat Hybrid Cloud Console, go to the Clusters navigation menu item and then select the Register cluster button.
  3. Enter your cluster’s details in the provided fields on the Register disconnected cluster page.
  4. From the Subscription settings section of the page, select the subcription settings that apply to your Red Hat subscription offering.
  5. To register your disconnected cluster, select the Register cluster button.

4.3.4. Updating the global cluster pull secret

You can update the global pull secret for your cluster by either replacing the current pull secret or appending a new pull secret.

The procedure is required when users use a separate registry to store images than the registry used during installation.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Optional: To append a new pull secret to the existing pull secret, complete the following steps:

    1. Enter the following command to download the pull secret:

      $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> 1
      1
      Provide the path to the pull secret file.
    2. Enter the following command to add the new pull secret:

      $ oc registry login --registry="<registry>" \ 1
      --auth-basic="<username>:<password>" \ 2
      --to=<pull_secret_location> 3
      1
      Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>".
      2
      Provide the credentials of the new registry.
      3
      Provide the path to the pull secret file.

      Alternatively, you can perform a manual update to the pull secret file.

  2. Enter the following command to update the global pull secret for your cluster:

    $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> 1
    1
    Provide the path to the new pull secret file.

    This update is rolled out to all nodes, which can take some time depending on the size of your cluster.

    Note

    As of OpenShift Container Platform 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot.

4.4. Enabling remote health reporting

If you or your organization have disabled remote health reporting, you can enable this feature again. You can see that remote health reporting is disabled from the message "Insights not available" in the Status tile on the OpenShift Container Platform Web Console Overview page.

To enable remote health reporting, you must Modify the global cluster pull secret with a new authorization token.

Note

Enabling remote health reporting enables both Insights Operator and Telemetry.

4.4.1. Modifying your global cluster pull secret to enable remote health reporting

You can modify your existing global cluster pull secret to enable remote health reporting. If you have previously disabled remote health monitoring, you must first download a new pull secret with your console.openshift.com access token from Red Hat OpenShift Cluster Manager.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.
  • Access to OpenShift Cluster Manager.

Procedure

  1. Navigate to https://console.redhat.com/openshift/downloads.
  2. From Tokens Pull Secret, click Download.

    The file pull-secret.txt containing your cloud.openshift.com access token in JSON format downloads:

    {
      "auths": {
        "cloud.openshift.com": {
          "auth": "<your_token>",
          "email": "<email_address>"
        }
      }
    }
  3. Download the global cluster pull secret to your local file system.

    $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' > pull-secret
  4. Make a backup copy of your pull secret.

    $ cp pull-secret pull-secret-backup
  5. Open the pull-secret file in a text editor.
  6. Append the cloud.openshift.com JSON entry from pull-secret.txt into auths.
  7. Save the file.
  8. Update the secret in your cluster.

    oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=pull-secret

It may take several minutes for the secret to update and your cluster to begin reporting.

Verification

  1. Navigate to the OpenShift Container Platform Web Console Overview page.
  2. Insights in the Status tile reports the number of issues found.

4.5. Using Insights to identify issues with your cluster

Insights repeatedly analyzes the data Insights Operator sends. Users of OpenShift Container Platform can display the report in the Insights Advisor service on Red Hat Hybrid Cloud Console.

4.5.1. About Red Hat Insights Advisor for OpenShift Container Platform

You can use Insights Advisor to assess and monitor the health of your OpenShift Container Platform clusters. Whether you are concerned about individual clusters, or with your whole infrastructure, it is important to be aware of the exposure of your cluster infrastructure to issues that can affect service availability, fault tolerance, performance, or security.

Using cluster data collected by the Insights Operator, Insights repeatedly compares that data against a library of recommendations. Each recommendation is a set of cluster-environment conditions that can leave OpenShift Container Platform clusters at risk. The results of the Insights analysis are available in the Insights Advisor service on Red Hat Hybrid Cloud Console. In the Console, you can perform the following actions:

  • See clusters impacted by a specific recommendation.
  • Use robust filtering capabilities to refine your results to those recommendations.
  • Learn more about individual recommendations, details about the risks they present, and get resolutions tailored to your individual clusters.
  • Share results with other stakeholders.

4.5.2. Understanding Insights Advisor recommendations

Insights Advisor bundles information about various cluster states and component configurations that can negatively affect the service availability, fault tolerance, performance, or security of your clusters. This information set is called a recommendation in Insights Advisor and includes the following information:

  • Name: A concise description of the recommendation
  • Added: When the recommendation was published to the Insights Advisor archive
  • Category: Whether the issue has the potential to negatively affect service availability, fault tolerance, performance, or security
  • Total risk: A value derived from the likelihood that the condition will negatively affect your infrastructure, and the impact on operations if that were to happen
  • Clusters: A list of clusters on which a recommendation is detected
  • Description: A brief synopsis of the issue, including how it affects your clusters
  • Link to associated topics: More information from Red Hat about the issue

4.5.3. Displaying potential issues with your cluster

This section describes how to display the Insights report in Insights Advisor on OpenShift Cluster Manager.

Note that Insights repeatedly analyzes your cluster and shows the latest results. These results can change, for example, if you fix an issue or a new issue has been detected.

Prerequisites

Procedure

  1. Navigate to Advisor Recommendations on OpenShift Cluster Manager.

    Depending on the result, Insights Advisor displays one of the following:

    • No matching recommendations found, if Insights did not identify any issues.
    • A list of issues Insights has detected, grouped by risk (low, moderate, important, and critical).
    • No clusters yet, if Insights has not yet analyzed the cluster. The analysis starts shortly after the cluster has been installed, registered, and connected to the internet.
  2. If any issues are displayed, click the > icon in front of the entry for more details.

    Depending on the issue, the details can also contain a link to more information from Red Hat about the issue.

4.5.4. Displaying all Insights Advisor recommendations

The Recommendations view, by default, only displays the recommendations that are detected on your clusters. However, you can view all of the recommendations in the advisor archive.

Prerequisites

Procedure

  1. Navigate to Advisor Recommendations on OpenShift Cluster Manager.
  2. Click the X icons next to the Clusters Impacted and Status filters.

    You can now browse through all of the potential recommendations for your cluster.

4.5.5. Advisor recommendation filters

The Insights advisor service can return a large number of recommendations. To focus on your most critical recommendations, you can apply filters to the Advisor recommendations list to remove low-priority recommendations.

By default, filters are set to only show enabled recommendations that are impacting one or more clusters. To view all or disabled recommendations in the Insights library, you can customize the filters.

To apply a filter, select a filter type and then set its value based on the options that are available in the drop-down list. You can apply multiple filters to the list of recommendations.

You can set the following filter types:

  • Name: Search for a recommendation by name.
  • Total risk: Select one or more values from Critical, Important, Moderate, and Low indicating the likelihood and the severity of a negative impact on a cluster.
  • Impact: Select one or more values from Critical, High, Medium, and Low indicating the potential impact to the continuity of cluster operations.
  • Likelihood: Select one or more values from Critical, High, Medium, and Low indicating the potential for a negative impact to a cluster if the recommendation comes to fruition.
  • Category: Select one or more categories from Service Availability, Performance, Fault Tolerance, Security, and Best Practice to focus your attention on.
  • Status: Click a radio button to show enabled recommendations (default), disabled recommendations, or all recommendations.
  • Clusters impacted: Set the filter to show recommendations currently impacting one or more clusters, non-impacting recommendations, or all recommendations.
  • Risk of change: Select one or more values from High, Moderate, Low, and Very low indicating the risk that the implementation of the resolution could have on cluster operations.

4.5.5.1. Filtering Insights advisor recommendations

As an OpenShift Container Platform cluster manager, you can filter the recommendations that are displayed on the recommendations list. By applying filters, you can reduce the number of reported recommendations and concentrate on your highest priority recommendations.

The following procedure demonstrates how to set and remove Category filters; however, the procedure is applicable to any of the filter types and respective values.

Prerequisites

You are logged in to the OpenShift Cluster Manager Hybrid Cloud Console.

Procedure

  1. Go to Red Hat Hybrid Cloud Console OpenShift Advisor recommendations.
  2. In the main, filter-type drop-down list, select the Category filter type.
  3. Expand the filter-value drop-down list and select the checkbox next to each category of recommendation you want to view. Leave the checkboxes for unnecessary categories clear.
  4. Optional: Add additional filters to further refine the list.

Only recommendations from the selected categories are shown in the list.

Verification

  • After applying filters, you can view the updated recommendations list. The applied filters are added next to the default filters.

4.5.5.2. Removing filters from Insights Advisor recommendations

You can apply multiple filters to the list of recommendations. When ready, you can remove them individually or completely reset them.

Removing filters individually

  • Click the X icon next to each filter, including the default filters, to remove them individually.

Removing all non-default filters

  • Click Reset filters to remove only the filters that you applied, leaving the default filters in place.

4.5.6. Disabling Insights Advisor recommendations

You can disable specific recommendations that affect your clusters, so that they no longer appear in your reports. It is possible to disable a recommendation for a single cluster or all of your clusters.

Note

Disabling a recommendation for all of your clusters also applies to any future clusters.

Prerequisites

Procedure

  1. Navigate to Advisor Recommendations on OpenShift Cluster Manager.
  2. Optional: Use the Clusters Impacted and Status filters as needed.
  3. Disable an alert by using one of the following methods:

    • To disable an alert:

      1. Click the Options menu kebab for that alert, and then click Disable recommendation.
      2. Enter a justification note and click Save.
    • To view the clusters affected by this alert before disabling the alert:

      1. Click the name of the recommendation to disable. You are directed to the single recommendation page.
      2. Review the list of clusters in the Affected clusters section.
      3. Click Actions Disable recommendation to disable the alert for all of your clusters.
      4. Enter a justification note and click Save.

4.5.7. Enabling a previously disabled Insights Advisor recommendation

When a recommendation is disabled for all clusters, you no longer see the recommendation in the Insights Advisor. You can change this behavior.

Prerequisites

Procedure

  1. Navigate to Advisor Recommendations on OpenShift Cluster Manager.
  2. Filter the recommendations to display on the disabled recommendations:

    1. From the Status drop-down menu, select Status.
    2. From the Filter by status drop-down menu, select Disabled.
    3. Optional: Clear the Clusters impacted filter.
  3. Locate the recommendation to enable.
  4. Click the Options menu kebab , and then click Enable recommendation.

4.5.8. Displaying the Insights status in the web console

Insights repeatedly analyzes your cluster and you can display the status of identified potential issues of your cluster in the OpenShift Container Platform web console. This status shows the number of issues in the different categories and, for further details, links to the reports in OpenShift Cluster Manager.

Prerequisites

  • Your cluster is registered in OpenShift Cluster Manager.
  • Remote health reporting is enabled, which is the default.
  • You are logged in to the OpenShift Container Platform web console.

Procedure

  1. Navigate to Home Overview in the OpenShift Container Platform web console.
  2. Click Insights on the Status card.

    The pop-up window lists potential issues grouped by risk. Click the individual categories or View all recommendations in Insights Advisor to display more details.

4.6. Using the Insights Operator

The Insights Operator periodically gathers configuration and component failure status and, by default, reports that data every two hours to Red Hat. This information enables Red Hat to assess configuration and deeper failure data than is reported through Telemetry. Users of OpenShift Container Platform can display the report in the Insights Advisor service on Red Hat Hybrid Cloud Console.

Additional resources

4.6.1. Understanding Insights Operator alerts

The Insights Operator declares alerts through the Prometheus monitoring system to the Alertmanager. You can view these alerts in the Alerting UI in the OpenShift Container Platform web console by using one of the following methods:

  • In the Administrator perspective, click Observe Alerting.
  • In the Developer perspective, click Observe <project_name> Alerts tab.

Currently, Insights Operator sends the following alerts when the conditions are met:

Table 4.1. Insights Operator alerts
AlertDescription

InsightsDisabled

Insights Operator is disabled.

SimpleContentAccessNotAvailable

Simple content access is not enabled in Red Hat Subscription Management.

InsightsRecommendationActive

Insights has an active recommendation for the cluster.

4.6.2. Disabling Insights Operator alerts

To prevent the Insights Operator from sending alerts to the cluster Prometheus instance, you edit the support secret. If the support secret doesn’t exist, you must create it when you first add custom configurations. Note that configurations within the support secret take precedence over the default settings defined in the pod.yaml file. To prevent the Insights Operator from sending alerts to the cluster Prometheus instance, you edit the support secret. Note that this secret is created by default. The configurations stored in the support secret take precedence over any default settings specified in the pod.yaml file.

Prerequisites

  • Remote health reporting is enabled, which is the default.
  • You are logged in to the OpenShift Container Platform web console as cluster-admin.

Procedure

  1. Navigate to Workloads Secrets.
  2. On the Secrets page, select All Projects from the Project list, and then set Show default projects to on.
  3. Select the openshift-config project from the Projects list.
  4. Search for the support secret by using the Search by name field.

    • If the secret exists:
  5. Click the Options menu kebab , and then click Edit Secret.
  6. Click Add Key/Value.

    1. In the Key field, enter disableInsightsAlerts.
    2. In the Value field, enter True.

      • If the secret does not exist:
    3. Click Create Key/value secret.

      1. In the Secret name field, enter support.
      2. In the Key field, enter disableInsightsAlerts.
      3. In the Value field, enter True.
    4. Click Create.

After you save the changes, Insights Operator no longer sends alerts to the cluster Prometheus instance.

4.6.3. Enabling Insights Operator alerts

When alerts are disabled, the Insights Operator no longer sends alerts to the cluster Prometheus instance. You can change this behavior.

Prerequisites

  • Remote health reporting is enabled, which is the default.
  • You are logged in to the OpenShift Container Platform web console as cluster-admin.

Procedure

  1. Navigate to Workloads Secrets.
  2. On the Secrets page, select All Projects from the Project list, and then set Show default projects to ON.
  3. Select the openshift-config project from the Projects list.
  4. Search for the support secret by using the Search by name field.
  5. Click the Options menu kebab , and then click Edit Secret.
  6. For the disableInsightsAlerts key, set the Value field to false.

After you save the changes, Insights Operator again sends alerts to the cluster Prometheus instance.

4.6.4. Downloading your Insights Operator archive

Insights Operator stores gathered data in an archive located in the openshift-insights namespace of your cluster. You can download and review the data that is gathered by the Insights Operator.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Find the name of the running pod for the Insights Operator:

    $ oc get pods --namespace=openshift-insights -o custom-columns=:metadata.name --no-headers  --field-selector=status.phase=Running
  2. Copy the recent data archives collected by the Insights Operator:

    $ oc cp openshift-insights/<insights_operator_pod_name>:/var/lib/insights-operator ./insights-data 1
    1
    Replace <insights_operator_pod_name> with the pod name output from the preceding command.

The recent Insights Operator archives are now available in the insights-data directory.

4.6.5. Viewing Insights Operator gather durations

You can view the time it takes for the Insights Operator to gather the information contained in the archive. This helps you to understand Insights Operator resource usage and issues with Insights Advisor.

Prerequisites

  • A recent copy of your Insights Operator archive.

Procedure

  1. From your archive, open /insights-operator/gathers.json.

    The file contains a list of Insights Operator gather operations:

        {
          "name": "clusterconfig/authentication",
          "duration_in_ms": 730, 1
          "records_count": 1,
          "errors": null,
          "panic": null
        }
    1
    duration_in_ms is the amount of time in milliseconds for each gather operation.
  2. Inspect each gather operation for abnormalities.

4.6.6. Disabling the Insights Operator gather operations

You can disable the Insights Operator gather operations. Disabling the gather operations gives you the ability to increase privacy for your organization as Insights Operator will no longer gather and send Insights cluster reports to Red Hat. This will disable Insights analysis and recommendations for your cluster without affecting other core functions that require communication with Red Hat such as cluster transfers. You can view a list of attempted gather operations for your cluster from the /insights-operator/gathers.json file in your Insights Operator archive. Be aware that some gather operations only occur when certain conditions are met and might not appear in your most recent archive.

Important

The InsightsDataGather custom resource is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Note

If you enable Technology Preview in your cluster, the Insights Operator runs gather operations in individual pods. This is part of the Technology Preview feature set for the Insights Operator and supports the new data gathering features.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.

Procedure

  1. Navigate to Administration CustomResourceDefinitions.
  2. On the CustomResourceDefinitions page, use the Search by name field to find the InsightsDataGather resource definition and click it.
  3. On the CustomResourceDefinition details page, click the Instances tab.
  4. Click cluster, and then click the YAML tab.
  5. Disable the gather operations by performing one of the following edits to the InsightsDataGather configuration file:

    1. To disable all the gather operations, enter all under the disabledGatherers key:

      apiVersion: config.openshift.io/v1alpha1
      kind: InsightsDataGather
      metadata:
      ....
      
      spec: 1
        gatherConfig:
          disabledGatherers:
            - all 2
      1
      The spec parameter specifies gather configurations.
      2
      The all value disables all gather operations.
    2. To disable individual gather operations, enter their values under the disabledGatherers key:

      spec:
        gatherConfig:
          disabledGatherers:
            - clusterconfig/container_images 1
            - clusterconfig/host_subnets
            - workloads/workload_info
      1
      Example individual gather operation
  6. Click Save.

    After you save the changes, the Insights Operator gather configurations are updated and the operations will no longer occur.

Note

Disabling gather operations degrades Insights Advisor’s ability to offer effective recommendations for your cluster.

4.6.7. Enabling the Insights Operator gather operations

You can enable the Insights Operator gather operations, if the gather operations have been disabled.

Important

The InsightsDataGather custom resource is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.

Procedure

  1. Navigate to Administration CustomResourceDefinitions.
  2. On the CustomResourceDefinitions page, use the Search by name field to find the InsightsDataGather resource definition and click it.
  3. On the CustomResourceDefinition details page, click the Instances tab.
  4. Click cluster, and then click the YAML tab.
  5. Enable the gather operations by performing one of the following edits:

    • To enable all disabled gather operations, remove the gatherConfig stanza:

      apiVersion: config.openshift.io/v1alpha1
      kind: InsightsDataGather
      metadata:
      ....
      
      spec:
        gatherConfig: 1
          disabledGatherers: all
      1
      Remove the gatherConfig stanza to enable all gather operations.
    • To enable individual gather operations, remove their values under the disabledGatherers key:

      spec:
        gatherConfig:
          disabledGatherers:
            - clusterconfig/container_images 1
            - clusterconfig/host_subnets
            - workloads/workload_info
      1
      Remove one or more gather operations.
  6. Click Save.

    After you save the changes, the Insights Operator gather configurations are updated and the affected gather operations start.

Note

Disabling gather operations degrades Insights Advisor’s ability to offer effective recommendations for your cluster.

4.6.8. Running an Insights Operator gather operation

You can run Insights Operator data gather operations on demand. The following procedures describe how to run the default list of gather operations using the OpenShift web console or CLI. You can customize the on demand gather function to exclude any gather operations you choose. Disabling gather operations from the default list degrades Insights Advisor’s ability to offer effective recommendations for your cluster. If you have previously disabled Insights Operator gather operations in your cluster, this procedure will override those parameters.

Important

The DataGather custom resource is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Note

If you enable Technology Preview in your cluster, the Insights Operator runs gather operations in individual pods. This is part of the Technology Preview feature set for the Insights Operator and supports the new data gathering features.

4.6.8.1. Running an Insights Operator gather operation using the web console

You can run an Insights Operator gather operation using the OpenShift Container Platform web console.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.

Procedure

  1. Navigate to Administration CustomResourceDefinitions.
  2. On the CustomResourceDefinitions page, use the Search by name field to find the DataGather resource definition and click it.
  3. On the CustomResourceDefinition details page, click the Instances tab.
  4. Click Create DataGather.
  5. To create a new DataGather operation, edit the configuration file:

    apiVersion: insights.openshift.io/v1alpha1
    kind: DataGather
    metadata:
      name: <your_data_gather> 1
    spec:
     gatherers: 2
       - name: workloads
         state: Disabled
    1
    Replace the <your_data_gather> with a unique name for your gather operation.
    2
    Enter individual gather operations to disable under the gatherers parameter. This example disables the workloads data gather operation and will run the remainder of the default operations. To run the complete list of default gather operations, leave the spec parameter empty. You can find the complete list of gather operations in the Insights Operator documentation.
  6. Click Save.

Verification

  1. Navigate to Workloads Pods.
  2. On the Pods page, select the Project pulldown menu, and then turn on Show default projects.
  3. Select the openshift-insights project from the Project pulldown menu.
  4. Check that your new gather operation is prefixed with your chosen name under the list of pods in the openshift-insights project. Upon completion, the Insights Operator automatically uploads the data to Red Hat for processing.

4.6.8.2. Running an Insights Operator gather operation using the OpenShift CLI

You can run an Insights Operator gather operation using the OpenShift Container Platform command line interface.

Prerequisites

  • You are logged in to OpenShift Container Platform as a user with the cluster-admin role.

Procedure

  • Enter the following command to run the gather operation:

    $ oc apply -f <your_datagather_definition>.yaml

    Replace <your_datagather_definition>.yaml with a configuration file using the following parameters:

    apiVersion: insights.openshift.io/v1alpha1
    kind: DataGather
    metadata:
      name: <your_data_gather> 1
    spec:
     gatherers: 2
       - name: workloads
         state: Disabled
    1
    Replace the <your_data_gather> with a unique name for your gather operation.
    2
    Enter individual gather operations to disable under the gatherers parameter. This example disables the workloads data gather operation and will run the remainder of the default operations. To run the complete list of default gather operations, leave the spec parameter empty. You can find the complete list of gather operations in the Insights Operator documentation.

Verification

  • Check that your new gather operation is prefixed with your chosen name under the list of pods in the openshift-insights project. Upon completion, the Insights Operator automatically uploads the data to Red Hat for processing.

4.6.9. Configuring Insights Operator

You can configure Insights Operator to meet the needs of your organization. The Insights Operator is configured using a combination of the default configurations in the pod.yaml file in the Insights Operator Config directory and the configurations stored in the support secret in the openshift-config namespace. The support secret does not exist by default and must be created when adding custom configurations for the first time. Configurations in the support secret override the defaults set in the pod.yaml file.

The table below describes the available configuration attributes:

Table 4.2. Insights Operator configurable attributes
Attribute nameDescriptionValue typeDefault value

enableGlobalObfuscation

Enables the global obfuscation of IP addresses and the cluster domain name

Boolean

false

scaInterval

Specifies the frequency of the simple content access entitlements download

Time interval

8h

scaPullDisabled

Disables the simple content access entitlements download

Boolean

false

clusterTransferInterval

Specifies how often Insights Operator checks OpenShift Cluster Manager for available cluster transfers

Time interval

24h

disableInsightsAlerts

Disables Insights Operator alerts to the cluster Prometheus instance

Boolean

False

httpProxy, httpsProxy, noProxy

Set custom proxy for Insights Operator

URL

no default

This procedure describes how to set custom Insights Operator configurations.

Important

Red Hat recommends you consult Red Hat Support before making changes to the default Insights Operator configuration.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with cluster-admin role.

Procedure

  1. Navigate to Workloads Secrets.
  2. On the Secrets page, select All Projects from the Project list, and then set Show default projects to on.
  3. Select the openshift-config project from the Project list.
  4. Search for the support secret using the Search by name field. If it does not exist, click Create Key/value secret to create it.
  5. Click the Options menu kebab for the secret, and then click Edit Secret.
  6. Click Add Key/Value.
  7. Enter an attribute name with an appropriate value (see table above), and click Save.
  8. Repeat the above steps for any additional configurations.

4.7. Using remote health reporting in a restricted network

You can manually gather and upload Insights Operator archives to diagnose issues from a restricted network.

To use the Insights Operator in a restricted network, you must:

  • Create a copy of your Insights Operator archive.
  • Upload the Insights Operator archive to console.redhat.com.

Additionally, you can choose to obfuscate the Insights Operator data before upload.

4.7.1. Running an Insights Operator gather operation

You must run a gather operation to create an Insights Operator archive.

Prerequisites

  • You are logged in to OpenShift Container Platform as cluster-admin.

Procedure

  1. Create a file named gather-job.yaml using this template:

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: insights-operator-job
      annotations:
        config.openshift.io/inject-proxy: insights-operator
    spec:
      backoffLimit: 6
      ttlSecondsAfterFinished: 600
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: operator
          nodeSelector:
            beta.kubernetes.io/os: linux
            node-role.kubernetes.io/master: ""
          tolerations:
          - effect: NoSchedule
            key: node-role.kubernetes.io/master
            operator: Exists
          - effect: NoExecute
            key: node.kubernetes.io/unreachable
            operator: Exists
            tolerationSeconds: 900
          - effect: NoExecute
            key: node.kubernetes.io/not-ready
            operator: Exists
            tolerationSeconds: 900
          volumes:
          - name: snapshots
            emptyDir: {}
          - name: service-ca-bundle
            configMap:
              name: service-ca-bundle
              optional: true
          initContainers:
          - name: insights-operator
            image: quay.io/openshift/origin-insights-operator:latest
            terminationMessagePolicy: FallbackToLogsOnError
            volumeMounts:
            - name: snapshots
              mountPath: /var/lib/insights-operator
            - name: service-ca-bundle
              mountPath: /var/run/configmaps/service-ca-bundle
              readOnly: true
            ports:
            - containerPort: 8443
              name: https
            resources:
              requests:
                cpu: 10m
                memory: 70Mi
            args:
            - gather
            - -v=4
            - --config=/etc/insights-operator/server.yaml
          containers:
            - name: sleepy
              image: quay.io/openshift/origin-base:latest
              args:
                - /bin/sh
                - -c
                - sleep 10m
              volumeMounts: [{name: snapshots, mountPath: /var/lib/insights-operator}]
  2. Copy your insights-operator image version:

    $ oc get -n openshift-insights deployment insights-operator -o yaml

    Example output

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: insights-operator
      namespace: openshift-insights
    # ...
    spec:
      template:
    # ...
        spec:
          containers:
          - args:
    # ...
            image: registry.ci.openshift.org/ocp/4.15-2023-10-12-212500@sha256:a0aa581400805ad0... 1
    # ...

    1
    Specifies your insights-operator image version.
  3. Paste your image version in gather-job.yaml:

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: insights-operator-job
    # ...
    spec:
    # ...
      template:
        spec:
        initContainers:
        - name: insights-operator
          image: image: registry.ci.openshift.org/ocp/4.15-2023-10-12-212500@sha256:a0aa581400805ad0... 1
          terminationMessagePolicy: FallbackToLogsOnError
          volumeMounts:
    1
    Replace any existing value with your insights-operator image version.
  4. Create the gather job:

    $ oc apply -n openshift-insights -f gather-job.yaml
  5. Find the name of the job pod:

    $ oc describe -n openshift-insights job/insights-operator-job

    Example output

    Name:             insights-operator-job
    Namespace:        openshift-insights
    # ...
    Events:
      Type    Reason            Age    From            Message
      ----    ------            ----   ----            -------
      Normal  SuccessfulCreate  7m18s  job-controller  Created pod: insights-operator-job-<your_job>

    where
    insights-operator-job-<your_job> is the name of the pod.
  6. Verify that the operation has finished:

    $ oc logs -n openshift-insights insights-operator-job-<your_job> insights-operator

    Example output

    I0407 11:55:38.192084       1 diskrecorder.go:34] Wrote 108 records to disk in 33ms

  7. Save the created archive:

    $ oc cp openshift-insights/insights-operator-job-<your_job>:/var/lib/insights-operator ./insights-data
  8. Clean up the job:

    $ oc delete -n openshift-insights job insights-operator-job

4.7.2. Uploading an Insights Operator archive

You can manually upload an Insights Operator archive to console.redhat.com to diagnose potential issues.

Prerequisites

  • You are logged in to OpenShift Container Platform as cluster-admin.
  • You have a workstation with unrestricted internet access.
  • You have created a copy of the Insights Operator archive.

Procedure

  1. Download the dockerconfig.json file:

    $ oc extract secret/pull-secret -n openshift-config --to=.
  2. Copy your "cloud.openshift.com" "auth" token from the dockerconfig.json file:

    {
      "auths": {
        "cloud.openshift.com": {
          "auth": "<your_token>",
          "email": "asd@redhat.com"
        }
    }
  3. Upload the archive to console.redhat.com:

    $ curl -v -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/<cluster_id>" -H "Authorization: Bearer <your_token>" -F "upload=@<path_to_archive>; type=application/vnd.redhat.openshift.periodic+tar" https://console.redhat.com/api/ingress/v1/upload

    where <cluster_id> is your cluster ID, <your_token> is the token from your pull secret, and <path_to_archive> is the path to the Insights Operator archive.

    If the operation is successful, the command returns a "request_id" and "account_number":

    Example output

    * Connection #0 to host console.redhat.com left intact
    {"request_id":"393a7cf1093e434ea8dd4ab3eb28884c","upload":{"account_number":"6274079"}}%

Verification steps

  1. Log in to https://console.redhat.com/openshift.
  2. Click the Clusters menu in the left pane.
  3. To display the details of the cluster, click the cluster name.
  4. Open the Insights Advisor tab of the cluster.

    If the upload was successful, the tab displays one of the following:

    • Your cluster passed all recommendations, if Insights Advisor did not identify any issues.
    • A list of issues that Insights Advisor has detected, prioritized by risk (low, moderate, important, and critical).

4.7.3. Enabling Insights Operator data obfuscation

You can enable obfuscation to mask sensitive and identifiable IPv4 addresses and cluster base domains that the Insights Operator sends to console.redhat.com.

Warning

Although this feature is available, Red Hat recommends keeping obfuscation disabled for a more effective support experience.

Obfuscation assigns non-identifying values to cluster IPv4 addresses, and uses a translation table that is retained in memory to change IP addresses to their obfuscated versions throughout the Insights Operator archive before uploading the data to console.redhat.com.

For cluster base domains, obfuscation changes the base domain to a hardcoded substring. For example, cluster-api.openshift.example.com becomes cluster-api.<CLUSTER_BASE_DOMAIN>.

The following procedure enables obfuscation using the support secret in the openshift-config namespace.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as cluster-admin.

Procedure

  1. Navigate to Workloads Secrets.
  2. Select the openshift-config project.
  3. Search for the support secret using the Search by name field. If it does not exist, click Create Key/value secret to create it.
  4. Click the Options menu kebab , and then click Edit Secret.
  5. Click Add Key/Value.
  6. Create a key named enableGlobalObfuscation with a value of true, and click Save.
  7. Navigate to Workloads Pods
  8. Select the openshift-insights project.
  9. Find the insights-operator pod.
  10. To restart the insights-operator pod, click the Options menu kebab , and then click Delete Pod.

Verification

  1. Navigate to Workloads Secrets.
  2. Select the openshift-insights project.
  3. Search for the obfuscation-translation-table secret using the Search by name field.

If the obfuscation-translation-table secret exists, then obfuscation is enabled and working.

Alternatively, you can inspect /insights-operator/gathers.json in your Insights Operator archive for the value "is_global_obfuscation_enabled": true.

Additional resources

4.8. Importing simple content access entitlements with Insights Operator

Insights Operator periodically imports your simple content access entitlements from OpenShift Cluster Manager and stores them in the etc-pki-entitlement secret in the openshift-config-managed namespace. Simple content access is a capability in Red Hat subscription tools which simplifies the behavior of the entitlement tooling. This feature makes it easier to consume the content provided by your Red Hat subscriptions without the complexity of configuring subscription tooling.

Insights Operator imports simple content access entitlements every eight hours, but can be configured or disabled using the support secret in the openshift-config namespace.

Note

Simple content access must be enabled in Red Hat Subscription Management for the importing to function.

Additional resources

4.8.1. Configuring simple content access import interval

You can configure how often the Insights Operator imports the simple content access entitlements by using the support secret in the openshift-config namespace. The entitlement import normally occurs every eight hours, but you can shorten this interval if you update your simple content access configuration in Red Hat Subscription Management.

This procedure describes how to update the import interval to one hour.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.

Procedure

  1. Navigate to Workloads Secrets.
  2. Select the openshift-config project.
  3. Search for the support secret by using the Search by name field. If it does not exist, click Create Key/value secret to create it.

    • If the secret exists:

      1. Click the Options menu kebab , and then click Edit Secret.
      2. Click Add Key/Value.

        1. In the Key field, enter scaInterval.
        2. In the Value field, enter 1h.
    • If the secret does not exist:

      1. Click Create Key/value secret.

        1. In the Secret name field, enter support.
        2. In the Key field, enter scaInterval.
        3. In the Value field, enter 1h.
      2. Click Create.
    Note

    The interval 1h can also be entered as 60m for 60 minutes.

4.8.2. Disabling simple content access import

You can disable the importing of simple content access entitlements by using the support secret in the openshift-config namespace.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as cluster-admin.

Procedure

  1. Navigate to Workloads Secrets.
  2. Select the openshift-config project.
  3. Search for the support secret using the Search by name field.

    • If the secret exists:

      1. Click the Options menu kebab , and then click Edit Secret.
      2. Click Add Key/Value.

        1. In the Key field, enter scaPullDisabled.
        2. In the Value field, enter true.
    • If the secret does not exist:

      1. Click Create Key/value secret.

        1. In the Secret name field, enter support.
        2. In the Key field, enter scaPullDisabled.
        3. In the Value field, enter true.
      2. Click Create.

    The simple content access entitlement import is now disabled.

    Note

    To enable the simple content access import again, edit the support secret and delete the scaPullDisabled key.

4.8.3. Enabling a previously disabled simple content access import

If the importing of simple content access entitlements is disabled, the Insights Operator does not import simple content access entitlements. You can change this behavior.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.

Procedure

  1. Navigate to Workloads Secrets.
  2. Select the openshift-config project.
  3. Search for the support secret by using the Search by name field.
  4. Click the Options menu kebab , and then click Edit Secret.
  5. For the scaPullDisabled key, set the Value field to false.

    The simple content access entitlement import is now disabled.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.