Troubleshooting

See the following procedure to start using the must-gather command:

Complete the following steps to run the must-gather command in a disconnected environment:

If you experience issues with one of the currently supported releases, or the product documentation, go to Red Hat Support where you can troubleshoot further, view Knowledgebase articles, connect with the Support Team, or open a case. You must log in with your Red Hat credentials.

If you experience issues with hosted control plane clusters, you can run the must-gather command to gather information to help you with troubleshooting.

More than ten minutes passed since you installed MultiClusterHub and one or more components from the status.components field of the MultiClusterHub resource report ProgressDeadlineExceeded. Resource constraints on the cluster might be the issue.

Check the pods in the namespace where Multiclusterhub was installed. You might see Pending with a status similar to the following:

reason: Unschedulable
message: '0/6 nodes are available: 3 Insufficient cpu, 3 node(s) had taint {node-role.kubernetes.io/master:
        }, that the pod didn't tolerate.'

In this case, the worker nodes resources are not sufficient in the cluster to run the product.

If you have this problem, then your cluster needs to be updated with either larger or more worker nodes. See Sizing your cluster for guidelines on sizing your cluster.

If your pods do not start after you install Red Hat Advanced Cluster Management, it is likely that Red Hat Advanced Cluster Management was previously installed, and not all of the pieces were removed before you attempted this installation.

In this case, the pods do not start after completing the installation process.

If you have this problem, complete the following steps:

After you attempt to list ManagedClusterSet and ManagedClusterSetBinding custom resource definitions, the following error message appears:

Error from server: request to convert CR from an invalid group/version: cluster.open-cluster-management.io/v1beta1

The previous message indicates that the migration of ManagedClusterSets and ManagedClusterSetBindings custom resource definitions from v1beta1 to v1beta2 failed.

To resolve this error, you must initiate the API migration manually. Completet the following steps:

oc get managedclusterset
oc get managedclustersetbinding -A

After you complete the procedure for creating a cluster, you cannot access it from the Red Hat Advanced Cluster Management console, and it shows a status of offline.

After you complete the procedure for importing a cluster, you cannot access it from the Red Hat Advanced Cluster Management for Kubernetes console.

There can be a few reasons why an imported cluster is not available after an attempt to import it. If the cluster import fails, complete the following steps, until you find the reason for the failed import:

After importing a cluster by using the Red Hat Advanced Cluster Management console, the cluster appears in the console with a status of Pending import.

An error log shows up when importing an OpenShift Container Platform cluster into Red Hat Advanced Cluster Management MultiClusterHub:

error log:
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Error from server (AlreadyExists): error when creating "STDIN": customresourcedefinitions.apiextensions.k8s.io "klusterlets.operator.open-cluster-management.io" already exists
The cluster cannot be imported because its Klusterlet CRD already exists.
Either the cluster was already imported, or it was not detached completely during a previous detach process.
Detach the existing cluster before trying the import again."

Check if there are any Red Hat Advanced Cluster Management-related resources on the cluster that you want to import to new the Red Hat Advanced Cluster Management MultiClusterHub by running the following commands:

oc get all -n open-cluster-management-agent
oc get all -n open-cluster-management-agent-addon

Remove the klusterlet custom resource by using the following command:

oc get klusterlet | grep klusterlet | awk '{print $1}' | xargs oc patch klusterlet --type=merge -p '{"metadata":{"finalizers": []}}'

Run the following commands to remove pre-existing resources:

oc delete namespaces open-cluster-management-agent open-cluster-management-agent-addon --wait=false
oc get crds | grep open-cluster-management.io | awk '{print $1}' | xargs oc delete crds --wait=false
oc get crds | grep open-cluster-management.io | awk '{print $1}' | xargs oc patch crds --type=merge -p '{"metadata":{"finalizers": []}}'

After creating a new Red Hat OpenShift Container Platform cluster on Red Hat OpenStack Platform using self-signed certificates, the cluster fails with an error message that indicates an unknown authority error.

The deployment of the managed cluster fails and returns the following error message:

x509: certificate signed by unknown authority

Verify that the following files are configured correctly:

After you attempt to import a Red Hat OpenShift Container Platform version 3.11 cluster, the import fails with a log message that resembles the following content:

customresourcedefinition.apiextensions.k8s.io/klusterlets.operator.open-cluster-management.io configured
clusterrole.rbac.authorization.k8s.io/klusterlet configured
clusterrole.rbac.authorization.k8s.io/open-cluster-management:klusterlet-admin-aggregate-clusterrole configured
clusterrolebinding.rbac.authorization.k8s.io/klusterlet configured
namespace/open-cluster-management-agent configured
secret/open-cluster-management-image-pull-credentials unchanged
serviceaccount/klusterlet configured
deployment.apps/klusterlet unchanged
klusterlet.operator.open-cluster-management.io/klusterlet configured
Error from server (BadRequest): error when creating "STDIN": Secret in version "v1" cannot be handled as a Secret:
v1.Secret.ObjectMeta:
v1.ObjectMeta.TypeMeta: Kind: Data: decode base64: illegal base64 data at input byte 1313, error found in #10 byte of ...|dhruy45="},"kind":"|..., bigger context ...|tye56u56u568yuo7i67i67i67o556574i"},"kind":"Secret","metadata":{"annotations":{"kube|...

This often occurs because the installed version of the kubectl command-line tool is 1.11, or earlier. Run the following command to see which version of the kubectl command-line tool you are running:

kubectl version

If the returned data lists version 1.11, or earlier, complete one of the fixes in Resolving the problem: OpenShift Container Platform version 3.11 cluster import failure.

You can resolve this issue by completing one of the following procedures:

After you complete the procedure for updating a certificate secret, one or more of your clusters that were online now display offline status in the console.

After updating the information for a custom API server certificate, clusters that were imported and running before the new certificate are now in an offline state.

The errors that indicate that the certificate is the problem are found in the logs for the pods in the open-cluster-management-agent namespace of the offline managed cluster. The following examples are similar to the errors that are displayed in the logs:

See the following work-agent log:

E0917 03:04:05.874759       1 manifestwork_controller.go:179] Reconcile work test-1-klusterlet-addon-workmgr fails with err: Failed to update work status with err Get "https://api.aaa-ocp.dev02.location.com:6443/apis/cluster.management.io/v1/namespaces/test-1/manifestworks/test-1-klusterlet-addon-workmgr": x509: certificate signed by unknown authority
E0917 03:04:05.874887       1 base_controller.go:231] "ManifestWorkAgent" controller failed to sync "test-1-klusterlet-addon-workmgr", err: Failed to update work status with err Get "api.aaa-ocp.dev02.location.com:6443/apis/cluster.management.io/v1/namespaces/test-1/manifestworks/test-1-klusterlet-addon-workmgr": x509: certificate signed by unknown authority
E0917 03:04:37.245859       1 reflector.go:127] k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156: Failed to watch *v1.ManifestWork: failed to list *v1.ManifestWork: Get "api.aaa-ocp.dev02.location.com:6443/apis/cluster.management.io/v1/namespaces/test-1/manifestworks?resourceVersion=607424": x509: certificate signed by unknown authority

See the following registration-agent log:

I0917 02:27:41.525026       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAvailableConditionUpdated' update managed cluster "test-1" available condition to "True", due to "Managed cluster is available"
E0917 02:58:26.315984       1 reflector.go:127] k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156: Failed to watch *v1beta1.CertificateSigningRequest: Get "https://api.aaa-ocp.dev02.location.com:6443/apis/cluster.management.io/v1/managedclusters?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dtest-1&resourceVersion=607408&timeout=9m33s&timeoutSeconds=573&watch=true"": x509: certificate signed by unknown authority
E0917 02:58:26.598343       1 reflector.go:127] k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156: Failed to watch *v1.ManagedCluster: Get "https://api.aaa-ocp.dev02.location.com:6443/apis/cluster.management.io/v1/managedclusters?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dtest-1&resourceVersion=607408&timeout=9m33s&timeoutSeconds=573&watch=true": x509: certificate signed by unknown authority
E0917 02:58:27.613963       1 reflector.go:127] k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156: Failed to watch *v1.ManagedCluster: failed to list *v1.ManagedCluster: Get "https://api.aaa-ocp.dev02.location.com:6443/apis/cluster.management.io/v1/managedclusters?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dtest-1&resourceVersion=607408&timeout=9m33s&timeoutSeconds=573&watch=true"": x509: certificate signed by unknown authority

If your managed cluster is the local-cluster, or your managed cluster was created by using Red Hat Advanced Cluster Management for Kubernetes, you must wait 10 minutes or longer to reimport your managed cluster.

To reimport your managed cluster immediately, you can delete your managed cluster import secret on the hub cluster and reimport it by using Red Hat Advanced Cluster Management. Run the following command:

oc delete secret -n <cluster_name> <cluster_name>-import

Replace <cluster_name> with the name of the managed cluster that you want to import.

If you want to reimport a managed cluster that was imported by using Red Hat Advanced Cluster Management, complete the following steps to import the managed cluster again:

Note: The previous steps do not detach the managed cluster from the hub cluster. The steps update the required manifests with current settings on the managed cluster, including the new certificate information.

After removing a managed cluster, the namespace is not removed.

Complete the following steps to remove the namespace manually:

When importing a hive cluster for management, an auto-import-secret already exists error is displayed.

This problem occurs when you attempt to import a cluster that was previously managed by Red Hat Advanced Cluster Management. When this happens, the secrets conflict when you try to reimport the cluster.

To work around this problem, complete the following steps:

You can configure a VolSync ReplicationSource or ReplicationDestination to use snapshots. Also, you can configure the storageclass and volumesnapshotclass in the ReplicationSource and ReplicationDestination. There is a parameter on the cinder volumesnapshotclass called force-create with a default value of false. This force-create parameter on the volumesnapshotclass means cinder does not allow the volumesnapshot to be taken of a PVC in use. As a result, the volumesnapshot is in an error state.

You must meet the following prerequisites to run the must-gather command:

Complete the following procedure to collect information by using the must-gather command:

The following information is collected from the must-gather command:

Sometimes you need to dump the output in the multicluster global hub database to debug a problem. The PostgreSQL database provides the pg_dump command line tool to dump the contents of the database. To dump data from localhost database server, run the following command:

pg_dump hoh > hoh.sql

To dump the multicluster global hub database located on a remote server with compressed format, use the command-line options to control the connection details, as shown in the following example:

pg_dump -h my.host.com -p 5432 -U postgres -F t hoh -f hoh-$(date +%d-%m-%y_%H-%M).tar

To restore a PostgreSQL database, you can use the psql or pg_restore command line tools. The psql tool is used to restore plain text files created by pg_dump:

psql -h another.host.com -p 5432 -U postgres -d hoh < hoh.sql

The pg_restore tool is used to restore a PostgreSQL database from an archive created by pg_dump in one of the non-plain-text formats (custom, tar, or directory):

pg_restore -h another.host.com -p 5432 -U postgres -d hoh hoh-$(date +%d-%m-%y_%H-%M).tar

If you have installed an early version of multicluster global hub before GA, then you need to upgrade your tables to be compatible with the current multicluster global hub operator. The main purpose of the upgrade is to convert the event.local_policies, event.local_root_policies, and history.local_compliance tables into partitioned tables.

The following example shows the conversion of the event.local_policies table with the date set for 2023-08. The upgrade steps of the other two tables are similar.

When the network that connects the managed cluster to the hub cluster is unstable, the status of the managed cluster that is reported by the hub cluster cycles between offline and available.

The connection between the hub cluster and managed cluster is maintained through a lease that is validated at the leaseDurationSeconds interval value. If the lease is not validated within five consecutive attempts of the leaseDurationSeconds value, then the cluster is marked offline.

For example, the cluster is marked offline after five minutes with a leaseDurationSeconds interval of 60 seconds. This configuration can be inadequate for reasons such as connectivity issues or latency, causing instability.

The five validation attempts is default and cannot be changed, but you can change the leaseDurationSeconds interval.

Determine the amount of time, in minutes, that you want the cluster to be marked as offline, then multiply that value by 60 to convert to seconds. Then divide by the default five number of attempts. The result is your leaseDurationSeconds value.

After creating a new cluster by using the Red Hat Advanced Cluster Management for Kubernetes console, the cluster does not progress beyond the status of Pending or displays Failed status.

If the cluster displays Failed status, navigate to the details page for the cluster and follow the link to the logs provided. If no logs are found or the cluster displays Pending status, continue with the following procedure to check for logs:

After you identify the errors in the logs, determine how to resolve the errors before you destroy the cluster and create it again.

The following example provides a possible log error of selecting an unsupported zone, and the actions that are required to resolve it:

No subnets provided for zones

When you created your cluster, you selected one or more zones within a region that are not supported. Complete one of the following actions when you recreate your cluster to resolve the issue:

After determining the issues from the log, destroy the cluster and recreate it.

See Cluster creation introduction for more information about creating a cluster.

If you hit the Gateway Time-out error when you query some time-consuming metrics in the Grafana explorer, it is possible that the timeout is caused by the Grafana in the open-cluster-management-observability namespace.

If you have this problem, complete the following steps:

Refresh the Grafana page and try to query the metrics again. The Gateway Time-out error is no longer displayed.

All managed clusters are selected with a placement rule, but the local-cluster is not. The placement rule user is not granted permission to get the managedcluster resources in the local-cluster namespace.

Deprecated: PlacementRule

To resolve this issue, you need to grant the managedcluster administrative permission in the local-cluster namespace. Complete the following steps:

If one or more of your application resources in the Subscription YAML file uses the deprecated API, you might receive an error similar to the following error:

failed to install release: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for
kind "Deployment" in version "extensions/v1beta1"

Or with new Kubernetes API version in your YAML file named old.yaml for instance, you might receive the following error:

error: unable to recognize "old.yaml": no matches for kind "Deployment" in version "deployment/v1beta1"

After deploying a Klusterlet on managed cluster, the KlusterletRegistrationDegraded or KlusterletWorkDegraded condition displays a status of True.

See the following list of degraded statuses and how you can attempt to resolve those issues:

The subscription of an Object storage channel cannot pick up the updated secret automatically. This prevents the subscription operator from reconciliation and deploys resources from Object storage to the managed cluster.

You need to manually input the credentials to create a secret, then refer to the secret within a channel.

After you run the command, you can go to the Application console to verify that the resource is deployed to the managed cluster. Or you can log in to the managed cluster to see if the application resource is created at the given namespace.

If the observability status is stuck in an Installing status after you install and create the Observability custom resource definition (CRD), it is possible that there is no value defined for the spec:storageConfig:storageClass parameter. Alternatively, the observability component automatically finds the default storageClass, but if there is no value for the storage, the component remains stuck with the Installing status.

If you have this problem, complete the following steps:

storageclass.kubernetes.io/is-default-class: "true"

The observability component status is updated to a Ready status when the installation is complete. If the installation fails to complete, the Fail status is displayed.

The endpoint-observability-operator-x pod checks if the prometheus-k8s service is available in the openshift-monitoring namespace. If the service is not present in the openshift-monitoring namespace, then the metrics-collector is not deployed. You might receive the following error message: Failed to get prometheus resource.

If you have this problem, complete the following steps:

There might be a managed cluster, where the metrics are unavailable. If this is the case, you might receive the following error from the metrics-collector deployment:

error: response status code is 500 Internal Server Error, response body is x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "observability-client-ca-certificate")

If you have this problem, complete the following steps:

The metrics-collector deployment is recreated and the observability-controller-open-cluster-management.io-observability-signer-client-cert secret is updated.

An error message resembling the following appears in the search-api logs: ERROR: could not resize shared memory segment "/PostgreSQL.1083654800" to 25031264 bytes: No space left on device (SQLSTATE 53100)

To resolve the issue, update the PostgreSQL resources found in the search-postgres ConfigMap. Complete the following steps to update the resources:

See Search customization and configurations for more information on adding environment variables.

The Thanos compactor halts because there is no space left on your persistent volume claim (PVC). You receive the following message:

ts=2024-01-24T15:34:51.948653839Z caller=compact.go:491 level=error msg="critical error detected; halting" err="compaction: group 0@5827190780573537664: compact blocks [ /var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE]: 2 errors: populate block: add series: write series data: write /var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE.tmp-for-creation/index: no space left on device; write /var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE.tmp-for-creation/index: no space left on device"

To resolve the problem, increase the storage space of the Thanos compactor PVC. Complete the following steps:

The Thanos compactor halts because you have corrupted blocks. You might receive the following output where the 01HKZYEZ2DVDQXF1STVEXAMPLE block is corrupted:

ts=2024-01-24T15:34:51.948653839Z caller=compact.go:491 level=error msg="critical error detected; halting" err="compaction: group 0@15699422364132557315: compact blocks [/var/thanos/compact/compact/0@15699422364132557315/01HKZGQGJCKQWF3XMA8EXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HKZQK7TD06J2XWGR5EXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HKZYEZ2DVDQXF1STVEXAMPLE /var/thanos/compact/compact/0@15699422364132557315/01HM05APAHXBQSNC0N5EXAMPLE]: populate block: chunk iter: cannot populate chunk 8 from block 01HKZYEZ2DVDQXF1STVEXAMPLE: segment index 0 out of range"

Add the thanos bucket verify command to the object storage configuration. Complete the following steps:

Your Submariner network is not communicating after installation.

If the network connectivity is not established after deploying Submariner, begin the troubleshooting steps. Note that it might take several minutes for the processes to complete when you deploy Submariner.

When Submariner does not run correctly after deployment, complete the following steps:

You add the Submariner add-on to the clusters in your cluster set, the following status is shown in the Gateway nodes, Agent status, and Connection status for the clusters:

It will also show which clusters are connected, and which ones are disconnected.

Red Hat Advanced Cluster Management shows a FinishedWithErrors status and one or more of the Velero restore resources created by the Red Hat Advanced Cluster Management restore show a PartiallyFailed status.

If you restore from a backup that is empty, you can safely ignore the FinishedWithErrors status.

Red Hat Advanced Cluster Management for Kubernetes restore shows a cumulative status for all Velero restore resources. If one status is PartiallyFailed and the others are Completed, the cumulative status you see is PartiallyFailed to notify you that there is at least one issue.

To resolve the issue, check the status for all individual Velero restore resources with a PartiallyFailed status and view the logs for more details. You can get the log from the object storage directly, or download it from the OADP Operator by using the DownloadRequest custom resource.

To create a DownloadRequest from the console, complete the following steps:

Resources backed up in the acm-resources-generic-schedule backup do not appear on the restored hub cluster. If you check the backup operator log, you see a message that resembles the following:

_2023-06-08T13:42:48.066572033Z 2023-06-08T13:42:48.066Z    INFO    Deleting resource DRPlacementControl [c1-helloworld-placement-1-drpc.c1-helloworld]    {"controller": "restore", "controllerGroup": "cluster.open-cluster-management.io", "controllerKind": "Restore", "restore":
{"name":"restore-acm","namespace":"open-cluster-management-backup"}

The resources are removed if the following conditions occur:

To resolve the issue, run the restore operation again and set the cleanupBeforeRestore: value to None.`

When the managed cluster and hub cluster are the same cluster the certificate data is redacted, so the contents are not parsed as a template JSON string. You might receive the following error messages:

message: >-
            [spec.tls.caCertificate: Invalid value: "redacted ca certificate
            data": failed to parse CA certificate: data does not contain any
            valid RSA or ECDSA certificates, spec.tls.certificate: Invalid
            value: "redacted certificate data": data does not contain any valid
            RSA or ECDSA certificates, spec.tls.key: Invalid value: "": no key specified]

Configure your certificate policy to retrieve the hub cluster and managed cluster fromSecret values. Use the autoindent function to update your certificate policy with the following content:

                 tls:
                    certificate: |
                      {{ print "{{hub fromSecret "open-cluster-management" "minio-cert" "tls.crt" hub}}" | base64dec | autoindent }}