Release notes

After you upgrade from 2.4.x to 2.5.x, and then to 2.6.x, deprecated resources in the managed cluster namespace might remain. You need to manually delete these deprecated resources if version 2.6.x was upgraded from 2.4.x:

Note: You need to wait 30 minutes or more before you upgrade from version 2.5.x to version 2.6.x.

You can delete from the console, or you can run a command similar to the following example for the resources you want to delete:

oc delete -n <managed cluster namespace> managedclusteraddons.addon.open-cluster-management.io <resource-name>

See the list of deprecated resources that might remain:

managedclusteraddons.addon.open-cluster-management.io:
policy-controller
manifestworks.work.open-cluster-management.io:
-klusterlet-addon-appmgr
-klusterlet-addon-certpolicyctrl
-klusterlet-addon-crds
-klusterlet-addon-iampolicyctrl
-klusterlet-addon-operator
-klusterlet-addon-policyctrl
-klusterlet-addon-workmgr

After upgrading Red Hat Advanced Cluster Management to a new version, a few pods that belong to a StatefulSet might remain in a failed state. This infrequent event is caused by a known Kubernetes issue.

As a workaround for this problem, delete the failed pod. Kubernetes automatically relaunches it with the correct settings.

When an OpenShift Container Platform cluster is in the upgrade stage, the cluster pods are restarted and the cluster might remain in upgrade failed status for a variation of 1-5 minutes. This behavior is expected and resolves after a few minutes.

After installing Red Hat Advanced Cluster Management for Kubernetes in the Red Hat OpenShift Container Platform console, a pop-up window with the following message appears:

MultiClusterEngine required

Create a MultiClusterEngine instance to use this Operator.

The Create MultiClusterEngine button in the pop-up window message might not work. To work around the issue, select Create instance in the MultiClusterEngine tile in the Provided APIs section.

The search-postgres pod is in CrashLoopBackoff state. If Red Hat Advanced Cluster Management is deployed in a cluster with nodes that have the hugepages parameter enabled and the search-postgres pod gets scheduled in these nodes, then the pod does not start.

Complete the following steps to increase the memory of the search-postgres pod:

The following value appears, 512Mi.

There are known issues with dark theme styling for older versions of Firefox. Upgrade to the latest version for the best console compatibility.

For more information, see Supported browsers.

When you update the storage size in the searchcustomization CR, the PVC configuration does not change. If you need to update the storage size, update the PVC (<storageclassname>-search-redisgraph-0) with the following command:

oc edit pvc <storageclassname>-search-redisgraph-0

If an environment becomes large and requires more tests for scaling, the search queries can timeout which results in a parsing error message being displayed. This error is displayed after 30 seconds of waiting for a search query.

Extend the timeout time with the following command:

kubectl annotate route multicloud-console haproxy.router.openshift.io/timeout=Xs

When you edit namespace bindings for a cluster set with the admin role or bind role, you might encounter an error that resembles the following message:

ResourceError: managedclustersetbindings.cluster.open-cluster-management.io "<cluster-set>" is forbidden: User "<user>" cannot create/delete resource "managedclustersetbindings" in API group "cluster.open-cluster-management.io" in the namespace "<namespace>".

To resolve the issue, make sure you also have permission to create or delete a ManagedClusterSetBinding resource in the namespace you want to bind. The role bindings only allow you to bind the cluster set to the namespace.

After provisioning a hosted control plane cluster, you might not be able to scroll horizontally in the cluster overview of the Red Hat Advanced Cluster Management console if the ClusterVersionUpgradeable parameter is too long. You cannot view the hidden data as a result.

To work around the issue, zoom out by using your browser zoom controls, increase your Red Hat Advanced Cluster Management console window size, or copy and paste the text to a different location.

When you add multiple label expressions or attempt to enter your cluster selector for your ApplicationSet, you might receive the following message repeatedly, "Expand to enter expression". You can enter your cluster selection despite this issue.

When you deploy ApplicationSet resources using the Argo CD pull model, the application is displayed incorrectly from the Topology page. Deploying ApplicationSet applications using the Argo CD pull model is not supported.

The hub cluster application set deploys to target managed clusters, but the local cluster, which is a managed hub cluster, is excluded as a target managed cluster.

Both the Argo CD controller and the propagation controller might reconcile on the same application resource and cause the duplicate instances of application deployment on the managed clusters, but from the different deployment models.

For deploying applications by using the pull model, the Argo CD controllers ignore these application resources when the Argo CD argocd.argoproj.io/skip-reconcile annotation is added to the template section of the ApplicationSet.

The argocd.argoproj.io/skip-reconcile annotation is only available in the GitOps operator version 1.9.0, or later. To prevent conflicts, wait until the hub cluster and all the managed clusters are upgraded to GitOps operator version 1.9.0 before implementing the pull model.

All the resources listed in the MulticlusterApplicationSetReport are actually deployed on the managed clusters. If a resource fails to deploy, the resource is not included in the resource list, but the cause is listed in the error message.

For large environments with over 1000 managed clusters and Argo CD application sets that are deployed to hundreds of managed clusters, Argo CD application creation on the hub cluster might take several minutes. You can set the requeueAfterSeconds to zero in the clusterDecisionResource generator of the application set, as it is displayed in the following example file:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cm-allclusters-app-set
  namespace: openshift-gitops
spec:
  generators:
  - clusterDecisionResource:
      configMapRef: ocm-placement-generator
      labelSelector:
        matchLabels:
          cluster.open-cluster-management.io/placement: app-placement
      requeueAfterSeconds: 0

You cannot specify allow and deny lists with ObjectBucket channel type in the subscription-admin role. In other channel types, the allow and deny lists in the subscription indicates which Kubernetes resources can be deployed, and which Kubernetes resources should not be deployed.

The application-manager add-on that is running on the managed clusters is now handled by the subscription operator, when it was previously handled by the klusterlet operator. The subscription operator is not managed the multicluster-hub, so changes to the multicluster_operators_subscription image in the multicluster-hub image manifest ConfigMap do not take effect automatically.

If the image that is used by the subscription operator is overrided by changing the multicluster_operators_subscription image in the multicluster-hub image manifest ConfigMap, the application-manager add-on on the managed clusters does not use the new image until the subscription operator pod is restarted. You need to restart the pod.

The policy.open-cluster-management.io/v1 resources are no longer deployed by an application subscription by default for Red Hat Advanced Cluster Management version 2.4.

A subscription administrator needs to deploy the application subscription to change this default behavior.

See Creating an allow and deny list as subscription administrator for information. policy.open-cluster-management.io/v1 resources that were deployed by existing application subscriptions in previous Red Hat Advanced Cluster Management versions remain, but are no longer reconciled with the source repository unless the application subscriptions are deployed by a subscription administrator.

Ansible hook stand-alone mode is not supported. To deploy Ansible hook on the hub cluster with a subscription, you might use the following subscription YAML:

apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  name: sub-rhacm-gitops-demo
  namespace: hello-openshift
annotations:
  apps.open-cluster-management.io/github-path: myapp
  apps.open-cluster-management.io/github-branch: master
spec:
  hooksecretref:
      name: toweraccess
  channel: rhacm-gitops-demo/ch-rhacm-gitops-demo
  placement:
     local: true

However, this configuration might never create the Ansible instance, since the spec.placement.local:true has the subscription running on standalone mode. You need to create the subscription in hub mode.

After applying both, you should see the Ansible instance created in your hub cluster.

If applications are not deploying after an update to a placement rule, verify that the application-manager pod is running. The application-manager is the subscription container that needs to run on managed clusters.

You can run oc get pods -n open-cluster-management-agent-addon |grep application-manager to verify.

You can also search for kind:pod cluster:yourcluster in the console and see if the application-manager is running.

If you cannot verify, attempt to import the cluster again and verify again.

Learn about Red Hat OpenShift Container Platform SCC at Managing Security Context Constraints (SCC), which is an additional configuration required on the managed cluster.

Different deployments have different security context and different service accounts. The subscription operator cannot create an SCC CR automatically.. Administrators control permissions for pods. A Security Context Constraints (SCC) CR is required to enable appropriate permissions for the relative service accounts to create pods in the non-default namespace. To manually create an SCC CR in your namespace, complete the following steps:

Creating more than one channel in the same namespace can cause errors with the hub cluster.

For instance, namespace charts-v1 is used by the installer as a Helm type channel, so do not create any additional channels in charts-v1. Ensure that you create your channel in a unique namespace. All channels need an individual namespace, except GitHub channels, which can share a namespace with another GitHub channel.

Ansible jobs fail to run when you select an incompatible option. Ansible Automation Platform only works when the -cluster-scoped channel options are chosen. This affects all components that need to perform Ansible jobs.

The Red Hat Ansible Automation Platform operator cannot access Ansible Automation Platform outside of a proxy-enabled OpenShift Container Platform cluster. To resolve, you can install the Ansible Automation Platform within the proxy. See install steps that are provided by Ansible Automation Platform.

An application name cannot exceed 37 characters. The application deployment displays the following error if the characters exceed this amount.

status:
  phase: PropagationFailed
  reason: 'Deployable.apps.open-cluster-management.io "_long_lengthy_name_" is invalid: metadata.labels: Invalid value: "_long_lengthy_name_": must be no more than 63 characters/n'

See the following limitations to various Application tables in the console:

The Console and Topology for Application changes for the 2.8. There is no filtering capability from the console Topology page.

The allow and deny list feature does not work in Object storage application subscriptions.

When various hub clusters deploy Red Hat Advanced Cluster Management observability using the same S3 storage, duplicate local-clusters can be detected and displayed within the Kubernetes/Service-Level Overview/API Server dashboard. The duplicate clusters affect the results within the following panels: Top Clusters, Number of clusters that has exceeded the SLO, and Number of clusters that are meeting the SLO. The local-clusters are unique clusters associated with the shared S3 storage. To prevent multiple local-clusters from displaying within the dashboard, it is recommended for each unique hub cluster to deploy observability with a S3 bucket specifically for the hub cluster.

The observability endpoint operator fails if you create a pull-secret to deploy to the MultiClusterObservability CustomResource (CR) and there is no pull-secret in the open-cluster-management-observability namespace. When you import a new cluster, or import a Hive cluster that is created with Red Hat Advanced Cluster Management, you need to manually create a pull-image secret on the managed cluster.

For more information, see Enabling observability.

Red Hat Advanced Cluster Management observability does not display data from a ROKS cluster on some panels within built-in dashboards. This is because ROKS does not expose any API server metrics from servers they manage. The following Grafana dashboards contain panels that do not support ROKS clusters: Kubernetes/API server, Kubernetes/Compute Resources/Workload, Kubernetes/Compute Resources/Namespace(Workload)

For ROKS clusters, Red Hat Advanced Cluster Management observability does not display data in the etcd panel of the dashboard.

By default, Prometheus on OpenShift uses ephemeral storage. Prometheus loses all metrics data whenever it is restarted.

When observability is enabled or disabled on OpenShift Container Platform managed clusters that are managed by Red Hat Advanced Cluster Management, the observability endpoint operator updates the cluster-monitoring-config ConfigMap by adding additional alertmanager configuration that restarts the local Prometheus automatically.

Observability receive pods report the following error message:

Error on ingesting out-of-order samples

The error message means that the time series data sent by a managed cluster, during a metrics collection interval is older than the time series data it sent in the previous collection interval. When this problem happens, data is discarded by the Thanos receivers and this might create a gap in the data shown in Grafana dashboards. If the error is seen frequently, it is recommended to increase the metrics collection interval to a higher value. For example, you can increase the interval to 60 seconds.

The problem is only noticed when the time series interval is set to a lower value, such as 30 seconds. Note, this problem is not seen when the metrics collection interval is set to the default value of 300 seconds.

If you have a grafana-dev instance deployed in earlier versions before 2.6, and you upgrade the environment to 2.6, the grafana-dev does not work. You must delete the existing grafana-dev instance by running the following command:

./setup-grafana-dev.sh --clean

Recreate the instance with the following command:

./setup-grafana-dev.sh --deploy

The klusterlet-addon-search pod fails because the memory limit is reached. You must update the memory request and limit by customizing the klusterlet-addon-search deployment on your managed cluster. Edit the ManagedclusterAddon custom resource named search-collector, on your hub cluster. Add the following annotations to the search-collector and update the memory, addon.open-cluster-management.io/search_memory_request=512Mi and addon.open-cluster-management.io/search_memory_limit=1024Mi.

For example, if you have a managed cluster named foobar, run the following command to change the memory request to 512Mi and the memory limit to 1024Mi:

oc annotate managedclusteraddon search-collector -n foobar \
addon.open-cluster-management.io/search_memory_request=512Mi \
addon.open-cluster-management.io/search_memory_limit=1024Mi

The Grafana dashboard shows an empty label list if the disableHubSelfManagement parameter is set to true in the mulitclusterengine custom resource. You must set the parameter to false or remove the parameter to see the label list. See disableHubSelfManagement for more details.

Endpoint url cannot have fully qualified paths

endpoint: example.com:443

A proxy configuration in a managed cluster that you configure by using the addonDeploymentConfig is not detected by the metrics collector. As a workaround, you can enable the proxy by removing the managed cluster ManifestWork. Removing the ManifestWork forces the changes in the addonDeploymentConfig to be applied.

A proxy configuration in a managed cluster does not work when a custom CA bundle is required.

When you use an external identity provider to log in to Red Hat Advanced Cluster Management, you might not be able to log out of Red Hat Advanced Cluster Management. This occurs when you use Red Hat Advanced Cluster Management, installed with IBM Cloud and Keycloak as the identity providers.

You must log out of the external identity provider before you attempt to log out of Red Hat Advanced Cluster Management.

When you install the gatekeeper operator on Red Hat OpenShift Container Platform version 4.9, the installation fails. Before you upgrade OpenShift Container Platform to version 4.9.0., you must upgrade the gatekeeper operator to version 0.2.0. See Upgrading gatekeeper and the gatekeeper operator for more information.

When you have a configuration policy that is configured with mustnothave for the complianceType parameter and enforce for the remediationAction parameter, the policy is listed as compliant after a deletion request is made to the Kubernetes API. Therefore, the Kubernetes object can be stuck in a Terminating state while the policy is listed as compliant.

While installation into an ARM environment is supported, operators that are deployed with policies might not support ARM environments. The following policies that install operators do not support ARM environments:

When you remove the config-policy-controller add-on from a managed cluster by disabling the policy controller in the KlusterletAddonConfig or by detaching the cluster, the ConfigurationPolicy CRD might get stuck in a terminating state. If the ConfigurationPolicy CRD is stuck in a terminating state, new policies might not be added to the cluster if the add-on is reinstalled later. You can also receive the following error:

template-error; Failed to create policy template: create not allowed while custom resource definition is terminating

Use the following command to check if the CRD is stuck:

oc get crd configurationpolicies.policy.open-cluster-management.io -o=jsonpath='{.metadata.deletionTimestamp}'

If a deletion timestamp is on the resource, the CRD is stuck. To resolve the issue, remove all finalizers from configuration policies that remain on the cluster. Use the following command on the managed cluster and replace <cluster-namespace> with the managed cluster namespace:

oc get configurationpolicy -n <cluster-namespace> -o name | xargs oc patch -n <cluster-namespace> --type=merge -p '{"metadata":{"finalizers": []}}'

The configuration policy resources are automatically removed from the cluster and the CRD exits its terminating state. If the add-on has already been reinstalled, the CRD is recreated automatically without a deletion timestamp.

When you modify an existing configuration policy, pruneObjectBehavior does not work. View the following reasons why pruneObjectBehavior might not work:

If a policy is set to remediationAction: enforce and is repeatedly updated, the Red Hat Advanced Cluster Management console shows repeated violations with successful updates. See the following two possible causes and solutions for the error:

For example, the stringData map in a Secret resource is converted by Kubernetes to data with base64 encoded values. Instead of using stringData, use data directly with base64 encoded values instead of strings.

The support of pod security policies is removed from OpenShift Container Platform 4.12 and later, and from Kubernetes v1.25 and later. If you apply a PodSecurityPolicy resource, you might receive the following non-compliant message:

violation - couldn't find mapping resource with kind PodSecurityPolicy, please check if you have CRD deployed

When you create a policy with identical policy template names, you receive inconsistent results that are not detected, but you might not know the cause. For example, defining a policy with multiple configuration policies named create-pod causes inconsistent results. Best practice: Avoid using duplicate names for policy templates.

When you disable governance deployments in the MultiClusterHub object, the deployments are not cleaned without errors. Complete the following steps to disable governance so that the deployments also get cleaned up:

When there are templating errors, such as incorrect syntax in a configuration policy, objects are deleted. Recreate your deleted object with the correct syntax.

If you have a PolicyAutomation that is set to Run once mode and disabled, an extra Ansible job is created. You can delete the extra Ansible job. Complete the following steps:

See the following known issues and limitations that might occur while using networking features.

graceful termination failed, controllers failed with error: the server could not find the requested resource (post clustermanagementaddons.addon.open-cluster-management.io)

apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
kind: SubmarinerConfig
metadata:
   name: submariner
   namespace: <managed-cluster-namespace>
spec:
   insecureBrokerConnection: true

GDPR establishes a stronger data protection regulatory framework for processing personal data of individuals. GDPR brings:

As a platform, Red Hat Advanced Cluster Management for Kubernetes deals with several categories of technical data that could be considered as personal data, such as an administrator user ID and password, service user IDs and passwords, IP addresses, and Kubernetes node names. The Red Hat Advanced Cluster Management for Kubernetes platform also deals with information about users who manage the platform. Applications that run on the platform might introduce other categories of personal data unknown to the platform.

Information on how this technical data is collected/created, stored, accessed, secured, logged, and deleted is described in later sections of this document.

Customers can submit online comments/feedback/requests for information about in a variety of ways, primarily:

Typically, only the client name and email address are used, to enable personal replies for the subject of the contact, and the use of personal data conforms to the Red Hat Online Privacy Statement.

The Red Hat Advanced Cluster Management for Kubernetes platform authentication manager accepts user credentials from the console and forwards the credentials to the backend OIDC provider, which validates the user credentials against the enterprise directory. The OIDC provider then returns an authentication cookie (auth-cookie) with the content of a JSON Web Token (JWT) to the authentication manager. The JWT token persists information such as the user ID and email address, in addition to group membership at the time of the authentication request. This authentication cookie is then sent back to the console. The cookie is refreshed during the session. It is valid for 12 hours after you sign out of the console or close your web browser.

For all subsequent authentication requests made from the console, the front-end NGINX server decodes the available authentication cookie in the request and validates the request by calling the authentication manager.

The Red Hat Advanced Cluster Management for Kubernetes platform CLI requires the user to provide credentials to log in.

The kubectl and oc CLI also requires credentials to access the cluster. These credentials can be obtained from the management console and expire after 12 hours. Access through service accounts is supported.

Red Hat Advanced Cluster Management for Kubernetes platform supports role-based access control (RBAC). In the role mapping stage, the user name that is provided in the authentication stage is mapped to a user or group role. The roles are used when authorizing which administrative activities can be carried out by the authenticated user.

Red Hat Advanced Cluster Management for Kubernetes platform roles control access to cluster configuration actions, to catalog and Helm resources, and to Kubernetes resources. Several IAM (Identity and Access Management) roles are provided, including Cluster Administrator, Administrator, Operator, Editor, Viewer. A role is assigned to users or user groups when you add them to a team. Team access to resources can be controlled by namespace.

Pod security policies are used to set up cluster-level control over what a pod can do or what it can access.