Chapter 1. Installing and upgrading

The maximum number of clusters that Red Hat Advanced Cluster Management can manage varies based on several factors, including:

The managed clusters are single-node OpenShift virtual machines hosted on Red Hat Enterprise Linux hypervisors. Virtual machines are used to achieve high-density counts of clusters per single bare metal machine in the testbed. Sushy-emulator is used with libvirt for virtual machines to have an accessible bare metal cluster by using Redfish APIs. The following operators are a part of the test installation, Topology Aware Lifecycle Manager, Local Storage Operator, and Red Hat OpenShift GitOps. The following table shows the lab environment scaling information:

The scalability of the Search component depends on the performance of the data store. The query run time is an important variable when analyzing the search performance.

You need to plan your environment if you want to enable and use the observability service. The resource consumption later is for the OpenShift Container Platform project, where observability components are installed. Values that you plan to use are sums for all observability components.

Note: Data is based on the results from a lab environment at the time of testing. Your results might vary, depending on your environment, network speed, and changes to the product.

The tests performed on large scaled environment show the following data for backup and restore:

The total backup time is 10m.

Total restore time is 50m18s.

The number of backup file are pruned using the veleroTtl parameter option set when the BackupSchedule is created. Any backups with a creation time older than the specified TTL (time to live) are expired and automatically deleted from the storage location by Velero.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: BackupSchedule
 metadata:
 name:schedule-acm
 namespace:open-cluster-management-backup
spec:
veleroSchedule:0 */1 * * *
veleroTtl:120h

Each Red Hat Advanced Cluster Management for Kubernetes cluster is unique and the following guidelines give sample deployment sizes for you. Recommendations are classified by size and purpose. Red Hat Advanced Cluster Management applies the following dimensions for sizing and placement of supporting services:

Important: For OpenShift Container Platform, distribute the master nodes of the cluster across three availability zones.

Before you install Red Hat Advanced Cluster Management, see the following requirements:

Verify that a Red Hat Advanced Cluster Management hub cluster is not already installed on your OpenShift Container Platform cluster. You cannot have more than one hub cluster.

You can only have one single Red Hat Advanced Cluster Management hub cluster installation on each OpenShift Container Platform cluster. Continue with the following steps if there is no Red Hat Advanced Cluster Management hub cluster installed:

You can proceed to install Red Hat Advanced Cluster Management from the console or the CLI.

Note: For installing the Red Hat Advanced Cluster Management hub cluster on infrastructure nodes, see the Installing the Red Hat Advanced Cluster Management hub cluster on infrastructure nodes section of this procedure.

Best practice: From the Administrator view in your console, install the OperatorHub web console interface that is provided with OpenShift Container Platform.

To gain access to the console, see the Accessing your console topic in Additional resources.

Install the operator and the objects. Complete the following steps:

Notes:

You can now configure your OpenShift Container Platform cluster to contain infrastructure nodes to run approved management components. Running components on infrastructure nodes avoids allocating OpenShift Container Platform subscription quota for the nodes that are running those management components. See Configuring infrastructure nodes for Red Hat Advanced Cluster Management for that procedure.

Learn about sizing, scaling, and advanced configuration.

Follow the procedures that are described in Creating infrastructure machine sets in the OpenShift Container Platform documentation. Infrastructure nodes are configured with a Kubernetes taints and labels to keep non-management workloads from running on them.

You must meet the following requirements before you install Red Hat Advanced Cluster Management for Kubernetes:

While you are connected, verify that a Red Hat Advanced Cluster Management hub cluster is not already installed on your OpenShift Container Platform cluster.

You can only have one single Red Hat Advanced Cluster Management hub cluster installation on each OpenShift Container Platform cluster.

Best practice: Use your existing mirror registry for the Operator Lifecycle Manager operator related content.

Installing Red Hat Advanced Cluster Management for Kubernetes in a disconnected environment requires a local mirror image registry. Because you have already completed the installation of the OpenShift Container Platform cluster in your disconnected environment, you already set up a mirror registry for use during the Red Hat OpenShift Container Platform cluster installation.

If you do not already have a local image registry, create one by completing the procedure that is described in Mirroring images for a disconnected installation of the Red Hat OpenShift Container Platform documentation.

Because Red Hat Advanced Cluster Management for Kubernetes is packaged as an operator, you need Operator Lifecycle Manager to install.

In disconnected environments, Operator Lifecycle Manager cannot access the standard operator sources that Red Hat provided operators can because they are hosted on image registries that are not accessible from a disconnected cluster. Instead, a cluster administrator can enable the installation and upgrade of operators in a disconnected environment by using mirrored image registries and operator catalogs.

To prepare your disconnected cluster for installing Red Hat Advanced Cluster Management for Kubernetes, follow the procedure that is described in Using Operator Lifecycle Manager on restricted networks in the OpenShift Container Platform documentation.

Operator Lifecycle Manager polls catalog sources for available packages on a regular timed interval. After Operator Lifecycle Manager polls the catalog source for your mirrored catalog, you can verify that the required packages are available from on your disconnected cluster by querying the available PackageManifest resources.

Run the following command, directed at your disconnected cluster:

oc -n openshift-marketplace get packagemanifests

The list that is displayed should include entries showing that the following packages are supplied by the catalog source for your mirror catalog:

In order to have your cluster obtain container images for the Red Hat Advanced Cluster Management for Kubernetes operator from your mirror registry, rather than from the internet-hosted registries, you must configure an ImageContentSourcePolicy on your disconnected cluster to redirect image references to your mirror registry.

If you mirrored your catalog using the oc adm catalog mirror command, the needed image content source policy configuration is in the imageContentSourcePolicy.yaml file inside of the manifests-* directory that is created by that command.

If you used the oc-mirror plug-in to mirror your catalog instead, the imageContentSourcePolicy.yaml file is within the oc-mirror-workspace/results-* directory create by the oc-mirror plug-in.

In either case, you can apply the policies to your disconnected command using an oc apply or oc replace command such as:

oc replace -f ./<path>/imageContentSourcePolicy.yaml

The required image content source policy statements can vary based on how you created your mirror registry, but are similar to this example:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  labels:
    operators.openshift.org/catalog: "true"
  name: operator-0
spec:
  repositoryDigestMirrors:
  - mirrors:
    - myregistry.example.com:5000/rhacm2
    source: registry.redhat.io/rhacm2
  - mirrors:
    - myregistry.example.com:5000/multicluster-engine
    source: registry.redhat.io/multicluster-engine
  - mirrors:
    - myregistry.example.com:5000/openshift4
    source: registry.redhat.io/openshift4
  - mirrors:
    - myregistry.example.com:5000/redhat
    source: registry.redhat.io/redhat

After you have configured Operator Lifecycle Manager and Red Hat OpenShift Container Platform as previously described, you can install Red Hat Advanced Cluster Management for Kubernetes by using either the OperatorHub console or a CLI. Follow the same guidance described in the Installing while connected online topic.

Important: Creating the MultiClusterHub resource is the beginning of the installation process of your hub cluster.

Because operator installation on a cluster requires the use of a non-default catalog source for the mirror catalog, a special annotation is needed in the MultiClusterHub resource to provide the name of the mirror catalog source to the operator. The following example displays the required mce-subscription-spec annotation:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
   namespace: open-cluster-management
   name: hub
   annotations:
      installer.open-cluster-management.io/mce-subscription-spec: '{"source": "my-mirror-catalog-source"}'
spec: {}

The mce-subscription-spec annotation is required because multicluster engine operator is automatically installed during the Red Hat Advanced Cluster Management installation. If you are creating the resource with a CLI, include the mce-subscription-spec annotation in the YAML that you apply with the oc apply command to create the MultiClusterHub resource.

If you create the resource by using the OperatorHub console, switch to the YAML view and insert the annotation as previously displayed. Important: There is no field in the OperatorHub console for the annotation in the Field view panel to create the MultiClusterHub resource.

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: redhat-operators
  namespace: openshift-marketplace
spec:
   displayName: Red Hat Operators
   priority: -100

The following example displays the spec.overrides default template that you can use to enable or disable the component:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace> 1
spec:
  overrides:
    components:
    - name: <name> 2
      enabled: true

Alternatively, you can run the following command. Replace namespace with the name of your project and name with the name of the component:

oc patch MultiClusterHub multiclusterhub -n <namespace> --type=json -p='[{"op": "add", "path": "/spec/overrides/components/-","value":{"name":"<name>","enabled":true}}]'

Note: When the console component is disabled, the Red Hat OpenShift Container Platform console is disabled.

If you plan to import Kubernetes clusters that were not created by OpenShift Container Platform or Red Hat Advanced Cluster Management, generate a secret that has your OpenShift Container Platform pull secret information to access the entitled content from the distribution registry.

The secret requirements for OpenShift Container Platform clusters are automatically resolved by OpenShift Container Platform and Red Hat Advanced Cluster Management, so you do not have to create the secret if you are not importing other types of Kubernetes clusters to be managed. Your OpenShift Container Platform pull secret is associated with your Red Hat Customer Portal ID, and is the same across all Kubernetes providers.

Important: These secrets are namespace-specific, so make sure that you are in the namespace that you use for your hub cluster.

The following example displays the spec.imagePullSecret template to use if you want to use a custom pull secret. Replace secret with the name of your pull secret:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  imagePullSecret: <secret>

The Red Hat Advanced Cluster Management hub cluster has two availabilities: High and Basic. By default, the hub cluster has an availability of High, which gives hub cluster components a replicaCount of 2. This provides better support in cases of failover but consumes more resources than the Basic availability, which gives components a replicaCount of 1.

Important: Set spec.availabilityConfig to Basic if you are using multicluster engine operator on a single-node OpenShift cluster.

The following example shows the spec.availabilityConfig template with Basic availability:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  availabilityConfig: "Basic"

You can define a set of node selectors in the Red Hat Advanced Cluster Management hub cluster to install to specific nodes on your cluster. The following example shows spec.nodeSelector to assign Red Hat Advanced Cluster Management pods to nodes with the label node-role.kubernetes.io/infra:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  nodeSelector:
    node-role.kubernetes.io/infra: ""

To define a set of node selectors for the multicluster engine operator hub cluster, see nodeSelector in the multicluster engine operator documentation.

You can define a list of tolerations to allow the Red Hat Advanced Cluster Management hub cluster to tolerate specific taints defined on the cluster.

The following example shows a spec.tolerations that matches a node-role.kubernetes.io/infra taint:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  tolerations:
  - key: node-role.kubernetes.io/infra
    effect: NoSchedule
    operator: Exists

The previous infra-node toleration is set on pods by default without specifying any tolerations in the configuration. Customizing tolerations in the configuration replaces this default.

To define a list of tolerations for the multicluster engine operator hub cluster, see tolreations in the multicluster engine operator documentation.

By default, the Red Hat Advanced Cluster Management hub cluster is automatically imported and managed by itself. This managed hub cluster is named, local-cluster. The setting that specifies whether a hub cluster manages itself is in the multiclusterengine custom resource. Changing this setting in Red Hat Advanced Cluster Management automatically changes the setting in the multiclusterengine custom resource.

Note: On a Red Hat Advanced Cluster Management hub cluster that is managing a multicluster engine operator cluster, any earlier manual configurations are replaced by this action.

If you do not want the Red Hat Advanced Cluster Management hub cluster to manage itself, you need to change the setting for spec.disableHubSelfManagement from false to true. If the setting is not included in the YAML file that defines the custom resource, you need to add it. The hub cluster can only be managed with this option.

Setting this option to true and attempting to manage the hub manually leads to unexpected behavior.

The following example shows the default template to use if you want to disable the hub cluster self-management feature. Replace namespace with the name of your project:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  disableHubSelfManagement: true

To enable the default local-cluster, return the setting to false, or remove this setting.

If you want to ensure that you use the same release image for all of your clusters, you can create your own custom list of release images that are available when you create a cluster.

See the following instructions in Maintaining a custom list of release images when connected to manage your available release images and to set the spec.disableUpdateClusterImageSets attribute, which stops the custom image list from being overwritten.

The following example shows the default template that disables updates to the cluster image set. Replace namespace with the name of your project:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  disableUpdateClusterImageSets: true

By default, Red Hat OpenShift Container Platform uses the Ingress Operator to create an internal CA.

The following example shows the default template used to provide a customized OpenShift Container Platform default ingress CA certificate to Red Hat Advanced Cluster Management. Replace namespace with the name of your project. Replace the spec.customCAConfigmap value with the name of your ConfigMap:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  customCAConfigmap: <configmap>

By default, the Red Hat Advanced Cluster Management hub cluster includes the full list of supported SSL ciphers.

The following example shows the default spec.ingress.sslCiphers template that is used to list sslCiphers for the management ingress. Replace namespace with the name of your project:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  ingress:
    sslCiphers:
    - "ECDHE-ECDSA-AES128-GCM-SHA256"
    - "ECDHE-RSA-AES128-GCM-SHA256"

The enableClusterBackup field is no longer supported and is replaced by this component.

The following example shows the spec.overrides default template used to enable ClusterBackup. Replace namespace with the name of your project:

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: <namespace>
spec:
  overrides:
    components:
    - name: cluster-backup
      enabled: true

Alternatively, you can run the following command. Replace namespace with the name of your project.

oc patch MultiClusterHub multiclusterhub -n <namespace> --type=json -p='[{"op": "add", "path": "/spec/overrides/components/-","value":{"name":"cluster-backup","enabled":true}}]'

If you are Managing cluster pools (Technology Preview), you need further configuration to stop automatic management of these cluster pools after upgrade.

Set cluster.open-cluster-management.io/createmanagedcluster: "false" in the ClusterClaim metadata.annotations.

All existing cluster claims are automatically imported when the product is upgraded unless you change this setting.

Red Hat Advanced Cluster Management uses the related multicluster engine operator functionality to provide foundational services that were delivered as part of the product. Red Hat Advanced Cluster Management automatically installs and manages the required multicluster engine operator and MulticlusterEngine resource instance as part of the hub cluster installation and upgrade.

In connected network environments, the cluster administrator can install or upgrade Red Hat Advanced Cluster Management without special mirror catalogs and catalog sources. However, because installation of any Operator Lifecycle Manager operator in a disconnected environment involves the use of special mirror catalogs and catalog sources, as described in the earlier sections, some additional steps are necessary after installation.

Important: If you begin an upgrade before you add the annotation, the upgrade begins but stalls when the operator attempts to install a subscription to multicluster-engine in the background. The status of the MulticlusterHub resource continues to display upgrading during this time.

To resolve this issue, run oc edit to add the mce-subscription-spec annotation as shown previously.

Install in disconnected network environments

You must have the following prerequisites before you can use OpenShift Update Service to upgrade your disconnected clusters:

You must mirror both the image that you want to upgrade to and the current image that you are upgrading from to your local mirror registry. Complete the following steps to mirror the images:

Tip: You can use the output of the last line of this script when you create your ImageContentSourcePolicy.

To deploy the operator for OpenShift Update Service in your OpenShift Container Platform environment, complete the following steps:

OpenShift Update Service uses graph data information to determine the available upgrades. In a connected environment, OpenShift Update Service pulls the graph data information for available upgrades directly from the Cincinnati graph data GitHub repository. Because you are configuring a disconnected environment, you must make the graph data available in a local repository by using an init container. Complete the following steps to create a graph data init container:

If you are using a secure external container registry to store your mirrored OpenShift Container Platform release images, OpenShift Update Service requires access to this registry to build an upgrade graph. Complete the following steps to configure your CA certificate to work with the OpenShift Update Service pod:

The OpenShift Update Service Operator watches the image.config.openshift.io API and the ConfigMap you created in the openshift-config namespace for changes, then restart the deployment if the CA cert has changed.

When you finish deploying the OpenShift Update Service instance on your hub cluster, this instance is located where the images for the cluster upgrades are mirrored and made available to the disconnected managed cluster. Complete the following steps to deploy the instance:

Note: The steps in this section only apply if you have mirrored your releases into your mirrored registry. Deprecated: PlacementRule

OpenShift Container Platform has a default image registry value that specifies where it finds the upgrade packages. In a disconnected environment, you can create a policy to replace that value with the path to your local image registry where you mirrored your release images.

For these steps, the policy is named policy-mirror. Complete the following steps to create the policy:

Push the Catalogsource policy to the managed cluster to change the default location from a connected location to your disconnected local registry.

Push the ClusterVersion policy to the managed cluster to change the default location where it retrieves its upgrades.

You can view a list of available upgrades for your managed cluster by completing the following steps:

You can use the Red Hat Advanced Cluster Management console to select a channel for your cluster upgrades on OpenShift Container Platform version 4.6 or later. Those versions must be available on the mirror registry. Complete the steps in Selecting a channel to specify a channel for your upgrades.

After configuring the disconnected registry, Red Hat Advanced Cluster Management and OpenShift Update Service use the disconnected registry to determine if upgrades are available. If no available upgrades are displayed, make sure that you have the release image of the current level of the cluster and at least one later level mirrored in the local repository. If the release image for the current version of the cluster is not available, no upgrades are available.

Complete the following steps to upgrade:

The managed cluster is updated to the selected version.

If your cluster upgrade fails, the Operator generally retries the upgrade a few times, stops, and reports the status of the failing component. In some cases, the upgrade process continues to cycle through attempts to complete the process. Rolling your cluster back to a previous version following a failed upgrade is not supported. Contact Red Hat support for assistance if your cluster upgrade fails.

Delete the MultiClusterHub custom resource and remove artifacts. Complete the following steps:

When you use the Red Hat OpenShift Container Platform console to uninstall, you remove the MultiClusterHub resource to delete the object. Wait for the status, then you uninstall the operator. Complete the following steps to uninstall by using the console:

Remove all artifacts that remain by running the clean-up script. You need to clean up artifacts if you plan to reinstall Red Hat Advanced Cluster Management with an older version of Red Hat Advanced Cluster Management on the same cluster.