Chapter 2. Working with hosted clusters in Red Hat Advanced Cluster Management
If you have multicluster engine operator clusters that are hosting multiple hosted clusters, you can bring those hosted clusters to a Red Hat Advanced Cluster Management hub cluster to manage with Red Hat Advanced Cluster Management components, such as Application lifecycle and Governance.
Begin with learning from the About hosted control planes with multicluster engine operator documentation. Then, see the following procedures for managing hosted clusters with Red Hat Advanced Cluster Management:
2.1. Integrated hosted control plane fleet management architecture pattern 링크 복사링크가 클립보드에 복사되었습니다!
As your Red Hat Advanced Cluster Management for Kubernetes and multicluster engine for Kubernetes operator hosted control plane deployment scales, you can strategically manage resource requirements and maintain centralized visibility across your fleet.
Consider the following guidance for an architecture pattern for managing hosted control plane clusters at scale.
Red Hat Advanced Cluster Management for Kubernetes or multicluster engine for Kubernetes operator enables hosted control plane capabilities. While deployments often begin with a single hub cluster, scaling the number of hosted control plane clusters introduces several challenges. See the following examples:
- Resource consumption: Resource requirements grow proportionally with the number of planes.
- Scaling complexity: You must choose between scaling existing clusters or provisioning new ones.
- Management overhead: Managing multiple hosting clusters individually becomes complex.
- Visibility gaps: Maintaining centralized visibility and governance across disparate clusters is difficult.
To address these challenges, implement a hierarchical management pattern, which is an architecture that separates the management logic from the hosting infrastructure. Learn more about this pattern in the sections that follow.
Learn about the benefits of this architecture pattern in the following table:
| Benefit | Description |
|---|---|
| Centralized management | Provides a single pane of glass for the entire fleet. |
| Scalability | Allows you to add multicluster engine operator clusters as needed to increase capacity. |
| Resource optimization | Focuses multicluster engine operator clusters on hosting and the Red Hat Advanced Cluster Management hub cluster on management. |
| Operational efficiency | Enables unified policy management and governance. |
| Cost effectiveness | Allows you to right-size each component for a specific role. |
2.1.1. Central Red Hat Advanced Cluster Management hub cluster 링크 복사링크가 클립보드에 복사되었습니다!
The central Red Hat Advanced Cluster Management hub cluster serves as the primary management and visibility layer for the following functions:
- Provides a single console for end-to-end fleet management
- Manages operational policies across the entire infrastructure
- Enables comprehensive monitoring and governance capabilities
2.1.2. multicluster engine operator hosting infrastructure 링크 복사링크가 클립보드에 복사되었습니다!
Deploy multiple multicluster engine operator clusters to serve as the physical hosting infrastructure for control planes. See the following functions of the multicluster engine operator hosting cluster:
- Provides hosted control plane components with a smaller footprint than a full Red Hat Advanced Cluster Management installation.
- Includes the management console and BareMetal infrastructure operators.
- Optimizes for hosting workloads rather than high-level fleet management.
2.1.3. Implementation considerations 링크 복사링크가 클립보드에 복사되었습니다!
Learn about the process and gather other important information for fleet management architecture.
- The central Red Hat Advanced Cluster Management hub cluster manages both the multicluster engine operator clusters and the attached hosted control plane clusters
- Policies and governance are applied consistently across the entire infrastructure.
- Monitoring and alerting are centralized while hosting capacity remains distributed.
See the following process overview:
- Deploy the central Red Hat Advanced Cluster Management hub cluster: Establish this cluster as your primary management interface.
- Deploy multicluster engine operator clusters: Use these as the hosting infrastructure for your control planes.
- Configure discovery: Ensure the Red Hat Advanced Cluster Management hub cluster can manage both the multicluster engine operator clusters and the attached hosted control plane clusters.
For detailed instructions on discovery configuration, see Discovering multicluster engine operator hosted clusters in Red Hat Advanced Cluster Management
Important: A Red Hat Advanced Cluster Management cluster cannot manage another Red Hat Advanced Cluster Management cluster.
If you have already deployed multiple Red Hat Advanced Cluster Management hub clusters to host control planes, use the following process for this pattern architecture:
- Convert Red Hat Advanced Cluster Management hosting clusters to multicluster engine operator: Uninstall Red Hat Advanced Cluster Management and install multicluster engine operator on the clusters that are designated for hosting infrastructure.
- Preserve hosted control plane clusters: Ensure the conversion process does not disrupt existing hosted control plane clusters.
- Designate the hub cluster: Select one Red Hat Advanced Cluster Management cluster to serve as the central management hub cluster.
To uninstall Red Hat Advanced Cluster Management from a cluster, follow the Uninstalling documentation. Then see the Installing multicluster engine operator documentation for installation instructions.
If you have multicluster engine operator clusters that are hosting multiple hosted clusters, you can bring those hosted clusters to a Red Hat Advanced Cluster Management hub cluster to manage with Red Hat Advanced Cluster Management components, such as Application lifecycle and Governance.
Those hosted clusters can be automatically discovered and imported as managed clusters.
Note: Since the hosted control planes run on the managed multicluster engine operator cluster nodes, the number of hosted control planes that the cluster can host is determined by the resource availability of managed multicluster engine operator cluster nodes, as well as the number of managed multicluster engine operator clusters. You can add more nodes or managed clusters to host more hosted control planes.
Required access: Cluster administrator
Prerequisites
- You need one or more multicluster engine operator clusters.
- You need a Red Hat Advanced Cluster Management cluster that is set as your hub cluster.
-
Install the
clusteradmCLI by running the following command:
curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash
curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash
multicluster engine operator has a local-cluster, which is a hub cluster that is managed. The following default addons are enabled for this local-cluster in the open-cluster-management-agent-addon namespace:
-
cluster-proxy -
managed-serviceaccount -
work-manager
Next you can configure add-ons. When your multicluster engine operator is imported into Red Hat Advanced Cluster Management, Red Hat Advanced Cluster Management enables the same set of add-ons to manage the multicluster engine operator.
Install those add-ons in a different multicluster engine operator namespace so that the multicluster engine operator can self-manage with the local-cluster add-ons while Red Hat Advanced Cluster Management manages multicluster engine operator at the same time. Complete the following procedure:
- Log in to your Red Hat Advanced Cluster Management with the CLI.
Update the
hypershift-addon-deploy-configinstall add-ons in theopen-cluster-management-agent-addon-discoverynamespace. Run the following command:oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"configureMceImport","value":"true"}]}}'oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"configureMceImport","value":"true"}]}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command to verify that the add-ons for the Red Hat Advanced Cluster Management
local-clusterare re-installed into the namespace that you specified:oc get deployment -n open-cluster-management-agent-addon-discovery
oc get deployment -n open-cluster-management-agent-addon-discoveryCopy to Clipboard Copied! Toggle word wrap Toggle overflow See the following output example:
NAME READY STATUS RESTARTS AGE application-manager-6b7f74b8f7-7sd25 1/1 Running 0 1d15h cluster-proxy-proxy-agent-7985ddfdb6-kng5p 3/3 Running 0 1d15h klusterlet-addon-workmgr-55fd575b4b-rs5vz 1/1 Running 0 1d15h managed-serviceaccount-addon-agent-54bd989b94-g6gz9 1/1 Running 0 1d15h
NAME READY STATUS RESTARTS AGE application-manager-6b7f74b8f7-7sd25 1/1 Running 0 1d15h cluster-proxy-proxy-agent-7985ddfdb6-kng5p 3/3 Running 0 1d15h klusterlet-addon-workmgr-55fd575b4b-rs5vz 1/1 Running 0 1d15h managed-serviceaccount-addon-agent-54bd989b94-g6gz9 1/1 Running 0 1d15hCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command to check the status of your configuration:
oc get configmap hypershift-addon-deploy-config-info -n multicluster-engine -o yaml
oc get configmap hypershift-addon-deploy-config-info -n multicluster-engine -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow
2.2.2. Importing multicluster engine operator manually 링크 복사링크가 클립보드에 복사되었습니다!
After configuring Red Hat Advanced Cluster Management add-ons to install in a different namespace than multicluster engine operator add-ons, you can start importing multicluster engine operator clusters. To manually import an multicluster engine operator cluster from your Red Hat Advanced Cluster Management cluster, complete the following procedure:
From your Red Hat Advanced Cluster Management cluster, create a
ManagedClusterresource manually to import an multicluster engine operator cluster. See the following file example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
The
mce-import-klusterlet-configannotation references theKlusterletConfigresource that you created in the previous step to install the Red Hat Advanced Cluster Management klusterlet with a different name in multicluster engine operator. -
Replace
<mname>with your cluster name.
-
The
-
Run
oc apply -f <filename>.yamlto apply the file. Optional: Create a
KlusterletAddonConfigresource for each multicluster engine operator cluster to enable additional Red Hat Advanced Cluster Management add-ons. Use the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Replace
<host-name>with your managed cluster name. -
Replace
<host-namespace>with your managed cluster namespace.
-
Replace
-
Run
oc apply -f <filename>.yamlto apply the file. Create the
auto-import-secretsecret that references thekubeconfigof the multicluster engine operator cluster. Go to Importing a cluster by using the auto import secret in Importing a managed cluster by using the CLI to add the auto import secret to complete the multicluster engine operator auto-import process.After you create the auto import secret in the multicluster engine operator managed cluster namespace in the Red Hat Advanced Cluster Management cluster, the managed cluster is registered.
Alternatively, extract the import manifest for your multicluster engine operator cluster. Run the following command. Replace
<name>with the name of your multicluster engine operator cluster:oc get secret mce-hosting-east-import -n <name> -o jsonpath={.data.import\\.yaml} | base64 --decode > import.yamloc get secret mce-hosting-east-import -n <name> -o jsonpath={.data.import\\.yaml} | base64 --decode > import.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the
import.yamlfile to your multicluster engine operator cluster. Run the following command:oc apply -f import.yaml
oc apply -f import.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command to get the status:
oc get managedcluster
oc get managedclusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow See following example output with the status and example URLs of managed clusters:
NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE local-cluster true https://<api.acm-hub.com:port> True True 44h mce-a true https://<api.mce-a.com:port> True True 27s
NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE local-cluster true https://<api.acm-hub.com:port> True True 44h mce-a true https://<api.mce-a.com:port> True True 27sCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Important: Do not enable any other Red Hat Advanced Cluster Management add-ons for the imported multicluster engine operator.
2.2.3. Discovering hosted clusters 링크 복사링크가 클립보드에 복사되었습니다!
After all your multicluster engine operator clusters are imported into Red Hat Advanced Cluster Management, you need to enable the hypershift-addon for those managed multicluster engine operator clusters to discover the hosted clusters.
Default add-ons are installed into a different namespace in the previous procedures. Similarly, you install the hypershift-addon into a different namespace in multicluster engine operator so that the add-ons agent for multicluster engine operator local-cluster and the agent for Red Hat Advanced Cluster Management can work in multicluster engine operator.
Important: For all the following commands, replace <managed-cluster-names> with comma-separated managed cluster names for multicluster engine operator.
Run the following command to set the
agentInstallNamespacenamespace of the add-on toopen-cluster-management-agent-addon-discovery:oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"agentInstallNamespace":"open-cluster-management-agent-addon-discovery"}}'oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"agentInstallNamespace":"open-cluster-management-agent-addon-discovery"}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command to disable metrics and to disable the HyperShift operator management:
oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value": "true"},{"name":"disableHOManagement","value": "true"}]}}'oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value": "true"},{"name":"disableHOManagement","value": "true"}]}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: Configure your naming convention. By default, imported hosted clusters use the
<mce-cluster-name>-<hosted-cluster-name>naming pattern, but you can customize your naming pattern.Remove the default prefix by running the following command:
oc patch addondeploymentconfig hypershift-addon-deploy-config \ -n multicluster-engine \ --type=merge \ -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value":"true"},{"name":"disableHOManagement","value":"true"},{"name":"discoveryPrefix","value":"custom-prefix"}]}}'oc patch addondeploymentconfig hypershift-addon-deploy-config \ -n multicluster-engine \ --type=merge \ -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value":"true"},{"name":"disableHOManagement","value":"true"},{"name":"discoveryPrefix","value":"custom-prefix"}]}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Change
custom-prefixto your new prefix. The hosted cluster names are created with the<custom-prefix>-<hosted-cluster-name>pattern. Run the following command:oc patch addondeploymentconfig hypershift-addon-deploy-config \ -n multicluster-engine \ --type=merge \ -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value":"true"},{"name":"disableHOManagement","value":"true"},{"name":"discoveryPrefix","value":"custom-prefix"}]}}'oc patch addondeploymentconfig hypershift-addon-deploy-config \ -n multicluster-engine \ --type=merge \ -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value":"true"},{"name":"disableHOManagement","value":"true"},{"name":"discoveryPrefix","value":"custom-prefix"}]}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow - If you need to remove the discovery prefix entirely, first ensure all hosted clusters are detached from the respective clusters. Important: Using an empty string as the custom prefix can cause klusterlet naming collisions within the multicluster engine operator cluster. Run the following command:
oc patch addondeploymentconfig hypershift-addon-deploy-config \ -n multicluster-engine \ --type=json \ -p='[{"op":"add","path":"/spec/customizedVariables/-","value":{"name":"autoImportDisabled","value":"true"}}]'oc patch addondeploymentconfig hypershift-addon-deploy-config \ -n multicluster-engine \ --type=json \ -p='[{"op":"add","path":"/spec/customizedVariables/-","value":{"name":"autoImportDisabled","value":"true"}}]'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command to enable the
hypershift-addonfor multicluster engine operator:clusteradm addon enable --names hypershift-addon --clusters <managed-cluster-names>
clusteradm addon enable --names hypershift-addon --clusters <managed-cluster-names>Copy to Clipboard Copied! Toggle word wrap Toggle overflow You can get the multicluster engine operator managed cluster names by running the following command in Red Hat Advanced Cluster Management.
oc get managedcluster
oc get managedclusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow Log into multicluster engine operator clusters and verify that the
hypershift-addonis installed in the namespace that you specified. Run the following command:oc get deployment -n open-cluster-management-agent-addon-discovery
oc get deployment -n open-cluster-management-agent-addon-discoveryCopy to Clipboard Copied! Toggle word wrap Toggle overflow See the following example output that lists the add-ons:
NAME READY UP-TO-DATE AVAILABLE AGE cluster-proxy-proxy-agent 1/1 1 1 24h klusterlet-addon-workmgr 1/1 1 1 24h hypershift-addon-agent 1/1 1 1 24h managed-serviceaccount-addon-agent 1/1 1 1 24h
NAME READY UP-TO-DATE AVAILABLE AGE cluster-proxy-proxy-agent 1/1 1 1 24h klusterlet-addon-workmgr 1/1 1 1 24h hypershift-addon-agent 1/1 1 1 24h managed-serviceaccount-addon-agent 1/1 1 1 24hCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Red Hat Advanced Cluster Management deploys the hypershift-addon, which is the discovery agent that discovers hosted clusters from multicluster engine operator. The agent creates the corresponding DiscoveredCluster custom resource in the multicluster engine operator managed cluster namespace in the Red Hat Advanced Cluster Management hub cluster when the hosted cluster kube-apiserver becomes available.
You can view your discovered clusters in the console.
Log in to hub cluster console and click Fleet Management > Infrastructure > Clusters.
Note: For OpenShift Container Platform versions earlier than version 4.20, select All Clusters from the cluster switcher.
-
Find the Discovered clusters tab to view all discovered hosted clusters from multicluster engine operator with type
MultiClusterEngineHCP.
Next, visit Automating import for discovered hosted clusters to learn how to automatically import clusters.
2.3. Automating import for discovered hosted clusters 링크 복사링크가 클립보드에 복사되었습니다!
Automate the import of hosted clusters by using the DiscoveredCluster resource for faster cluster management, without manually importing individual clusters.
When you automatically import a discovered hosted cluster into Red Hat Advanced Cluster Management, all Red Hat Advanced Cluster Management add-ons are enabled so that you can start managing the hosted clusters with the available management tools.
The hosted cluster is also auto-imported into multicluster engine operator. Through the multicluster engine operator console, you can manage the hosted cluster lifecycle. However, you cannot manage the hosted cluster lifecycle from the Red Hat Advanced Cluster Management console.
Required access: Cluster administrator
2.3.1. Prerequisites 링크 복사링크가 클립보드에 복사되었습니다!
- You need Red Hat Advanced Cluster Management installed. See the Red Hat Advanced Cluster Management Installing and upgrading documentation.
- You need to learn about Policies. See the introduction to Governance in the Red Hat Advanced Cluster Management documentation.
2.3.2. Configuring settings for automatic import 링크 복사링크가 클립보드에 복사되었습니다!
Discovered hosted clusters from managed multicluster engine operator clusters are represented in DiscoveredCluster custom resources, which are located in the managed multicluster engine operator cluster namespace in Red Hat Advanced Cluster Management. See the following DiscoveredCluster resource and namespace example:
Discovered hosted clusters are not automatically imported into Red Hat Advanced Cluster Management until the spec.importAsManagedCluster field is changed from false to true. Learn how to use a Red Hat Advanced Cluster Management policy to automatically set this field to true for all type.MultiClusterEngineHCP within DiscoveredCluster resources so that discovered hosted clusters are immediately and automatically imported into Red Hat Advanced Cluster Management.
Configure your Policy to import all your discovered hosted clusters.
- Log in to your hub cluster from the CLI to complete the following procedure:
Create a YAML file for your
DiscoveredClustercustom resource and edit the configuration that is referenced in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Run
oc apply -f <filename>.yaml -n <namespace>to apply the file.
2.3.3. Creating the placement definition 링크 복사링크가 클립보드에 복사되었습니다!
You need to create a placement definition that specifies the managed cluster for the policy deployment. Complete the following procedure:
Create the
Placementdefinition that selects only thelocal-cluster, which is a hub cluster that is managed. Use the following YAML sample:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Run
oc apply -f placement.yaml -n <namespace>, wherenamespacematches the namespace that you used for the policy that you previously created.
2.3.4. Binding the import policy to a placement definition 링크 복사링크가 클립보드에 복사되었습니다!
After you create the policy and the placement, you need to connect the two resources. Complete the following steps:
Connect the resources by using a
PlacementBindingresource. See the following example whereplacementRefreferences thePlacementthat you created, andsubjectsreferences thePolicythat you created:Copy to Clipboard Copied! Toggle word wrap Toggle overflow To verify, run the following command:
oc get policies.policy.open-cluster-management.io policy-mce-hcp-autoimport -n <namespace>
oc get policies.policy.open-cluster-management.io policy-mce-hcp-autoimport -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Important: You can detach a hosted cluster from Red Hat Advanced Cluster Management by using the Detach option in the Red Hat Advanced Cluster Management console, or by removing the corresponding ManagedCluster custom resource from the command line.
For best results, detach the managed hosted cluster before destroying the hosted cluster.
When a discovered cluster is detached, the following annotation is added to the DiscoveredCluster resource to prevent the policy to import the discovered cluster again.
annotations:
discovery.open-cluster-management.io/previously-auto-imported: "true"
annotations:
discovery.open-cluster-management.io/previously-auto-imported: "true"
If you want the detached discovered cluster to be reimported, remove this annotation.
2.4. Automating import for discovered Red Hat OpenShift Service on AWS clusters 링크 복사링크가 클립보드에 복사되었습니다!
Automate the import of Red Hat OpenShift Service on AWS clusters by using Red Hat Advanced Cluster Management policy enforcement for faster cluster management, without manually importing individual clusters.
Required access: Cluster administrator
2.4.1. Prerequisites 링크 복사링크가 클립보드에 복사되었습니다!
- You need Red Hat Advanced Cluster Management installed. See the Red Hat Advanced Cluster Management Installing and upgrading documentation.
- You need to learn about Policies. See the introduction to Governance in the Red Hat Advanced Cluster Management documentation.
2.4.2. Creating the automatic import policy 링크 복사링크가 클립보드에 복사되었습니다!
The following policy and procedure is an example of how to import all your discovered Red Hat OpenShift Service on AWS clusters automatically.
Log in to your hub cluster from the CLI to complete the following procedure:
Create a YAML file with the following example and apply the changes that are referenced:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- To enable automatic import, change the
spec.remediationActiontoenforce. - 2
- Optional: Specify a value here to select a subset of the matching Red Hat OpenShift Service on AWS clusters, which are based on discovered cluster names. The
rosa-filterhas no value by default, so the filter does not restrict cluster names without a subset value.
-
Run
oc apply -f <filename>.yaml -n <namespace>to apply the file.
2.4.3. Creating the placement definition 링크 복사링크가 클립보드에 복사되었습니다!
You need to create a placement definition that specifies the managed cluster for the policy deployment.
Create the placement definition that selects only the
local-cluster, which is a hub cluster that is managed. Use the following YAML sample:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Run
oc apply -f placement.yaml -n <namespace>, wherenamespacematches the namespace that you used for the policy that you previously created.
2.4.4. Binding the import policy to a placement definition 링크 복사링크가 클립보드에 복사되었습니다!
After you create the policy and the placement, you need to connect the two resources.
Connect the resources by using a
PlacementBinding. See the following example whereplacementRefreferences thePlacementthat you created, andsubjectsreferences thePolicythat you created:Copy to Clipboard Copied! Toggle word wrap Toggle overflow To verify, run the following command:
oc get policies.policy.open-cluster-management.io policy-rosa-autoimport -n <namespace>
oc get policies.policy.open-cluster-management.io policy-rosa-autoimport -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow