Chapter 13. Creating and managing ManagedClusterSets (Technology Preview)

A ManagedClusterSet is a group of managed clusters. With a managed cluster set, you can manage access to all of the managed clusters in the group together. You can also create a ManagedClusterSetBinding resource to bind a ManagedClusterSet resource to a namespace.

Creating a ManagedClusterSet
- Creating a ManagedClusterSet by using the console
- Creating a ManagedClusterSet by using the command line
Adding clusters to a ManagedClusterSet
- Adding clusters to a ManagedClusterSet by using the console
- Adding clusters to a ManagedClusterSet by using the command line
Removing a managed cluster from a ManagedClusterSet
- Removing a managed cluster from a ManagedClusterSet by using the console
- Removing clusters from a ManagedClusterSet by using the command line
Creating a ManagedClusterSetBinding resource
- Creating a ManagedClusterSetBinding by using the console
- Creating a ManagedClusterSetBinding by using the command line
Assigning users or groups Role-Based Access Control permissions to your ManagedClusterSet

13.1. Creating a ManagedClusterSet

You can group managed clusters together in a managed cluster set to limit the user access on managed clusters.

Required access: Cluster administrator

A ManagedClusterSet is a cluster-scoped resource, so you must have cluster administration permissions for the cluster where you are creating the ManagedClusterSet. A managed cluster cannot be included in more than one ManagedClusterSet. You can create a managed cluster set from either the Red Hat Advanced Cluster Management for Kubernetes console or from the command-line interface.

13.1.1. Creating a ManagedClusterSet by using the console

Complete the following steps to create a managed cluster set by using the Red Hat Advanced Cluster Management console:

In the main console navigation, select Infrastructure > Clusters.
Select the Cluster sets tab.
Select Create cluster set.
Enter the name of the cluster set that you want to create.
Select Create to create the cluster set resource.

13.1.2. Creating a ManagedClusterSet by using the command line

Add the following definition of the managed cluster set to your yaml file to create a managed cluster set by using the command line:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: ManagedClusterSet
metadata:
  name: <clusterset1>

Replace clusterset1 with the name of your managed cluster set.

13.2. Adding clusters to a ManagedClusterSet

After you create your ManagedClusterSet, you must add one or more managed clusters. You can add managed clusters to your managed cluster set by using either the console or the command line.

13.2.1. Adding clusters to a ManagedClusterSet by using the console

Complete the following steps to add a cluster to a managed cluster set by using the Red Hat Advanced Cluster Management console:

If you just created your managed cluster set, select Manage resource assignments to go directly to the Manage resource assignments page. Continue with step 6 of this procedure.
If your cluster already exists, access the cluster page by selecting Infrastructure > Clusters in the main navigation.
Select the Cluster sets tab to view the available cluster sets.
Select the name of the cluster set that you want to add to the managed cluster sets to view the cluster set details.
Select Actions > Manage resource assignments.
On the Manage resource assignments page, select the checkbox for the resources that you want to add to the cluster set.
Select Review to review your changes.
Select Save to save your changes.
Note: If you move a managed cluster from one managed cluster set to another, you must have the required RBAC permission available on both managed cluster sets.

13.2.2. Adding clusters to a ManagedClusterSet by using the command line

Complete the following steps to add a cluster to a managed cluster set by using the command line:

Ensure that there is an RBAC ClusterRole entry that allows you to create on a virtual subresource of managedclustersets/join. Without this permission, you cannot assign a managed cluster to a ManagedClusterSet.
If this entry does not exist, add it to your yaml file. A sample entry resembles the following content:
```
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: clusterrole1
rules:
  - apiGroups: ["cluster.open-cluster-management.io"]
    resources: ["managedclustersets/join"]
    resourceNames: ["<clusterset1>"]
    verbs: ["create"]
```
Replace clusterset1 with the name of your ManagedClusterSet.
Note: If you are moving a managed cluster from one ManagedClusterSet to another, you must have that permission available on both managed cluster sets.
Find the definition of the managed cluster in the yaml file. The section of the managed cluster definition where you add a label resembles the following content:
```
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: cluster1
spec:
  hubAcceptsClient: true
```
In this example, cluster1 is the name of the managed cluster.
Add a label that specifies the name of the ManagedClusterSet in the format: cluster.open-cluster-management.io/clusterset: clusterset1.
Your code resembles the following example:
```
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: cluster1
  labels:
    cluster.open-cluster-management.io/clusterset: clusterset1
spec:
  hubAcceptsClient: true
```
In this example, cluster1 is the cluster that is added to the managed cluster set names clusterset1.
Note: If the managed cluster was previously assigned to a managed cluster set that was deleted, the managed cluster might have a managed cluster set already specified to a cluster set that does not exist. If so, replace the name with the new one.

13.3. Removing a managed cluster from a ManagedClusterSet

You might want to remove a managed cluster from a managed cluster set to move it to a different managed cluster set, or remove it from the management settings of the set. You can remove a managed cluster from a managed cluster set by using the console or the command-line interface.

13.3.1. Removing a managed cluster from a ManagedClusterSet by using the console

Complete the following steps to remove a cluster from a managed cluster set by using the Red Hat Advanced Cluster Management console:

If you just created your managed cluster set, select Manage resource assignments to go directly to the Manage resource assignments page. Continue with step 6 of this procedure.
If your cluster already exists, access the cluster page by selecting Infrastructure > Clusters in the main navigation.
Select the Cluster sets tab to view the available cluster sets.
Select the name of the cluster set that you want to remove from the managed cluster set to view the cluster set details.
Select Actions > Manage resource assignments.
On the Manage resource assignments page, select the checkbox for the resources that you want to remove from the cluster set.
This step removes a resource that is already a member of the cluster set, or adds a resource that is not already a member of the cluster set. You can see if the resource is already a member of a cluster set by viewing the details of the managed cluster.
Select Review and review your changes, then select Save.

Note: If you are moving a managed cluster from one managed cluster set to another, you must have the required RBAC permissions on both managed cluster sets.

13.3.2. Removing clusters from a ManagedClusterSet by using the command line

To remove a managed cluster from a managed cluster set by using the command line, complete the following steps:

Run the following command to display a list of managed clusters in the managed cluster set:
```
oc get managedclusters -l cluster.open-cluster-management.io/clusterset=<clusterset1>
```
Replace clusterset1 with the name of the managed cluster set.
Locate the entry for the cluster that you want to remove.
Remove the label from the the yaml entry for the cluster that you want to remove. See the following code for an example of the label:
```
labels:
   cluster.open-cluster-management.io/clusterset: clusterset1
```

Note: If you are moving a managed cluster from one managed cluster set to another, you must have the required RBAC permission available on both managed cluster sets.

13.4. Creating a ManagedClusterSetBinding resource

Create a ManagedClusterSetBinding resource to bind a ManagedClusterSet resource to a namespace. Applications and policies that are created in the same namespace can only access managed clusters that are included in the bound managed cluster set resource.

Access permissions to the namespace automatically apply to a managed cluster set that is bound to that namespace. If you have access permissions to access the namespace to which the managed cluster set is bound, you automatically have permissions to access any managed cluster set that is bound to that namespace. However, if you only have permissions to access the managed cluster set, you do not automatically have permissions to access other managed cluster sets on the namespace. If you do not see a managed cluster set, you might not have the required permissions to view it.

You can create a managed cluster set binding by using the console or the command line.

13.4.1. Creating a ManagedClusterSetBinding by using the console

Complete the following steps to remove a cluster from a managed cluster set by using the Red Hat Advanced Cluster Management console:

Access the cluster page by selecting Infrastructure > Clusters in the main navigation.
Select the Cluster sets tab to view the available cluster sets.
Select the name of the cluster set that you want to create a binding for to view the cluster set details.
Select Actions > Edit namespace bindings.
On the Edit namespace bindings page, select the namespace to which you want to bind the cluster set from the drop-down menu. The existing namespaces that have bindings to the cluster set are already selected.
Select Save to submit your changes.

13.4.2. Creating a ManagedClusterSetBinding by using the command line

To create a managed cluster set binding by using the command line, complete the following steps:

Create the ManagedClusterSetBinding resource in your yaml file. When you create a managed cluster set binding, the name of the managed cluster set binding must match the name of the managed cluster set to bind. Your ManagedClusterSetBinding resource might resemble the following information:
```
apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: ManagedClusterSetBinding
metadata:
  namespace: project1
  name: clusterset1
spec:
  clusterSet: clusterset1
```

Ensure that you have the bind permission on the target managed cluster set. View the following example of a ClusterRole resource, which contains rules that allow the user to bind to clusterset1:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: clusterrole1
rules:
  - apiGroups: ["cluster.open-cluster-management.io"]
    resources: ["managedclustersets/bind"]
    resourceNames: ["clusterset1"]
    verbs: ["create"]

13.5. Assigning users or groups Role-Based Access Control permissions to your ManagedClusterSet

You can assign users or groups to your cluster set that are provided by the configured identity providers on the hub cluster.

Required access: Cluster administrator

The ManagedClusterSet API offers two levels of RBAC permissions:

Cluster set admin
- Full access permissions to all of the cluster and cluster pool resources that are assigned to the managed cluster set.
- Permission to create clusters, import clusters, and create cluster pools. The permissions must be assigned to the managed cluster set when the managed cluster set is created.
Cluster set view
- Read only permissions to all of the cluster and cluster pool resources that are assigned to the managed cluster set.
- No permission to create clusters, import clusters, or create cluster pools.

Complete the following steps to assign users or groups to your managed cluster set from the Red Hat Advanced Cluster Management console:

From the main navigation menu of the console, select Infrastructure > Clusters.
Select the Cluster sets tab.
Select your target cluster set.
Select the Access management tab.
Select Add user or group.
Search for, and select the user or group that you want to provide access.
Select the Cluster set admin or Cluster set view role to give to the selected user or user group. See Overview of roles for more information about the role permissions.
Select Add to submit the changes.

Your user or group is displayed in the table. It might take a few seconds for the permission assignments for all of the managed cluster set resources to be propagated to your user or group.

For more information about role-based actions, see Role-based access control.

See Using ManagedClustersSets with Placement for placement information.

13.6. Using ManagedClusterSets with Placement

A Placement resource is a namespace-scoped resource that defines a rule to select a set of ManagedClusters from the ManagedClusterSets, which are bound to the placement namespace.

Required access: Cluster administrator, Cluster set administrator

13.6.1. Placement overview

See the following information about how placement with managed clusters works:

Kubernetes clusters are registered with the hub cluster as cluster-scoped ManagedClusters.
The ManagedClusters are organized into cluster-scoped ManagedClusterSets.
The ManagedClusterSets are bound to workload namespaces.
The namespace-scoped Placements specify a portion of ManagedClusterSets that select a working set of the potential ManagedClusters.
Placements select from that working set by using label and claim selectors.
Important: Placement does not select a ManagedCluster if there is no ManagedClusterSet bound to the placement namespace.

The Placement specification includes the following fields:

ClusterSets represents the ManagedClusterSets from which the ManagedClusters are selected.
- If not specified, ManagedClusters is selected from the ManagedClusterSets that are bound to the placement namespace.
- If specified, ManagedClusters is selected from the intersection of this set and the ManagedClusterSets that are bound to the placement namespace.
NumberOfClusters represents the desired number of ManagedClusters to be selected, which meet the placement requirements.
If not specified, all ManagedClusters that meet the placement requirements are selected.
Predicates represents a slice of predicates to select ManagedClusters with label and claim selector. The predicates are ORed.

13.6.2. Placement examples

You need to bind at least one ManagedClusterSet to a namespace by creating a ManagedClusterSetBinding in that namespace. Note: You need role-based access to CREATE on the virtual sub-resource of managedclustersets/bind. See the following examples:

You can select ManagedClusters with the labelSelector. See the following sample where the labelSelector only matches clusters with label vendor: OpenShift:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: Placement
metadata:
  name: placement1
  namespace: ns1
spec:
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchLabels:
            vendor: OpenShift

You can select ManagedClusters with claimSelector. See the following sample where claimSelector only matches clusters with region.open-cluster-management.io with us-west-1:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: Placement
metadata:
  name: placement2
  namespace: ns1
spec:
  predicates:
    - requiredClusterSelector:
        claimSelector:
          matchExpressions:
            - key: region.open-cluster-management.io
              operator: In
              values:
                - us-west-1

You can select ManagedClusters from particular clusterSets. See the following sample where claimSelector only matches clusterSets: clusterset1 clusterset2:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: Placement
metadata:
  name: placement3
  namespace: ns1
spec:
  clusterSets:
    - clusterset1
    - clusterset2
  predicates:
    - requiredClusterSelector:
        claimSelector:
          matchExpressions:
            - key: region.open-cluster-management.io
              operator: In
              values:
                - us-west-1

Select desired number of ManagedClusters. See the following sample where numberOfClusters is 3:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: Placement
metadata:
  name: placement4
  namespace: ns1
spec:
  numberOfClusters: 3
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchLabels:
            vendor: OpenShift
        claimSelector:
          matchExpressions:
            - key: region.open-cluster-management.io
              operator: In
              values:
                - us-west-1

13.6.3. Placement decision

One or multiple PlacementDecisions with label cluster.open-cluster-management.io/placement={placement name} are created to represent the ManagedClusters selected by a Placement.

If a ManagedCluster is selected and added to a PlacementDecision, components that consume this Placement might apply the workload on this ManagedCluster. After the ManagedCluster is no longer selected and it is removed from the PlacementDecisions, the workload that is applied on this ManagedCluster should be removed accordingly.

See the following PlacementDecision sample:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: PlacementDecision
metadata:
  labels:
    cluster.open-cluster-management.io/placement: placement1
  name: placement1-kbc7q
  namespace: ns1
  ownerReferences:
    - apiVersion: cluster.open-cluster-management.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Placement
      name: placement1
      uid: 05441cf6-2543-4ecc-8389-1079b42fe63e
status:
  decisions:
    - clusterName: cluster1
      reason: ''
    - clusterName: cluster2
      reason: ''
    - clusterName: cluster3
      reason: ''