Chapter 1. multicluster engine operator with Red Hat Advanced Cluster Management integration

From your Red Hat Advanced Cluster Management cluster, create a ManagedCluster resource manually to import an multicluster engine operator cluster. See the following file example:

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  annotations:
    agent.open-cluster-management.io/klusterlet-config: mce-import-klusterlet-config 1
  labels:
    cloud: auto-detect
    vendor: auto-detect
  name: mce-a 2
spec:
  hubAcceptsClient: true
  leaseDurationSeconds: 60

Create the auto-import-secret secret that references the kubeconfig of the multicluster engine operator cluster. Go to Importing a cluster by using the auto import secret in Importing a managed cluster by using the CLI to add the auto import secret to complete the multicluster engine operator auto-import process.

After you create the auto import secret in the multicluster engine operator managed cluster namespace in the Red Hat Advanced Cluster Management cluster, the managed cluster is registered.

Run the following command to get the status:

oc get managedcluster

See following example output with the status and example URLs of managed clusters:

NAME           HUB ACCEPTED   MANAGED CLUSTER URLS            JOINED   AVAILABLE   AGE
local-cluster  true           https://<api.acm-hub.com:port>  True     True        44h
mce-a          true           https://<api.mce-a.com:port>    True     True        27s

Run the following command to set the agentInstallNamespace namespace of the add-on to open-cluster-management-agent-addon-discovery:

oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"agentInstallNamespace":"open-cluster-management-agent-addon-discovery"}}'

Run the following command to disable metrics and to disable the HyperShift operator management:

oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value": "true"},{"name":"disableHOManagement","value": "true"}]}}'

Optional: You can set a naming convention by appending the previous command with {"name":"discoveryPrefix","value": ""}]}}'. When a discovered hosted cluster is automatically imported into the Red Hat Advanced Cluster Management hub cluster, it becomes a managed cluster with the following naming convention:`<mce-cluster-name>`-<hosted-cluster-name>. You can also set the discoveryPrefix to some other string to use it as a prefix that replaces <mce-cluster-name>. Run the following command with the prefix:

oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value": "true"},{"name":"disableHOManagement","value": "true"},{"name":"discoveryPrefix","value": ""}]}}'

Run the following command to enable the hypershift-addon for multicluster engine operator:

clusteradm addon enable --names hypershift-addon --clusters <managed-cluster-names>

You can get the multicluster engine operator managed cluster names by running the following command in Red Hat Advanced Cluster Management.

oc get managedcluster

Log into multicluster engine operator clusters and verify that the hypershift-addon is installed in the namespace that you specified. Run the following command:

oc get deployment -n open-cluster-management-agent-addon-discovery

See the following example output that lists the add-ons:

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
cluster-proxy-proxy-agent            1/1     1            1           24h
klusterlet-addon-workmgr            1/1     1            1           24h
hypershift-addon-agent              1/1     1            1           24h
managed-serviceaccount-addon-agent  1/1     1            1           24h

Create a YAML file for your DiscoveredCluster custom resource and edit the configuration that is referenced in the following example:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-mce-hcp-autoimport
  namespace: open-cluster-management-global-set
  annotations:
    policy.open-cluster-management.io/standards: NIST SP 800-53
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/description: Discovered clusters that are of
      type MultiClusterEngineHCP can be automatically imported into ACM as managed clusters.
      This policy configure those discovered clusters so they are automatically imported.
      Fine tuning MultiClusterEngineHCP clusters to be automatically imported
      can be done by configure filters at the configMap or add annotation to the discoverd cluster.
spec:
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: mce-hcp-autoimport-config
        spec:
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  name: discovery-config
                  namespace: open-cluster-management-global-set
                data:
                  rosa-filter: ""
          remediationAction: enforce 1
          severity: low
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-mce-hcp-autoimport
        spec:
          remediationAction: enforce
          severity: low
          object-templates-raw: |
            {{- /* find the MultiClusterEngineHCP DiscoveredClusters */ -}}
            {{- range $dc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }}
              {{- /* Check for the flag that indicates the import should be skipped */ -}}
              {{- $skip := "false" -}}
              {{- range $key, $value := $dc.metadata.annotations }}
                {{- if and (eq $key "discovery.open-cluster-management.io/previously-auto-imported")
                           (eq $value "true") }}
                  {{- $skip = "true" }}
                {{- end }}
              {{- end }}
              {{- /* if the type is MultiClusterEngineHCP and the status is Active */ -}}
              {{- if and (eq $dc.spec.status "Active")
                         (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "mce-hcp-filter") $dc.spec.displayName)
                         (eq $dc.spec.type "MultiClusterEngineHCP")
                         (eq $skip "false") }}
            - complianceType: musthave
              objectDefinition:
                apiVersion: discovery.open-cluster-management.io/v1
                kind: DiscoveredCluster
                metadata:
                  name: {{ $dc.metadata.name }}
                  namespace: {{ $dc.metadata.namespace }}
                spec:
                  importAsManagedCluster: true 2
              {{- end }}
            {{- end }}

Create the Placement definition that selects only the local-cluster, which is a hub cluster that is managed. Use the following YAML sample:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: policy-mce-hcp-autoimport-placement
  namespace: open-cluster-management-global-set
spec:
  tolerations:
    - key: cluster.open-cluster-management.io/unreachable
      operator: Exists
    - key: cluster.open-cluster-management.io/unavailable
      operator: Exists
  clusterSets:
    - global
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchExpressions:
            - key: local-cluster
              operator: In
              values:
                - "true"

Connect the resources by using a PlacementBinding resource. See the following example where placementRef references the Placement that you created, and subjects references the Policy that you created:

apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: policy-mce-hcp-autoimport-placement-binding
  namespace: open-cluster-management-global-set
placementRef:
  name: policy-mce-hcp-autoimport-placement
  apiGroup: cluster.open-cluster-management.io
  kind: Placement
subjects:
  - name: policy-mce-hcp-autoimport
    apiGroup: policy.open-cluster-management.io
    kind: Policy

To verify, run the following command:

oc get policies.policy.open-cluster-management.io policy-mce-hcp-autoimport -n <namespace>

Create a YAML file with the following example and apply the changes that are referenced:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-rosa-autoimport
  annotations:
    policy.open-cluster-management.io/standards: NIST SP 800-53
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/description: OpenShift Service on AWS discovered clusters can be automatically imported into
Red Hat Advanced Cluster Management as managed clusters with this policy. You can select and configure those managed clusters so you can import. Configure filters or add an annotation if you do not want all of your OpenShift Service on AWS clusters to be automatically imported.
spec:
  remediationAction: inform 1
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: rosa-autoimport-config
        spec:
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  name: discovery-config
                  namespace: open-cluster-management-global-set
                data:
                  rosa-filter: "" 2
          remediationAction: enforce
          severity: low
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-rosa-autoimport
        spec:
          remediationAction: enforce
          severity: low
          object-templates-raw: |
            {{- /* find the ROSA DiscoveredClusters */ -}}
            {{- range $dc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }}
              {{- /* Check for the flag that indicates the import should be skipped */ -}}
              {{- $skip := "false" -}}
              {{- range $key, $value := $dc.metadata.annotations }}
                {{- if and (eq $key "discovery.open-cluster-management.io/previously-auto-imported")
                           (eq $value "true") }}
                  {{- $skip = "true" }}
                {{- end }}
              {{- end }}
              {{- /* if the type is ROSA and the status is Active */ -}}
              {{- if and (eq $dc.spec.status "Active")
                         (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") $dc.spec.displayName)
                         (eq $dc.spec.type "ROSA")
                         (eq $skip "false") }}
            - complianceType: musthave
              objectDefinition:
                apiVersion: discovery.open-cluster-management.io/v1
                kind: DiscoveredCluster
                metadata:
                  name: {{ $dc.metadata.name }}
                  namespace: {{ $dc.metadata.namespace }}
                spec:
                  importAsManagedCluster: true
              {{- end }}
            {{- end }}
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-rosa-managedcluster-status
        spec:
          remediationAction: enforce
          severity: low
          object-templates-raw: |
            {{- /* Use the same DiscoveredCluster list to check ManagedCluster status */ -}}
            {{- range $dc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }}
              {{- /* Check for the flag that indicates the import should be skipped */ -}}
              {{- $skip := "false" -}}
              {{- range $key, $value := $dc.metadata.annotations }}
                {{- if and (eq $key "discovery.open-cluster-management.io/previously-auto-imported")
                           (eq $value "true") }}
                  {{- $skip = "true" }}
                {{- end }}
              {{- end }}
              {{- /* if the type is ROSA and the status is Active */ -}}
              {{- if and (eq $dc.spec.status "Active")
                         (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") $dc.spec.displayName)
                         (eq $dc.spec.type "ROSA")
                         (eq $skip "false") }}
            - complianceType: musthave
              objectDefinition:
                apiVersion: cluster.open-cluster-management.io/v1
                kind: ManagedCluster
                metadata:
                  name: {{ $dc.spec.displayName }}
                  namespace: {{ $dc.spec.displayName }}
                status:
                  conditions:
                    - type: ManagedClusterConditionAvailable
                      status: "True"
              {{- end }}
            {{- end }}

Create the placement definition that selects only the local-cluster, which is a hub cluster that is managed. Use the following YAML sample:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement-openshift-plus-hub
spec:
  predicates:
  - requiredClusterSelector:
      labelSelector:
        matchExpressions:
        - key: name
      	    operator: In
      	    values:
      	    - local-cluster

Connect the resources by using a PlacementBinding. See the following example where placementRef references the Placement that you created, and subjects references the Policy that you created:

apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: binding-policy-rosa-autoimport
placementRef:
  apiGroup: cluster.open-cluster-management.io
  kind: Placement
  name: placement-policy-rosa-autoimport
subjects:
- apiGroup: policy.open-cluster-management.io
  kind: Policy
  name: policy-rosa-autoimport

To verify, run the following command:

oc get policies.policy.open-cluster-management.io policy-rosa-autoimport -n <namespace>