multicluster engine operator with Red Hat Advanced Cluster Management

From your Red Hat Advanced Cluster Management cluster, create a ManagedCluster resource manually to import an multicluster engine operator cluster. See the following file example:

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  annotations:
    agent.open-cluster-management.io/klusterlet-config: mce-import-klusterlet-config 1
  name: mce-a 2
spec:
  hubAcceptsClient: true
  leaseDurationSeconds: 60

Create the auto-import-secret secret that references the kubeconfig of the multicluster engine operator cluster. Go to Importing a cluster by using the auto import secret in Importing a managed cluster by using the CLI to add the auto import secret to complete the multicluster engine operator auto-import process.

After you create the auto import secret in the multicluster engine operator managed cluster namespace in the Red Hat Advanced Cluster Management cluster, the managed cluster is registered.

Run the following command to get the status:

oc get managedcluster

See following example output with the status and example URLs of managed clusters:

NAME           HUB ACCEPTED   MANAGED CLUSTER URLS            JOINED   AVAILABLE   AGE
local-cluster  true           https://<api.acm-hub.com:port>  True     True        44h
mce-a          true           https://<api.mce-a.com:port>    True     True        27s

Run the following command to set the agentInstallNamespace namespace of the add-on to open-cluster-management-agent-addon-discovery:

oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"agentInstallNamespace":"open-cluster-management-agent-addon-discovery"}}'

Run the following command to disable metrics and to disable the HyperShift operator management:

oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value": "true"},{"name":"disableHOManagement","value": "true"}]}}'

Run the following command to enable the hypershift-addon for multicluster engine operator:

clusteradm addon enable --names hypershift-addon --clusters <managed-cluster-names>

You can get the multicluster engine operator managed cluster names by running the following command in Red Hat Advanced Cluster Management.

oc get managedcluster

Log into multicluster engine operator clusters and verify that the hypershift-addon is installed in the namespace that you specified. Run the following command:

oc get deployment -n open-cluster-management-agent-addon-discovery

See the following example output that lists the add-ons:

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
cluster-proxy-proxy-agent            1/1     1            1           24h
klusterlet-addon-workmgr            1/1     1            1           24h
hypershift-addon-agent              1/1     1            1           24h
managed-serviceaccount-addon-agent  1/1     1            1           24h

Create a YAML file for your DiscoveredCluster custom resource and edit the configuration that is referenced in the following example:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-mce-hcp-autoimport
  namespace: open-cluster-management-global-set
  annotations:
    policy.open-cluster-management.io/standards: NIST SP 800-53
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/description: Discovered clusters that are of
      type MultiClusterEngineHCP can be automatically imported into ACM as managed clusters.
      This policy configure those discovered clusters so they are automatically imported.
      Fine tuning MultiClusterEngineHCP clusters to be automatically imported
      can be done by configure filters at the configMap or add annotation to the discoverd cluster.
spec:
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: mce-hcp-autoimport-config
        spec:
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  name: discovery-config
                  namespace: open-cluster-management-global-set
                data:
                  rosa-filter: ""
          remediationAction: enforce 1
          severity: low
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-mce-hcp-autoimport
        spec:
          remediationAction: enforce
          severity: low
          object-templates-raw: |
            {{- /* find the MultiClusterEngineHCP DiscoveredClusters */ -}}
            {{- range $dc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }}
              {{- /* Check for the flag that indicates the import should be skipped */ -}}
              {{- $skip := "false" -}}
              {{- range $key, $value := $dc.metadata.annotations }}
                {{- if and (eq $key "discovery.open-cluster-management.io/previously-auto-imported")
                           (eq $value "true") }}
                  {{- $skip = "true" }}
                {{- end }}
              {{- end }}
              {{- /* if the type is MultiClusterEngineHCP and the status is Active */ -}}
              {{- if and (eq $dc.spec.status "Active")
                         (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "mce-hcp-filter") $dc.spec.displayName)
                         (eq $dc.spec.type "MultiClusterEngineHCP")
                         (eq $skip "false") }}
            - complianceType: musthave
              objectDefinition:
                apiVersion: discovery.open-cluster-management.io/v1
                kind: DiscoveredCluster
                metadata:
                  name: {{ $dc.metadata.name }}
                  namespace: {{ $dc.metadata.namespace }}
                spec:
                  importAsManagedCluster: true 2
              {{- end }}
            {{- end }}

Create the Placement definition that selects only the local-cluster, which is a hub cluster that is managed. Use the following YAML sample:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: policy-mce-hcp-autoimport-placement
  namespace: open-cluster-management-global-set
spec:
  tolerations:
    - key: cluster.open-cluster-management.io/unreachable
      operator: Exists
    - key: cluster.open-cluster-management.io/unavailable
      operator: Exists
  clusterSets:
    - global
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchExpressions:
            - key: local-cluster
              operator: In
              values:
                - "true"

Connect the resources by using a PlacementBinding resource. See the following example where placementRef points to the Placement that you created, and subjects points to the Policy that you created:

apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: policy-mce-hcp-autoimport-placement-binding
  namespace: open-cluster-management-global-set
placementRef:
  name: policy-mce-hcp-autoimport-placement
  apiGroup: cluster.open-cluster-management.io
  kind: Placement
subjects:
  - name: policy-mce-hcp-autoimport
    apiGroup: policy.open-cluster-management.io
    kind: Policy

To verify, run the following command:

oc get policy policy-mce-hcp-autoimport -n <namespace>

Create a YAML file with the following example and apply the changes that are referenced:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-rosa-autoimport
  annotations:
    policy.open-cluster-management.io/standards: NIST SP 800-53
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/description: OpenShift Service on AWS discovered clusters can be automatically imported into
Red Hat Advanced Cluster Management as managed clusters with this policy. You can select and configure those managed clusters so you can import. Configure filters or add an annotation if you do not want all of your OpenShift Service on AWS clusters to be automatically imported.
spec:
  remediationAction: inform 1
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: rosa-autoimport-config
        spec:
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  name: discovery-config
                  namespace: open-cluster-management-global-set
                data:
                  rosa-filter: "" 2
          remediationAction: enforce
          severity: low
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-rosa-autoimport
        spec:
          remediationAction: enforce
          severity: low
          object-templates-raw: |
            {{- /* find the ROSA DiscoveredClusters */ -}}
            {{- range $dc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }}
              {{- /* Check for the flag that indicates the import should be skipped */ -}}
              {{- $skip := "false" -}}
              {{- range $key, $value := $dc.metadata.annotations }}
                {{- if and (eq $key "discovery.open-cluster-management.io/previously-auto-imported")
                           (eq $value "true") }}
                  {{- $skip = "true" }}
                {{- end }}
              {{- end }}
              {{- /* if the type is ROSA and the status is Active */ -}}
              {{- if and (eq $dc.spec.status "Active")
                         (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") $dc.spec.displayName)
                         (eq $dc.spec.type "ROSA")
                         (eq $skip "false") }}
            - complianceType: musthave
              objectDefinition:
                apiVersion: discovery.open-cluster-management.io/v1
                kind: DiscoveredCluster
                metadata:
                  name: {{ $dc.metadata.name }}
                  namespace: {{ $dc.metadata.namespace }}
                spec:
                  importAsManagedCluster: true
              {{- end }}
            {{- end }}
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: policy-rosa-managedcluster-status
        spec:
          remediationAction: enforce
          severity: low
          object-templates-raw: |
            {{- /* Use the same DiscoveredCluster list to check ManagedCluster status */ -}}
            {{- range $dc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }}
              {{- /* Check for the flag that indicates the import should be skipped */ -}}
              {{- $skip := "false" -}}
              {{- range $key, $value := $dc.metadata.annotations }}
                {{- if and (eq $key "discovery.open-cluster-management.io/previously-auto-imported")
                           (eq $value "true") }}
                  {{- $skip = "true" }}
                {{- end }}
              {{- end }}
              {{- /* if the type is ROSA and the status is Active */ -}}
              {{- if and (eq $dc.spec.status "Active")
                         (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") $dc.spec.displayName)
                         (eq $dc.spec.type "ROSA")
                         (eq $skip "false") }}
            - complianceType: musthave
              objectDefinition:
                apiVersion: cluster.open-cluster-management.io/v1
                kind: ManagedCluster
                metadata:
                  name: {{ $dc.spec.displayName }}
                  namespace: {{ $dc.spec.displayName }}
                status:
                  conditions:
                    - type: ManagedClusterConditionAvailable
                      status: "True"
              {{- end }}
            {{- end }}

Create the placement definition that selects only the local-cluster, which is a hub cluster that is managed. Use the following YAML sample:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement-openshift-plus-hub
spec:
  predicates:
  - requiredClusterSelector:
      labelSelector:
        matchExpressions:
        - key: name
      	    operator: In
      	    values:
      	    - local-cluster

Connect the resources by using a PlacementBinding. See the following example where placementRef points to the Placement that you created, and subjects points to the Policy that you created:

apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: binding-policy-rosa-autoimport
placementRef:
  apiGroup: cluster.open-cluster-management.io
  kind: Placement
  name: placement-policy-rosa-autoimport
subjects:
- apiGroup: policy.open-cluster-management.io
  kind: Policy
  name: policy-rosa-autoimport

To verify, run the following command:

oc get policy policy-rosa-autoimport -n <namespace>

Set the enabled field to true in the siteconfig entry of spec.overrides.components in the Multiclusterhub resource by running the following command:

oc patch multiclusterhubs.operator.open-cluster-management.io multiclusterhub -n rhacm --type json --patch '[{"op": "add", "path":"/spec/overrides/components/-", "value": {"name":"siteconfig","enabled": true}}]'

Verify that the SiteConfig operator is enabled by running the following command on the hub cluster:

oc -n rhacm get po | grep siteconfig

See the following example output:

siteconfig-controller-manager-6fdd86cc64-sdg87                    2/2     Running   0             43s

Optional: Verify that you have the default installation templates by running the following command on the hub cluster:

oc -n rhacm get cm

See the following list of templates in the output example:

NAME                                DATA   AGE
ai-cluster-templates-v1             5      97s
ai-node-templates-v1                2      97s
...
ibi-cluster-templates-v1            3      97s
ibi-node-templates-v1               3      97s
...

Create a YAML file for the target namespace. See the following example file that is named clusterinstance-namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: example-sno

Apply your file to create the resource. Run the following command on the hub cluster:

oc apply -f clusterinstance-namespace.yaml

Create a YAML file to pull images. See the following example of a file that is named pull-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: pull-secret
  namespace: example-sno 1
data:
  .dockerconfigjson: <encoded_docker_configuration> 2
type: kubernetes.io/dockerconfigjson

Apply the file to create the resource. Run the following command on the hub cluster:

oc apply -f pull-secret.yaml

Create a YAML file for the BMC secret. See the following sample file that is named example-bmc-secret.yaml:

apiVersion: v1
data:
  password: <password>
  username: <username>
kind: Secret
metadata:
  name: example-bmh-secret
  namespace: "example-sno" 1
type: Opaque

Apply the file to create the resource. Run the following command on the hub cluster:

oc apply -f example-bmc-secret.yaml

Create a YAML file for an extra manifest ConfigMap object, for example named enable-crun.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: enable-crun
  namespace: example-sno 1
data:
  enable-crun-master.yaml: |
    apiVersion: machineconfiguration.openshift.io/v1
    kind: ContainerRuntimeConfig
    metadata:
      name: enable-crun-master
    spec:
      machineConfigPoolSelector:
        matchLabels:
          pools.operator.machineconfiguration.openshift.io/master: ""
      containerRuntimeConfig:
        defaultRuntime: crun
  enable-crun-worker.yaml: |
    apiVersion: machineconfiguration.openshift.io/v1
    kind: ContainerRuntimeConfig
    metadata:
      name: enable-crun-worker
    spec:
      machineConfigPoolSelector:
        matchLabels:
          pools.operator.machineconfiguration.openshift.io/worker: ""
      containerRuntimeConfig:
        defaultRuntime: crun

Create the resource by running the following command on the hub cluster:

oc apply -f enable-crun.yaml

In the example-sno namespace, create the ClusterInstance custom resource that is named clusterinstance-ibi.yaml in the following example:

apiVersion: siteconfig.open-cluster-management.io/v1alpha1
kind: ClusterInstance
metadata:
  name: "example-clusterinstance"
  namespace: "example-sno" 1
spec:
  holdInstallation: false
  extraManifestsRefs: 2
    - name: extra-machine-configs
    - name: enable-crun
  pullSecretRef:
    name: "pull-secret" 3
  [...]
  clusterName: "example-sno" 4
  [...]
  templateRefs: 5
    - name: ibi-cluster-templates-v1
      namespace: rhacm
  [...]
  nodes:
      [...]
      bmcCredentialsName: 6
        name: "example-bmh-secret"
      [...]
      templateRefs: 7
        - name: ibi-node-templates-v1
          namespace: rhacm
      [...]

Apply the file and create the resource by running the following command:

oc apply -f clusterinstance-ibi.yaml

After you create the custom resource, the SiteConfig operator starts reconciling the ClusterInstance custom resource, then validates and renders the installation manifests.

The SiteConfig operator continues to monitor for changes in the ClusterDeployment custom resources to update the cluster installation progress of the corresponding ClusterInstance custom resource.

Monitor the process by running the following command:

oc get clusterinstance <cluster_name> -n <target_namespace> -o yaml

See the following example output from the status.conditions section for successful manifest generation:

message: Applied site config manifests
reason: Completed
status: "True"
type: RenderedTemplatesApplied

Check the manifests that SiteConfig operator rendered by running the following command:

oc get clusterinstance <cluster_name> -n <target_namespace> -o jsonpath='{.status.manifestsRendered}'

Delete the ClusterInstance custom resource by running the following command:

oc delete clusterinstance <cluster_name> -n <target_namespace>

Verify that the deletion was successful by running the following command:

oc get clusterinstance <cluster_name> -n <target_namespace>

Add an annotation in the extraAnnotations field of the worker node entry in the ClusterInstance custom resource that is used to provision your cluster:

spec:
   ...
   nodes:
   - hostName: "worker-node2.example.com"
      role: "worker"
      ironicInspect: ""
      extraAnnotations:
        BareMetalHost:
          bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true"
...

Apply the changes. See the following options:

oc apply -f <clusterinstance>.yaml

Verify that the annotation is applied to the BaremetalHost worker resource by running the following command on the hub cluster:

oc get bmh -n <clusterinstance_namespace> worker-node2.example.com -ojsonpath='{.metadata.annotations}' | jq

See the following example output for successful application of the annotation:

Update the node object that you want to delete in your existing ClusterInstance custom resource with the following configuration:

...
spec:
   ...
   nodes:
     - hostName: "worker-node2.example.com"
       ...
       pruneManifests:
         - apiVersion: metal3.io/v1alpha1
           kind: BareMetalHost
...

Apply the changes. See the following options.

oc apply -f <clusterinstance>.yaml

Verify that the BareMetalHost resources are removed by running the following command on the hub cluster:

oc get bmh -n <clusterinstance_namespace> --watch --kubeconfig <hub_cluster_kubeconfig_filename>

See the following example output:

NAME                        STATE                        CONSUMER         ONLINE   ERROR   AGE
master-node1.example.com    provisioned                  true             81m
worker-node2.example.com    deprovisioning               true             44m
worker-node2.example.com    powering off before delete   true             20h
worker-node2.example.com    deleting                     true             50m

Verify that the Agent resources are removed by running the following command on the hub cluster:

oc get agents -n <clusterinstance_namespace> --kubeconfig <hub_cluster_kubeconfig_filename>

See the following example output:

NAME                       CLUSTER                  APPROVED   ROLE     STAGE
master-node1.example.com   <managed_cluster_name>   true       master   Done
master-node2.example.com   <managed_cluster_name>   true       master   Done
master-node3.example.com   <managed_cluster_name>   true       master   Done
worker-node1.example.com   <managed_cluster_name>   true       worker   Done

Verify that the Node resources are removed by running the following command on the managed cluster:

oc get nodes --kubeconfig <managed_cluster_kubeconfig_filename>

See the following example output:

NAME                       STATUS                        ROLES                  AGE   VERSION
worker-node2.example.com   NotReady,SchedulingDisabled   worker                 19h   v1.30.5
worker-node1.example.com   Ready                         worker                 19h   v1.30.5
master-node1.example.com   Ready                         control-plane,master   19h   v1.30.5
master-node2.example.com   Ready                         control-plane,master   19h   v1.30.5
master-node3.example.com   Ready                         control-plane,master   19h   v1.30.5

Define a new node object in the existing ClusterInstance custom resource:

spec:
  ...
  nodes:
    - hostName: "<host_name>"
      role: "worker"
      templateRefs:
        - name: ai-node-templates-v1
          namespace: rhacm
      bmcAddress: "<bmc_address>"
      bmcCredentialsName:
        name: "<bmc_credentials_name>"
      bootMACAddress: "<boot_mac_address>"
...

Apply the changes. See the following options:

oc apply -f <clusterinstance>.yaml

Verify that a new BareMetalHost resource is added by running the following command on the hub cluster:

oc get bmh -n <clusterinstance_namespace> --watch --kubeconfig <hub_cluster_kubeconfig_filename>

See the following example output:

NAME                        STATE          CONSUMER   ONLINE   ERROR   AGE
master-node1.example.com    provisioned               true             81m
worker-node2.example.com    provisioning              true             44m

Verify that a new Agent resource is added by running the following command on the hub cluster:

oc get agents -n <clusterinstance_namespace> --kubeconfig <hub_cluster_kubeconfig_filename>

See the following example output:

NAME                       CLUSTER                   APPROVED    ROLE     STAGE
master-node1.example.com   <managed_cluster_name>    true        master   Done
master-node2.example.com   <managed_cluster_name>    true        master   Done
master-node3.example.com   <managed_cluster_name>    true        master   Done
worker-node1.example.com   <managed_cluster_name>    false       worker
worker-node2.example.com   <managed_cluster_name>    true        worker   Starting installation
worker-node2.example.com   <managed_cluster_name>    true        worker   Installing
worker-node2.example.com   <managed_cluster_name>    true        worker   Writing image to disk
worker-node2.example.com   <managed_cluster_name>    true        worker   Waiting for control plane
worker-node2.example.com   <managed_cluster_name>    true        worker   Rebooting
worker-node2.example.com   <managed_cluster_name>    true        worker   Joined
worker-node2.example.com   <managed_cluster_name>    true        worker   Done

Verify that a new Node resource is added by running the following command on the managed cluster:

oc get nodes --kubeconfig <managed_cluster_kubeconfig_filename>

See the following example output:

NAME                       STATUS    ROLES                  AGE   VERSION
worker-node2.example.com   Ready     worker                 1h    v1.30.5
worker-node1.example.com   Ready     worker                 19h   v1.30.5
master-node1.example.com   Ready     control-plane,master   19h   v1.30.5
master-node2.example.com   Ready     control-plane,master   19h   v1.30.5
master-node3.example.com   Ready     control-plane,master   19h   v1.30.5