Chapter 4. Installing managed clusters with RHACM and SiteConfig resources

1. A Discovery image ISO file is generated and booted on the target host.
2. When the ISO file successfully boots on the target host it reports the host hardware information to RHACM.
3. After all hosts are discovered, OpenShift Container Platform is installed.
4. When OpenShift Container Platform finishes installing, the hub installs the klusterlet service on the target cluster.
5. The requested add-on services are installed on the target cluster.

Procedure

1. Create a YAML secret file containing credentials for the host Baseboard Management Controller (BMC) and a pull secret required for installing OpenShift and all add-on cluster Operators:

   1. Save the following YAML as the file example-sno-secret.yaml:

      apiVersion: v1
      kind: Secret
      metadata:
        name: example-sno-bmc-secret
        namespace: example-sno 1
      data: 2
        password: <base64_password>
        username: <base64_username>
      type: Opaque
      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: pull-secret
        namespace: example-sno  3
      data:
        .dockerconfigjson: <pull_secret> 4
      type: kubernetes.io/dockerconfigjson

      1

      Must match the namespace configured in the related SiteConfig CR

      2

      Base64-encoded values for password and username

      3

      Must match the namespace configured in the related SiteConfig CR

      4

      Base64-encoded pull secret

2. Add the relative path to example-sno-secret.yaml to the kustomization.yaml file that you use to install the cluster.

Procedure

1. Create the InfraEnv CR and edit the spec.kernelArguments specification to configure kernel arguments.

   1. Save the following YAML in an InfraEnv-example.yaml file:

      Note

      The InfraEnv CR in this example uses template syntax such as {{ .Cluster.ClusterName }} that is populated based on values in the SiteConfig CR. The SiteConfig CR automatically populates values for these templates during deployment. Do not edit the templates manually.

      apiVersion: agent-install.openshift.io/v1beta1
      kind: InfraEnv
      metadata:
        annotations:
          argocd.argoproj.io/sync-wave: "1"
        name: "{{ .Cluster.ClusterName }}"
        namespace: "{{ .Cluster.ClusterName }}"
      spec:
        clusterRef:
          name: "{{ .Cluster.ClusterName }}"
          namespace: "{{ .Cluster.ClusterName }}"
        kernelArguments:
          - operation: append 1
            value: audit=0 2
          - operation: append
            value: trace=1
        sshAuthorizedKey: "{{ .Site.SshPublicKey }}"
        proxy: "{{ .Cluster.ProxySettings }}"
        pullSecretRef:
          name: "{{ .Site.PullSecretRef.Name }}"
        ignitionConfigOverride: "{{ .Cluster.IgnitionConfigOverride }}"
        nmStateConfigLabelSelector:
          matchLabels:
            nmstate-label: "{{ .Cluster.ClusterName }}"
        additionalNTPSources: "{{ .Cluster.AdditionalNTPSources }}"

      1

      Specify the append operation to add a kernel argument.

      2

      Specify the kernel argument you want to configure. This example configures the audit kernel argument and the trace kernel argument.

2. Commit the InfraEnv-example.yaml CR to the same location in your Git repository that has the SiteConfig CR and push your changes. The following example shows a sample Git repository structure:

   ~/example-ztp/install
             └── site-install
                  ├── siteconfig-example.yaml
                  ├── InfraEnv-example.yaml
                  ...

3. Edit the spec.clusters.crTemplates specification in the SiteConfig CR to reference the InfraEnv-example.yaml CR in your Git repository:

   clusters:
     crTemplates:
       InfraEnv: "InfraEnv-example.yaml"

   When you are ready to deploy your cluster by committing and pushing the SiteConfig CR, the build pipeline uses the custom InfraEnv-example CR in your Git repository to configure the infrastructure environment, including the custom kernel arguments.

1. Begin an SSH session with the target host:

   $ ssh -i /path/to/privatekey core@<host_name>

2. View the system’s kernel arguments by using the following command:

   $ cat /proc/cmdline

Procedure

1. Create the required managed cluster secrets on the hub cluster. These resources must be in a namespace with a name matching the cluster name. For example, in out/argocd/example/siteconfig/example-sno.yaml, the cluster name and namespace is example-sno.

   1. Export the cluster namespace by running the following command:

      $ export CLUSTERNS=example-sno

   2. Create the namespace:

      $ oc create namespace $CLUSTERNS

2. Create pull secret and BMC Secret CRs for the managed cluster. The pull secret must contain all the credentials necessary for installing OpenShift Container Platform and all required Operators. See "Creating the managed bare-metal host secrets" for more information.

   Note

   The secrets are referenced from the SiteConfig custom resource (CR) by name. The namespace must match the SiteConfig namespace.

3. Create a SiteConfig CR for your cluster in your local clone of the Git repository:

   1. Choose the appropriate example for your CR from the out/argocd/example/siteconfig/ folder. The folder includes example files for single node, three-node, and standard clusters:

      • example-sno.yaml
      • example-3node.yaml
      • example-standard.yaml

   2. Change the cluster and host details in the example file to match the type of cluster you want. For example:

      Example single-node OpenShift SiteConfig CR

      # example-node1-bmh-secret & assisted-deployment-pull-secret need to be created under same namespace example-sno
      ---
      apiVersion: ran.openshift.io/v1
      kind: SiteConfig
      metadata:
        name: "example-sno"
        namespace: "example-sno"
      spec:
        baseDomain: "example.com"
        pullSecretRef:
          name: "assisted-deployment-pull-secret"
        clusterImageSetNameRef: "openshift-4.16"
        sshPublicKey: "ssh-rsa AAAA..."
        clusters:
          - clusterName: "example-sno"
            networkType: "OVNKubernetes"
            # installConfigOverrides is a generic way of passing install-config
            # parameters through the siteConfig.  The 'capabilities' field configures
            # the composable openshift feature.  In this 'capabilities' setting, we
            # remove all the optional set of components.
            # Notes:
            # - OperatorLifecycleManager is needed for 4.15 and later
            # - NodeTuning is needed for 4.13 and later, not for 4.12 and earlier
            # - Ingress is needed for 4.16 and later
            installConfigOverrides: |
              {
                "capabilities": {
                  "baselineCapabilitySet": "None",
                  "additionalEnabledCapabilities": [
                    "NodeTuning",
                    "OperatorLifecycleManager",
                    "Ingress"
                  ]
                }
              }
            # It is strongly recommended to include crun manifests as part of the additional install-time manifests for 4.13+.
            # The crun manifests can be obtained from source-crs/optional-extra-manifest/ and added to the git repo ie.sno-extra-manifest.
            # extraManifestPath: sno-extra-manifest
            clusterLabels:
              # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples
              du-profile: "latest"
              # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples in ../policygentemplates:
              # ../policygentemplates/common-ranGen.yaml will apply to all clusters with 'common: true'
              common: true
              # ../policygentemplates/group-du-sno-ranGen.yaml will apply to all clusters with 'group-du-sno: ""'
              group-du-sno: ""
              # ../policygentemplates/example-sno-site.yaml will apply to all clusters with 'sites: "example-sno"'
              # Normally this should match or contain the cluster name so it only applies to a single cluster
              sites: "example-sno"
            clusterNetwork:
              - cidr: 1001:1::/48
                hostPrefix: 64
            machineNetwork:
              - cidr: 1111:2222:3333:4444::/64
            serviceNetwork:
              - 1001:2::/112
            additionalNTPSources:
              - 1111:2222:3333:4444::2
            # Initiates the cluster for workload partitioning. Setting specific reserved/isolated CPUSets is done via PolicyTemplate
            # please see Workload Partitioning Feature for a complete guide.
            cpuPartitioningMode: AllNodes
            # Optionally; This can be used to override the KlusterletAddonConfig that is created for this cluster:
            #crTemplates:
            #  KlusterletAddonConfig: "KlusterletAddonConfigOverride.yaml"
            nodes:
              - hostName: "example-node1.example.com"
                role: "master"
                # Optionally; This can be used to configure desired BIOS setting on a host:
                #biosConfigRef:
                #  filePath: "example-hw.profile"
                bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1"
                bmcCredentialsName:
                  name: "example-node1-bmh-secret"
                bootMACAddress: "AA:BB:CC:DD:EE:11"
                # Use UEFISecureBoot to enable secure boot
                bootMode: "UEFI"
                rootDeviceHints:
                  deviceName: "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0"
                # disk partition at `/var/lib/containers` with ignitionConfigOverride. Some values must be updated. See DiskPartitionContainer.md for more details
                ignitionConfigOverride: |
                  {
                    "ignition": {
                      "version": "3.2.0"
                    },
                    "storage": {
                      "disks": [
                        {
                          "device": "/dev/disk/by-id/wwn-0x6b07b250ebb9d0002a33509f24af1f62",
                          "partitions": [
                            {
                              "label": "var-lib-containers",
                              "sizeMiB": 0,
                              "startMiB": 250000
                            }
                          ],
                          "wipeTable": false
                        }
                      ],
                      "filesystems": [
                        {
                          "device": "/dev/disk/by-partlabel/var-lib-containers",
                          "format": "xfs",
                          "mountOptions": [
                            "defaults",
                            "prjquota"
                          ],
                          "path": "/var/lib/containers",
                          "wipeFilesystem": true
                        }
                      ]
                    },
                    "systemd": {
                      "units": [
                        {
                          "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target",
                          "enabled": true,
                          "name": "var-lib-containers.mount"
                        }
                      ]
                    }
                  }
                nodeNetwork:
                  interfaces:
                    - name: eno1
                      macAddress: "AA:BB:CC:DD:EE:11"
                  config:
                    interfaces:
                      - name: eno1
                        type: ethernet
                        state: up
                        ipv4:
                          enabled: false
                        ipv6:
                          enabled: true
                          address:
                            # For SNO sites with static IP addresses, the node-specific,
                            # API and Ingress IPs should all be the same and configured on
                            # the interface
                            - ip: 1111:2222:3333:4444::aaaa:1
                              prefix-length: 64
                    dns-resolver:
                      config:
                        search:
                          - example.com
                        server:
                          - 1111:2222:3333:4444::2
                    routes:
                      config:
                        - destination: ::/0
                          next-hop-interface: eno1
                          next-hop-address: 1111:2222:3333:4444::1
                          table-id: 254

      Note

      For more information about BMC addressing, see the "Additional resources" section. The installConfigOverrides and ignitionConfigOverride fields are expanded in the example for ease of readability.

   3. You can inspect the default set of extra-manifest MachineConfig CRs in out/argocd/extra-manifest. It is automatically applied to the cluster when it is installed.

   4. Optional: To provision additional install-time manifests on the provisioned cluster, create a directory in your Git repository, for example, sno-extra-manifest/, and add your custom manifest CRs to this directory. If your SiteConfig.yaml refers to this directory in the extraManifestPath field, any CRs in this referenced directory are appended to the default set of extra manifests.

      Enabling the crun OCI container runtime

      For optimal cluster performance, enable crun for master and worker nodes in single-node OpenShift, single-node OpenShift with additional worker nodes, three-node OpenShift, and standard clusters.

      Enable crun in a ContainerRuntimeConfig CR as an additional Day 0 install-time manifest to avoid the cluster having to reboot.

      The enable-crun-master.yaml and enable-crun-worker.yaml CR files are in the out/source-crs/optional-extra-manifest/ folder that you can extract from the ztp-site-generate container. For more information, see "Customizing extra installation manifests in the GitOps ZTP pipeline".

4. Add the SiteConfig CR to the kustomization.yaml file in the generators section, similar to the example shown in out/argocd/example/siteconfig/kustomization.yaml.

5. Commit the SiteConfig CR and associated kustomization.yaml changes in your Git repository and push the changes.

   The ArgoCD pipeline detects the changes and begins the managed cluster deployment.

1. The Assisted Service Operator installs OpenShift Container Platform on the cluster. You can monitor the progress of cluster installation from the RHACM dashboard or from the command line by running the following commands:

   1. Export the cluster name:

      $ export CLUSTER=<clusterName>

   2. Query the AgentClusterInstall CR for the managed cluster:

      $ oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.conditions[?(@.type=="Completed")]}' | jq

   3. Get the installation events for the cluster:

      $ curl -sk $(oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.debugInfo.eventsURL}')  | jq '.[-2,-1]'

Procedure

1. Check that the installation CRs were created by using the following command:

   $ oc get AgentClusterInstall -n <cluster_name>

   If no object is returned, use the following steps to troubleshoot the ArgoCD pipeline flow from SiteConfig files to the installation CRs.

2. Verify that the ManagedCluster CR was generated using the SiteConfig CR on the hub cluster:

   $ oc get managedcluster

3. If the ManagedCluster is missing, check if the clusters application failed to synchronize the files from the Git repository to the hub cluster:

   $ oc describe -n openshift-gitops application clusters

   1. Check for the Status.Conditions field to view the error logs for the managed cluster. For example, setting an invalid value for extraManifestPath: in the SiteConfig CR raises the following error:

      Status:
        Conditions:
          Last Transition Time:  2021-11-26T17:21:39Z
          Message:               rpc error: code = Unknown desc = `kustomize build /tmp/https___git.com/ran-sites/siteconfigs/ --enable-alpha-plugins` failed exit status 1: 2021/11/26 17:21:40 Error could not create extra-manifest ranSite1.extra-manifest3 stat extra-manifest3: no such file or directory 2021/11/26 17:21:40 Error: could not build the entire SiteConfig defined by /tmp/kust-plugin-config-913473579: stat extra-manifest3: no such file or directory Error: failure in plugin configured via /tmp/kust-plugin-config-913473579; exit status 1: exit status 1
          Type:  ComparisonError

   2. Check the Status.Sync field. If there are log errors, the Status.Sync field could indicate an Unknown error:

      Status:
        Sync:
          Compared To:
            Destination:
              Namespace:  clusters-sub
              Server:     https://kubernetes.default.svc
            Source:
              Path:             sites-config
              Repo URL:         https://git.com/ran-sites/siteconfigs/.git
              Target Revision:  master
          Status:               Unknown

Procedure

1. Disable TLS in the Provisioning resource by running the following command:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"disableVirtualMediaTLS": true}}'

2. Continue the steps to deploy your single-node OpenShift cluster.

Procedure

1. Remove a site and the associated CRs by removing the associated SiteConfig and PolicyGenerator or PolicyGentemplate files from the kustomization.yaml file.

2. Add the following syncOptions field to your SiteConfig application.

   kind: Application
   spec:
     syncPolicy:
       syncOptions:
       - PrunePropagationPolicy=background

   When you run the GitOps ZTP pipeline again, the generated CRs are removed.

3. Optional: If you want to permanently remove a site, you should also remove the SiteConfig and site-specific PolicyGenerator or PolicyGentemplate files from the Git repository.
4. Optional: If you want to remove a site temporarily, for example when redeploying a site, you can leave the SiteConfig and site-specific PolicyGenerator or PolicyGentemplate CRs in the Git repository.

Procedure

1. Remove the affected PolicyGenerator or PolicyGentemplate files from the Git repository, commit and push to the remote repository.
2. Wait for the changes to synchronize through the application and the affected policies to be removed from the hub cluster.

3. Add the updated PolicyGenerator or PolicyGentemplate files back to the Git repository, and then commit and push to the remote repository.

   Note

   Removing GitOps Zero Touch Provisioning (ZTP) policies from the Git repository, and as a result also removing them from the hub cluster, does not affect the configuration of the managed cluster. The policy and CRs managed by that policy remains in place on the managed cluster.

4. Optional: As an alternative, after making changes to PolicyGenerator or PolicyGentemplate CRs that result in obsolete policies, you can remove these policies from the hub cluster manually. You can delete policies from the RHACM console using the Governance tab or by running the following command:

   $ oc delete policy -n <namespace> <policy_name>

Procedure

1. Detach all clusters from Red Hat Advanced Cluster Management (RHACM) on the hub cluster.

2. Delete the kustomization.yaml file in the deployment directory using the following command:

   $ oc delete -k out/argocd/deployment

3. Commit and push your changes to the site repository.