Chapter 4. Installing managed clusters with RHACM and SiteConfig resources

Procedure

1. Create a YAML secret file containing credentials for the host Baseboard Management Controller (BMC) and a pull secret required for installing OpenShift and all add-on cluster Operators:

   1. Save the following YAML as the file example-sno-secret.yaml:

      apiVersion: v1
      kind: Secret
      metadata:
        name: example-sno-bmc-secret
        namespace: example-sno 

      1

      
      data: 

      2

      
        password: <base64_password>
        username: <base64_username>
      type: Opaque
      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: pull-secret
        namespace: example-sno  

      3

      
      data:
        .dockerconfigjson: <pull_secret> 

      4

      
      type: kubernetes.io/dockerconfigjson

      1

      Must match the namespace configured in the related SiteConfig CR

      2

      Base64-encoded values for password and username

      3

      Must match the namespace configured in the related SiteConfig CR

      4

      Base64-encoded pull secret

2. Add the relative path to example-sno-secret.yaml to the kustomization.yaml file that you use to install the cluster.

Procedure

1. Create the InfraEnv CR and edit the spec.kernelArguments specification to configure kernel arguments.

   1. Save the following YAML in an InfraEnv-example.yaml file:

      Note

      The InfraEnv CR in this example uses template syntax such as {{ .Cluster.ClusterName }} that is populated based on values in the SiteConfig CR. The SiteConfig CR automatically populates values for these templates during deployment. Do not edit the templates manually.

      apiVersion: agent-install.openshift.io/v1beta1
      kind: InfraEnv
      metadata:
        annotations:
          argocd.argoproj.io/sync-wave: "1"
        name: "{{ .Cluster.ClusterName }}"
        namespace: "{{ .Cluster.ClusterName }}"
      spec:
        clusterRef:
          name: "{{ .Cluster.ClusterName }}"
          namespace: "{{ .Cluster.ClusterName }}"
        kernelArguments:
          - operation: append 

      1

      
            value: audit=0 

      2

      
          - operation: append
            value: trace=1
        sshAuthorizedKey: "{{ .Site.SshPublicKey }}"
        proxy: "{{ .Cluster.ProxySettings }}"
        pullSecretRef:
          name: "{{ .Site.PullSecretRef.Name }}"
        ignitionConfigOverride: "{{ .Cluster.IgnitionConfigOverride }}"
        nmStateConfigLabelSelector:
          matchLabels:
            nmstate-label: "{{ .Cluster.ClusterName }}"
        additionalNTPSources: "{{ .Cluster.AdditionalNTPSources }}"

      1

      Specify the append operation to add a kernel argument.

      2

      Specify the kernel argument you want to configure. This example configures the audit kernel argument and the trace kernel argument.

2. Commit the InfraEnv-example.yaml CR to the same location in your Git repository that has the SiteConfig CR and push your changes. The following example shows a sample Git repository structure:

   ~/example-ztp/install
             └── site-install
                  ├── siteconfig-example.yaml
                  ├── InfraEnv-example.yaml
                  ...

3. Edit the spec.clusters.crTemplates specification in the SiteConfig CR to reference the InfraEnv-example.yaml CR in your Git repository:

   clusters:
     crTemplates:
       InfraEnv: "InfraEnv-example.yaml"

   When you are ready to deploy your cluster by committing and pushing the SiteConfig CR, the build pipeline uses the custom InfraEnv-example CR in your Git repository to configure the infrastructure environment, including the custom kernel arguments.

1. Begin an SSH session with the target host:

   $ ssh -i /path/to/privatekey core@<host_name>

2. View the system’s kernel arguments by using the following command:

   $ cat /proc/cmdline

Procedure

1. Create the required managed cluster secrets on the hub cluster. These resources must be in a namespace with a name matching the cluster name. For example, in out/argocd/example/siteconfig/example-sno.yaml, the cluster name and namespace is example-sno.

   1. Export the cluster namespace by running the following command:

      $ export CLUSTERNS=example-sno

   2. Create the namespace:

      $ oc create namespace $CLUSTERNS

2. Create pull secret and BMC Secret CRs for the managed cluster. The pull secret must contain all the credentials necessary for installing OpenShift Container Platform and all required Operators. See "Creating the managed bare-metal host secrets" for more information.

   Note

   The secrets are referenced from the SiteConfig custom resource (CR) by name. The namespace must match the SiteConfig namespace.

3. Create a SiteConfig CR for your cluster in your local clone of the Git repository:

   1. Choose the appropriate example for your CR from the out/argocd/example/siteconfig/ folder. The folder includes example files for single node, three-node, and standard clusters:

      • example-sno.yaml
      • example-3node.yaml
      • example-standard.yaml

   2. Change the cluster and host details in the example file to match the type of cluster you want. For example:

      Example single-node OpenShift SiteConfig CR

      # example-node1-bmh-secret & assisted-deployment-pull-secret need to be created under same namespace example-sno
      ---
      apiVersion: ran.openshift.io/v1
      kind: SiteConfig
      metadata:
        name: "example-sno"
        namespace: "example-sno"
      spec:
        baseDomain: "example.com"
        pullSecretRef:
          name: "assisted-deployment-pull-secret"
        clusterImageSetNameRef: "openshift-4.18"
        sshPublicKey: "ssh-rsa AAAA..."
        clusters:
          - clusterName: "example-sno"
            networkType: "OVNKubernetes"
            # installConfigOverrides is a generic way of passing install-config
            # parameters through the siteConfig.  The 'capabilities' field configures
            # the composable openshift feature.  In this 'capabilities' setting, we
            # remove all the optional set of components.
            # Notes:
            # - OperatorLifecycleManager is needed for 4.15 and later
            # - NodeTuning is needed for 4.13 and later, not for 4.12 and earlier
            # - Ingress is needed for 4.16 and later
            installConfigOverrides: |
              {
                "capabilities": {
                  "baselineCapabilitySet": "None",
                  "additionalEnabledCapabilities": [
                    "NodeTuning",
                    "OperatorLifecycleManager",
                    "Ingress"
                  ]
                }
              }
            # It is strongly recommended to include crun manifests as part of the additional install-time manifests for 4.13+.
            # The crun manifests can be obtained from source-crs/optional-extra-manifest/ and added to the git repo ie.sno-extra-manifest.
            # extraManifestPath: sno-extra-manifest
            clusterLabels:
              # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples
              du-profile: "latest"
              # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples in ../policygentemplates:
              # ../acmpolicygenerator/common-ranGen.yaml will apply to all clusters with 'common: true'
              common: true
              # ../policygentemplates/group-du-sno-ranGen.yaml will apply to all clusters with 'group-du-sno: ""'
              group-du-sno: ""
              # ../policygentemplates/example-sno-site.yaml will apply to all clusters with 'sites: "example-sno"'
              # Normally this should match or contain the cluster name so it only applies to a single cluster
              sites: "example-sno"
            clusterNetwork:
              - cidr: 1001:1::/48
                hostPrefix: 64
            machineNetwork:
              - cidr: 1111:2222:3333:4444::/64
            serviceNetwork:
              - 1001:2::/112
            additionalNTPSources:
              - 1111:2222:3333:4444::2
            # Initiates the cluster for workload partitioning. Setting specific reserved/isolated CPUSets is done via PolicyTemplate
            # please see Workload Partitioning Feature for a complete guide.
            cpuPartitioningMode: AllNodes
            # Optionally; This can be used to override the KlusterletAddonConfig that is created for this cluster:
            #crTemplates:
            #  KlusterletAddonConfig: "KlusterletAddonConfigOverride.yaml"
            nodes:
              - hostName: "example-node1.example.com"
                role: "master"
                # Optionally; This can be used to configure desired BIOS setting on a host:
                #biosConfigRef:
                #  filePath: "example-hw.profile"
                bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1"
                bmcCredentialsName:
                  name: "example-node1-bmh-secret"
                bootMACAddress: "AA:BB:CC:DD:EE:11"
                # Use UEFISecureBoot to enable secure boot.
                bootMode: "UEFISecureBoot"
                rootDeviceHints:
                  deviceName: "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0"
                #crTemplates:
                #  BareMetalHost: "bmhOverride.yaml"
                # disk partition at `/var/lib/containers` with ignitionConfigOverride. Some values must be updated. See DiskPartitionContainer.md for more details
                ignitionConfigOverride: |
                  {
                    "ignition": {
                      "version": "3.2.0"
                    },
                    "storage": {
                      "disks": [
                        {
                          "device": "/dev/disk/by-id/wwn-0x6b07b250ebb9d0002a33509f24af1f62",
                          "partitions": [
                            {
                              "label": "var-lib-containers",
                              "sizeMiB": 0,
                              "startMiB": 250000
                            }
                          ],
                          "wipeTable": false
                        }
                      ],
                      "filesystems": [
                        {
                          "device": "/dev/disk/by-partlabel/var-lib-containers",
                          "format": "xfs",
                          "mountOptions": [
                            "defaults",
                            "prjquota"
                          ],
                          "path": "/var/lib/containers",
                          "wipeFilesystem": true
                        }
                      ]
                    },
                    "systemd": {
                      "units": [
                        {
                          "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target",
                          "enabled": true,
                          "name": "var-lib-containers.mount"
                        }
                      ]
                    }
                  }
                nodeNetwork:
                  interfaces:
                    - name: eno1
                      macAddress: "AA:BB:CC:DD:EE:11"
                  config:
                    interfaces:
                      - name: eno1
                        type: ethernet
                        state: up
                        ipv4:
                          enabled: false
                        ipv6:
                          enabled: true
                          address:
                            # For SNO sites with static IP addresses, the node-specific,
                            # API and Ingress IPs should all be the same and configured on
                            # the interface
                            - ip: 1111:2222:3333:4444::aaaa:1
                              prefix-length: 64
                    dns-resolver:
                      config:
                        search:
                          - example.com
                        server:
                          - 1111:2222:3333:4444::2
                    routes:
                      config:
                        - destination: ::/0
                          next-hop-interface: eno1
                          next-hop-address: 1111:2222:3333:4444::1
                          table-id: 254

      Note

      For more information about BMC addressing, see the "Additional resources" section. The installConfigOverrides and ignitionConfigOverride fields are expanded in the example for ease of readability.

      Note

      To override the default BareMetalHost CR for a node, you can reference the override CR in the node-level crTemplates field in the SiteConfig CR. Ensure that you set the argocd.argoproj.io/sync-wave: "3" annotation in your override BareMetalHost CR.

   3. You can inspect the default set of extra-manifest MachineConfig CRs in out/argocd/extra-manifest. It is automatically applied to the cluster when it is installed.

   4. Optional: To provision additional install-time manifests on the provisioned cluster, create a directory in your Git repository, for example, sno-extra-manifest/, and add your custom manifest CRs to this directory. If your SiteConfig.yaml refers to this directory in the extraManifestPath field, any CRs in this referenced directory are appended to the default set of extra manifests.

      Enabling the crun OCI container runtime

      For optimal cluster performance, enable crun for master and worker nodes in single-node OpenShift, single-node OpenShift with additional worker nodes, three-node OpenShift, and standard clusters.

      Enable crun in a ContainerRuntimeConfig CR as an additional Day 0 install-time manifest to avoid the cluster having to reboot.

      The enable-crun-master.yaml and enable-crun-worker.yaml CR files are in the out/source-crs/optional-extra-manifest/ folder that you can extract from the ztp-site-generate container. For more information, see "Customizing extra installation manifests in the GitOps ZTP pipeline".

4. Add the SiteConfig CR to the kustomization.yaml file in the generators section, similar to the example shown in out/argocd/example/siteconfig/kustomization.yaml.

5. Commit the SiteConfig CR and associated kustomization.yaml changes in your Git repository and push the changes.

   The ArgoCD pipeline detects the changes and begins the managed cluster deployment.

1. The Assisted Service Operator installs OpenShift Container Platform on the cluster. You can monitor the progress of cluster installation from the RHACM dashboard or from the command line by running the following commands:

   1. Export the cluster name:

      $ export CLUSTER=<clusterName>

   2. Query the AgentClusterInstall CR for the managed cluster:

      $ oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.conditions[?(@.type=="Completed")]}' | jq

   3. Get the installation events for the cluster:

      $ curl -sk $(oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.debugInfo.eventsURL}')  | jq '.[-2,-1]'

Procedure

1. Check that the installation CRs were created by using the following command:

   $ oc get AgentClusterInstall -n <cluster_name>

   If no object is returned, use the following steps to troubleshoot the ArgoCD pipeline flow from SiteConfig files to the installation CRs.

2. Verify that the ManagedCluster CR was generated using the SiteConfig CR on the hub cluster:

   $ oc get managedcluster

3. If the ManagedCluster is missing, check if the clusters application failed to synchronize the files from the Git repository to the hub cluster:

   $ oc get applications.argoproj.io -n openshift-gitops clusters -o yaml

   1. To identify error logs for the managed cluster, inspect the status.operationState.syncResult.resources field. For example, if an invalid value is assigned to the extraManifestPath in the SiteConfig CR, an error similar to the following is generated:

      syncResult:
        resources:
        - group: ran.openshift.io
          kind: SiteConfig
          message: The Kubernetes API could not find ran.openshift.io/SiteConfig for
            requested resource spoke-sno/spoke-sno. Make sure the "SiteConfig" CRD is
            installed on the destination cluster

   2. To see a more detailed SiteConfig error, complete the following steps:

      1. In the Argo CD dashboard, click the SiteConfig resource that Argo CD is trying to sync.

      2. Check the DESIRED MANIFEST tab to find the siteConfigError field.

         siteConfigError: >- Error: could not build the entire SiteConfig defined by /tmp/kust-plugin-config-1081291903: stat sno-extra-manifest: no such file or directory

   3. Check the Status.Sync field. If there are log errors, the Status.Sync field could indicate an Unknown error:

      Status:
        Sync:
          Compared To:
            Destination:
              Namespace:  clusters-sub
              Server:     https://kubernetes.default.svc
            Source:
              Path:             sites-config
              Repo URL:         https://git.com/ran-sites/siteconfigs/.git
              Target Revision:  master
          Status:               Unknown

Procedure

1. Disable TLS in the Provisioning resource by running the following command:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"disableVirtualMediaTLS": true}}'

2. Continue the steps to deploy your single-node OpenShift cluster.

Procedure

1. Remove a site and the associated CRs by removing the associated SiteConfig and PolicyGenerator or PolicyGentemplate files from the kustomization.yaml file.

2. Add the following syncOptions field to your SiteConfig application.

   kind: Application
   spec:
     syncPolicy:
       syncOptions:
       - PrunePropagationPolicy=background

   When you run the GitOps ZTP pipeline again, the generated CRs are removed.

3. Optional: If you want to permanently remove a site, you should also remove the SiteConfig and site-specific PolicyGenerator or PolicyGentemplate files from the Git repository.
4. Optional: If you want to remove a site temporarily, for example when redeploying a site, you can leave the SiteConfig and site-specific PolicyGenerator or PolicyGentemplate CRs in the Git repository.

Procedure

1. Remove the affected PolicyGenerator or PolicyGentemplate files from the Git repository, commit and push to the remote repository.
2. Wait for the changes to synchronize through the application and the affected policies to be removed from the hub cluster.

3. Add the updated PolicyGenerator or PolicyGentemplate files back to the Git repository, and then commit and push to the remote repository.

   Note

   Removing GitOps Zero Touch Provisioning (ZTP) policies from the Git repository, and as a result also removing them from the hub cluster, does not affect the configuration of the managed cluster. The policy and CRs managed by that policy remains in place on the managed cluster.

4. Optional: As an alternative, after making changes to PolicyGenerator or PolicyGentemplate CRs that result in obsolete policies, you can remove these policies from the hub cluster manually. You can delete policies from the RHACM console using the Governance tab or by running the following command:

   $ oc delete policy -n <namespace> <policy_name>

Procedure

1. Detach all clusters from Red Hat Advanced Cluster Management (RHACM) on the hub cluster.

2. Delete the kustomization.yaml file in the deployment directory using the following command:

   $ oc delete -k out/argocd/deployment

3. Commit and push your changes to the site repository.