Chapter 5. Manually installing a single-node OpenShift cluster with GitOps ZTP

Procedure

1. Create an output folder by running the following command:

   $ mkdir -p ./out

2. Log in to the Ecosystem container registry with your credentials by running the following command:

   $ podman login registry.redhat.io

3. Extract the reference and example CRs from the ztp-site-generate container image by running the following command:

   $ podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.22 extract /home/ztp --tar | tar x -C ./out

   The ./out directory contains the reference PolicyGenerator and ClusterInstance CRs in the out/argocd/example/ folder.

   Example output

   out
    └── argocd
         └── example
              ├── acmpolicygenerator
              │     ├── {policy-prefix}common-ranGen.yaml
              │     ├── {policy-prefix}example-sno-site.yaml
              │     ├── {policy-prefix}group-du-sno-ranGen.yaml
              │     ├── ...
              │     ├── kustomization.yaml
              │     └── ns.yaml
              └── clusterinstance
                    ├── example-sno.yaml
                    ├── example-3node.yaml
                    ├── example-standard.yaml
                    └── ...

4. Create a ClusterInstance CR for your cluster.

   Use the example ClusterInstance CRs in the out/argocd/example/clusterinstance/ folder that you previously extracted from the ztp-site-generate container as a reference. The folder includes example files for single node, three-node, and standard clusters:

   • example-sno.yaml
   • example-3node.yaml

   • example-standard.yaml

     Change the cluster and host details in the example file to match the type of cluster you want to install. For example:

     Example single-node OpenShift ClusterInstance CR

     # example-node1-bmh-secret & assisted-deployment-pull-secret need to be created under same namespace example-ai-sno
     ---
     apiVersion: siteconfig.open-cluster-management.io/v1alpha1
     kind: ClusterInstance
     metadata:
       name: "example-ai-sno"
       namespace: "example-ai-sno"
     spec:
       baseDomain: "example.com"
       pullSecretRef:
         name: "assisted-deployment-pull-secret"
       clusterImageSetNameRef: "openshift-4.22"
       sshPublicKey: "ssh-rsa AAAA..."
       clusterName: "example-ai-sno"
       networkType: "OVNKubernetes"
       # installConfigOverrides is a generic way of passing install-config
       # parameters through the siteConfig.  The 'capabilities' field configures
       # the composable openshift feature.  In this 'capabilities' setting, we
       # remove all the optional set of components.
       # Notes:
       # - OperatorLifecycleManager is needed for 4.15 and later
       # - NodeTuning is needed for 4.13 and later, not for 4.12 and earlier
       # - Ingress is needed for 4.16 and later
       installConfigOverrides: |
         {
           "capabilities": {
             "baselineCapabilitySet": "None",
             "additionalEnabledCapabilities": [
               "NodeTuning",
               "OperatorLifecycleManager",
               "Ingress"
             ]
           }
         }
       # Include references to extraManifest ConfigMaps.
       extraManifestsRefs:
         - name: sno-extra-manifest-configmap
       extraLabels:
         ManagedCluster:
           # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples
           du-profile: "latest"
           # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples in ../policygentemplates:
           # ../policygentemplates/common-ranGen.yaml will apply to all clusters with 'common: true'
           common: "true"
           # ../policygentemplates/group-du-sno-ranGen.yaml will apply to all clusters with 'group-du-sno: ""'
           group-du-sno: ""
           # ../policygentemplates/example-sno-site.yaml will apply to all clusters with 'sites: "example-sno"'
           # Normally this should match or contain the cluster name so it only applies to a single cluster
           sites : "example-sno"
       clusterNetwork:
         - cidr: 1001:1::/48
           hostPrefix: 64
       machineNetwork:
         - cidr: 1111:2222:3333:4444::/64
       serviceNetwork:
         - cidr: 1001:2::/112
       additionalNTPSources:
         - 1111:2222:3333:4444::2
       # Initiates the cluster for workload partitioning. Setting specific reserved/isolated CPUSets is done via PolicyTemplate
       # please see Workload Partitioning Feature for a complete guide.
       cpuPartitioningMode: AllNodes
       templateRefs:
         - name: ai-cluster-templates-v1
           namespace: open-cluster-management
       nodes:
         - hostName: "example-node1.example.com"
           role: "master"
           bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1"
           bmcCredentialsName:
             name: "example-node1-bmh-secret"
           bootMACAddress: "AA:BB:CC:DD:EE:11"
           # Use UEFISecureBoot to enable secure boot, UEFI to disable.
           bootMode: "UEFISecureBoot"
           rootDeviceHints:
             deviceName: "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0"
           # disk partition at `/var/lib/containers` with ignitionConfigOverride. Some values must be updated. See DiskPartitionContainer.md in argocd folder for more details
           ignitionConfigOverride: |
             {
               "ignition": {
                 "version": "3.2.0"
               },
               "storage": {
                 "disks": [
                   {
                     "device": "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0",
                     "partitions": [
                       {
                         "label": "var-lib-containers",
                         "sizeMiB": 0,
                         "startMiB": 250000
                       }
                     ],
                     "wipeTable": false
                   }
                 ],
                 "filesystems": [
                   {
                     "device": "/dev/disk/by-partlabel/var-lib-containers",
                     "format": "xfs",
                     "mountOptions": [
                       "defaults",
                       "prjquota"
                     ],
                     "path": "/var/lib/containers",
                     "wipeFilesystem": true
                   }
                 ]
               },
               "systemd": {
                 "units": [
                   {
                     "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target",
                     "enabled": true,
                     "name": "var-lib-containers.mount"
                   }
                 ]
               }
             }
           nodeNetwork:
             interfaces:
               - name: eno1
                 macAddress: "AA:BB:CC:DD:EE:11"
             config:
               interfaces:
                 - name: eno1
                   type: ethernet
                   state: up
                   ipv4:
                     enabled: false
                   ipv6:
                     enabled: true
                     address:
                     # For SNO sites with static IP addresses, the node-specific,
                     # API and Ingress IPs should all be the same and configured on
                     # the interface
                     - ip: 1111:2222:3333:4444::aaaa:1
                       prefix-length: 64
               dns-resolver:
                 config:
                   search:
                   - example.com
                   server:
                   - 1111:2222:3333:4444::2
               routes:
                 config:
                 - destination: ::/0
                   next-hop-interface: eno1
                   next-hop-address: 1111:2222:3333:4444::1
                   table-id: 254
           templateRefs:
             - name: ai-node-templates-v1
               namespace: open-cluster-management

     Note

     Optional: To provision additional install-time manifests on the provisioned cluster, create the extra manifest CRs and apply them to the hub cluster. Then reference them in the extraManifestsRefs field of the ClusterInstance CR. For more information, see "Customizing extra installation manifests in the GitOps ZTP pipeline".

5. Optional: Generate Day 2 configuration CRs from the reference PolicyGenerator CRs:

   1. Create an output folder for the configuration CRs by running the following command:

      $ mkdir -p ./ref

   2. Generate the configuration CRs by running the following command:

      $ podman run -it --rm -v `pwd`/out/argocd/example/policygentemplates:/resources:Z -v `pwd`/ref:/output:Z,U registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.22 generator config -N . /output

      The command generates example group and cluster-specific configuration CRs in the ./ref folder. You can apply these CRs to the cluster after installation is complete.

Procedure

1. Create a YAML secret file containing credentials for the host Baseboard Management Controller (BMC) and a pull secret required for installing OpenShift and all add-on cluster Operators:

   1. Save the following YAML as the file example-sno-secret.yaml:

      apiVersion: v1
      kind: Secret
      metadata:
        name: example-sno-bmc-secret
        namespace: example-sno
      data:
        password: <base64_password>
        username: <base64_username>
      type: Opaque
      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: pull-secret
        namespace: example-sno
      data:
        .dockerconfigjson: <pull_secret>
      type: kubernetes.io/dockerconfigjson

      where:

      namespace

      Must match the namespace configured in the related ClusterInstance CR.

      password, username

      Base64-encoded values for password and username.

      .dockerconfigjson

      Base64-encoded pull secret.

2. Add the relative path to example-sno-secret.yaml to the kustomization.yaml file that you use to install the cluster.

Procedure

1. Edit the spec.kernelArguments specification in the InfraEnv CR to configure kernel arguments:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <cluster_name>
     namespace: <cluster_name>
   spec:
     kernelArguments:
       - operation: append
         value: audit=0
       - operation: append
         value: trace=1
     clusterRef:
       name: <cluster_name>
       namespace: <cluster_name>
     pullSecretRef:
       name: pull-secret

   where:

   operation

   Specify the append operation to add a kernel argument.

   value

   Specify the kernel argument you want to configure. This example configures the audit kernel argument and the trace kernel argument.

   Note

   The ClusterInstance CR generates the InfraEnv resource as part of the day-0 installation CRs.

1. Begin an SSH session with the target host:

   $ ssh -i /path/to/privatekey core@<host_name>

2. View the system’s kernel arguments by using the following command:

   $ cat /proc/cmdline

Procedure

1. Create a ClusterImageSet for each specific cluster version to be deployed, for example clusterImageSet-4.22.yaml. A ClusterImageSet has the following format:

   apiVersion: hive.openshift.io/v1
   kind: ClusterImageSet
   metadata:
     name: openshift-4.22.0
   spec:
      releaseImage: quay.io/openshift-release-dev/ocp-release:4.22.0-x86_64

   where:

   name

   The descriptive version that you want to deploy.

   releaseImage

   Specifies the releaseImage to deploy and determines the operating system image version. The discovery ISO is based on the image version as set by releaseImage, or the latest version if the exact version is unavailable.

2. Apply the clusterImageSet CR:

   $ oc apply -f clusterImageSet-4.22.yaml

3. Create the Namespace CR in the cluster-namespace.yaml file:

   apiVersion: v1
   kind: Namespace
   metadata:
        name: <cluster_name>
        labels:
           name: <cluster_name>

   where:

   name

   The name of the managed cluster to provision.

4. Apply the Namespace CR by running the following command:

   $ oc apply -f cluster-namespace.yaml

5. Apply the ClusterInstance CR that you configured to the hub cluster by running the following command:

   $ oc apply -f clusterinstance.yaml

   The SiteConfig Operator processes the ClusterInstance CR and automatically generates the required installation CRs, including BareMetalHost, AgentClusterInstall, ClusterDeployment, InfraEnv, and NMStateConfig. The assisted service then begins the cluster installation.

Procedure

1. Verify that the InfraEnv CR does not have a clusterRef field set:

   $ oc get infraenv <infraenv_name> -n <namespace> -o jsonpath='{.spec.clusterRef}'

   If the command returns an empty result, the InfraEnv is configured for late binding. If a clusterRef value is returned, the BareMetalHost cluster-reference annotation is ignored and you must use the standard binding flow.

2. Verify that the Agent CRs exist and are not bound to a cluster:

   $ oc get agents -n <namespace>

   Example output

   NAME                                   CLUSTER   APPROVED   ROLE          STAGE
   aaaaaaaa-1111-2222-3333-bbbbbbbbbbbb             false      auto-assign
   cccccccc-4444-5555-6666-dddddddddddd             false      auto-assign

   Agents that are not bound to a cluster have an empty CLUSTER column.

3. Add the bmac.agent-install.openshift.io/cluster-reference annotation to the BareMetalHost resource to bind it to the target ClusterDeployment CR:

   $ oc annotate bmh <bmh_name> -n <namespace> \
       bmac.agent-install.openshift.io/cluster-reference=<cluster_namespace>/<cluster_name>

   Replace <cluster_namespace>/<cluster_name> with the namespace and name of the target ClusterDeployment CR.

   Alternatively, you can set the annotation declaratively in the BareMetalHost YAML manifest:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: worker-01
     namespace: my-infraenv-ns
     annotations:
       bmac.agent-install.openshift.io/cluster-reference: my-cluster-ns/my-cluster
     labels:
       infraenvs.agent-install.openshift.io: my-infraenv
   spec:
     online: true
     bootMACAddress: 00:00:5E:00:53:01
     bmc:
       address: redfish-virtualmedia://192.0.2.10/redfish/v1/Systems/1
       credentialsName: worker-01-bmc-secret

   The bmac.agent-install.openshift.io/cluster-reference annotation value uses the format <namespace>/<name>, referencing the target ClusterDeployment.

4. Verify that the Agent CR is bound to the correct ClusterDeployment CR by running the following command:

   $ oc get agents -n <namespace>

   Example output

   NAME                                   CLUSTER      APPROVED   ROLE          STAGE
   aaaaaaaa-1111-2222-3333-bbbbbbbbbbbb   my-cluster   false      auto-assign

   The CLUSTER column shows the name of the target ClusterDeployment.

5. Verify the Bound condition on the Agent CR:

   $ oc get agent <agent_name> -n <namespace> \
       -o jsonpath='{.status.conditions[?(@.type=="Bound")]}'

   A successfully bound agent shows status: "True" and reason: "Bound" in the output.

   Note

   Agents discovered by using a BareMetalHost resource are automatically approved for installation when the bootMACAddress in the BareMetalHost matches a NIC in the inventory of the discovered host. You do not need to manually approve these agents.

6. Optional: To unbind a host from a cluster before installation starts, set the annotation value to an empty string:

   $ oc annotate bmh <bmh_name> -n <namespace> \
       bmac.agent-install.openshift.io/cluster-reference="" --overwrite

   The bare metal agent controller (BMAC) clears the spec.clusterDeploymentName field on the corresponding Agent CR, and the host returns to the unbound pool.

   Important

   You cannot unbind a host after cluster installation has started. If you try to unbind a host that is already installed or in an error or canceled state, the Agent CR Bound condition is set to False with the reason UnbindingPendingUserAction. For hosts discovered by using a BareMetalHost resource, the assisted controllers automatically use the BareMetalHost to boot the host back into the discovery ISO to complete the unbinding process. For hosts discovered without a BareMetalHost resource, you must manually reboot the host with the discovery ISO.

Procedure

1. Check the status of the managed cluster:

   $ oc get managedcluster

   True indicates the managed cluster is ready.

2. Check the agent status:

   $ oc get agent -n <cluster_name>

3. Use the describe command to provide an in-depth description of the agent’s condition. Statuses to be aware of include BackendError, InputError, ValidationsFailing, InstallationFailed, and AgentIsConnected. These statuses are relevant to the Agent and AgentClusterInstall custom resources.

   $ oc describe agent -n <cluster_name>

4. Check the cluster provisioning status:

   $ oc get agentclusterinstall -n <cluster_name>

5. Use the describe command to provide an in-depth description of the cluster provisioning status:

   $ oc describe agentclusterinstall -n <cluster_name>

6. Check the status of the managed cluster’s add-on services:

   $ oc get managedclusteraddon -n <cluster_name>

7. Retrieve the authentication information of the kubeconfig file for the managed cluster:

   $ oc get secret -n <cluster_name> <cluster_name>-admin-kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > <directory>/<cluster_name>-kubeconfig

Procedure

1. Check the status of the managed cluster:

   $ oc get managedcluster

   Example output

   NAME            HUB ACCEPTED   MANAGED CLUSTER URLS   JOINED   AVAILABLE   AGE
   SNO-cluster     true                                   True     True      2d19h

   If the status in the AVAILABLE column is True, the managed cluster is being managed by the hub.

   If the status in the AVAILABLE column is Unknown, the managed cluster is not being managed by the hub. Use the following steps to continue checking to get more information.

2. Check the AgentClusterInstall install status:

   $ oc get clusterdeployment -n <cluster_name>

   Example output

   NAME        PLATFORM            REGION   CLUSTERTYPE   INSTALLED    INFRAID    VERSION  POWERSTATE AGE
   Sno0026    agent-baremetal                               false                          Initialized
   2d14h

   If the status in the INSTALLED column is false, the installation was unsuccessful.

3. If the installation failed, enter the following command to review the status of the AgentClusterInstall resource:

   $ oc describe agentclusterinstall -n <cluster_name> <cluster_name>

4. Resolve the errors and reset the cluster:

   1. Remove the cluster’s managed cluster resource:

      $ oc delete managedcluster <cluster_name>

   2. Remove the cluster’s namespace:

      $ oc delete namespace <cluster_name>

      This deletes all of the namespace-scoped custom resources created for this cluster. You must wait for the ManagedCluster CR deletion to complete before proceeding.

   3. Recreate the custom resources for the managed cluster.