Chapter 19. Deploying distributed units at scale in a disconnected environment

Procedure

1. Create a NMStateConfig custom resource:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: NMStateConfig
   metadata:
    name: <cluster_name>
    namespace: <cluster_name>
    labels:
      sno-cluster-<cluster-name>: <cluster_name>
   spec:
    config:
      interfaces:
        - name: eth0
          type: ethernet
          state: up
          ipv4:
            enabled: true
            address:
              - ip: <ip_address> 1
                prefix-length: <public_network_prefix> 2
            dhcp: false
      dns-resolver:
        config:
          server:
            - <dns_resolver> 3
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: <gateway> 4
            next-hop-interface: eth0
            table-id: 254
    interfaces:
      - name: "eth0" 5
        macAddress: <mac_address> 6

   1

   The static IP address of the target bare-metal host.

   2

   The static IP address’s subnet prefix for the target bare-metal host.

   3

   The DNS server for the target bare-metal host.

   4

   The gateway for the target bare-metal host.

   5

   Must match the name specified in the interfaces section.

   6

   The mac address of the interface.

2. When creating the BareMetalHost custom resource, ensure that one of its mac addresses matches a mac address in the NMStateConfig target bare-metal host.

3. When creating the InfraEnv custom resource, reference the label from the NMStateConfig custom resource in the InfraEnv custom resource:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <cluster_name>
     namespace: <cluster_name>
   spec:
     clusterRef:
       name: <cluster_name>
       namespace: <cluster_name>
     sshAuthorizedKey: <public_key>
     agentLabels: 1
       location: "<label-name>"
     pullSecretRef:
       name: assisted-deployment-pull-secret
     nmStateConfigLabelSelector:
       matchLabels:
         sno-cluster-<cluster-name>: <cluster_name> # Match this label

   1

   Sets a label to match. The labels apply when the agents boot.

1. A Discovery image ISO file is generated and booted on the target machine.
2. When the ISO file successfully boots on the target machine it reports the hardware information of the target machine.
3. After all hosts are discovered, OpenShift Container Platform is installed.
4. When OpenShift Container Platform finishes installing, the hub installs the klusterlet service on the target cluster.
5. The requested add-on services are installed on the target cluster.

Procedure

1. Check the status of the managed cluster:

   $ oc get managedcluster

   True indicates the managed cluster is ready.

2. Check the agent status:

   $ oc get agent -n <cluster_name>

3. Use the describe command to provide an in-depth description of the agent’s condition. Statuses to be aware of include BackendError, InputError, ValidationsFailing, InstallationFailed, and AgentIsConnected. These statuses are relevant to the Agent and AgentClusterInstall custom resources.

   $ oc describe agent -n <cluster_name>

4. Check the cluster provisioning status:

   $ oc get agentclusterinstall -n <cluster_name>

5. Use the describe command to provide an in-depth description of the cluster provisioning status:

   $ oc describe agentclusterinstall -n <cluster_name>

6. Check the status of the managed cluster’s add-on services:

   $ oc get managedclusteraddon -n <cluster_name>

7. Retrieve the authentication information of the kubeconfig file for the managed cluster:

   $ oc get secret -n <cluster_name> <cluster_name>-admin-kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > <directory>/<cluster_name>-kubeconfig

Procedure

1. Create a ConfigMap containing the mirror registry config:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: assisted-installer-mirror-config
     namespace: assisted-installer
     labels:
       app: assisted-service
   data:
     ca-bundle.crt: <certificate> 1
     registries.conf: |  2
       unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
   
       [[registry]]
         location = <mirror_registry_url>  3
         insecure = false
         mirror-by-digest-only = true

   1

   The mirror registry’s certificate used when creating the mirror registry.

   2

   The configuration for the mirror registry.

   3

   The URL of the mirror registry.

   This updates mirrorRegistryRef in the AgentServiceConfig custom resource, as shown below:

   Example output

   apiVersion: agent-install.openshift.io/v1beta1
   kind: AgentServiceConfig
   metadata:
     name: agent
     namespace: assisted-installer
   spec:
     databaseStorage:
       volumeName: <db_pv_name>
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: <db_storage_size>
     filesystemStorage:
       volumeName: <fs_pv_name>
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: <fs_storage_size>
     mirrorRegistryRef:
       name: 'assisted-installer-mirror-config'
     osImages:
       - openshiftVersion: <ocp_version>
         rootfs: <rootfs_url> 1
         url: <iso_url> 2

   1 2

   Must match the URLs of the HTTPD server.

2. For disconnected installations, you must deploy an NTP clock that is reachable through the disconnected network. You can do this by configuring chrony to act as server, editing the /etc/chrony.conf file, and adding the following allowed IPv6 range:

   # Allow NTP client access from local network.
   #allow 192.168.0.0/16
   local stratum 10
   bindcmdaddress ::
   allow 2620:52:0:1310::/64

Procedure

1. In the AgentClusterInstall custom resource, modify the IP addresses in clusterNetwork and serviceNetwork for IPv6 addresses:

   apiVersion: extensions.hive.openshift.io/v1beta1
   kind: AgentClusterInstall
   metadata:
     # Only include the annotation if using OVN, otherwise omit the annotation
     annotations:
       agent-install.openshift.io/install-config-overrides: '{"networking":{"networkType":"OVNKubernetes"}}'
     name: <cluster_name>
     namespace: <cluster_name>
   spec:
     clusterDeploymentRef:
       name: <cluster_name>
     imageSetRef:
       name: <cluster_image_set>
     networking:
       clusterNetwork:
       - cidr: "fd01::/48"
         hostPrefix: 64
       machineNetwork:
       - cidr: <machine_network_cidr>
       serviceNetwork:
       - "fd02::/112"
     provisionRequirements:
       controlPlaneAgents: 1
       workerAgents: 0
     sshPublicKey: <public_key>

2. Update the NMStateConfig custom resource with the IPv6 addresses you defined.

Procedure

1. Check the status of the managed cluster:

   $ oc get managedcluster

   Example output

   NAME            HUB ACCEPTED   MANAGED CLUSTER URLS   JOINED   AVAILABLE   AGE
   SNO-cluster     true                                   True     True      2d19h

   If the status in the AVAILABLE column is True, the managed cluster is being managed by the hub.

   If the status in the AVAILABLE column is Unknown, the managed cluster is not being managed by the hub. Use the following steps to continue checking to get more information.

2. Check the AgentClusterInstall install status:

   $ oc get clusterdeployment -n <cluster_name>

   Example output

   NAME        PLATFORM            REGION   CLUSTERTYPE   INSTALLED    INFRAID    VERSION  POWERSTATE AGE
   Sno0026    agent-baremetal                               false                          Initialized
   2d14h

   If the status in the INSTALLED column is false, the installation was unsuccessful.

3. If the installation failed, enter the following command to review the status of the AgentClusterInstall resource:

   $ oc describe agentclusterinstall -n <cluster_name> <cluster_name>

4. Resolve the errors and reset the cluster:

   1. Remove the cluster’s managed cluster resource:

      $ oc delete managedcluster <cluster_name>

   2. Remove the cluster’s namespace:

      $ oc delete namespace <cluster_name>

      This deletes all of the namespace-scoped custom resources created for this cluster. You must wait for the ManagedCluster CR deletion to complete before proceeding.

   3. Recreate the custom resources for the managed cluster.

Procedure

1. Configure the kustomization.yaml file to reference the policyGenerator.yaml file. The following example shows the PolicyGenerator definition:

   apiVersion: policyGenerator/v1
   kind: PolicyGenerator
   metadata:
     name: acm-policy
     namespace: acm-policy-generator
   # The arguments should be given and defined as below with same order --policyGenTempPath= --sourcePath= --outPath= --stdout --customResources
   argsOneLiner: ./ranPolicyGenTempExamples ./sourcePolicies ./out true false

   Where:

   • policyGenTempPath is the path to the policyGenTemp files.
   • sourcePath: is the path to the source policies.
   • outPath: is the path to save the generated ACM policies.
   • stdout: If true, prints the generated policies to the console.
   • customResources: If true generates the CRs from the sourcePolicies files without ACM policies.

2. Test PolicyGen by running the following commands:

   $ cd cnf-features-deploy/ztp/ztp-policy-generator/

   $ XDG_CONFIG_HOME=./ kustomize build --enable-alpha-plugins

   An out directory is created with the expected policies, as shown in this example:

   out
   ├── common
   │   ├── common-log-sub-ns-policy.yaml
   │   ├── common-log-sub-oper-policy.yaml
   │   ├── common-log-sub-policy.yaml
   │   ├── common-pao-sub-catalog-policy.yaml
   │   ├── common-pao-sub-ns-policy.yaml
   │   ├── common-pao-sub-oper-policy.yaml
   │   ├── common-pao-sub-policy.yaml
   │   ├── common-policies-placementbinding.yaml
   │   ├── common-policies-placementrule.yaml
   │   ├── common-ptp-sub-ns-policy.yaml
   │   ├── common-ptp-sub-oper-policy.yaml
   │   ├── common-ptp-sub-policy.yaml
   │   ├── common-sriov-sub-ns-policy.yaml
   │   ├── common-sriov-sub-oper-policy.yaml
   │   └── common-sriov-sub-policy.yaml
   ├── groups
   │   ├── group-du
   │   │   ├── group-du-mc-chronyd-policy.yaml
   │   │   ├── group-du-mc-mount-ns-policy.yaml
   │   │   ├── group-du-mcp-du-policy.yaml
   │   │   ├── group-du-mc-sctp-policy.yaml
   │   │   ├── group-du-policies-placementbinding.yaml
   │   │   ├── group-du-policies-placementrule.yaml
   │   │   ├── group-du-ptp-config-policy.yaml
   │   │   └── group-du-sriov-operconfig-policy.yaml
   │   └── group-sno-du
   │       ├── group-du-sno-policies-placementbinding.yaml
   │       ├── group-du-sno-policies-placementrule.yaml
   │       ├── group-sno-du-console-policy.yaml
   │       ├── group-sno-du-log-forwarder-policy.yaml
   │       └── group-sno-du-log-policy.yaml
   └── sites
       └── site-du-sno-1
           ├── site-du-sno-1-policies-placementbinding.yaml
           ├── site-du-sno-1-policies-placementrule.yaml
           ├── site-du-sno-1-sriov-nn-fh-policy.yaml
           ├── site-du-sno-1-sriov-nnp-mh-policy.yaml
           ├── site-du-sno-1-sriov-nw-fh-policy.yaml
           ├── site-du-sno-1-sriov-nw-mh-policy.yaml
           └── site-du-sno-1-.yaml

   The common policies are flat because they will be applied to all clusters. However, the groups and sites have subdirectories for each group and site as they will be applied to different clusters.

Procedure

1. Create a secret for authenticating to the site Baseboard Management Controller (BMC). Ensure the secret name matches the name used in the SiteConfig. In this example, the secret name is test-sno-bmh-secret:

   apiVersion: v1
   kind: Secret
   metadata:
     name: test-sno-bmh-secret
     namespace: test-sno
   data:
     password: dGVtcA==
     username: cm9vdA==
   type: Opaque

2. Create the pull secret for the site. The pull secret must contain all credentials necessary for installing OpenShift and all add-on Operators. In this example, the secret name is assisted-deployment-pull-secret:

   apiVersion: v1
   kind: Secret
   metadata:
     name: assisted-deployment-pull-secret
     namespace: test-sno
   type: kubernetes.io/dockerconfigjson
   data:
     .dockerconfigjson: <Your pull secret base64 encoded>

Procedure

1. Create one or more SiteConfig custom resources, site-config.yaml files, that contains the site-plan data for the clusters. For example:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "test-sno"
     namespace: "test-sno"
   spec:
     baseDomain: "clus2.t5g.lab.eng.bos.redhat.com"
     pullSecretRef:
       name: "assisted-deployment-pull-secret"
     clusterImageSetNameRef: "openshift-4.9"
     sshPublicKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDB3dwhI5X0ZxGBb9VK7wclcPHLc8n7WAyKjTNInFjYNP9J+Zoc/ii+l3YbGUTuqilDwZN5rVIwBux2nUyVXDfaM5kPd9kACmxWtfEWTyVRootbrNWwRfKuC2h6cOd1IlcRBM1q6IzJ4d7+JVoltAxsabqLoCbK3svxaZoKAaK7jdGG030yvJzZaNM4PiTy39VQXXkCiMDmicxEBwZx1UsA8yWQsiOQ5brod9KQRXWAAST779gbvtgXR2L+MnVNROEHf1nEjZJwjwaHxoDQYHYKERxKRHlWFtmy5dNT6BbvOpJ2e5osDFPMEd41d2mUJTfxXiC1nvyjk9Irf8YJYnqJgBIxi0IxEllUKH7mTdKykHiPrDH5D2pRlp+Donl4n+sw6qoDc/3571O93+RQ6kUSAgAsvWiXrEfB/7kGgAa/BD5FeipkFrbSEpKPVu+gue1AQeJcz9BuLqdyPUQj2VUySkSg0FuGbG7fxkKeF1h3Sga7nuDOzRxck4I/8Z7FxMF/e8DmaBpgHAUIfxXnRqAImY9TyAZUEMT5ZPSvBRZNNmLbfex1n3NLcov/GEpQOqEYcjG5y57gJ60/av4oqjcVmgtaSOOAS0kZ3y9YDhjsaOcpmRYYijJn8URAH7NrW8EZsvAoF6GUt6xHq5T258c6xSYUm5L0iKvBqrOW9EjbLw== root@cnfdc2.clus2.t5g.lab.eng.bos.redhat.com"
     clusters:
     - clusterName: "test-sno"
       clusterType: "sno"
       clusterProfile: "du"
       clusterLabels:
         group-du-sno: ""
         common: true
         sites : "test-sno"
       clusterNetwork:
         - cidr: 1001:db9::/48
           hostPrefix: 64
       machineNetwork:
         - cidr: 2620:52:0:10e7::/64
       serviceNetwork:
         - 1001:db7::/112
       additionalNTPSources:
         - 2620:52:0:1310::1f6
       nodes:
         - hostName: "test-sno.clus2.t5g.lab.eng.bos.redhat.com"
           bmcAddress: "idrac-virtualmedia+https://[2620:52::10e7:f602:70ff:fee4:f4e2]/redfish/v1/Systems/System.Embedded.1"
           bmcCredentialsName:
             name: "test-sno-bmh-secret"
           bmcDisableCertificateVerification: true 1
           bootMACAddress: "0C:42:A1:8A:74:EC"
           bootMode: "UEFI"
           rootDeviceHints:
             hctl: '0:1:0'
           cpuset: "0-1,52-53"
           nodeNetwork:
             interfaces:
               - name: eno1
                 macAddress: "0C:42:A1:8A:74:EC"
             config:
               interfaces:
                 - name: eno1
                   type: ethernet
                   state: up
                   macAddress: "0C:42:A1:8A:74:EC"
                   ipv4:
                     enabled: false
                   ipv6:
                     enabled: true
                     address:
                     - ip: 2620:52::10e7:e42:a1ff:fe8a:900
                       prefix-length: 64
               dns-resolver:
                 config:
                   search:
                   - clus2.t5g.lab.eng.bos.redhat.com
                   server:
                   - 2620:52:0:1310::1f6
               routes:
                 config:
                 - destination: ::/0
                   next-hop-interface: eno1
                   next-hop-address: 2620:52:0:10e7::fc
                   table-id: 254

   1

   If you are using UEFI SecureBoot, add this line to prevent failures due to invalid or local certificates.

2. Save the files and push them to the zero touch provisioning (ZTP) Git repository accessible from the hub cluster and defined as a source repository of the ArgoCD application.

Procedure

1. Create the PolicyGenTemplates and save them to the zero touch provisioning (ZTP) Git repository accessible from the hub cluster and defined as a source repository of the ArgoCD application.

2. ArgoCD detects that the application is out of sync. Upon sync, either automatic or manual, ArgoCD applies the new PolicyGenTemplate to the hub cluster and launches the associated resource hooks. These hooks are responsible for generating the policy wrapped configuration CRs that apply to the spoke cluster and perform the following actions:

   1. Create the Red Hat Advanced Cluster Management (RHACM) (ACM) policies according to the basic distributed unit (DU) profile and required customizations.
   2. Apply the generated policies to the hub cluster.

Procedure

1. Monitor the progress of cluster installation using the following commands:

   $ export CLUSTER=<cluster_name>

   $ oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.conditions[?(@.type=="Completed")]}' | jq

   $ curl -sk $(oc get agentclusterinstall -n $CLUSTER $CLUSTER -o jsonpath='{.status.debugInfo.eventsURL}') | jq '.[-2,-1]'

2. Use the Red Hat Advanced Cluster Management (RHACM) (ACM) dashboard to monitor the progress of policy reconciliation.

Procedure

1. Check the synchronization of the SiteConfig to the hub cluster using either of the following commands:

   $ oc get siteconfig -A

   or

   $ oc get siteconfig -n clusters-sub

   If the SiteConfig is missing, one of the following situations has occurred:

   • The clusters application failed to synchronize the CR from the Git repository to the hub. Use the following command to verify this:

     $ oc describe -n openshift-gitops application clusters

     Check for Status: Synced and that the Revision: is the SHA of the commit you pushed to the subscribed repository.

   • The pre-sync hook failed, possibly due to a failure to pull the container image. Check the ArgoCD dashboard for the status of the pre-sync job in the clusters application.

2. Verify the post hook job ran:

   $ oc describe job -n clusters-sub siteconfig-post

   • If successful, the returned output indicates succeeded: 1.
   • If the job fails, ArgoCD retries it. In some cases, the first pass will fail and the second pass will indicate that the job passed.

3. Check for errors in the post hook job:

   $ oc get pod -n clusters-sub

   Note the name of the siteconfig-post-xxxxx pod:

   $ oc logs -n clusters-sub siteconfig-post-xxxxx

   If the logs indicate errors, correct the conditions and push the corrected SiteConfig or PolicyGenTemplate to the Git repository.

Procedure

1. Check the synchronization of the PolicyGenTemplate to the hub cluster:

   $ oc get policygentemplate -A

   or

   $ oc get policygentemplate -n $NS

   If the PolicyGenTemplate is not synchronized, one of the following situations has occurred:

   • The clusters application failed to synchronize the CR from the Git repository to the hub. Use the following command to verify this:

     $ oc describe -n openshift-gitops application clusters

     Check for Status: Synced and that the Revision: is the SHA of the commit you pushed to the subscribed repository.

   • The pre-sync hook failed, possibly due to a failure to pull the container image. Check the ArgoCD dashboard for the status of the pre-sync job in the clusters application.

2. Ensure the policies were copied to the cluster namespace. When ACM recognizes that policies apply to a ManagedCluster, ACM applies the policy CR objects to the cluster namespace:

   $ oc get policy -n <cluster_name>

   ACM copies all applicable common, group, and site policies here. The policy names are <policyNamespace> and <policyName>.

3. Check the placement rule for any policies not copied to the cluster namespace. The matchSelector in the PlacementRule for those policies should match the labels on the ManagedCluster:

   $ oc get placementrule -n $NS

4. Make a note of the PlacementRule name for the missing common, group, or site policy:

    oc get placementrule -n $NS <placmentRuleName> -o yaml

   • The status decisions value should include your cluster name.

   • The key value of the matchSelector in the spec should match the labels on your managed cluster. Check the labels on ManagedCluster:

      oc get ManagedCluster $CLUSTER -o jsonpath='{.metadata.labels}' | jq

     Example

     apiVersion: apps.open-cluster-management.io/v1
     kind: PlacementRule
     metadata:
       name: group-test1-policies-placementrules
       namespace: group-test1-policies
     spec:
       clusterSelector:
         matchExpressions:
         - key: group-test1
           operator: In
           values:
           - ""
     status:
       decisions:
       - clusterName: <cluster_name>
         clusterNamespace: <cluster_name>

5. Ensure all policies are compliant:

    oc get policy -n $CLUSTER

   If the Namespace, OperatorGroup, and Subscription policies are compliant but the Operator configuration policies are not it is likely that the Operators did not install.