Chapter 5. Deploying hosted control planes in a disconnected environment

Procedure

1. Create a config map that resembles the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: <config_map_name> 1
     namespace: <config_map_namespace> 2
   data: 3
     <registry_name>..<port>: | 4
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
     <registry_name>..<port>: |
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
     <registry_name>..<port>: |
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----

   1

   Specify the name of the config map.

   2

   Specify the namespace for the config map.

   3

   In the data field, specify the registry names and the registry certificate content. Replace <port> with the port where the registry server is running; for example, 5000.

   4

   Ensure that the data in the config map is defined by using | only instead of other methods, such as | -. If you use other methods, issues can occur when the pod reads the certificates.

2. Patch the cluster-wide object, image.config.openshift.io to include the following specification:

   spec:
     additionalTrustedCA:
       - name: registry-config

   As a result of this patch, the control plane nodes can retrieve images from the private registry and the HyperShift Operator can extract the OpenShift Container Platform payload for hosted cluster deployments.

   The process to patch the object might take several minutes to be completed.

Procedure

1. In the hc.spec.additionalTrustBundle file, add the following specification:

   spec:
     additionalTrustBundle:
       - name: user-ca-bundle 1

   1

   The user-ca-bundle entry is a config map that you create in the next step.

2. In the same namespace where the HostedCluster object is created, create the user-ca-bundle config map. The config map resembles the following example:

   apiVersion: v1
   data:
     ca-bundle.crt: |
       // Registry1 CA
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
   
       // Registry2 CA
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
   
       // Registry3 CA
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
   
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: <hosted_cluster_namespace> 1

   1

   Specify the namespace where the HostedCluster object is created.

Procedure

1. Enter the following command:

   $ hcp create cluster kubevirt \
     --name <hosted-cluster-name> \ 1
     --node-pool-replicas <worker-count> \ 2
     --pull-secret <path-to-pull-secret> \ 3
     --memory <value-for-memory> \ 4
     --cores <value-for-cpu> \ 5
     --etcd-storage-class=<etcd-storage-class> 6

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the worker count, for example, 2.

   3

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   4

   Specify a value for memory, for example, 6Gi.

   5

   Specify a value for CPU, for example, 2.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   Note

   You can use the --release-image flag to set up the hosted cluster with a specific OpenShift Container Platform release.

   A default node pool is created for the cluster with two virtual machine worker replicas according to the --node-pool-replicas flag.

2. After a few moments, verify that the hosted control plane pods are running by entering the following command:

   $ oc -n clusters-<hosted-cluster-name> get pods

   Example output

   NAME                                                  READY   STATUS    RESTARTS   AGE
   capi-provider-5cc7b74f47-n5gkr                        1/1     Running   0          3m
   catalog-operator-5f799567b7-fd6jw                     2/2     Running   0          69s
   certified-operators-catalog-784b9899f9-mrp6p          1/1     Running   0          66s
   cluster-api-6bbc867966-l4dwl                          1/1     Running   0          66s
   .
   .
   .
   redhat-operators-catalog-9d5fd4d44-z8qqk              1/1     Running   0          66s

   A hosted cluster that has worker nodes that are backed by KubeVirt virtual machines typically takes 10-15 minutes to be fully provisioned.

3. To check the status of the hosted cluster, see the corresponding HostedCluster resource by entering the following command:

   $ oc get --namespace clusters hostedclusters

   See the following example output, which illustrates a fully provisioned HostedCluster object:

   NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
   clusters    example   4.x.0     example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

   Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

Procedure

1. Configure the custom registries by adding the following config map, which contains the disconnected details to customize the deployment:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: custom-registries
     namespace: multicluster-engine
     labels:
       app: assisted-service
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
     registries.conf: |
       unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
   
       [[registry]]
       prefix = ""
       location = "registry.redhat.io/openshift4"
       mirror-by-digest-only = true
   
       [[registry.mirror]]
         location = "registry.dns.base.domain.name:5000/openshift4" 1
   
       [[registry]]
       prefix = ""
       location = "registry.redhat.io/rhacm2"
       mirror-by-digest-only = true
       # ...
       # ...

   1

   Replace dns.base.domain.name with the DNS base domain name.

   The object contains two fields:

   • Custom CAs: This field contains the Certificate Authorities (CAs) that are loaded into the various processes of the deployment.
   • Registries: The Registries.conf field contains information about images and namespaces that need to be consumed from a mirror registry rather than the original source registry.

2. Configure the Assisted Service by adding the AssistedServiceConfig object, as shown in the following example:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: AgentServiceConfig
   metadata:
     annotations:
       unsupported.agent-install.openshift.io/assisted-service-configmap: assisted-service-config 1
     name: agent
     namespace: multicluster-engine
   spec:
     mirrorRegistryRef:
       name: custom-registries 2
     databaseStorage:
       storageClassName: lvms-vg1
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: 10Gi
     filesystemStorage:
       storageClassName: lvms-vg1
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: 20Gi
     osImages: 3
     - cpuArchitecture: x86_64 4
       openshiftVersion: "4.14"
       rootFSUrl: http://registry.dns.base.domain.name:8080/images/rhcos-414.92.202308281054-0-live-rootfs.x86_64.img 5
       url: http://registry.dns.base.domain.name:8080/images/rhcos-414.92.202308281054-0-live.x86_64.iso
       version: 414.92.202308281054-0
     - cpuArchitecture: x86_64
      openshiftVersion: "4.15"
      rootFSUrl: http://registry.dns.base.domain.name:8080/images/rhcos-415.92.202403270524-0-live-rootfs.x86_64.img
      url: http://registry.dns.base.domain.name:8080/images/rhcos-415.92.202403270524-0-live.x86_64.iso
      version: 415.92.202403270524-0

   1

   The metadata.annotations["unsupported.agent-install.openshift.io/assisted-service-configmap"] annotation references the config map name that the Operator consumes to customize behavior.

   2

   The spec.mirrorRegistryRef.name annotation points to the config map that contains disconnected registry information that the Assisted Service Operator consumes. This config map adds those resources during the deployment process.

   3

   The spec.osImages field contains different versions available for deployment by this Operator. This field is mandatory. This example assumes that you already downloaded the RootFS and LiveISO files.

   4

   Add a cpuArchitecture subsection for every OpenShift Container Platform release that you want to deploy. In this example, cpuArchitecture subsections are included for 4.14 and 4.15.

   5

   In the rootFSUrl and url fields, replace dns.base.domain.name with the DNS base domain name.

3. Deploy all of the objects by concatenating them into a single file and applying them to the management cluster. To do so, enter the following command:

   $ oc apply -f agentServiceConfig.yaml

   The command triggers two pods.

   Example output

   assisted-image-service-0                               1/1     Running   2             11d 1
   assisted-service-668b49548-9m7xw                       2/2     Running   5             11d 2

   1

   The assisted-image-service pod is responsible for creating the Red Hat Enterprise Linux CoreOS (RHCOS) boot image template, which is customized for each cluster that you deploy.

   2

   The assisted-service refers to the Operator.

Procedure

1. Create a config map that resembles the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: <config_map_name> 1
     namespace: <config_map_namespace> 2
   data: 3
     <registry_name>..<port>: | 4
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
     <registry_name>..<port>: |
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
     <registry_name>..<port>: |
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----

   1

   Specify the name of the config map.

   2

   Specify the namespace for the config map.

   3

   In the data field, specify the registry names and the registry certificate content. Replace <port> with the port where the registry server is running; for example, 5000.

   4

   Ensure that the data in the config map is defined by using | only instead of other methods, such as | -. If you use other methods, issues can occur when the pod reads the certificates.

2. Patch the cluster-wide object, image.config.openshift.io to include the following specification:

   spec:
     additionalTrustedCA:
       - name: registry-config

   As a result of this patch, the control plane nodes can retrieve images from the private registry and the HyperShift Operator can extract the OpenShift Container Platform payload for hosted cluster deployments.

   The process to patch the object might take several minutes to be completed.

Procedure

1. In the hc.spec.additionalTrustBundle file, add the following specification:

   spec:
     additionalTrustBundle:
       - name: user-ca-bundle 1

   1

   The user-ca-bundle entry is a config map that you create in the next step.

2. In the same namespace where the HostedCluster object is created, create the user-ca-bundle config map. The config map resembles the following example:

   apiVersion: v1
   data:
     ca-bundle.crt: |
       // Registry1 CA
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
   
       // Registry2 CA
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
   
       // Registry3 CA
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
   
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: <hosted_cluster_namespace> 1

   1

   Specify the namespace where the HostedCluster object is created.

Procedure

1. Create a YAML file with the following information about the namespaces:

   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     creationTimestamp: null
     name: <hosted_cluster_namespace>-<hosted_cluster_name> 1
   spec: {}
   status: {}
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     creationTimestamp: null
     name: <hosted_cluster_namespace> 2
   spec: {}
   status: {}

   1

   Replace <hosted_cluster_name> with your hosted cluster.

   2

   Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

2. Create a YAML file with the following information about the config maps and secrets to include in the HostedCluster deployment:

   ---
   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: <hosted_cluster_namespace> 1
   ---
   apiVersion: v1
   data:
     .dockerconfigjson: xxxxxxxxx
   kind: Secret
   metadata:
     creationTimestamp: null
     name: <hosted_cluster_name>-pull-secret 2
     namespace: <hosted_cluster_namespace> 3
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: sshkey-cluster-<hosted_cluster_name> 4
     namespace: <hosted_cluster_namespace> 5
   stringData:
     id_rsa.pub: ssh-rsa xxxxxxxxx
   ---
   apiVersion: v1
   data:
     key: nTPtVBEt03owkrKhIdmSW8jrWRxU57KO/fnZa8oaG0Y=
   kind: Secret
   metadata:
     creationTimestamp: null
     name: <hosted_cluster_name>-etcd-encryption-key 6
     namespace: <hosted_cluster_namespace> 7
   type: Opaque

   1 3 5 7

   Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

   2 4 6

   Replace <hosted_cluster_name> with your hosted cluster.

3. Create a YAML file that contains the RBAC roles so that Assisted Service agents can be in the same HostedControlPlane namespace as the hosted control plane and still be managed by the cluster API:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     creationTimestamp: null
     name: capi-provider-role
     namespace: <hosted_cluster_namespace>-<hosted_cluster_name> 1 2
   rules:
   - apiGroups:
     - agent-install.openshift.io
     resources:
     - agents
     verbs:
     - '*'

   1

   Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

   2

   Replace <hosted_cluster_name> with your hosted cluster.

4. Create a YAML file with information about the HostedCluster object, replacing values as necessary:

   apiVersion: hypershift.openshift.io/v1beta1
   kind: HostedCluster
   metadata:
     name: <hosted_cluster_name> 1
     namespace: <hosted_cluster_namespace> 2
   spec:
     additionalTrustBundle:
       name: "user-ca-bundle"
     olmCatalogPlacement: guest
     imageContentSources: 3
     - source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
       mirrors:
       - registry.<dns.base.domain.name>:5000/openshift/release 4
     - source: quay.io/openshift-release-dev/ocp-release
       mirrors:
       - registry.<dns.base.domain.name>:5000/openshift/release-images 5
     - mirrors:
     ...
     ...
     autoscaling: {}
     controllerAvailabilityPolicy: SingleReplica
     dns:
       baseDomain: <dns.base.domain.name> 6
     etcd:
       managed:
         storage:
           persistentVolume:
             size: 8Gi
           restoreSnapshotURL: null
           type: PersistentVolume
       managementType: Managed
     fips: false
     networking:
       clusterNetwork:
       - cidr: 10.132.0.0/14
       - cidr: fd01::/48
       networkType: OVNKubernetes
       serviceNetwork:
       - cidr: 172.31.0.0/16
       - cidr: fd02::/112
     platform:
       agent:
         agentNamespace: <hosted_cluster_namespace>-<hosted_cluster_name> 7 8
       type: Agent
     pullSecret:
       name: <hosted_cluster_name>-pull-secret 9
     release:
       image: registry.<dns.base.domain.name>:5000/openshift/release-images:4.x.y-x86_64 10 11
     secretEncryption:
       aescbc:
         activeKey:
           name: <hosted_cluster_name>-etcd-encryption-key 12
       type: aescbc
     services:
     - service: APIServer
       servicePublishingStrategy:
         type: LoadBalancer
     - service: OAuthServer
       servicePublishingStrategy:
         type: Route
     - service: OIDC
       servicePublishingStrategy:
         type: Route
     - service: Konnectivity
       servicePublishingStrategy:
         type: Route
     - service: Ignition
       servicePublishingStrategy:
         type: Route
     sshKey:
       name: sshkey-cluster-<hosted_cluster_name> 13
   status:
     controlPlaneEndpoint:
       host: ""
       port: 0

   1 7 9 12 13

   Replace <hosted_cluster_name> with your hosted cluster.

   2 8

   Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

   3

   The imageContentSources section contains mirror references for user workloads within the hosted cluster.

   4 5 6 10

   Replace <dns.base.domain.name> with the DNS base domain name.

   11

   Replace 4.x.y with the supported OpenShift Container Platform version you want to use.

5. Add an annotation in the HostedCluster object that points to the HyperShift Operator release in the OpenShift Container Platform release:

   1. Obtain the image payload by entering the following command:

      $ oc adm release info registry.<dns.base.domain.name>:5000/openshift-release-dev/ocp-release:4.x.y-x86_64 | grep hypershift

      where <dns.base.domain.name> is the DNS base domain name and 4.x.y is the supported OpenShift Container Platform version you want to use.

      Example output

      hypershift        sha256:31149e3e5f8c5e5b5b100ff2d89975cf5f7a73801b2c06c639bf6648766117f8

   2. By using the OpenShift Container Platform Images namespace, check the digest by entering the following command:

      podman pull registry.<dns.base.domain.name>:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:31149e3e5f8c5e5b5b100ff2d89975cf5f7a73801b2c06c639bf6648766117f8

      where <dns.base.domain.name> is the DNS base domain name.

      Example output

      podman pull registry.dns.base.domain.name:5000/openshift/release@sha256:31149e3e5f8c5e5b5b100ff2d89975cf5f7a73801b2c06c639bf6648766117f8
      Trying to pull registry.dns.base.domain.name:5000/openshift/release@sha256:31149e3e5f8c5e5b5b100ff2d89975cf5f7a73801b2c06c639bf6648766117f8...
      Getting image source signatures
      Copying blob d8190195889e skipped: already exists
      Copying blob c71d2589fba7 skipped: already exists
      Copying blob d4dc6e74b6ce skipped: already exists
      Copying blob 97da74cc6d8f skipped: already exists
      Copying blob b70007a560c9 done
      Copying config 3a62961e6e done
      Writing manifest to image destination
      Storing signatures
      3a62961e6ed6edab46d5ec8429ff1f41d6bb68de51271f037c6cb8941a007fde

      The release image that is set in the HostedCluster object must use the digest rather than the tag; for example, quay.io/openshift-release-dev/ocp-release@sha256:e3ba11bd1e5e8ea5a0b36a75791c90f29afb0fdbe4125be4e48f69c76a5c47a0.

6. Create all of the objects that you defined in the YAML files by concatenating them into a file and applying them against the management cluster. To do so, enter the following command:

   $ oc apply -f 01-4.14-hosted_cluster-nodeport.yaml

   Example output

   NAME                                                  READY   STATUS    RESTARTS   AGE
   capi-provider-5b57dbd6d5-pxlqc                        1/1     Running   0          3m57s
   catalog-operator-9694884dd-m7zzv                      2/2     Running   0          93s
   cluster-api-f98b9467c-9hfrq                           1/1     Running   0          3m57s
   cluster-autoscaler-d7f95dd5-d8m5d                     1/1     Running   0          93s
   cluster-image-registry-operator-5ff5944b4b-648ht      1/2     Running   0          93s
   cluster-network-operator-77b896ddc-wpkq8              1/1     Running   0          94s
   cluster-node-tuning-operator-84956cd484-4hfgf         1/1     Running   0          94s
   cluster-policy-controller-5fd8595d97-rhbwf            1/1     Running   0          95s
   cluster-storage-operator-54dcf584b5-xrnts             1/1     Running   0          93s
   cluster-version-operator-9c554b999-l22s7              1/1     Running   0          95s
   control-plane-operator-6fdc9c569-t7hr4                1/1     Running   0          3m57s
   csi-snapshot-controller-785c6dc77c-8ljmr              1/1     Running   0          77s
   csi-snapshot-controller-operator-7c6674bc5b-d9dtp     1/1     Running   0          93s
   csi-snapshot-webhook-5b8584875f-2492j                 1/1     Running   0          77s
   dns-operator-6874b577f-9tc6b                          1/1     Running   0          94s
   etcd-0                                                3/3     Running   0          3m39s
   hosted-cluster-config-operator-f5cf5c464-4nmbh        1/1     Running   0          93s
   ignition-server-6b689748fc-zdqzk                      1/1     Running   0          95s
   ignition-server-proxy-54d4bb9b9b-6zkg7                1/1     Running   0          95s
   ingress-operator-6548dc758b-f9gtg                     1/2     Running   0          94s
   konnectivity-agent-7767cdc6f5-tw782                   1/1     Running   0          95s
   kube-apiserver-7b5799b6c8-9f5bp                       4/4     Running   0          3m7s
   kube-controller-manager-5465bc4dd6-zpdlk              1/1     Running   0          44s
   kube-scheduler-5dd5f78b94-bbbck                       1/1     Running   0          2m36s
   machine-approver-846c69f56-jxvfr                      1/1     Running   0          92s
   oauth-openshift-79c7bf44bf-j975g                      2/2     Running   0          62s
   olm-operator-767f9584c-4lcl2                          2/2     Running   0          93s
   openshift-apiserver-5d469778c6-pl8tj                  3/3     Running   0          2m36s
   openshift-controller-manager-6475fdff58-hl4f7         1/1     Running   0          95s
   openshift-oauth-apiserver-dbbc5cc5f-98574             2/2     Running   0          95s
   openshift-route-controller-manager-5f6997b48f-s9vdc   1/1     Running   0          95s
   packageserver-67c87d4d4f-kl7qh                        2/2     Running   0          93s

   When the hosted cluster is available, the output looks like the following example.

   Example output

   NAMESPACE   NAME         VERSION   KUBECONFIG                PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
   clusters    hosted-dual            hosted-admin-kubeconfig   Partial    True          False         The hosted control plane is available

Procedure

1. Create a YAML file with the following information about the NodePool object, replacing values as necessary:

   apiVersion: hypershift.openshift.io/v1beta1
   kind: NodePool
   metadata:
     creationTimestamp: null
     name: <hosted_cluster_name> \1
     namespace: <hosted_cluster_namespace> \2
   spec:
     arch: amd64
     clusterName: <hosted_cluster_name>
     management:
       autoRepair: false \3
       upgradeType: InPlace \4
     nodeDrainTimeout: 0s
     platform:
       type: Agent
     release:
       image: registry.<dns.base.domain.name>:5000/openshift/release-images:4.x.y-x86_64 \5
     replicas: 2 6
   status:
     replicas: 2

   1

   Replace <hosted_cluster_name> with your hosted cluster.

   2

   Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

   3

   The autoRepair field is set to false because the node will not be re-created if it is removed.

   4

   The upgradeType is set to InPlace, which indicates that the same bare metal node is reused during an upgrade.

   5

   All of the nodes included in this NodePool are based on the following OpenShift Container Platform version: 4.x.y-x86_64. Replace the <dns.base.domain.name> value with your DNS base domain name and the 4.x.y value with the supported OpenShift Container Platform version you want to use.

   6

   You can set the replicas value to 2 to create two node pool replicas in your hosted cluster.

2. Create the NodePool object by entering the following command:

   $ oc apply -f 02-nodepool.yaml

   Example output

   NAMESPACE   NAME          CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION                              UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
   clusters    hosted-dual   hosted    0                               False         False        4.x.y-x86_64

Procedure

1. Create a YAML file with the following information about the InfraEnv resource, replacing values as necessary:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <hosted_cluster_name>
     namespace: <hosted-cluster-namespace>-<hosted_cluster_name> 1 2
   spec:
     pullSecretRef: 3
       name: pull-secret
     sshAuthorizedKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDk7ICaUE+/k4zTpxLk4+xFdHi4ZuDi5qjeF52afsNkw0w/glILHhwpL5gnp5WkRuL8GwJuZ1VqLC9EKrdmegn4MrmUlq7WTsP0VFOZFBfq2XRUxo1wrRdor2z0Bbh93ytR+ZsDbbLlGngXaMa0Vbt+z74FqlcajbHTZ6zBmTpBVq5RHtDPgKITdpE1fongp7+ZXQNBlkaavaqv8bnyrP4BWahLP4iO9/xJF9lQYboYwEEDzmnKLMW1VtCE6nJzEgWCufACTbxpNS7GvKtoHT/OVzw8ArEXhZXQUS1UY8zKsX2iXwmyhw5Sj6YboA8WICs4z+TrFP89LmxXY0j6536TQFyRz1iB4WWvCbH5n6W+ABV2e8ssJB1AmEy8QYNwpJQJNpSxzoKBjI73XxvPYYC/IjPFMySwZqrSZCkJYqQ023ySkaQxWZT7in4KeMu7eS2tC+Kn4deJ7KwwUycx8n6RHMeD8Qg9flTHCv3gmab8JKZJqN3hW1D378JuvmIX4V0= 4

   1

   Replace <hosted_cluster_name> with your hosted cluster.

   2

   Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

   3

   The pullSecretRef refers to the config map reference in the same namespace as the InfraEnv, where the pull secret is used.

   4

   The sshAuthorizedKey represents the SSH public key that is placed in the boot image. The SSH key allows access to the worker nodes as the core user.

2. Create the InfraEnv resource by entering the following command:

   $ oc apply -f 03-infraenv.yaml

   Example output

   NAMESPACE              NAME     ISO CREATED AT
   clusters-hosted-dual   hosted   2023-09-11T15:14:10Z

Procedure

1. If this is not your first attempt to create worker nodes, you must first delete your previous setup. To do so, delete the plan by entering the following command:

   $ kcli delete plan <hosted_cluster_name> 1

   1

   Replace <hosted_cluster_name> with the name of your hosted cluster.

   1. When you are prompted to confirm whether you want to delete the plan, type y.
   2. Confirm that you see a message stating that the plan was deleted.

2. Create the virtual machines by entering the following commands:

   1. Enter the following command to create the first virtual machine:

      $ kcli create vm \
        -P start=False \1
        -P uefi_legacy=true \2
        -P plan=<hosted_cluster_name> \3
        -P memory=8192 -P numcpus=16 \4
        -P disks=[200,200] \5
        -P nets=["{\"name\": \"<network>\", \"mac\": \"aa:aa:aa:aa:11:01\"}"] \6
        -P uuid=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa1101 \
        -P name=<hosted_cluster_name>-worker0 7

      1

      Include start=False if you do not want the virtual machine (VM) to automatically start upon creation.

      2

      Include uefi_legacy=true to indicate that you will use UEFI legacy boot to ensure compatibility with previous UEFI implementations.

      3

      Replace <hosted_cluster_name> with the name of your hosted cluster. The plan=<hosted_cluster_name> statement indicates the plan name, which identifies a group of machines as a cluster.

      4

      Include the memory=8192 and numcpus=16 parameters to specify the resources for the VM, including the RAM and CPU.

      5

      Include disks=[200,200] to indicate that you are creating two thin-provisioned disks in the VM.

      6

      Include nets=[{"name": "<network>", "mac": "aa:aa:aa:aa:02:13"}] to provide network details, including the network name to connect to, the type of network (ipv4, ipv6, or dual), and the MAC address of the primary interface.

      7

      Replace <hosted_cluster_name> with the name of your hosted cluster.

   2. Enter the following command to create the second virtual machine:

      $ kcli create vm \
        -P start=False \1
        -P uefi_legacy=true \2
        -P plan=<hosted_cluster_name> \3
        -P memory=8192 -P numcpus=16 \4
        -P disks=[200,200] \5
        -P nets=["{\"name\": \"<network>\", \"mac\": \"aa:aa:aa:aa:11:02\"}"] \6
        -P uuid=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa1102
        -P name=<hosted_cluster_name>-worker1 7

      1

      Include start=False if you do not want the virtual machine (VM) to automatically start upon creation.

      2

      Include uefi_legacy=true to indicate that you will use UEFI legacy boot to ensure compatibility with previous UEFI implementations.

      3

      Replace <hosted_cluster_name> with the name of your hosted cluster. The plan=<hosted_cluster_name> statement indicates the plan name, which identifies a group of machines as a cluster.

      4

      Include the memory=8192 and numcpus=16 parameters to specify the resources for the VM, including the RAM and CPU.

      5

      Include disks=[200,200] to indicate that you are creating two thin-provisioned disks in the VM.

      6

      Include nets=[{"name": "<network>", "mac": "aa:aa:aa:aa:02:13"}] to provide network details, including the network name to connect to, the type of network (ipv4, ipv6, or dual), and the MAC address of the primary interface.

      7

      Replace <hosted_cluster_name> with the name of your hosted cluster.

   3. Enter the following command to create the third virtual machine:

      $ kcli create vm \
        -P start=False \1
        -P uefi_legacy=true \2
        -P plan=<hosted_cluster_name> \3
        -P memory=8192 -P numcpus=16 \4
        -P disks=[200,200] \5
        -P nets=["{\"name\": \"<network>\", \"mac\": \"aa:aa:aa:aa:11:03\"}"] \6
        -P uuid=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa1103
        -P name=<hosted_cluster_name>-worker2 7

      1

      Include start=False if you do not want the virtual machine (VM) to automatically start upon creation.

      2

      Include uefi_legacy=true to indicate that you will use UEFI legacy boot to ensure compatibility with previous UEFI implementations.

      3

      Replace <hosted_cluster_name> with the name of your hosted cluster. The plan=<hosted_cluster_name> statement indicates the plan name, which identifies a group of machines as a cluster.

      4

      Include the memory=8192 and numcpus=16 parameters to specify the resources for the VM, including the RAM and CPU.

      5

      Include disks=[200,200] to indicate that you are creating two thin-provisioned disks in the VM.

      6

      Include nets=[{"name": "<network>", "mac": "aa:aa:aa:aa:02:13"}] to provide network details, including the network name to connect to, the type of network (ipv4, ipv6, or dual), and the MAC address of the primary interface.

      7

      Replace <hosted_cluster_name> with the name of your hosted cluster.

3. Enter the restart ksushy command to restart the ksushy tool to ensure that the tool detects the VMs that you added:

   $ systemctl restart ksushy

   Example output

   +---------------------+--------+-------------------+----------------------------------------------------+-------------+---------+
   |         Name        | Status |         Ip        |                       Source                       |     Plan    | Profile |
   +---------------------+--------+-------------------+----------------------------------------------------+-------------+---------+
   |    hosted-worker0   |  down  |                   |                                                    | hosted-dual |  kvirt  |
   |    hosted-worker1   |  down  |                   |                                                    | hosted-dual |  kvirt  |
   |    hosted-worker2   |  down  |                   |                                                    | hosted-dual |  kvirt  |
   +---------------------+--------+-------------------+----------------------------------------------------+-------------+---------+

1. Create a YAML file with the following information:

   Because you have at least one secret that holds the bare metal host credentials, you need to create at least two objects for each worker node.

   apiVersion: v1
   kind: Secret
   metadata:
     name: <hosted_cluster_name>-worker0-bmc-secret \1
     namespace: <hosted_cluster_namespace>-<hosted_cluster_name> \2
   data:
     password: YWRtaW4= \3
     username: YWRtaW4= \4
   type: Opaque
   # ...
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: <hosted_cluster_name>-worker0
     namespace: <hosted_cluster_namespace>-<hosted_cluster_name> \5
     labels:
       infraenvs.agent-install.openshift.io: <hosted_cluster_name> \6
     annotations:
       inspect.metal3.io: disabled
       bmac.agent-install.openshift.io/hostname: <hosted_cluster_name>-worker0 \7
   spec:
     automatedCleaningMode: disabled \8
     bmc:
       disableCertificateVerification: true \9
       address: redfish-virtualmedia://[192.168.126.1]:9000/redfish/v1/Systems/local/<hosted_cluster_name>-worker0 \10
       credentialsName: <hosted_cluster_name>-worker0-bmc-secret \11
     bootMACAddress: aa:aa:aa:aa:02:11 \12
     online: true 13

   1

   Replace <hosted_cluster_name> with your hosted cluster.

   2 5

   Replace <hosted_cluster_name> with your hosted cluster. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

   3

   Specify the password of the baseboard management controller (BMC) in Base64 format.

   4

   Specify the user name of the BMC in Base64 format.

   6

   Replace <hosted_cluster_name> with your hosted cluster. The infraenvs.agent-install.openshift.io field serves as the link between the Assisted Installer and the BareMetalHost objects.

   7

   Replace <hosted_cluster_name> with your hosted cluster. The bmac.agent-install.openshift.io/hostname field represents the node name that is adopted during deployment.

   8

   The automatedCleaningMode field prevents the node from being erased by the Metal3 Operator.

   9

   The disableCertificateVerification field is set to true to bypass certificate validation from the client.

   10

   Replace <hosted_cluster_name> with your hosted cluster. The address field denotes the BMC address of the worker node.

   11

   Replace <hosted_cluster_name> with your hosted cluster. The credentialsName field points to the secret where the user and password credentials are stored.

   12

   The bootMACAddress field indicates the interface MAC address that the node starts from.

   13

   The online field defines the state of the node after the BareMetalHost object is created.

2. Deploy the BareMetalHost object by entering the following command:

   $ oc apply -f 04-bmh.yaml

   During the process, you can view the following output:

   • This output indicates that the process is trying to reach the nodes:

     Example output

     NAMESPACE         NAME             STATE         CONSUMER   ONLINE   ERROR   AGE
     clusters-hosted   hosted-worker0   registering              true             2s
     clusters-hosted   hosted-worker1   registering              true             2s
     clusters-hosted   hosted-worker2   registering              true             2s

   • This output indicates that the nodes are starting:

     Example output

     NAMESPACE         NAME             STATE          CONSUMER   ONLINE   ERROR   AGE
     clusters-hosted   hosted-worker0   provisioning              true             16s
     clusters-hosted   hosted-worker1   provisioning              true             16s
     clusters-hosted   hosted-worker2   provisioning              true             16s

   • This output indicates that the nodes started successfully:

     Example output

     NAMESPACE         NAME             STATE         CONSUMER   ONLINE   ERROR   AGE
     clusters-hosted   hosted-worker0   provisioned              true             67s
     clusters-hosted   hosted-worker1   provisioned              true             67s
     clusters-hosted   hosted-worker2   provisioned              true             67s

3. After the nodes start, notice the agents in the namespace, as shown in this example:

   Example output

   NAMESPACE         NAME                                   CLUSTER   APPROVED   ROLE          STAGE
   clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0411             true       auto-assign
   clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0412             true       auto-assign
   clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0413             true       auto-assign

   The agents represent nodes that are available for installation. To assign the nodes to a hosted cluster, scale up the node pool.

Procedure

1. To scale up the node pool, enter the following command:

   $ oc -n <hosted_cluster_namespace> scale nodepool <hosted_cluster_name> --replicas 3

   where:

   • <hosted_cluster_namespace> is the name of the hosted cluster namespace.
   • <hosted_cluster_name> is the name of the hosted cluster.

2. After the scaling process is complete, notice that the agents are assigned to a hosted cluster:

   Example output

   NAMESPACE         NAME                                   CLUSTER   APPROVED   ROLE          STAGE
   clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0411   hosted    true       auto-assign
   clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0412   hosted    true       auto-assign
   clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0413   hosted    true       auto-assign

3. Also notice that the node pool replicas are set:

   Example output

   NAMESPACE   NAME     CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION       UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
   clusters    hosted   hosted    3                               False         False        4.x.y-x86_64                                     Minimum availability requires 3 replicas, current 0 available

   Replace 4.x.y with the supported OpenShift Container Platform version that you want to use.

4. Wait until the nodes join the cluster. During the process, the agents provide updates on their stage and status.