Chapter 3. Deploying hosted control planes

1. Create an Amazon Web Services (AWS) credential secret for the HyperShift Operator and name it hypershift-operator-external-dns-credentials in the local-cluster namespace.

2. See the following table to verify that the secret has the required fields:

   Table 3.2. Required fields for the AWS secret

   Field name	Description
   Optional or required	provider
   The DNS provider that manages the service-level DNS zone.	Required
   domain-filter	The service-level domain.
   Required	credentials
   The credential file that supports all external DNS types.	Optional when you use AWS keys
   aws-access-key-id	The credential access key id.
   Optional when you use the AWS DNS service	aws-secret-access-key
   The credential access key secret.	Optional when you use the AWS DNS service

3. To create an AWS secret, run the following command:

   $ oc create secret generic <secret_name> --from-literal=provider=aws --from-literal=domain-filter=<domain_name> --from-file=credentials=<path_to_aws_credentials_file> -n local-cluster

   Note

   Disaster recovery backup for the secret is not automatically enabled. To back up the secret for disaster recovery, add the hypershift-operator-external-dns-credentials by entering the following command:

   $ oc label secret hypershift-operator-external-dns-credentials -n local-cluster cluster.open-cluster-management.io/backup=""

Procedure

1. In the Route 53 management console, click Create hosted zone.
2. On the Hosted zone configuration page, type a domain name, verify that Publish hosted zone is selected as the type, and click Create hosted zone.
3. After the zone is created, on the Records tab, note the values in the Value/Route traffic to column.
4. In the main domain, create an NS record to redirect the DNS requests to the delegated zone. In the Value field, enter the values that you noted in the previous step.
5. Click Create records.

6. Verify that the DNS hosted zone is working by creating a test entry in the new subzone and testing it with a dig command, such as in the following example:

   $ dig +short test.user-dest-public.aws.kerberos.com

   Example output

   192.168.1.1

7. To create a hosted cluster that sets the hostname for the LoadBalancer and Route services, enter the following command:

   $ hcp create cluster aws --name=<hosted_cluster_name> --endpoint-access=PublicAndPrivate --external-dns-domain=<public_hosted_zone> ... 1

   1

   Replace <public_hosted_zone> with the public hosted zone that you created.

   Example services block for the hosted cluster

     platform:
       aws:
         endpointAccess: PublicAndPrivate
   ...
     services:
     - service: APIServer
       servicePublishingStrategy:
         route:
           hostname: api-example.service-provider-domain.com
         type: Route
     - service: OAuthServer
       servicePublishingStrategy:
         route:
           hostname: oauth-example.service-provider-domain.com
         type: Route
     - service: Konnectivity
       servicePublishingStrategy:
         type: Route
     - service: Ignition
       servicePublishingStrategy:
         type: Route

Procedure

1. To access your management cluster, enter the following command:

   $ export KUBECONFIG=<path_to_management_cluster_kubeconfig>

2. Verify that the External DNS Operator is running by entering the following command:

   $ oc get pod -n hypershift -lapp=external-dns

   Example output

   NAME                            READY   STATUS    RESTARTS   AGE
   external-dns-7c89788c69-rn8gp   1/1     Running   0          40s

3. To create a hosted cluster by using external DNS, enter the following command:

   $ hcp create cluster aws \
       --role-arn <arn_role> \ 1
       --instance-type <instance_type> \ 2
       --region <region> \ 3
       --auto-repair \
       --generate-ssh \
       --name <hosted_cluster_name> \ 4
       --namespace clusters \
       --base-domain <service_consumer_domain> \ 5
       --node-pool-replicas <node_replica_count> \ 6
       --pull-secret <path_to_your_pull_secret> \ 7
       --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 8
       --external-dns-domain=<service_provider_domain> \ 9
       --endpoint-access=PublicAndPrivate 10
       --sts-creds <path_to_sts_credential_file> 11

   1

   Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.

   2

   Specify the instance type, for example, m6i.xlarge.

   3

   Specify the AWS region, for example, us-east-1.

   4

   Specify your hosted cluster name, for example, my-external-aws.

   5

   Specify the public hosted zone that the service consumer owns, for example, service-consumer-domain.com.

   6

   Specify the node replica count, for example, 2.

   7

   Specify the path to your pull secret file.

   8

   Specify the supported {ocp-short} version that you want to use, for example, 4.14.0-x86_64.

   9

   Specify the public hosted zone that the service provider owns, for example, service-provider-domain.com.

   10

   Set as PublicAndPrivate. You can use external DNS with Public or PublicAndPrivate configurations only.

   11

   Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

Procedure

1. Generate the kubeconfig file by entering the following command:

   $ hcp create kubeconfig --namespace <hosted_cluster_namespace> --name <hosted_cluster_name> > <hosted_cluster_name>.kubeconfig

2. After you save the kubeconfig file, access the hosted cluster by entering the following command:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

Procedure

1. Create a hosted cluster on an ARM64 OpenShift Container Platform cluster by entering the following command:

   $ hcp create cluster aws \
     --name <hosted_cluster_name> \1
     --node-pool-replicas <node_pool_replica_count> \2
     --base-domain <basedomain> \3
     --pull-secret <path_to_pull_secret> \4
     --sts-creds <path_to_sts_credential_file> \5
     --region <region> \6
     --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \7
     --role-arn <role_name> \8
     --multi-arch 9

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the node pool replica count, for example, 3.

   3

   Specify your base domain, for example, example.com.

   4

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   5

   Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

   6

   Specify the AWS region name, for example, us-east-1.

   7

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see "Extracting the OpenShift Container Platform release image digest".

   8

   Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.

   9

   The multi-arch flag indicates that the hosted cluster supports both amd64 and arm64 node pools. When you set the flag and supply a release image, the release image must be a multi-architecture release image. When you set the flag and supply a release stream, the release stream must be a multi-architecture release stream. If you do not set the multi-arch flag, the management cluster and node pool CPU architectures must match. Otherwise, a warning is displayed. A node pool supports only one CPU architecture within the same node pool.

2. Add a NodePool object to the hosted cluster by running the following command:

   $ hcp create nodepool aws \
     --cluster-name <hosted_cluster_name> \1
     --name <nodepool_name> \2
     --node-count <node_pool_replica_count> 3

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the node pool name.

   3

   Specify the node pool replica count, for example, 3.

Procedure

1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials. For instructions to open the console, see Accessing the web console.
2. In the console header, ensure that All Clusters is selected.
3. Click Infrastructure Clusters.

4. Click Create cluster Host inventory Hosted control plane.

   The Create cluster page is displayed.

5. On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation.

   Note

   As you enter details about the cluster, you might find the following tips useful:

   • If you want to use predefined values to automatically populate fields in the console, you can create a host inventory credential. For more information, see Creating a credential for an on-premises environment.
   • On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a host inventory credential, the pull secret is automatically populated.
   • On the Node pools page, the namespace contains the hosts for the node pool. If you created a host inventory by using the console, the console creates a dedicated namespace.
   • On the Networking page, you select an API server publishing strategy. The API server for the hosted cluster can be exposed either by using an existing load balancer or as a service of the NodePort type. A DNS entry must exist for the api.<hosted_cluster_name>.<base_domain> setting that points to the destination where the API server can be reached. This entry can be a record that points to one of the nodes in the management cluster or a record that points to a load balancer that redirects incoming traffic to the Ingress pods.

6. Review your entries and click Create.

   The Hosted cluster view is displayed.

7. Monitor the deployment of the hosted cluster in the Hosted cluster view.
8. If you do not see information about the hosted cluster, ensure that All Clusters is selected, then click the cluster name.
9. Wait until the control plane components are ready. This process can take a few minutes.
10. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Procedure

1. Create a YAML file to define Image Content Source Policies (ICSP). See the following example:

   - mirrors:
     - brew.registry.redhat.io
     source: registry.redhat.io
   - mirrors:
     - brew.registry.redhat.io
     source: registry.stage.redhat.io
   - mirrors:
     - brew.registry.redhat.io
     source: registry-proxy.engineering.redhat.com

2. Save the file as icsp.yaml. This file contains your mirror registries.

3. To create a hosted cluster by using your mirror registries, run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --image-content-sources icsp.yaml  \6
       --ssh-key  <path_to_ssh_key> \7
       --namespace <hosted_cluster_namespace> \8
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the icsp.yaml file that defines ICSP and your mirror registries.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

Procedure

1. Enter the following command:

   $ hcp create cluster kubevirt \
     --name <hosted-cluster-name> \ 1
     --node-pool-replicas <worker-count> \ 2
     --pull-secret <path-to-pull-secret> \ 3
     --memory <value-for-memory> \ 4
     --cores <value-for-cpu> \ 5
     --etcd-storage-class=<etcd-storage-class> 6

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the worker count, for example, 2.

   3

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   4

   Specify a value for memory, for example, 6Gi.

   5

   Specify a value for CPU, for example, 2.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   Note

   You can use the --release-image flag to set up the hosted cluster with a specific OpenShift Container Platform release.

   A default node pool is created for the cluster with two virtual machine worker replicas according to the --node-pool-replicas flag.

2. After a few moments, verify that the hosted control plane pods are running by entering the following command:

   $ oc -n clusters-<hosted-cluster-name> get pods

   Example output

   NAME                                                  READY   STATUS    RESTARTS   AGE
   capi-provider-5cc7b74f47-n5gkr                        1/1     Running   0          3m
   catalog-operator-5f799567b7-fd6jw                     2/2     Running   0          69s
   certified-operators-catalog-784b9899f9-mrp6p          1/1     Running   0          66s
   cluster-api-6bbc867966-l4dwl                          1/1     Running   0          66s
   .
   .
   .
   redhat-operators-catalog-9d5fd4d44-z8qqk              1/1     Running   0          66s

   A hosted cluster that has worker nodes that are backed by KubeVirt virtual machines typically takes 10-15 minutes to be fully provisioned.

3. To check the status of the hosted cluster, see the corresponding HostedCluster resource by entering the following command:

   $ oc get --namespace clusters hostedclusters

   See the following example output, which illustrates a fully provisioned HostedCluster object:

   NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
   clusters    example   4.x.0     example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

   Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

Procedure

1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials.
2. In the console header, ensure that All Clusters is selected.
3. Click Infrastructure > Clusters.
4. Click Create cluster > Red Hat OpenShift Virtualization > Hosted.

5. On the Create cluster page, follow the prompts to enter details about the cluster and node pools.

   Note
   • If you want to use predefined values to automatically populate fields in the console, you can create a OpenShift Virtualization credential. For more information, see Creating a credential for an on-premises environment.
   • On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a OpenShift Virtualization credential, the pull secret is automatically populated.

6. Review your entries and click Create.

   The Hosted cluster view is displayed.

7. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, and click the cluster name.
8. Wait until the control plane components are ready. This process can take a few minutes.
9. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Procedure

1. Enter the following command:

   $ hcp create cluster kubevirt \
     --name <hosted_cluster_name> \ 1
     --node-pool-replicas <worker_count> \ 2
     --pull-secret <path_to_pull_secret> \ 3
     --memory <value_for_memory> \ 4
     --cores <value_for_cpu> \ 5
     --base-domain <basedomain> 6

   1

   Specify the name of your hosted cluster.

   2

   Specify the worker count, for example, 2.

   3

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   4

   Specify a value for memory, for example, 6Gi.

   5

   Specify a value for CPU, for example, 2.

   6

   Specify the base domain, for example, hypershift.lab.

   As a result, the hosted cluster has an ingress wildcard that is configured for the cluster name and the base domain, for example, .apps.example.hypershift.lab. The hosted cluster remains in Partial status because after you create a hosted cluster with unique base domain, you must configure the required DNS records and load balancer.

2. View the status of your hosted cluster by entering the following command:

   $ oc get --namespace clusters hostedclusters

   Example output

   NAME            VERSION   KUBECONFIG                       PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
   example                   example-admin-kubeconfig         Partial    True        False         The hosted control plane is available

3. Access the cluster by entering the following commands:

   $ hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig

   $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get co

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.x.0     False       False         False      30m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.example.hypershift.lab): Get "https://console-openshift-console.apps.example.hypershift.lab": dial tcp: lookup console-openshift-console.apps.example.hypershift.lab on 172.31.0.10:53: no such host
   ingress                                    4.x.0     True        False         True       28m     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

   Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

Procedure

1. A NodePort service that exposes the hosted cluster ingress already exists. You can export the node ports and create the load balancer service that targets those ports.

   1. Get the HTTP node port by entering the following command:

      $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get services -n openshift-ingress router-nodeport-default -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}'

      Note the HTTP node port value to use in the next step.

   2. Get the HTTPS node port by entering the following command:

      $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get services -n openshift-ingress router-nodeport-default -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}'

      Note the HTTPS node port value to use in the next step.

2. Create the load balancer service by entering the following command:

   oc apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: <hosted_cluster_name>
     name: <hosted_cluster_name>-apps
     namespace: clusters-<hosted_cluster_name>
   spec:
     ports:
     - name: https-443
       port: 443
       protocol: TCP
       targetPort: <https_node_port> 1
     - name: http-80
       port: 80
       protocol: TCP
       targetPort: <http-node-port> 2
     selector:
       kubevirt.io: virt-launcher
     type: LoadBalancer

   1

   Specify the HTTPS node port value that you noted in the previous step.

   2

   Specify the HTTP node port value that you noted in the previous step.

Procedure

1. Get the external IP address by entering the following command:

   $ oc -n clusters-<hosted_cluster_name> get service <hosted-cluster-name>-apps -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

   Example output

   192.168.20.30

2. Configure a wildcard DNS entry that references the external IP address. View the following example DNS entry:

   *.apps.<hosted_cluster_name\>.<base_domain\>.

   The DNS entry must be able to route inside and outside of the cluster.

   DNS resolutions example

   dig +short test.apps.example.hypershift.lab
   
   192.168.20.30

3. Check that hosted cluster status has moved from Partial to Completed by entering the following command:

   $ oc get --namespace clusters hostedclusters

   Example output

   NAME            VERSION   KUBECONFIG                       PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
   example         4.x.0     example-admin-kubeconfig         Completed   True        False         The hosted control plane is available

   Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

Procedure

1. To create a node pool, enter the following information. In this example, the node pool has more CPUs assigned to the VMs:

   export NODEPOOL_NAME=${CLUSTER_NAME}-extra-cpu
   export WORKER_COUNT="2"
   export MEM="6Gi"
   export CPU="4"
   export DISK="16"
   
   $ hcp create nodepool kubevirt \
     --cluster-name $CLUSTER_NAME \
     --name $NODEPOOL_NAME \
     --node-count $WORKER_COUNT \
     --memory $MEM \
     --cores $CPU \
     --root-volume-size $DISK

2. Check the status of the node pool by listing nodepool resources in the clusters namespace:

   $ oc get nodepools --namespace clusters

   Example output

   NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
   example                   example         5               5               False         False        4.x.0
   example-extra-cpu         example         2                               False         False                  True              True             Minimum availability requires 2 replicas, current 0 available

   Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

3. After some time, you can check the status of the node pool by entering the following command:

   $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

   Example output

   NAME                      STATUS   ROLES    AGE     VERSION
   example-9jvnf             Ready    worker   97s     v1.27.4+18eadca
   example-n6prw             Ready    worker   116m    v1.27.4+18eadca
   example-nc6g4             Ready    worker   117m    v1.27.4+18eadca
   example-thp29             Ready    worker   4m17s   v1.27.4+18eadca
   example-twxns             Ready    worker   88s     v1.27.4+18eadca
   example-extra-cpu-zh9l5   Ready    worker   2m6s    v1.27.4+18eadca
   example-extra-cpu-zr8mj   Ready    worker   102s    v1.27.4+18eadca

4. Verify that the node pool is in the status that you expect by entering this command:

   $ oc get nodepools --namespace clusters

   Example output

   NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
   example                   example         5               5               False         False        4.x.0
   example-extra-cpu         example         2               2               False         False        4.x.0

   Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

Procedure

1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials.
2. In the console header, select All Clusters.
3. Click Infrastructure Clusters.

4. Click Create cluster Host inventory Hosted control plane.

   The Create cluster page is displayed.

5. On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation.

Procedure

1. Create a YAML file to define Image Content Source Policies (ICSP). See the following example:

   - mirrors:
     - brew.registry.redhat.io
     source: registry.redhat.io
   - mirrors:
     - brew.registry.redhat.io
     source: registry.stage.redhat.io
   - mirrors:
     - brew.registry.redhat.io
     source: registry-proxy.engineering.redhat.com

2. Save the file as icsp.yaml. This file contains your mirror registries.

3. To create a hosted cluster by using your mirror registries, run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --image-content-sources icsp.yaml  \6
       --ssh-key  <path_to_ssh_key> \7
       --namespace <hosted_cluster_namespace> \8
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the icsp.yaml file that defines ICSP and your mirror registries.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

Procedure

1. Run the following command:

   virt-install \
      --name "<vm_name>" \ 1
      --autostart \
      --ram=16384 \
      --cpu host \
      --vcpus=4 \
      --location "<path_to_kernel_initrd_image>,kernel=kernel.img,initrd=initrd.img" \ 2
      --disk <qcow_image_path> \ 3
      --network network:macvtap-net,mac=<mac_address> \ 4
      --graphics none \
      --noautoconsole \
      --wait=-1
      --extra-args "rd.neednet=1 nameserver=<nameserver>   coreos.live.rootfs_url=http://<http_server>/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs=console=tty1 console=ttyS1,115200n8" 5

   1

   Specify the name of the virtual machine.

   2

   Specify the location of the kernel_initrd_image file.

   3

   Specify the disk image path.

   4

   Specify the Mac address.

   5

   Specify the server name of the agents.

2. For ISO boot, download ISO from the InfraEnv resource and boot the nodes by running the following command:

   virt-install \
     --name "<vm_name>" \ 1
     --autostart \
     --memory=16384 \
     --cpu host \
     --vcpus=4 \
     --network network:macvtap-net,mac=<mac_address> \ 2
     --cdrom "<path_to_image.iso>" \ 3
     --disk <qcow_image_path> \
     --graphics none \
     --noautoconsole \
     --os-variant <os_version> \ 4
     --wait=-1

   1

   Specify the name of the virtual machine.

   2

   Specify the Mac address.

   3

   Specify the location of the image.iso file.

   4

   Specify the operating system version that you are using.

Procedure

1. Create a boot parameter file for the agents:

   Example parameter file

   rd.neednet=1 cio_ignore=all,!condev \
   console=ttysclp0 \
   ignition.firstboot ignition.platform.id=metal
   coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \1
   coreos.inst.persistent-kargs=console=ttysclp0
   ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \2
   rd.znet=qeth,<network_adaptor_range>,layer2=1
   rd.<disk_type>=<adapter> \3
   zfcp.allow_lun_scan=0
   ai.ip_cfg_override=1 \4
   random.trust_cpu=on rd.luks.options=discard

   1

   For the coreos.live.rootfs_url artifact, specify the matching rootfs artifact for the kernel and initramfs that you are starting. Only HTTP and HTTPS protocols are supported.

   2

   For the ip parameter, manually assign the IP address, as described in Installing a cluster with z/VM on IBM Z and IBM LinuxONE.

   3

   For installations on DASD-type disks, use rd.dasd to specify the DASD where Red Hat Enterprise Linux CoreOS (RHCOS) is to be installed. For installations on FCP-type disks, use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed.

   4

   Specify this parameter when you use an Open Systems Adapter (OSA) or HiperSockets.

2. Generate the .ins and initrd.img.addrsize files.

   The .ins file includes installation data and is on the FTP server. You can access the file from the HMC system. The .ins file contains details such as mapping of the location of installation data on the disk or FTP server, the memory locations where the data is to be copied.

   Note

   In OpenShift Container Platform 4.16, the .ins file and initrd.img.addrsize are not automatically generated as part of boot-artifacts from the installation program. You must manually generate these files.

   1. Run the following commands to get the size of the kernel and initrd:

      KERNEL_IMG_PATH='./kernel.img'
      INITRD_IMG_PATH='./initrd.img'
      CMDLINE_PATH='./generic.prm'
      kernel_size=$(stat -c%s $KERNEL_IMG_PATH )
      initrd_size=$(stat -c%s $INITRD_IMG_PATH)

   2. Round the kernel size up to the next MiB boundary. This value is the starting address of initrd.img.

      offset=$(( (kernel_size + 1048575) / 1048576 * 1048576 ))

   3. Create the kernel binary patch file that contains the initrd address and size by running the following commands:

      INITRD_IMG_NAME=$(echo $INITRD_IMG_PATH | rev | cut -d '/' -f 1 | rev)
      KERNEL_OFFSET=0x00000000
      KERNEL_CMDLINE_OFFSET=0x00010480
      INITRD_ADDR_SIZE_OFFSET=0x00010408
      OFFSET_HEX=$(printf '0x%08x\n' $offset)

   4. Convert the address and size to binary format by running the following command:

      $ printf "$(printf '%016x\n' $initrd_size)" | xxd -r -p > temp_size.bin

   5. Merge the address and size binaries by running the following command:

      $ cat temp_address.bin temp_size.bin > "$INITRD_IMG_NAME.addrsize"

   6. Clean up temporary files by running the following command:

      $ rm -rf temp_address.bin temp_size.bin

   7. Create the .ins file. The file is based on the paths of the kernel.img, initrd.img, initrd.img.addrsize, and cmdline files and the memory locations where the data is to be copied.

      $KERNEL_IMG_PATH $KERNEL_OFFSET
      $INITRD_IMG_PATH $OFFSET_HEX
      $INITRD_IMG_NAME.addrsize $INITRD_ADDR_SIZE_OFFSET
      $CMDLINE_PATH $KERNEL_CMDLINE_OFFSET

3. Transfer the initrd, kernel, generic.ins, and initrd.img.addrsize parameter files to the file server. For more information about how to transfer the files with FTP and boot, see Installing in an LPAR.
4. Start the machine.
5. Repeat the procedure for all other machines in the cluster.

Procedure

1. Update the parameter file to add the rootfs_url, network_adaptor and disk_type values.

   Example parameter file

   rd.neednet=1 cio_ignore=all,!condev \
   console=ttysclp0  \
   ignition.firstboot ignition.platform.id=metal \
   coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \1
   coreos.inst.persistent-kargs=console=ttysclp0
   ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \2
   rd.znet=qeth,<network_adaptor_range>,layer2=1
   rd.<disk_type>=<adapter> \3
   zfcp.allow_lun_scan=0
   ai.ip_cfg_override=1 \4

   1

   For the coreos.live.rootfs_url artifact, specify the matching rootfs artifact for the kernel and initramfs that you are starting. Only HTTP and HTTPS protocols are supported.

   2

   For the ip parameter, manually assign the IP address, as described in Installing a cluster with z/VM on IBM Z and IBM LinuxONE.

   3

   For installations on DASD-type disks, use rd.dasd to specify the DASD where Red Hat Enterprise Linux CoreOS (RHCOS) is to be installed. For installations on FCP-type disks, use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed.

   4

   Specify this parameter when you use an Open Systems Adapter (OSA) or HiperSockets.

2. Move initrd, kernel images, and the parameter file to the guest VM by running the following commands:

   vmur pun -r -u -N kernel.img $INSTALLERKERNELLOCATION/<image name>

   vmur pun -r -u -N generic.parm $PARMFILELOCATION/paramfilename

   vmur pun -r -u -N initrd.img $INSTALLERINITRAMFSLOCATION/<image name>

3. Run the following command from the guest VM console:

   cp ipl c

4. To list the agents and their properties, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agents

   Example output

   NAME    CLUSTER APPROVED    ROLE    STAGE
   50c23cda-cedc-9bbd-bcf1-9b3a5c75804d    auto-assign
   5e498cd3-542c-e54f-0c58-ed43e28b568a    auto-assign

5. Run the following command to approve the agent.

   $ oc -n <hosted_control_plane_namespace> patch agent \
     50c23cda-cedc-9bbd-bcf1-9b3a5c75804d -p \
     '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-zvm-0.hostedn.example.com"}}' \1
     --type merge

   1

   Optionally, you can set the agent ID <installation_disk_id> and <hostname> in the specification.

6. Run the following command to verify that the agents are approved:

   $ oc -n <hosted_control_plane_namespace> get agents

   Example output

   NAME                                            CLUSTER     APPROVED   ROLE          STAGE
   50c23cda-cedc-9bbd-bcf1-9b3a5c75804d             true       auto-assign
   5e498cd3-542c-e54f-0c58-ed43e28b568a             true       auto-assign