Chapter 3. Configuring multi-architecture compute machines on an OpenShift cluster

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your Azure account:

   $ az login

   Copy to Clipboard Toggle word wrap

2. Create a storage account and upload the aarch64 virtual hard disk (VHD) to your storage account. The OpenShift Container Platform installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:

   $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS 

   1

   Copy to Clipboard Toggle word wrap

   1

   The westus object is an example region.

3. Create a storage container using the storage account you generated:

   $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}

   Copy to Clipboard Toggle word wrap

4. You must use the OpenShift Container Platform installation program JSON file to extract the URL and aarch64 VHD name:

   1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:

      $ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')

      Copy to Clipboard Toggle word wrap

   2. Extract the aarch64 VHD name and set it to BLOB_NAME as the file name by running the following command:

      $ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd

      Copy to Clipboard Toggle word wrap

5. Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container with the following commands:

   $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`

   Copy to Clipboard Toggle word wrap

   $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

   Copy to Clipboard Toggle word wrap

6. Copy the RHCOS VHD into the storage container:

   $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
    --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
    --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

   Copy to Clipboard Toggle word wrap

   You can check the status of the copying process with the following command:

   $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

   Copy to Clipboard Toggle word wrap

   Example output

   {
    "completionTime": null,
    "destinationSnapshot": null,
    "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
    "incrementalCopy": null,
    "progress": "17179869696/17179869696",
    "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
    "status": "success", 

   1

   
    "statusDescription": null
   }

   Copy to Clipboard Toggle word wrap

   1

   If the status parameter displays the success object, the copying process is complete.

7. Create an image gallery using the following command:

   $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

   Copy to Clipboard Toggle word wrap

   Use the image gallery to create an image definition. In the following example command, rhcos-arm64 is the name of the image definition.

   $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2

   Copy to Clipboard Toggle word wrap

8. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

   $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

   Copy to Clipboard Toggle word wrap

9. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version. In the following example, 1.0.0 is the image version.

   $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

   Copy to Clipboard Toggle word wrap

10. Your arm64 boot image is now generated. You can access the ID of your image with the following command:

    $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0

    Copy to Clipboard Toggle word wrap

    The following example image ID is used in the recourseID parameter of the compute machine set:

    Example resourceID

    /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0

    Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your Azure account by running the following command:

   $ az login

   Copy to Clipboard Toggle word wrap

2. Create a storage account and upload the x86_64 virtual hard disk (VHD) to your storage account by running the following command. The OpenShift Container Platform installation program creates a resource group. However, the boot image can also be uploaded to a custom named resource group:

   $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS 

   1

   Copy to Clipboard Toggle word wrap

   1

   The westus object is an example region.

3. Create a storage container using the storage account you generated by running the following command:

   $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}

   Copy to Clipboard Toggle word wrap

4. Use the OpenShift Container Platform installation program JSON file to extract the URL and x86_64 VHD name:

   1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:

      $ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.x86_64."rhel-coreos-extensions"."azure-disk".url')

      Copy to Clipboard Toggle word wrap

   2. Extract the x86_64 VHD name and set it to BLOB_NAME as the file name by running the following command:

      $ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.x86_64."rhel-coreos-extensions"."azure-disk".release')-azure.x86_64.vhd

      Copy to Clipboard Toggle word wrap

5. Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container by running the following commands:

   $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`

   Copy to Clipboard Toggle word wrap

   $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

   Copy to Clipboard Toggle word wrap

6. Copy the RHCOS VHD into the storage container by running the following command:

   $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
    --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
    --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

   Copy to Clipboard Toggle word wrap

   You can check the status of the copying process by running the following command:

   $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

   Copy to Clipboard Toggle word wrap

   Example output

   {
    "completionTime": null,
    "destinationSnapshot": null,
    "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
    "incrementalCopy": null,
    "progress": "17179869696/17179869696",
    "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
    "status": "success", 

   1

   
    "statusDescription": null
   }

   Copy to Clipboard Toggle word wrap

   1

   If the status parameter displays the success object, the copying process is complete.

7. Create an image gallery by running the following command:

   $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

   Copy to Clipboard Toggle word wrap

8. Use the image gallery to create an image definition by running the following command:

   $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-x86_64 --publisher RedHat --offer x86_64 --sku x86_64 --os-type linux --architecture x64 --hyper-v-generation V2

   Copy to Clipboard Toggle word wrap

   In this example command, rhcos-x86_64 is the name of the image definition.

9. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

   $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

   Copy to Clipboard Toggle word wrap

10. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version by running the following command:

    $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

    Copy to Clipboard Toggle word wrap

    In this example, 1.0.0 is the image version.

11. Optional: Access the ID of the generated x86_64 boot image by running the following command:

    $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-x86_64 -e 1.0.0

    Copy to Clipboard Toggle word wrap

    The following example image ID is used in the recourseID parameter of the compute machine set:

    Example resourceID

    /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-x86_64/versions/1.0.0

    Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift CLI (oc).

2. Create a YAML file, and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

   Example MachineSet object for an Azure 64-bit ARM or 64-bit x86 compute node

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id>
       machine.openshift.io/cluster-api-machine-role: worker
       machine.openshift.io/cluster-api-machine-type: worker
     name: <infrastructure_id>-machine-set-0
     namespace: openshift-machine-api
   spec:
     replicas: 2
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id>
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
     template:
       metadata:
         labels:
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: worker
           machine.openshift.io/cluster-api-machine-type: worker
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
       spec:
         lifecycleHooks: {}
         metadata: {}
         providerSpec:
           value:
             acceleratedNetworking: true
             apiVersion: machine.openshift.io/v1beta1
             credentialsSecret:
               name: azure-cloud-credentials
               namespace: openshift-machine-api
             image:
               offer: ""
               publisher: ""
               resourceID: /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0 

   1

   
               sku: ""
               version: ""
             kind: AzureMachineProviderSpec
             location: <region>
             managedIdentity: <infrastructure_id>-identity
             networkResourceGroup: <infrastructure_id>-rg
             osDisk:
               diskSettings: {}
               diskSizeGB: 128
               managedDisk:
                 storageAccountType: Premium_LRS
               osType: Linux
             publicIP: false
             publicLoadBalancer: <infrastructure_id>
             resourceGroup: <infrastructure_id>-rg
             subnet: <infrastructure_id>-worker-subnet
             userDataSecret:
               name: worker-user-data
             vmSize: Standard_D4ps_v5 

   2

   
             vnet: <infrastructure_id>-vnet
             zone: "<zone>"

   Copy to Clipboard Toggle word wrap

   1

   Set the resourceID parameter to either arm64 or amd64 boot image.

   2

   Set the vmSize parameter to the instance type used in your installation. Some example instance types are Standard_D4ps_v5 or D8ps.

3. Create the compute machine set by running the following command:

   $ oc create -f <file_name> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: arm64-machine-set-0.yaml, or amd64-machine-set-0.yaml.

Verification

1. Verify that the new machines are running by running the following command:

   $ oc get machineset -n openshift-machine-api

   Copy to Clipboard Toggle word wrap

   The output must include the machine set that you created.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-machine-set-0                   2        2      2          2  10m

   Copy to Clipboard Toggle word wrap

2. You can check if the nodes are ready and schedulable by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift CLI (oc).

2. Create a YAML file, and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

   Example MachineSet object for an AWS 64-bit ARM or x86 compute node

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id> 

   1

   
     name: <infrastructure_id>-aws-machine-set-0 

   2

   
     namespace: openshift-machine-api
   spec:
     replicas: 1
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id> 

   3

   
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>-<zone> 

   4

   
     template:
       metadata:
         labels:
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: <role> 

   5

   
           machine.openshift.io/cluster-api-machine-type: <role> 

   6

   
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>-<zone> 

   7

   
       spec:
         metadata:
           labels:
             node-role.kubernetes.io/<role>: ""
         providerSpec:
           value:
             ami:
               id: ami-02a574449d4f4d280 

   8

   
             apiVersion: awsproviderconfig.openshift.io/v1beta1
             blockDevices:
               - ebs:
                   iops: 0
                   volumeSize: 120
                   volumeType: gp2
             credentialsSecret:
               name: aws-cloud-credentials
             deviceIndex: 0
             iamInstanceProfile:
               id: <infrastructure_id>-worker-profile 

   9

   
             instanceType: m6g.xlarge 

   10

   
             kind: AWSMachineProviderConfig
             placement:
               availabilityZone: us-east-1a 

   11

   
               region: <region> 

   12

   
             securityGroups:
               - filters:
                   - name: tag:Name
                     values:
                       - <infrastructure_id>-node 

   13

   
             subnet:
               filters:
                 - name: tag:Name
                   values:
                     - <infrastructure_id>-subnet-private-<zone>
             tags:
               - name: kubernetes.io/cluster/<infrastructure_id> 

   14

   
                 value: owned
               - name: <custom_tag_name>
                 value: <custom_tag_value>
             userDataSecret:
               name: worker-user-data

   Copy to Clipboard Toggle word wrap

   1 2 3 9 13 14

   Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command:

   $ oc get -o jsonpath="{.status.infrastructureName}{'\n'}" infrastructure cluster

   Copy to Clipboard Toggle word wrap

   4 7

   Specify the infrastructure ID, role node label, and zone.

   5 6

   Specify the role node label to add.

   8

   Specify a Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI) for your AWS region for the nodes. The RHCOS AMI must be compatible with the machine architecture.

   $ oc get configmap/coreos-bootimages \
   	  -n openshift-machine-config-operator \
   	  -o jsonpath='{.data.stream}' | jq \
   	  -r '.architectures.<arch>.images.aws.regions."<region>".image'

   Copy to Clipboard Toggle word wrap

   10

   Specify a machine type that aligns with the CPU architecture of the chosen AMI. For more information, see "Tested instance types for AWS 64-bit ARM"

   11

   Specify the zone. For example, us-east-1a. Ensure that the zone you select has machines with the required architecture.

   12

   Specify the region. For example, us-east-1. Ensure that the zone you select has machines with the required architecture.

3. Create the compute machine set by running the following command:

   $ oc create -f <file_name> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: aws-arm64-machine-set-0.yaml, or aws-amd64-machine-set-0.yaml.

Verification

1. View the list of compute machine sets by running the following command:

   $ oc get machineset -n openshift-machine-api

   Copy to Clipboard Toggle word wrap

   The output must include the machine set that you created.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-aws-machine-set-0                   2        2      2          2  10m

   Copy to Clipboard Toggle word wrap

2. You can check if the nodes are ready and schedulable by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift CLI (oc).

2. Create a YAML file, and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

   Example MachineSet object for a GCP 64-bit ARM or 64-bit x86 compute node

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id> 

   1

   
     name: <infrastructure_id>-w-a
     namespace: openshift-machine-api
   spec:
     replicas: 1
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id>
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
     template:
       metadata:
         creationTimestamp: null
         labels:
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: <role> 

   2

   
           machine.openshift.io/cluster-api-machine-type: <role>
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
       spec:
         metadata:
           labels:
             node-role.kubernetes.io/<role>: ""
         providerSpec:
           value:
             apiVersion: gcpprovider.openshift.io/v1beta1
             canIPForward: false
             credentialsSecret:
               name: gcp-cloud-credentials
             deletionProtection: false
             disks:
             - autoDelete: true
               boot: true
               image: <path_to_image> 

   3

   
               labels: null
               sizeGb: 128
               type: pd-ssd
             gcpMetadata: 

   4

   
             - key: <custom_metadata_key>
               value: <custom_metadata_value>
             kind: GCPMachineProviderSpec
             machineType: n1-standard-4 

   5

   
             metadata:
               creationTimestamp: null
             networkInterfaces:
             - network: <infrastructure_id>-network
               subnetwork: <infrastructure_id>-worker-subnet
             projectID: <project_name> 

   6

   
             region: us-central1 

   7

   
             serviceAccounts:
             - email: <infrastructure_id>-w@<project_name>.iam.gserviceaccount.com
               scopes:
               - https://www.googleapis.com/auth/cloud-platform
             tags:
               - <infrastructure_id>-worker
             userDataSecret:
               name: worker-user-data
             zone: us-central1-a

   Copy to Clipboard Toggle word wrap

   1

   Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. You can obtain the infrastructure ID by running the following command:

   $ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

   Copy to Clipboard Toggle word wrap

   2

   Specify the role node label to add.

   3

   Specify the path to the image that is used in current compute machine sets. You need the project and image name for your path to image.

   To access the project and image name, run the following command:

   $ oc get configmap/coreos-bootimages \
     -n openshift-machine-config-operator \
     -o jsonpath='{.data.stream}' | jq \
     -r '.architectures.aarch64.images.gcp'

   Copy to Clipboard Toggle word wrap

   Example output

     "gcp": {
       "release": "415.92.202309142014-0",
       "project": "rhcos-cloud",
       "name": "rhcos-415-92-202309142014-0-gcp-aarch64"
     }

   Copy to Clipboard Toggle word wrap

   Use the project and name parameters from the output to create the path to image field in your machine set. The path to the image should follow the following format:

   $ projects/<project>/global/images/<image_name>

   Copy to Clipboard Toggle word wrap

   4

   Optional: Specify custom metadata in the form of a key:value pair. For example use cases, see the GCP documentation for setting custom metadata.

   5

   Specify a machine type that aligns with the CPU architecture of the chosen OS image. For more information, see "Tested instance types for GCP on 64-bit ARM infrastructures".

   6

   Specify the name of the GCP project that you use for your cluster.

   7

   Specify the region. For example, us-central1. Ensure that the zone you select has machines with the required architecture.

3. Create the compute machine set by running the following command:

   $ oc create -f <file_name> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: gcp-arm64-machine-set-0.yaml, or gcp-amd64-machine-set-0.yaml.

Verification

1. View the list of compute machine sets by running the following command:

   $ oc get machineset -n openshift-machine-api

   Copy to Clipboard Toggle word wrap

   The output must include the machine set that you created.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-gcp-machine-set-0                   2        2      2          2  10m

   Copy to Clipboard Toggle word wrap

2. You can check if the nodes are ready and schedulable by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

   Copy to Clipboard Toggle word wrap
2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files.

3. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

   Copy to Clipboard Toggle word wrap

4. You can access the ISO image for booting your new machine by running to following command:

   RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.<architecture>.artifacts.metal.formats.iso.disk.location')

   Copy to Clipboard Toggle word wrap

5. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection with a LOM interface.

6. Boot the RHCOS ISO image without specifying any options, or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   You can interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you must use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 

   1

   2

   Copy to Clipboard Toggle word wrap

   1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

   Copy to Clipboard Toggle word wrap

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Ensure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 

     1

     
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img 

     2

     Copy to Clipboard Toggle word wrap

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

   • For iPXE (x86_64 + aarch64):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 

     1

     2

     
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img 

     3

     
     boot

     Copy to Clipboard Toggle word wrap

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your HTTP server.

     Note

     This configuration does not enable serial console access on machines with a graphical console To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     Note

     To network boot the CoreOS kernel on aarch64 architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and GRUB as second stage) on aarch64:

     menuentry 'Install CoreOS' {
         linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 

     1

     2

     
         initrd rhcos-<version>-live-initramfs.<architecture>.img 

     3

     
     }

     Copy to Clipboard Toggle word wrap

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server. The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file on your HTTP Server.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your TFTP server.

2. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.33.4
   master-1  Ready     master  63m  v1.33.4
   master-2  Ready     master  64m  v1.33.4

   Copy to Clipboard Toggle word wrap

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   Copy to Clipboard Toggle word wrap

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Copy to Clipboard Toggle word wrap
     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   Copy to Clipboard Toggle word wrap

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

     Copy to Clipboard Toggle word wrap

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.33.4
   master-1  Ready     master  73m  v1.33.4
   master-2  Ready     master  74m  v1.33.4
   worker-0  Ready     worker  11m  v1.33.4
   worker-1  Ready     worker  11m  v1.33.4

   Copy to Clipboard Toggle word wrap

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

   Copy to Clipboard Toggle word wrap
2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

3. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<http_server>/worker.ign

   Copy to Clipboard Toggle word wrap

4. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   Copy to Clipboard Toggle word wrap

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   Copy to Clipboard Toggle word wrap

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

   Copy to Clipboard Toggle word wrap
5. Move the downloaded RHEL live kernel, initramfs, and rootfs files to an HTTP or HTTPS server that is accessible from the RHCOS guest you want to add.

6. Create a parameter file for the guest. The following parameters are specific for the virtual machine:

   • Optional: To specify a static IP address, add an ip= parameter with the following entries, with each separated by a colon:

     1. The IP address for the machine.
     2. An empty string.
     3. The gateway.
     4. The netmask.
     5. The machine host and domain name in the form hostname.domainname. If you omit this value, RHCOS obtains the hostname through a reverse DNS lookup.
     6. The network interface name. If you omit this value, RHCOS applies the IP configuration to all available interfaces.
     7. The value none.
   • For coreos.inst.ignition_url=, specify the URL to the worker.ign file. Only HTTP and HTTPS protocols are supported.
   • For coreos.live.rootfs_url=, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.

   • For installations on DASD-type disks, complete the following tasks:

     1. For coreos.inst.install_dev=, specify /dev/dasda.
     2. Use rd.dasd= to specify the DASD where RHCOS is to be installed.

     3. You can adjust further parameters if required.

        The following is an example parameter file, additional-worker-dasd.parm:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/dasda \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        rd.dasd=0.0.3490 \
        zfcp.allow_lun_scan=0

        Copy to Clipboard Toggle word wrap

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

   • For installations on FCP-type disks, complete the following tasks:

     1. Use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed. For multipathing, repeat this step for each additional path.

        Note

        When you install with multiple paths, you must enable multipathing directly after the installation, not at a later point in time, as this can cause problems.

     2. Set the install device as: coreos.inst.install_dev=/dev/sda.

        Note

        If additional LUNs are configured with NPIV, FCP requires zfcp.allow_lun_scan=0. If you must enable zfcp.allow_lun_scan=1 because you use a CSI driver, for example, you must configure your NPIV so that each node cannot access the boot partition of another node.

     3. You can adjust further parameters if required.

        Important

        Additional postinstallation steps are required to fully enable multipathing. For more information, see “Enabling multipathing with kernel arguments on RHCOS" in Machine configuration.

        The following is an example parameter file, additional-worker-fcp.parm for a worker node with multipathing:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/sda \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        zfcp.allow_lun_scan=0 \
        rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.1987,0x50050763071bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763071bc5e3,0x4008400B00000000

        Copy to Clipboard Toggle word wrap

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

7. Transfer the initramfs, kernel, parameter files, and RHCOS images to z/VM, for example, by using FTP. For details about how to transfer the files with FTP and boot from the virtual reader, see Booting the installation on IBM Z® to install RHEL in z/VM.

8. Punch the files to the virtual reader of the z/VM guest virtual machine.

   See PUNCH in IBM® Documentation.

   Tip

   You can use the CP PUNCH command or, if you use Linux, the vmur command to transfer files between two z/VM guest virtual machines.

9. Log in to CMS on the bootstrap machine.

10. IPL the bootstrap machine from the reader by running the following command:

    $ ipl c

    Copy to Clipboard Toggle word wrap

    See IPL in IBM® Documentation.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.33.4
   master-1  Ready     master  63m  v1.33.4
   master-2  Ready     master  64m  v1.33.4

   Copy to Clipboard Toggle word wrap

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   Copy to Clipboard Toggle word wrap

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Copy to Clipboard Toggle word wrap
     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   Copy to Clipboard Toggle word wrap

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

     Copy to Clipboard Toggle word wrap

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.33.4
   master-1  Ready     master  73m  v1.33.4
   master-2  Ready     master  74m  v1.33.4
   worker-0  Ready     worker  11m  v1.33.4
   worker-1  Ready     worker  11m  v1.33.4

   Copy to Clipboard Toggle word wrap

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

   Copy to Clipboard Toggle word wrap
2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

3. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<http_server>/worker.ign

   Copy to Clipboard Toggle word wrap

4. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   Copy to Clipboard Toggle word wrap

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   Copy to Clipboard Toggle word wrap

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

   Copy to Clipboard Toggle word wrap
5. Move the downloaded RHEL live kernel, initramfs, and rootfs files to an HTTP or HTTPS server that is accessible from the RHCOS guest you want to add.

6. Create a parameter file for the guest. The following parameters are specific for the virtual machine:

   • Optional: To specify a static IP address, add an ip= parameter with the following entries, with each separated by a colon:

     1. The IP address for the machine.
     2. An empty string.
     3. The gateway.
     4. The netmask.
     5. The machine host and domain name in the form hostname.domainname. If you omit this value, RHCOS obtains the hostname through a reverse DNS lookup.
     6. The network interface name. If you omit this value, RHCOS applies the IP configuration to all available interfaces.
     7. The value none.
   • For coreos.inst.ignition_url=, specify the URL to the worker.ign file. Only HTTP and HTTPS protocols are supported.
   • For coreos.live.rootfs_url=, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.

   • For installations on DASD-type disks, complete the following tasks:

     1. For coreos.inst.install_dev=, specify /dev/dasda.
     2. Use rd.dasd= to specify the DASD where RHCOS is to be installed.

     3. You can adjust further parameters if required.

        The following is an example parameter file, additional-worker-dasd.parm:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/dasda \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        rd.dasd=0.0.3490 \
        zfcp.allow_lun_scan=0

        Copy to Clipboard Toggle word wrap

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

   • For installations on FCP-type disks, complete the following tasks:

     1. Use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed. For multipathing, repeat this step for each additional path.

        Note

        When you install with multiple paths, you must enable multipathing directly after the installation, not at a later point in time, as this can cause problems.

     2. Set the install device as: coreos.inst.install_dev=/dev/sda.

        Note

        If additional LUNs are configured with NPIV, FCP requires zfcp.allow_lun_scan=0. If you must enable zfcp.allow_lun_scan=1 because you use a CSI driver, for example, you must configure your NPIV so that each node cannot access the boot partition of another node.

     3. You can adjust further parameters if required.

        Important

        Additional postinstallation steps are required to fully enable multipathing. For more information, see “Enabling multipathing with kernel arguments on RHCOS" in Machine configuration.

        The following is an example parameter file, additional-worker-fcp.parm for a worker node with multipathing:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/sda \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        zfcp.allow_lun_scan=0 \
        rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.1987,0x50050763071bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763071bc5e3,0x4008400B00000000

        Copy to Clipboard Toggle word wrap

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

7. Transfer the initramfs, kernel, parameter files, and RHCOS images to the LPAR, for example with FTP. For details about how to transfer the files with FTP and boot, see Booting the installation on IBM Z® to install RHEL in an LPAR.
8. Boot the machine

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.33.4
   master-1  Ready     master  63m  v1.33.4
   master-2  Ready     master  64m  v1.33.4

   Copy to Clipboard Toggle word wrap

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   Copy to Clipboard Toggle word wrap

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Copy to Clipboard Toggle word wrap
     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   Copy to Clipboard Toggle word wrap

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

     Copy to Clipboard Toggle word wrap

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.33.4
   master-1  Ready     master  73m  v1.33.4
   master-2  Ready     master  74m  v1.33.4
   worker-0  Ready     worker  11m  v1.33.4
   worker-1  Ready     worker  11m  v1.33.4

   Copy to Clipboard Toggle word wrap

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

   Copy to Clipboard Toggle word wrap
2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

3. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

   Copy to Clipboard Toggle word wrap

4. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

    $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   Copy to Clipboard Toggle word wrap

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   Copy to Clipboard Toggle word wrap

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

   Copy to Clipboard Toggle word wrap
5. Move the downloaded RHEL live kernel, initramfs, and rootfs files to an HTTP or HTTPS server before you launch virt-install.

6. Create the new KVM guest nodes using the RHEL kernel, initramfs, and Ignition files; the new disk image; and adjusted parm line arguments.

   $ virt-install \
      --connect qemu:///system \
      --name <vm_name> \
      --autostart \
      --os-variant rhel9.4 \ 

   1

   
      --cpu host \
      --vcpus <vcpus> \
      --memory <memory_mb> \
      --disk <vm_name>.qcow2,size=<image_size> \
      --network network=<virt_network_parm> \
      --location <media_location>,kernel=<rhcos_kernel>,initrd=<rhcos_initrd> \ 

   2

   
      --extra-args "rd.neednet=1" \
      --extra-args "coreos.inst.install_dev=/dev/vda" \
      --extra-args "coreos.inst.ignition_url=http://<http_server>/worker.ign " \ 

   3

   
      --extra-args "coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img" \ 

   4

   
      --extra-args "ip=<ip>::<gateway>:<netmask>:<hostname>::none" \ 

   5

   
      --extra-args "nameserver=<dns>" \
      --extra-args "console=ttysclp0" \
      --noautoconsole \
      --wait

   Copy to Clipboard Toggle word wrap

   1

   For os-variant, specify the RHEL version for the RHCOS compute machine. rhel9.4 is the recommended version. To query the supported RHEL version of your operating system, run the following command:

   $ osinfo-query os -f short-id

   Copy to Clipboard Toggle word wrap

   Note

   The os-variant is case sensitive.

   2

   For --location, specify the location of the kernel/initrd on the HTTP or HTTPS server.

   3

   Specify the location of the worker.ign config file. Only HTTP and HTTPS protocols are supported.

   4

   Specify the location of the rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported

   5

   Optional: For hostname, specify the fully qualified hostname of the client machine.

   Note

   If you are using HAProxy as a load balancer, update your HAProxy rules for ingress-router-443 and ingress-router-80 in the /etc/haproxy/haproxy.cfg configuration file.

7. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.33.4
   master-1  Ready     master  63m  v1.33.4
   master-2  Ready     master  64m  v1.33.4

   Copy to Clipboard Toggle word wrap

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   Copy to Clipboard Toggle word wrap

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Copy to Clipboard Toggle word wrap
     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   Copy to Clipboard Toggle word wrap

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 

     1

     Copy to Clipboard Toggle word wrap

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

     Copy to Clipboard Toggle word wrap

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.33.4
   master-1  Ready     master  73m  v1.33.4
   master-2  Ready     master  74m  v1.33.4
   worker-0  Ready     worker  11m  v1.33.4
   worker-1  Ready     worker  11m  v1.33.4

   Copy to Clipboard Toggle word wrap

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.