Chapter 4. Deploying hosted control planes

Procedure

1. Create an S3 bucket that has public access to host OIDC discovery documents for your clusters.

   1. Enter the following command:

      $ aws s3api create-bucket --bucket <bucket_name> \
        --create-bucket-configuration LocationConstraint=<region> \
        --region <region> 

      1

      where:

      <bucket_name>

      Specifies the name of the S3 bucket you are creating.

      <region>

      Specifies that you want to create the bucket in a region other than the us-east-1 region. Include this line and replace <region> with the region you want to use. To create a bucket in the us-east-1 region, omit this line.

   2. Enter the following command:

      $ aws s3api delete-public-access-block --bucket <bucket_name>

      Replace <bucket_name> with the name of the S3 bucket you are creating.

   3. Enter the following command:

      $ echo '{
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": "*",
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket_name>/*"
              }
          ]
      }' | envsubst > policy.json

      Replace <bucket_name> with the name of the S3 bucket you are creating.

   4. Enter the following command:

      $ aws s3api put-bucket-policy --bucket <bucket_name> \
        --policy file://policy.json

      Replace <bucket_name> with the name of the S3 bucket you are creating.

      Note

      If you are using a Mac computer, you must export the bucket name in order for the policy to work.

2. Create an OIDC S3 secret named hypershift-operator-oidc-provider-s3-credentials for the HyperShift Operator.
3. Save the secret in the local-cluster namespace.

4. See the following table to verify that the secret contains the following fields:

   Expand

   Table 4.1. Required fields for the AWS secret

   Field name	Description
   bucket	Contains an S3 bucket with public access to host OIDC discovery documents for your hosted clusters.
   credentials	A reference to a file that contains the credentials of the default profile that can access the bucket. By default, HyperShift only uses the default profile to operate the bucket.
   region	Specifies the region of the S3 bucket.

   Show more

5. To create an AWS secret, run the following command:

   $ oc create secret generic <secret_name> \
     --from-file=credentials=<path>/.aws/credentials \
     --from-literal=bucket=<s3_bucket> \
     --from-literal=region=<region> \
     -n local-cluster

   Note

   Disaster recovery backup for the secret is not automatically enabled. To add the label that enables the hypershift-operator-oidc-provider-s3-credentials secret to be backed up for disaster recovery, run the following command:

   $ oc label secret hypershift-operator-oidc-provider-s3-credentials \
     -n local-cluster cluster.open-cluster-management.io/backup=true

Procedure

1. Get the Amazon Resource Name (ARN) of your user by running the following command:

   $ aws sts get-caller-identity --query "Arn" --output text

   Example output

   arn:aws:iam::1234567890:user/<aws_username>

   Use this output as the value for the <arn> value in the next step.

2. Create a JSON file that contains the trust relationship configuration for your role. See the following example:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": "<arn>"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }

   Replace <arn> with the ARN of your user that you noted in the previous step.

3. Create the Identity and Access Management (IAM) role by running the following command:

   $ aws iam create-role \
     --role-name <name> \
     --assume-role-policy-document file://<file_name>.json \
     --query "Role.Arn"

   where:

   <name>

   Specifies the role name, for example, hcp-cli-role.

   <file_name>

   Specifies the name of the JSON file you created in the previous step.

   Example output

   arn:aws:iam::820196288204:role/myrole

4. Create a JSON file named policy.json that contains the following permission policies for your role:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "EC2",
               "Effect": "Allow",
               "Action": [
                   "ec2:CreateDhcpOptions",
                   "ec2:DeleteSubnet",
                   "ec2:ReplaceRouteTableAssociation",
                   "ec2:DescribeAddresses",
                   "ec2:DescribeInstances",
                   "ec2:DeleteVpcEndpoints",
                   "ec2:CreateNatGateway",
                   "ec2:CreateVpc",
                   "ec2:DescribeDhcpOptions",
                   "ec2:AttachInternetGateway",
                   "ec2:DeleteVpcEndpointServiceConfigurations",
                   "ec2:DeleteRouteTable",
                   "ec2:AssociateRouteTable",
                   "ec2:DescribeInternetGateways",
                   "ec2:DescribeAvailabilityZones",
                   "ec2:CreateRoute",
                   "ec2:CreateInternetGateway",
                   "ec2:RevokeSecurityGroupEgress",
                   "ec2:ModifyVpcAttribute",
                   "ec2:DeleteInternetGateway",
                   "ec2:DescribeVpcEndpointConnections",
                   "ec2:RejectVpcEndpointConnections",
                   "ec2:DescribeRouteTables",
                   "ec2:ReleaseAddress",
                   "ec2:AssociateDhcpOptions",
                   "ec2:TerminateInstances",
                   "ec2:CreateTags",
                   "ec2:DeleteRoute",
                   "ec2:CreateRouteTable",
                   "ec2:DetachInternetGateway",
                   "ec2:DescribeVpcEndpointServiceConfigurations",
                   "ec2:DescribeNatGateways",
                   "ec2:DisassociateRouteTable",
                   "ec2:AllocateAddress",
                   "ec2:DescribeSecurityGroups",
                   "ec2:RevokeSecurityGroupIngress",
                   "ec2:CreateVpcEndpoint",
                   "ec2:DescribeVpcs",
                   "ec2:DeleteSecurityGroup",
                   "ec2:DeleteDhcpOptions",
                   "ec2:DeleteNatGateway",
                   "ec2:DescribeVpcEndpoints",
                   "ec2:DeleteVpc",
                   "ec2:CreateSubnet",
                   "ec2:DescribeSubnets"
               ],
               "Resource": "*"
           },
           {
               "Sid": "ELB",
               "Effect": "Allow",
               "Action": [
                   "elasticloadbalancing:DeleteLoadBalancer",
                   "elasticloadbalancing:DescribeLoadBalancers",
                   "elasticloadbalancing:DescribeTargetGroups",
                   "elasticloadbalancing:DeleteTargetGroup"
               ],
               "Resource": "*"
           },
           {
               "Sid": "IAMPassRole",
               "Effect": "Allow",
               "Action": "iam:PassRole",
               "Resource": "arn:*:iam::*:role/*-worker-role",
               "Condition": {
                   "ForAnyValue:StringEqualsIfExists": {
                       "iam:PassedToService": "ec2.amazonaws.com"
                   }
               }
           },
           {
               "Sid": "IAM",
               "Effect": "Allow",
               "Action": [
                   "iam:CreateInstanceProfile",
                   "iam:DeleteInstanceProfile",
                   "iam:GetRole",
                   "iam:UpdateAssumeRolePolicy",
                   "iam:GetInstanceProfile",
                   "iam:TagRole",
                   "iam:RemoveRoleFromInstanceProfile",
                   "iam:CreateRole",
                   "iam:DeleteRole",
                   "iam:PutRolePolicy",
                   "iam:AddRoleToInstanceProfile",
                   "iam:CreateOpenIDConnectProvider",
                   "iam:ListOpenIDConnectProviders",
                   "iam:DeleteRolePolicy",
                   "iam:UpdateRole",
                   "iam:DeleteOpenIDConnectProvider",
                   "iam:GetRolePolicy"
               ],
               "Resource": "*"
           },
           {
               "Sid": "Route53",
               "Effect": "Allow",
               "Action": [
                   "route53:ListHostedZonesByVPC",
                   "route53:CreateHostedZone",
                   "route53:ListHostedZones",
                   "route53:ChangeResourceRecordSets",
                   "route53:ListResourceRecordSets",
                   "route53:DeleteHostedZone",
                   "route53:AssociateVPCWithHostedZone",
                   "route53:ListHostedZonesByName"
               ],
               "Resource": "*"
           },
           {
               "Sid": "S3",
               "Effect": "Allow",
               "Action": [
                   "s3:ListAllMyBuckets",
                   "s3:ListBucket",
                   "s3:DeleteObject",
                   "s3:DeleteBucket"
               ],
               "Resource": "*"
           }
       ]
   }

5. Attach the policy.json file that contains the permissions policies for your role by running the following command:

   $ aws iam put-role-policy \
     --role-name <role_name> \
     --policy-name <policy_name> \
     --policy-document file://policy.json

   where:

   <role_name>

   Specifies the name of your role.

   <policy_name>

   Specifies your policy name.

6. Retrieve STS credentials in a JSON file named sts-creds.json by running the following command:

   $ aws sts get-session-token --output json > sts-creds.json

   Example sts-creds.json file

   {
       "Credentials": {
           "AccessKeyId": "<access_key_id",
           "SecretAccessKey": "<secret_access_key>”,
           "SessionToken": "<session_token>",
           "Expiration": "<time_stamp>"
       }
   }

Procedure

1. Create an AWS credential secret for the HyperShift Operator and name it hypershift-operator-private-link-credentials. The secret must reside in the managed cluster namespace that is the namespace of the managed cluster being used as the management cluster. If you used local-cluster, create the secret in the local-cluster namespace.

2. See the following table to confirm that the secret contains the required fields:

   Expand

   Table 4.2. Required fields for the AWS secret

   Field name	Description	Optional or required
   region	Region for use with Private Link	Required
   aws-access-key-id	The credential access key id.	Required
   aws-secret-access-key	The credential access key secret.	Required

   Show more

3. To create an AWS secret, run the following command:

   $ oc create secret generic <secret_name> \
     --from-literal=aws-access-key-id=<aws_access_key_id> \
     --from-literal=aws-secret-access-key=<aws_secret_access_key> \
     --from-literal=region=<region> -n local-cluster

4. Disaster recovery backup for the secret is not automatically enabled. Run the following command to add the label that enables the hypershift-operator-private-link-credentials secret to be backed up for disaster recovery:

   $ oc label secret hypershift-operator-private-link-credentials \
     -n local-cluster \
     cluster.open-cluster-management.io/backup=""

Procedure

1. Create an Amazon Web Services (AWS) credential secret for the HyperShift Operator and name it hypershift-operator-external-dns-credentials in the local-cluster namespace.

2. Verify that the secret has the required fields. For your reference, the required fields are detailed in the following table.

   Expand

   Table 4.3. Required fields for the AWS secret

   Field name	Description	Optional or required
   provider	The DNS provider that manages the service-level DNS zone.	Required
   domain-filter	The service-level domain.	Required
   credentials	The credential file that supports all external DNS types.	Optional when you use AWS keys
   aws-access-key-id	The credential access key id.	Optional when you use the AWS DNS service
   aws-secret-access-key	The credential access key secret.	Optional when you use the AWS DNS service

   Show more

3. Create an AWS secret by running the following command:

   $ oc create secret generic <secret_name> \
     --from-literal=provider=aws \
     --from-literal=domain-filter=<domain_name> \
     --from-file=credentials=<path_to_aws_credentials_file> -n local-cluster

   Note

   Disaster recovery backup for the secret is not automatically enabled. To back up the secret for disaster recovery, add the hypershift-operator-external-dns-credentials by entering the following command:

   $ oc label secret hypershift-operator-external-dns-credentials \
     -n local-cluster \
     cluster.open-cluster-management.io/backup=""

Procedure

1. In the AWS Route 53 management console, click Create hosted zone.
2. On the Hosted zone configuration page, type a domain name, verify that Public hosted zone is selected as the type, and click Create hosted zone.
3. After the zone is created, on the Records tab, note the values in the Value/Route traffic to column.
4. In the main domain, create an NS record to redirect the DNS requests to the delegated zone. In the Value field, enter the values that you noted in the previous step.
5. Click Create records.

6. Verify that the DNS hosted zone is working by creating a test entry in the new subzone and testing it with a dig command, such as in the following example:

   $ dig +short test.user-dest-public.aws.kerberos.com

   Example output

   192.168.1.1

7. To create a hosted cluster that sets the hostname for the LoadBalancer and Route services, enter the following command:

   $ hcp create cluster aws --name=<hosted_cluster_name> \
     --endpoint-access=PublicAndPrivate \
     --external-dns-domain=<public_hosted_zone> ...

   Replace <public_hosted_zone> with the public hosted zone that you created.

   Example services block for the hosted cluster

     platform:
       aws:
         endpointAccess: PublicAndPrivate
   ...
     services:
     - service: APIServer
       servicePublishingStrategy:
         route:
           hostname: api-example.service-provider-domain.com
         type: Route
     - service: OAuthServer
       servicePublishingStrategy:
         route:
           hostname: oauth-example.service-provider-domain.com
         type: Route
     - service: Konnectivity
       servicePublishingStrategy:
         type: Route
     - service: Ignition
       servicePublishingStrategy:
         type: Route

   The Control Plane Operator creates the Services and Routes resources and annotates them with the external-dns.alpha.kubernetes.io/hostname annotation. For Services and Routes, the Control Plane Operator uses a value of the hostname parameter in the servicePublishingStrategy field for the service endpoints. To create the DNS records, you can use a mechanism, such as the external-dns deployment.

   You can configure service-level DNS indirection for public services only. You cannot set hostname for private services because they use the hypershift.local private zone.

   The following table shows when it is valid to set hostname for a service and endpoint combinations:

   Expand

   Table 4.4. Service and endpoint combinations to set hostname

   Service	Public	PublicAndPrivate	Private
   APIServer	Y	Y	N
   OAuthServer	Y	Y	N
   Konnectivity	Y	N	N
   Ignition	Y	N	N

   Show more

Procedure

1. On the hcp command-line interface (CLI), enter the following command to access your management cluster:

   $ export KUBECONFIG=<path_to_management_cluster_kubeconfig>

2. Verify that the External DNS Operator is running by entering the following command:

   $ oc get pod -n hypershift -lapp=external-dns

   Example output

   NAME                            READY   STATUS    RESTARTS   AGE
   external-dns-7c89788c69-rn8gp   1/1     Running   0          40s

3. To create a hosted cluster by using external DNS, enter the following command:

   $ hcp create cluster aws \
       --role-arn <arn_role> \
       --instance-type <instance_type> \
       --region <region> \
       --auto-repair \
       --generate-ssh \
       --name <hosted_cluster_name> \
       --namespace clusters \
       --base-domain <service_consumer_domain> \
       --node-pool-replicas <node_replica_count> \
       --pull-secret <path_to_your_pull_secret> \
       --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \
       --external-dns-domain=<service_provider_domain> \
       --endpoint-access=<endpoint_access_configuration> \
       --sts-creds <path_to_sts_credential_file>

   where:

   <arn_role>

   Specifies the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.

   <instance_types>

   Specifies the instance type, for example, m6i.xlarge.

   <region>

   Specifies the AWS region, for example, us-east-1.

   <hosted_cluster_name>

   Specifies your hosted cluster name, for example, my-external-aws.

   <service_consumer_domain>

   Specifies the public hosted zone that the service consumer owns, for example, service-consumer-domain.com.

   <node_replica_count>

   Specifies the node replica count, for example, 2.

   <path_to_your_pull_secret>

   Specifies the path to your pull secret file.

   <ocp_release_image>

   Specifies the supported OpenShift Container Platform version that you want to use, for example, 4.22.0-multi.

   <service_provider_domain>

   Specifies the public hosted zone that the service provider owns, for example, service-provider-domain.com.

   <endpoint_access_configuraton>

   Specifies the endpoint access configuration for the external DNS. Set as PublicAndPrivate. You can use external DNS with Public or PublicAndPrivate configurations only.

   <path_to_sts_credential_file>

   Specifies the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

Procedure

1. To create a hosted cluster on AWS, run the following command:

   $ hcp create cluster aws \
       --name <hosted_cluster_name> \
       --infra-id <infra_id> \
       --base-domain <basedomain> \
       --sts-creds <path_to_sts_credential_file> \
       --pull-secret <path_to_pull_secret> \
       --region <region> \
       --generate-ssh \
       --node-pool-replicas <node_pool_replica_count> \
       --namespace <hosted_cluster_namespace> \
       --role-arn <role_name> \
       --render-into <file_name>.yaml

   where:

   <hosted_cluster_name>

   Specifies the name of your hosted cluster.

   <infra_id>

   Specifies your infrastructure name. You must provide the same value for <hosted_cluster_name> and <infra_id>. Otherwise the cluster might not appear correctly in the multicluster engine for Kubernetes Operator console.

   <basedomain>

   Specifies your base domain, for example, example.com.

   <path_to_sts_credential_file>

   Specifies the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

   <path_to_pull_secret>

   Specifies the path to your pull secret, for example, /user/name/pullsecret.

   <region>

   Specifies the AWS region name, for example, us-east-1.

   <node_pool_replica_count>

   Specifies the node pool replica count, for example, 3.

   <hosted_cluster_namespace>

   Specifies that you want to create the HostedCluster and NodePool custom resource in a specific namespace. Otherwise, by default, all HostedCluster and NodePool custom resources are created in the clusters namespace.

   <role_name>

   Specifies the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.

   <file_name>

   Specifies whether the EC2 instance runs on shared or single tenant hardware. The --render-into flag renders Kubernetes resources into the YAML file that you specify in this field. Continue to the next step to edit the YAML file.

2. If you included the --render-into flag in the previous command, edit the specified YAML file. Edit the NodePool specification in the YAML file to indicate whether the EC2 instance should run on shared or single-tenant hardware, similar to the following example:

   Example YAML file

   apiVersion: hypershift.openshift.io/v1beta1
   kind: NodePool
   metadata:
     name: <nodepool_name>
   spec:
     platform:
       aws:
         placement:
           tenancy: "default"

   where:

   <nodepool_name>

   Specifies the name of the NodePool resource.

   spec.platform.aws.placement.tenancy

   Specifies a valid value for tenancy: "default", "dedicated", or "host". Use "default" when node pool instances run on shared hardware. Use "dedicated" when each node pool instance runs on single-tenant hardware. Use "host" when node pool instances run on your pre-allocated dedicated hosts.

3. If you use external load balancers, configure the ingress endpoint by editing the HostedCluster resource as shown in the following example. If you do not configure the endpoint, the default behavior is to randomize the node port that the service exposes the ingress on. To configure how the ingress controller publishes the default ingress route, set the endpointPublishingStrategy parameter and its underlying functions:

   #...
   spec:
     operatorConfiguration:
       ingressOperator:
         endpointPublishingStrategy:
           type: LoadBalancerService
           loadBalancer:
             scope: Internal
   #...

   The spec.operatorConfiguration.ingressOperator.endPointPublishingStrategy.type parameter specifies the endpoint for the load balancer. For AWS, use the LoadBalancerService type.

Verification

1. Verify the status of your hosted cluster to check that the value of AVAILABLE is True. Run the following command:

   $ oc get hostedclusters -n <hosted_cluster_namespace>

2. Get a list of your node pools by running the following command:

   $ oc get nodepools --namespace <hosted_cluster_namespace>

Procedure

1. Find the private IPs of nodes by entering the following command:

   $ aws ec2 describe-instances \
     --filter="Name=tag:kubernetes.io/cluster/<infra_id>,Values=owned" \
     | jq '.Reservations[] | .Instances[] | select(.PublicDnsName=="") \
     | .PrivateIpAddress'

2. Create a kubeconfig file for the hosted cluster that you can copy to a node by entering the following command:

   $ hcp create kubeconfig > <hosted_cluster_kubeconfig>

3. To SSH into one of the nodes through the bastion, enter the following command:

   $ ssh -o ProxyCommand="ssh ec2-user@<bastion_ip> \
     -W %h:%p" core@<node_ip>

4. From the SSH shell, copy the kubeconfig file contents to a file on the node by entering the following command:

   $ mv <path_to_kubeconfig_file> <new_file_name>

5. Export the kubeconfig file by entering the following command:

   $ export KUBECONFIG=<path_to_kubeconfig_file>

6. Observe the hosted cluster status by entering the following command:

   $ oc get clusteroperators clusterversion

Procedure

1. Set your environment variables as shown in the following example:

   PERSISTENT_RG_NAME="os4-common"
   LOCATION="eastus"
   AZURE_CREDS="/path/to/azure-creds.json"
   SUBSCRIPTION_ID="my-subscription-id"

2. Create a persistent resource group by entering the following command:

   $ az group create --name $PERSISTENT_RG_NAME --location $LOCATION

3. Configure an OIDC issuer URL by using the Cloud Credential Operator tool to complete the following steps:

   1. Set the OIDC issuer variables as shown in the following example:

      OIDC_STORAGE_ACCOUNT_NAME="yourstorageaccount"
      TENANT_ID="your-tenant-id"

   2. Create an RSA key pair and save the private and public key by entering the following command:

      $ ccoctl azure create-key-pair

   3. Set variables for the token issuer key paths as shown in the following example:

      SA_TOKEN_ISSUER_PRIVATE_KEY_PATH="/path/to/serviceaccount-signer.private"
      SA_TOKEN_ISSUER_PUBLIC_KEY_PATH="/path/to/serviceaccount-signer.public"

   4. Create an OIDC issuer by entering the following command:

      $ ccoctl azure create-oidc-issuer \
          --oidc-resource-group-name ${PERSISTENT_RG_NAME} \
          --tenant-id ${TENANT_ID} \
          --region ${LOCATION} \
          --name ${OIDC_STORAGE_ACCOUNT_NAME} \
          --subscription-id ${SUBSCRIPTION_ID} \
          --public-key-file ${SA_TOKEN_ISSUER_PUBLIC_KEY_PATH}

   5. Set the OIDC issuer URL as shown in the following example:

      OIDC_ISSUER_URL="https://${OIDC_STORAGE_ACCOUNT_NAME}.blob.core.windows.net/${OIDC_STORAGE_ACCOUNT_NAME}"

Procedure

1. Set environment variables as shown in the following example:

   CLUSTER_NAME="my-self-managed-cluster"
   INFRA_ID="${CLUSTER_NAME}-$(openssl rand -hex 4)"

2. On the hosted control planes command-line interface, hcp, enter the following command:

   $ hcp create iam azure \
       --name <my_cluster_name> \
       --infra-id <infra_id> \
       --azure-creds <azure_credentials_file> \
       --resource-group-name <resource_group> \
       --oidc-issuer-url <oidc_issuer_url> \
       --output-file <workload_identities_file> \
       --location <my_region> \
       --cloud <my_cloud_environment>

   where:

   <my_cluster_name>

   Specifies the name of the cluster you intend to create.

   <infra_id>

   Specifies the unique identifier for naming Azure resources. Typically, this identifier is the cluster name with a suffix.

   <azure_credentials_file>

   Specifies the Azure credentials file with permission to create managed identities and federated credentials.

   <resource_group>

   Specifies the name of the resource group where you intend to create identities.

   <oidc_issuer_url>

   Specifies the URL of the OIDC identity provider for Workload Identity federation.

   <workload_identities_file>

   Specifies the output file path, such as my-cluster-name-iam-output.json.

   You can also add these optional flags to the hcp create iam azure command:

   <my_region>

   Specifies the Azure region for the managed identities. The default value is eastus.

   <my_cloud_environment>

   Specifies the Azure cloud environment. The default value is AzurePublicCloud.

Procedure

1. Identify the Azure Virtual Network (VNet) of the management cluster.

   1. Obtain the infrastructure resource group of the management cluster as shown in the following example:

      $ MGMT_INFRA_RG=$(oc get infrastructure cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')

   2. Find the VNet in the infrastructure resource group as shown in the following example:

      $ MGMT_VNET_NAME=$(az network vnet list --resource-group "${MGMT_INFRA_RG}" --query "[0].name" -o tsv)

   3. Set the environment variable for the VNet by entering the following command:

      $ MGMT_VNET_RG="${MGMT_INFRA_RG}"

2. Create the NAT subnet.

   1. Check the existing address space and subnets to ensure that you do not choose an overlapping classless inter-domain routing (CIDR) range. Enter the following command:

      $ az network vnet show \
        --resource-group "${MGMT_VNET_RG}" \
        --name "${MGMT_VNET_NAME}" \
        --query '{addressSpace: addressSpace.addressPrefixes, subnets: subnets[].{name: name, prefix: addressPrefix}}' \
        -o json

   2. Create the subnet as shown in the following example:

      $ az network vnet subnet create \
        --resource-group "${MGMT_VNET_RG}" \
        --vnet-name "${MGMT_VNET_NAME}" \
        --name "${NAT_SUBNET_NAME}" \
        --address-prefixes 10.1.64.0/24 \
        --disable-private-link-service-network-policies true

      • The NAT subnet must be in the VNet of the management cluster because Private Link is created alongside the internal load balancer of the management cluster.
      • The 10.1.64.0/24 address prefix is an example only. Replace it with a CIDR range that does not overlap with any other subnet in the VNet of the management cluster. If the VNet uses 10.0.0.0/16, the NAT subnet must fall within that range, or you must expand the address space of the VNet.
      • The --disable-private-link-service-network-policies flag is required and must be set to true. Otherwise, Azure rejects the creation of Private Link on the subnet.

3. Obtain the NAT subnet resource ID for later use as shown in the following example:

   $ NAT_SUBNET_ID=$(az network vnet subnet show \
     --resource-group "${MGMT_VNET_RG}" \
     --vnet-name "${MGMT_VNET_NAME}" \
     --name "${NAT_SUBNET_NAME}" \
     --query id -o tsv)

Procedure

1. Obtain credentials so that the Operator can manage Private Link resources.

   1. Obtain the credentials file for Private Link management as shown in the following example:

      $ AZURE_PRIVATE_CREDS="<path_to_azure_private_credentials_json>"

   2. Obtain the infrastructure resource group of the management cluster as shown in the following example:

      $ MGMT_INFRA_RG=$(oc get infrastructure cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')

2. Set the external DNS configuration variables as shown in the following examples:

   $ SERVICE_PRINCIPAL_FILEPATH="<path_to_azure_mgmt_json_file>"

   $ DNS_ZONE_NAME="<my_subdomain_my_parent_dns_zone_com>"

3. Install the HyperShift Operator with private platform support as shown in the following example:

   $ hcp install \
     --pull-secret ${PULL_SECRET} \
     --private-platform Azure \
     --azure-private-creds ${AZURE_PRIVATE_CREDS} \
     --azure-pls-resource-group ${MGMT_INFRA_RG} \
     --external-dns-provider=azure \
     --external-dns-credentials ${SERVICE_PRINCIPAL_FILEPATH} \
     --external-dns-domain-filter ${DNS_ZONE_NAME}

   • --private-platform Azure specifies that Azure Private Link management is to be enabled in the Operator.

   • --azure-private-creds specifies the path to the Azure credentials file that is used for Private Link operations.

     Note

     Besides using the --azure-private-creds flag, you can use one of the following authentication methods. Be sure to use only one authentication method.

     • --azure-private-secret specifies an existing Kubernetes secret that has Azure credentials. This flag has a companion flag, --azure-private-secret-key. Its default value is credentials, but you can customize it for secrets that have non-standard key names.
     • --azure-pls-managed-identity-client-id specifies the client ID of a managed identity for Private Link operations through Workload Identity federation. If you specify this flag, you must also include the --azure-pls-subscription-id flag, which specifies the Azure subscription ID for Private Link operations.
   • --azure-pls-resource-group specifies the resource group where the Private Link resources are to be created. This resource group is the same as the resource group of the infrastructure for the management cluster.
   • --external-dns-credentials specifies the path to a file that contains DNS credentials. If preferred, you can use the --external-dns-secret flag instead to specify a Kubernetes secret that has DNS credentials.

Procedure

1. Set your environment variables:

   1. Set your prefix by entering the following command:

      $ PREFIX="<my_prefix>"

   2. Set the cluster name by entering the following command:

      $ CLUSTER_NAME="${PREFIX}-hc"

   3. Set the resource group name by entering the following command:

      $ RESOURCE_GROUP_NAME="${CLUSTER_NAME}-${PREFIX}"

   4. Set the location by entering the following command:

      $ LOCATION="<my_region>"

   5. Set the variable for the path to the Azure credentials file by entering the following command:

      $ AZURE_CREDS="<path_to_azure_credentials_json>"

   6. Set the variable for the OIDC issuer URL by entering the following command:

      $ OIDC_ISSUER_URL="<my_oidc_url_com>"

   7. Set the path to the Workload Identities file by entering the following command:

      $ WORKLOAD_IDENTITIES_FILE="<path_to_workload_identities_file_json>"

2. Create Workload Identities by entering the following command:

   $ hcp create iam azure \
     --name "${CLUSTER_NAME}" \
     --infra-id "${PREFIX}" \
     --azure-creds "${AZURE_CREDS}" \
     --location "${LOCATION}" \
     --resource-group-name "${RESOURCE_GROUP_NAME}" \
     --oidc-issuer-url "${OIDC_ISSUER_URL}" \
     --output-file "${WORKLOAD_IDENTITIES_FILE}"

   The command creates 8 Workload Identities. For Private and PublicAndPrivate clusters, the Control Plane Operator identity is used to create and manage private endpoints, private DNS zones, Azure Virtual Network (VNet) links, and DNS A records. The Control Plane Identity is assigned the Contributor role by default. To use a more restrictive role, use the --assign-custom-hcp-roles flag.

Procedure

1. Set your environment variables:

   1. Set the variable for the DNS zone resource group by entering the following command:

      $ DNS_ZONE_RG_NAME="os4-common"

   2. Set the variable for the base domain of your DNS zone by entering the following command:

      $ PARENT_DNS_ZONE="<your_base_domain_com>"

   3. Set the variable that points to the infrastructure output file by entering the following command:

      $ INFRA_OUTPUT_FILE="${PREFIX}-infra-output.json"

2. Create infrastructure by entering the following command:

   $ hcp create infra azure \
     --azure-creds "${AZURE_CREDS}" \
     --infra-id "${PREFIX}" \
     --name "${CLUSTER_NAME}" \
     --location "${LOCATION}" \
     --base-domain "${PARENT_DNS_ZONE}" \
     --dns-zone-rg-name "${DNS_ZONE_RG_NAME}" \
     --workload-identities-file "${WORKLOAD_IDENTITIES_FILE}" \
     --assign-identity-roles \
     --output-file "${INFRA_OUTPUT_FILE}"

3. Read the infrastructure output to get the resource IDs that you created, as shown in the following example:

   1. Read the resource group name by entering the following command:

      $ MANAGED_RG_NAME=$(yq -r -p yaml '.resourceGroupName' "${INFRA_OUTPUT_FILE}")

   2. Read the VNet ID by entering the following command:

      $ VNET_ID=$(yq -r -p yaml '.vnetID' "${INFRA_OUTPUT_FILE}")

   3. Read the subnet ID by entering the following command:

      $ SUBNET_ID=$(yq -r -p yaml '.subnetID' "${INFRA_OUTPUT_FILE}")

   4. Read the network security group ID by entering the following command:

      $ NSG_ID=$(yq -r -p yaml '.securityGroupID' "${INFRA_OUTPUT_FILE}")

      Note the resource IDs because you need them to create the private hosted cluster.

Procedure

1. Create the private hosted cluster by entering the following command:

   $ hcp create cluster azure \
     --name "$CLUSTER_NAME" \
     --namespace "clusters" \
     --azure-creds ${AZURE_CREDS} \
     --location ${LOCATION} \
     --node-pool-replicas 2 \
     --base-domain ${PARENT_DNS_ZONE} \
     --pull-secret ${PULL_SECRET} \
     --generate-ssh \
     --release-image ${RELEASE_IMAGE} \
     --resource-group-name "${MANAGED_RG_NAME}" \
     --vnet-id "${VNET_ID}" \
     --subnet-id "${SUBNET_ID}" \
     --network-security-group-id "${NSG_ID}" \
     --sa-token-issuer-private-key-path "${SA_TOKEN_ISSUER_PRIVATE_KEY_PATH}" \
     --oidc-issuer-url "${OIDC_ISSUER_URL}" \
     --dns-zone-rg-name ${DNS_ZONE_RG_NAME} \
     --assign-service-principal-roles \
     --workload-identities-file ${WORKLOAD_IDENTITIES_FILE} \
     --external-dns-domain ${DNS_ZONE_NAME} \
     --endpoint-access Private \
     --endpoint-access-private-nat-subnet-id "${NAT_SUBNET_ID}"

   • When you choose a value for the --external-dns-domain flag, ensure that the value does not match {cluster-name}.{base-domain}. If you use {cluster-name}.{base-domain} for the value, the Control Plane Operator creates a private DNS zone that can shadow the *.apps domain, which causes the console and ingress to become unreachable.

   • --endpoint-access accepts 3 values:

     • Public, which is the default value. When you specify Public, the API server is accessible through public endpoint only.
     • PublicAndPrivate, where the API server is accessible through both public and private endpoints.

     • Private, where the API server is accessible only through Private Link.

       After you create a cluster, you cannot change it between Public endpoint and non-public (PublicAndPrivate or Private) endpoint access.

       If you need to allow Private endpoint connections from Azure subscriptions other than the hosted cluster’s subscription, use the following flag: --endpoint-access-private-additional-allowed-subscriptions "sub-id-1, sub-id-2".

Verification

1. Check the Private Link resources by entering the following command:

   $ oc get azureprivatelinkservices -n clusters-${CLUSTER_NAME}

2. Check the status and connections by entering the following command:

   $ oc get azureprivatelinkservices -n clusters-${CLUSTER_NAME} -o yaml

   Expand

   Table 4.9. Setup progress conditions

   Condition	Description
   AzureInternalLoadBalancerAvailable	The internal load balancer has a front-end IP address.
   AzurePLSCreated	Private Link is created in the management cluster.
   AzurePrivateEndpointAvailable	Private endpoint is created in the hosted cluster VNet.
   AzurePrivateDNSAvailable	Private DNS zones and A records are created.
   AzurePrivateLinkServiceAvailable	All components are ready and private connectivity is available.

   Show more

3. Check the overall cluster status by entering the following command:

   $ oc get hostedcluster ${CLUSTER_NAME} -n clusters

4. Wait for the status by entering the following command:

   $ oc wait --for=condition=Available hostedcluster/${CLUSTER_NAME} -n clusters --timeout=30m

Procedure

1. Enter the following command to check the Private Link conditions:

   $ oc get azureprivatelinkservices \
     -n clusters-${CLUSTER_NAME} \
     -o jsonpath='{.items[0].status.conditions}' | jq .

2. Review the output and compare it to the following condition table:

   Expand

   Table 4.10. Private cluster stuck conditions

   Condition	Possible cause
   AzureInternalLoadBalancerAvailable = False	The private-router service has not received an internal load balancer IP address yet. Check the service status and Azure networking.
   AzurePLSCreated = False	Private Link creation failed. Check the NAT subnet policies, credentials, and the HyperShift Operator logs.
   AzurePrivateEndpointAvailable = False	Private endpoint creation failed or the connection was not approved. Check the Private Link auto-approval list and the Control Plane Operator logs.
   AzurePrivateDNSAvailable = False	The DNS zone or record creation failed. In the Azure subscription that stores the infrastructure resources for the hosted cluster, check the Control Plane Operator identity permissions.

   Show more

1. The Control Plane Operator removes the private endpoint, private DNS zones, VNet links, and A records.
2. The hcp destroy command removes role-based access control (RBAC) role assignments.
3. The HyperShift Operator removes Private Link.

Procedure

1. Create a namespace to store your hardware inventory by entering the following command:

   $ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig create \
     namespace <namespace_example>

   where:

   <directory_example>

   Is the name of the directory where the kubeconfig file for the management cluster is saved.

   <namespace_example>

   Is the name of the namespace that you are creating; for example, hardware-inventory.

   Example output

   namespace/hardware-inventory created

2. Copy the pull secret of the management cluster by entering the following command:

   $ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig \
     -n openshift-config get secret pull-secret -o yaml \
     | grep -vE "uid|resourceVersion|creationTimestamp|namespace" \
     | sed "s/openshift-config/<namespace_example>/g" \
     | oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig \
     -n <namespace> apply -f -

   where:

   <directory_example>

   Is the name of the directory where the kubeconfig file for the management cluster is saved.

   <namespace_example>

   Is the name of the namespace that you are creating; for example, hardware-inventory.

   Example output

   secret/pull-secret created

3. Create the InfraEnv resource by adding the following content to a YAML file:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: hosted
     namespace: <namespace_example>
   spec:
     additionalNTPSources:
     - <ip_address>
     pullSecretRef:
       name: pull-secret
     sshAuthorizedKey: <ssh_public_key>
   # ...

4. Apply the changes to the YAML file by entering the following command:

   $ oc apply -f <infraenv_config>.yaml

   Replace <infraenv_config> with the name of your file.

5. Verify that the InfraEnv resource was created by entering the following command:

   $ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig \
     -n <namespace_example> get infraenv hosted

6. Add bare-metal hosts by following one of two methods:

   • If you do not use the Metal3 Operator, obtain the discovery ISO from the InfraEnv resource and boot the hosts manually by completing the following steps:

     1. Download the live ISO by entering the following commands:

        $ oc get infraenv -A

        $ oc get infraenv <namespace_example> -o jsonpath='{.status.isoDownloadURL}' -n <namespace_example> <iso_url>

     2. Boot the ISO. The node communicates with the Assisted Service and registers as an agent in the same namespace as the InfraEnv resource.

     3. For each agent, set the installation disk ID and hostname, and approve it to indicate that the agent is ready for use.

        1. Enter the following command to get the agents for your hosted control plane namespace:

           $ oc -n <hosted_control_plane_namespace> get agents

           In this example, two agents are listed.

           Example output

           NAME                                   CLUSTER   APPROVED   ROLE          STAGE
           example-agent-1                        auto-assign
           example-agent-2                        auto-assign

        2. Enter the following command to set the installation disk ID and hostname for the first agent:

           $ oc -n <hosted_control_plane_namespace> \
             patch agent example-agent-1 \
             -p '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-0.example.krnl.es"}}' \
             --type merge

        3. Enter the following command to set the installation disk ID and hostname for the second agent:

           $ oc -n <hosted_control_plane_namespace> \
             patch agent example-agent-2 -p \
             '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-1.example.krnl.es"}}' \
             --type merge

   • If you use the Metal3 Operator, you can automate the bare-metal host registration by creating the following objects:

     1. Create a YAML file and add the following content to it:

        apiVersion: v1
        kind: Secret
        metadata:
          name: hosted-worker0-bmc-secret
          namespace: <namespace_example>
        data:
          password: <password>
          username: <username>
        type: Opaque
        ---
        apiVersion: v1
        kind: Secret
        metadata:
          name: hosted-worker1-bmc-secret
          namespace: <namespace_example>
        data:
          password: <password>
          username: <username>
        type: Opaque
        ---
        apiVersion: v1
        kind: Secret
        metadata:
          name: hosted-worker2-bmc-secret
          namespace: <namespace_example>
        data:
          password: <password>
          username: <username>
        type: Opaque
        ---
        apiVersion: metal3.io/v1alpha1
        kind: BareMetalHost
        metadata:
          name: hosted-worker0
          namespace: <namespace_example>
          labels:
            infraenvs.agent-install.openshift.io: hosted
          annotations:
            inspect.metal3.io: disabled
            bmac.agent-install.openshift.io/hostname: hosted-worker0
        spec:
          automatedCleaningMode: disabled
          bmc:
            disableCertificateVerification: True
            address: <bmc_address>
            credentialsName: hosted-worker0-bmc-secret
          bootMACAddress: aa:aa:aa:aa:02:01
          online: true
        ---
        apiVersion: metal3.io/v1alpha1
        kind: BareMetalHost
        metadata:
          name: hosted-worker1
          namespace: <namespace_example>
          labels:
            infraenvs.agent-install.openshift.io: hosted
          annotations:
            inspect.metal3.io: disabled
            bmac.agent-install.openshift.io/hostname: hosted-worker1
        spec:
          automatedCleaningMode: disabled
          bmc:
            disableCertificateVerification: True
            address: <bmc_address>
            credentialsName: hosted-worker1-bmc-secret
          bootMACAddress: aa:aa:aa:aa:02:02
          online: true
        ---
        apiVersion: metal3.io/v1alpha1
        kind: BareMetalHost
        metadata:
          name: hosted-worker2
          namespace: <namespace_example>
          labels:
            infraenvs.agent-install.openshift.io: hosted
          annotations:
            inspect.metal3.io: disabled
            bmac.agent-install.openshift.io/hostname: hosted-worker2
        spec:
          automatedCleaningMode: disabled
          bmc:
            disableCertificateVerification: True
            address: <bmc_address>
            credentialsName: hosted-worker2-bmc-secret
          bootMACAddress: aa:aa:aa:aa:02:03
          online: true
        ---
        apiVersion: rbac.authorization.k8s.io/v1
        kind: Role
        metadata:
          name: capi-provider-role
          namespace: <namespace_example>
        rules:
        - apiGroups:
          - agent-install.openshift.io
          resources:
          - agents
          verbs:
          - '*'

        where:

        <namespace_example>

        Is the your namespace.

        <password>

        Is the password for your secret.

        <username>

        Is the user name for your secret.

        <bmc_address>

        Is the BMC address for the BareMetalHost object.

        Note

        When you apply this YAML file, the following objects are created:

        • Secrets with credentials for the Baseboard Management Controller (BMCs)
        • The BareMetalHost objects
        • A role for the HyperShift Operator to be able to manage the agents

        Notice how the InfraEnv resource is referenced in the BareMetalHost objects by using the infraenvs.agent-install.openshift.io: hosted custom label. This ensures that the nodes are booted with the ISO generated.

     2. Apply the changes to the YAML file by entering the following command:

        $ oc apply -f <bare_metal_host_config>.yaml

        Replace <bare_metal_host_config> with the name of your file.

7. Enter the following command, and then wait a few minutes for the BareMetalHost objects to move to the Provisioning state:

   $ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig -n <namespace_example> get bmh

   Example output

   NAME             STATE          CONSUMER   ONLINE   ERROR   AGE
   hosted-worker0   provisioning              true             106s
   hosted-worker1   provisioning              true             106s
   hosted-worker2   provisioning              true             106s

8. Enter the following command to verify that nodes are booting and showing up as agents. This process can take a few minutes, and you might need to enter the command more than once.

   $ oc --kubeconfig ~/<directory_example>/mgmt-kubeconfig -n <namespace_example> get agent

   Example output

   NAME                                   CLUSTER   APPROVED   ROLE          STAGE
   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0201             true       auto-assign
   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0202             true       auto-assign
   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0203             true       auto-assign

Procedure

1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials. For instructions to open the console, see "Accessing the web console".
2. In the console header, ensure that All Clusters is selected.
3. Click Infrastructure Host inventory Create infrastructure environment.
4. After you create the InfraEnv resource, add bare-metal hosts from within the InfraEnv view by clicking Add hosts and selecting from the available options.

Procedure

1. Create a namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>

   Replace <hosted_cluster_namespace> with an identifier for your hosted cluster namespace. The HyperShift Operator creates the namespace. During the hosted cluster creation process on bare-metal infrastructure, a generated Cluster API provider role requires that the namespace already exists.

2. Create the configuration file for your hosted cluster by entering the following command:

   $ hcp create cluster agent \
     --name=<hosted_cluster_name> \
     --pull-secret=<path_to_pull_secret> \
     --agent-namespace=<hosted_control_plane_namespace> \
     --base-domain=<base_domain> \
     --api-server-address=api.<hosted_cluster_name>.<base_domain>
     --etcd-storage-class=<etcd_storage_class> \
     --ssh-key=<path_to_ssh_key> \
     --namespace=<hosted_cluster_namespace> \
     --control-plane-availability-policy=HighlyAvailable \
     --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image>-multi \
     --node-pool-replicas=<node_pool_replica_count> \
     --render \
     --render-sensitive \
     --ssh-key <home_directory>/<path_to_ssh_key>/<ssh_key> > hosted-cluster-config.yaml

   where:

   <hosted_cluster_name>

   Specifies the name of your hosted cluster, such as example.

   <path_to_pull_secret>

   Specifies the path to your pull secret, such as /user/name/pullsecret.

   <hosted_control_plane_namespace>

   Specifies your hosted control plane namespace, such as clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   <base_domain>

   Specifies your base domain, such as krnl.es.

   --api-server-address

   Specifies the IP address that gets used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   <etcd_storage_class>

   Specifies the etcd storage class name, such as lvm-storageclass.

   <path_to_ssh_key>

   Specifies the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   <hosted_cluster_namespace>

   Specifies your hosted cluster namespace.

   control-plane-availability-policy

   Specifies the availability policy for the hosted control plane components. Supported options are SingleReplica and HighlyAvailable. The default value is HighlyAvailable.

   <ocp_release_image>

   Specifies the supported OpenShift Container Platform version that you want to use, such as 4.22.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see "Extracting the OpenShift Container Platform release image digest".

   <node_pool_replica_count>

   Specifies the node pool replica count, such as 3. You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, you do not create node pools.

   <home_directory>/<path_to_ssh_key>/<ssh_key>

   Specifies the path to the SSH key, such as user/.ssh/id_rsa.

3. Configure the service publishing strategy. By default, hosted clusters use the NodePort service publishing strategy because node ports are always available without additional infrastructure. However, you can configure the service publishing strategy to use a load balancer.

   • If you are using the default NodePort strategy, configure the DNS to point to the hosted cluster compute nodes, not the management cluster nodes. For more information, see "DNS configurations on bare metal".

   • For production environments, use the LoadBalancer strategy because this strategy provides certificate handling and automatic DNS resolution. The following example demonstrates changing the service publishing LoadBalancer strategy in your hosted cluster configuration file:

     apiVersion: hypershift.openshift.io/v1beta1
     kind: HostedCluster
     metadata:
     # ...
     spec:
       services:
       - service: APIServer
         servicePublishingStrategy:
           type: LoadBalancer
       - service: Ignition
         servicePublishingStrategy:
           type: Route
       - service: Konnectivity
         servicePublishingStrategy:
           type: Route
       - service: OAuthServer
         servicePublishingStrategy:
           type: Route
       - service: OIDC
         servicePublishingStrategy:
           type: Route
       sshKey:
         name: <ssh_key>
     # ...

     Specify LoadBalancer as the API Server type. For all other services, specify Route as the type.

4. If you use external load balancers, configure the ingress endpoint as shown in the following example. If you do not configure the endpoint, the default behavior is to randomize the node port that the service exposes the ingress on. To configure how the ingress controller publishes the default ingress route, set the endpointPublishingStrategy parameter and its underlying functions by editing the HostedCluster resource:

   apiVersion: hypershift.openshift.io/v1beta1
   kind: HostedCluster
   metadata:
   #...
   spec:
     operatorConfiguration:
       ingressOperator:
         endpointPublishingStrategy:
           hostNetwork:
             httpPort: 80
             httpsPort: 443
             protocol: TCP
             statsPort: 1936
           type: HostNetwork
   #...

   The spec.operatorConfiguration.ingressOperator.endPointPublishingStrategy.type parameter specifies the endpoint for the load balancer. For bare-metal installations, use the HostNetwork type.

5. Apply the changes to the hosted cluster configuration file by entering the following command:

   $ oc apply -f hosted_cluster_config.yaml

6. Check for the creation of the hosted cluster, node pools, and pods by entering the following commands:

   $ oc get hostedcluster \
     <hosted_cluster_namespace> -n \
     <hosted_cluster_namespace> -o \
     jsonpath='{.status.conditions[?(@.status=="False")]}' | jq .

   $ oc get nodepool \
     <hosted_cluster_namespace> -n \
     <hosted_cluster_namespace> -o \
     jsonpath='{.status.conditions[?(@.status=="False")]}' | jq .

   $ oc get pods -n <hosted_cluster_namespace>

7. Confirm that the hosted cluster is ready. The status of Available: True indicates the readiness of the cluster and the node pool status shows AllMachinesReady: True. These statuses indicate the healthiness of all cluster Operators.

8. Install MetalLB in the hosted cluster:

   1. Extract the kubeconfig file from the hosted cluster and set the environment variable for hosted cluster access by entering the following commands:

      $ oc get secret \
        <hosted_cluster_namespace>-admin-kubeconfig \
        -n <hosted_cluster_namespace> \
        -o jsonpath='{.data.kubeconfig}' \
        | base64 -d > \
        kubeconfig-<hosted_cluster_namespace>.yaml

      $ export KUBECONFIG="/path/to/kubeconfig-<hosted_cluster_namespace>.yaml"

   2. Install the MetalLB Operator by creating the install-metallb-operator.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: metallb-system
      ---
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: metallb-operator
        namespace: metallb-system
      ---
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: metallb-operator
        namespace: metallb-system
      spec:
        channel: "stable"
        name: metallb-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        installPlanApproval: Automatic
      # ...

   3. Apply the file by entering the following command:

      $ oc apply -f install-metallb-operator.yaml

   4. Configure the MetalLB IP address pool by creating the deploy-metallb-ipaddresspool.yaml file:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        name: metallb
        namespace: metallb-system
      spec:
        autoAssign: true
        addresses:
        - 10.11.176.71-10.11.176.75
      ---
      apiVersion: metallb.io/v1beta1
      kind: L2Advertisement
      metadata:
        name: l2advertisement
        namespace: metallb-system
      spec:
        ipAddressPools:
        - metallb
      # ...

   5. Apply the configuration by entering the following command:

      $ oc apply -f deploy-metallb-ipaddresspool.yaml

   6. Verify the installation of MetalLB by checking the Operator status, the IP address pool, and the L2Advertisement resource by entering the following commands:

      $ oc get pods -n metallb-system

      $ oc get ipaddresspool -n metallb-system

      $ oc get l2advertisement -n metallb-system

9. Configure the load balancer for ingress:

   1. Create the ingress-loadbalancer.yaml file:

      apiVersion: v1
      kind: Service
      metadata:
        annotations:
          metallb.universe.tf/address-pool: metallb
        name: metallb-ingress
        namespace: openshift-ingress
      spec:
        ports:
          - name: http
            protocol: TCP
            port: 80
            targetPort: 80
          - name: https
            protocol: TCP
            port: 443
            targetPort: 443
        selector:
          ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
        type: LoadBalancer
      # ...

   2. Apply the configuration by entering the following command:

      $ oc apply -f ingress-loadbalancer.yaml

   3. Verify that the load balancer service works as expected by entering the following command:

      $ oc get svc metallb-ingress -n openshift-ingress

      Example output

      NAME              TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
      metallb-ingress   LoadBalancer   172.31.127.129   10.11.176.71   80:30961/TCP,443:32090/TCP   16h

10. Configure the DNS to work with the load balancer:

    1. Configure the DNS for the apps domain by pointing the *.apps.<hosted_cluster_namespace>.<base_domain> wildcard DNS record to the load balancer IP address.

    2. Verify the DNS resolution by entering the following command:

       $ nslookup console-openshift-console.apps.<hosted_cluster_namespace>.<base_domain> <load_balancer_ip_address>

       Example output

       Server:         10.11.176.1
       Address:        10.11.176.1#53
       
       Name:   console-openshift-console.apps.my-hosted-cluster.sample-base-domain.com
       Address: 10.11.176.71

Verification

1. Check the cluster Operators by entering the following command:

   $ oc get clusteroperators

   Ensure that all Operators show AVAILABLE: True, PROGRESSING: False, and DEGRADED: False.

2. Check the nodes by entering the following command:

   $ oc get nodes

   Ensure that each node has the READY status.

3. Test access to the console by entering the following URL in a web browser:

   https://console-openshift-console.apps.<hosted_cluster_namespace>.<base_domain>

Procedure

1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials. For instructions to open the console, see "Accessing the web console".
2. In the console header, ensure that All Clusters is selected.
3. Click Infrastructure Clusters.

4. Click Create cluster Host inventory Hosted control plane.

   The Create cluster page is displayed.

5. On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation.

   Note

   As you enter details about the cluster, you might find the following tips useful:

   • If you want to use predefined values to automatically populate fields in the console, you can create a host inventory credential. For more information, see "Creating a credential for an on-premises environment".
   • On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a host inventory credential, the pull secret is automatically populated.
   • On the Node pools page, the namespace contains the hosts for the node pool. If you created a host inventory by using the console, the console creates a dedicated namespace.
   • On the Networking page, you select an API server publishing strategy. The API server for the hosted cluster can be exposed either by using an existing load balancer or as a service of the NodePort type. A DNS entry must exist for the api.<hosted_cluster_name>.<base_domain> setting that points to the destination where the API server can be reached. This entry can be a record that points to one of the nodes in the management cluster or a record that points to a load balancer that redirects incoming traffic to the Ingress pods.

6. Review your entries and click Create.

   The Hosted cluster view is displayed.

7. Monitor the deployment of the hosted cluster in the Hosted cluster view.
8. If you do not see information about the hosted cluster, ensure that All Clusters is selected, then click the cluster name.
9. Wait until the control plane components are ready. This process can take a few minutes.
10. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Procedure

1. Create a YAML file to define Image Content Source Policies (ICSP). See the following example:

   - mirrors:
     - brew.registry.redhat.io
     source: registry.redhat.io
   - mirrors:
     - brew.registry.redhat.io
     source: registry.stage.redhat.io
   - mirrors:
     - brew.registry.redhat.io
     source: registry-proxy.engineering.redhat.com

2. Save the file as icsp.yaml. This file contains your mirror registries.

3. To create a hosted cluster by using your mirror registries, run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \
       --pull-secret=<path_to_pull_secret> \
       --agent-namespace=<hosted_control_plane_namespace> \
       --base-domain=<base_domain> \
       --api-server-address=api.<hosted_cluster_name>.<base_domain> \
       --image-content-sources <icsp.yaml>  \
       --ssh-key  <path_to_ssh_key> \
       --namespace <hosted_cluster_namespace> \
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image>

   where:

   <hosted_cluster_name>

   Specifies the name of your hosted cluster, for example, my-hosted-cluster.

   <path_to_pull_secret>

   Specifies the path to your pull secret, for example, /user/name/pullsecret.

   <hosted_control_plane_namespace>

   Specifies your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.

   <base_domain>

   Specifies your base domain, for example, krnl.es.

   <hosted_cluster_name>.<base_domain>

   Specifies the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   <icsp.yaml>

   Specifies the icsp.yaml file that defines ICSP and your mirror registries.

   <path_to_ssh_key>

   Specifies the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   <hosted_cluster_namespace>

   Specifies your hosted cluster namespace.

   <ocp_release_image>

   Specifies the supported OpenShift Container Platform version that you want to use, for example, 4.22.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see "Extracting the OpenShift Container Platform release image digest".

Procedure

1. Create a hosted cluster with the KubeVirt platform by entering the following command:

   $ hcp create cluster kubevirt \
     --name <hosted_cluster_name> \
     --node-pool-replicas <node_pool_replica_count> \
     --pull-secret <path_to_pull_secret> \
     --memory <value_for_memory> \
     --cores <value_for_cpu> \
     --etcd-storage-class=<etcd_storage_class> \
     --arch <architecture_of_the_nodepool> \
     --release-image <ocp_release_image_for_the_cluster> \
     --image-content-sources <path_to_image_content_sources_file> \
     --additional-trust-bundle <path_to_ca_bundle_file>

   • --name defines the name of your hosted cluster, for example, my-hosted-cluster.
   • --node-pool-replicas defines the node pool replica count, for example, 3. You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, no node pools are created.
   • --pull-secret defines the path to your pull secret, for example, /user/name/pullsecret.
   • --memory defines a value for memory, for example, 6Gi.
   • --cores defines a value for CPU, for example, 2.
   • --etcd-storage-class defines the etcd storage class name, for example, lvm-storageclass.
   • --arch defines the architecture of the node pool, for example, s390x. The default is amd64.
   • --release-image defines the OpenShift Container Platform release image for the cluster, for example, quay.io/openshift-release-dev/ocp-release:4.20.14-multi. You can use the --release-image flag to set up the hosted cluster with a specific OpenShift Container Platform release.
   • --image-content-sources specifies the path to a file with image content sources.
   • --additional-trust-bundle specifies the path to a file with user CA bundle.

   A default node pool is created for the cluster with a specific number of virtual machine worker replicas according to the --node-pool-replicas flag.

2. After a few moments, verify that the hosted control plane pods are running by entering the following command:

   $ oc -n clusters-<hosted-cluster-name> get pods

   Example output

   NAME                                                  READY   STATUS    RESTARTS   AGE
   capi-provider-5cc7b74f47-n5gkr                        1/1     Running   0          3m
   catalog-operator-5f799567b7-fd6jw                     2/2     Running   0          69s
   certified-operators-catalog-784b9899f9-mrp6p          1/1     Running   0          66s
   cluster-api-6bbc867966-l4dwl                          1/1     Running   0          66s
   .
   .
   .
   redhat-operators-catalog-9d5fd4d44-z8qqk              1/1     Running   0          66s

   A hosted cluster that has worker nodes that are backed by KubeVirt virtual machines typically takes 10-15 minutes to be fully provisioned.

Procedure

1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials.
2. In the console header, ensure that All Clusters is selected.
3. Click Infrastructure > Clusters.
4. Click Create cluster > Red Hat OpenShift Virtualization > Hosted.

5. On the Create cluster page, follow the prompts to enter details about the cluster and node pools.

   On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a OpenShift Virtualization credential, the pull secret is automatically populated.

   Important

   Avoid storing all hosted cluster information in a shared namespace. If you create a hosted cluster in a shared namespace and then back up and restore the hosted cluster, you might unintentionally change other hosted clusters. Either store hosted cluster information in a separate namespace or set up your hosted cluster to back up and restore resources based on labels.

6. On the Node pools page, expand the Networking options section and configure the networking options for your node pool:

   1. In the Additional networks field, enter a network name in the format of <namespace>/<name>; for example, my-namespace/network1. The namespace and the name must be valid DNS labels. Multiple networks are supported.
   2. By default, the Attach default pod network checkbox is selected. You can clear this checkbox only if additional networks exist.

7. Review your entries and click Create.

   The Hosted cluster view is displayed.

Verification

1. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, and click the cluster name.
2. Wait until the control plane components are ready. This process can take a few minutes.
3. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Procedure

1. Enter the following command:

   $ hcp create cluster kubevirt \
     --name <hosted_cluster_name> \ 

   1

   
     --node-pool-replicas <worker_count> \ 

   2

   
     --pull-secret <path_to_pull_secret> \ 

   3

   
     --memory <value_for_memory> \ 

   4

   
     --cores <value_for_cpu> \ 

   5

   
     --base-domain <basedomain> \ 

   6

   
     --arch <architecture_of_the_nodepool> \ 

   7

   
     --release-image <ocp_release_image_for_the_cluster> \ 

   8

   
     --image-content-sources <path_to_image_content_sources_file> \ 

   9

   
     --additional-trust-bundle <path_to_ca_bundle_file> 

   10

   1

   Specify the name of your hosted cluster.

   1 2

   Specify the worker count, for example, 2.

   3

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   4

   Specify a value for memory, for example, 6Gi.

   5

   Specify a value for CPU, for example, 2.

   6

   Specify the base domain, for example, hypershift.lab.

   7

   Specify the architecture of the node pool, for example, s390x. The default is amd64.

   8

   Specify the ocp release image for the cluster, for example, quay.io/openshift-release-dev/ocp-release:4.20.14-multi.

   9

   Specify the path to a file with image content sources.

   10

   Specify the path to a file with user CA bundle.

   As a result, the hosted cluster has an ingress wildcard that is configured for the cluster name and the base domain, for example, .apps.example.hypershift.lab. The hosted cluster remains in Partial status because after you create a hosted cluster with unique base domain, you must configure the required DNS records and load balancer.