Hosted control planes


OpenShift Container Platform 4.17

Using hosted control planes with OpenShift Container Platform

Red Hat OpenShift Documentation Team

Abstract

This document provides instructions for managing hosted control planes for OpenShift Container Platform. With hosted control planes, you create control planes as pods on a hosting cluster without the need for dedicated physical or virtual machines for each control plane.

Chapter 1. Hosted control planes release notes

Release notes contain information about new and deprecated features, changes, and known issues.

1.1. Hosted control planes release notes for OpenShift Container Platform 4.17

With this release, hosted control planes for OpenShift Container Platform 4.17 is available. Hosted control planes for OpenShift Container Platform 4.17 supports the multicluster engine for Kubernetes Operator version 2.7.

1.1.1. New features and enhancements

This release adds improvements related to the following concepts:

1.1.1.1. Custom taints and tolerations (Technology Preview)

For hosted control planes on OpenShift Virtualization, you can now apply tolerations to hosted control plane pods by using the hcp CLI -tolerations argument or by using the hc.Spec.Tolerations API file. This feature is available as a Technology Preview feature. For more information, see Custom taints and tolerations.

1.1.1.2. Support for NVIDIA GPU devices on OpenShift Virtualization (Technology Preview)

For hosted control planes on OpenShift Virtualization, you can attach one or more NVIDIA graphics processing unit (GPU) devices to node pools. This feature is available as a Technology Preview feature. For more information, see Attaching NVIDIA GPU devices by using the hcp CLI and Attaching NVIDIA GPU devices by using the NodePool resource.

1.1.1.3. Support for tenancy on AWS

When you create a hosted cluster on AWS, you can indicate whether the EC2 instance should run on shared or single-tenant hardware. For more information, see Creating a hosted cluster on AWS.

1.1.1.4. Support for OpenShift Container Platform versions in hosted clusters

You can deploy a range of supported OpenShift Container Platform versions in a hosted cluster. For more information, see Supported OpenShift Container Platform versions in a hosted cluster.

1.1.1.5. Hosted control planes on OpenShift Virtualization in a disconnected environment is Generally Available

In this release, hosted control planes on OpenShift Virtualization in a disconnected environment is Generally Available. For more information, see Deploying hosted control planes on OpenShift Virtualization in a disconnected environment.

1.1.1.6. Hosted control planes for an ARM64 OpenShift Container Platform cluster on AWS is Generally Available

In this release, hosted control planes for an ARM64 OpenShift Container Platform cluster on AWS is Generally Available. For more information, see Running hosted clusters on an ARM64 architecture.

1.1.1.7. Hosted control planes on IBM Z is Generally Available

In this release, hosted control planes on IBM Z is Generally Available. For more information, see Deploying hosted control planes on IBM Z.

1.1.1.8. Hosted control planes on IBM Power is Generally Available

In this release, hosted control planes on IBM Power is Generally Available. For more information, see Deploying hosted control planes on IBM Power.

1.1.2. Bug fixes

  • Previously, when a hosted cluster proxy was configured and it used an identity provider (IDP) that had an HTTP or HTTPS endpoint, the hostname of the IDP was unresolved before sending it through the proxy. Consequently, hostnames that could only be resolved by the data plane failed to resolve for IDPs. With this update, a DNS lookup is performed before sending IPD traffic through the konnectivity tunnel. As a result, IDPs with hostnames that can only be resolved by the data plane can be verified by the Control Plane Operator. (OCPBUGS-41371)
  • Previously, when the hosted cluster controllerAvailabilityPolicy was set to SingleReplica, podAntiAffinity on networking components blocked the availability of the components. With this release, the issue is resolved. (OCPBUGS-39313)
  • Previously, the AdditionalTrustedCA that was specified in the hosted cluster image configuration was not reconciled into the openshift-config namespace, as expected by the image-registry-operator, and the component did not become available. With this release, the issue is resolved. (OCPBUGS-39225)
  • Previously, Red Hat HyperShift periodic conformance jobs failed because of changes to the core operating system. These failed jobs caused the OpenShift API deployment to fail. With this release, an update recursively copies individual trusted certificate authority (CA) certificates instead of copying a single file, so that the periodic conformance jobs succeed and the OpenShift API runs as expected. (OCPBUGS-38941)
  • Previously, the Konnectivity proxy agent in a hosted cluster always sent all TCP traffic through an HTTP/S proxy. It also ignored host names in the NO_PROXY configuration because it only received resolved IP addresses in its traffic. As a consequence, traffic that was not meant to be proxied, such as LDAP traffic, was proxied regardless of configuration. With this release, proxying is completed at the source (control plane) and the Konnectivity agent proxying configuration is removed. As a result, traffic that is not meant to be proxied, such as LDAP traffic, is not proxied anymore. The NO_PROXY configuration that includes host names is honored. (OCPBUGS-38637)
  • Previously, the azure-disk-csi-driver-controller image was not getting appropriate override values when using registryOverride. This was intentional so as to avoid propagating the values to the azure-disk-csi-driver data plane images. With this update, the issue is resolved by adding a separate image override value. As a result, the azure-disk-csi-driver-controller can be used with registryOverride and no longer affects azure-disk-csi-driver data plane images. (OCPBUGS-38183)
  • Previously, the AWS cloud controller manager within a hosted control plane that was running on a proxied management cluster would not use the proxy for cloud API communication. With this release, the issue is fixed. (OCPBUGS-37832)
  • Previously, proxying for Operators that run in the control plane of a hosted cluster was performed through proxy settings on the Konnectivity agent pod that runs in the data plane. It was not possible to distinguish if proxying was needed based on application protocol.

    For parity with OpenShift Container Platform, IDP communication via HTTPS or HTTP should be proxied, but LDAP communication should not be proxied. This type of proxying also ignores NO_PROXY entries that rely on host names because by the time traffic reaches the Konnectivity agent, only the destination IP address is available.

    With this release, in hosted clusters, proxy is invoked in the control plane through konnectivity-https-proxy and konnectivity-socks5-proxy, and proxying traffic is stopped from the Konnectivity agent. As a result, traffic that is destined for LDAP servers is no longer proxied. Other HTTPS or HTTPS traffic is proxied correctly. The NO_PROXY setting is honored when you specify hostnames. (OCPBUGS-37052)

  • Previously, proxying for IDP communication occurred in the Konnectivity agent. By the time traffic reached Konnectivity, its protocol and hostname were no longer available. As a consequence, proxying was not done correctly for the OAUTH server pod. It did not distinguish between protocols that require proxying (http/s) and protocols that do not (ldap://). In addition, it did not honor the no_proxy variable that is configured in the HostedCluster.spec.configuration.proxy spec.

    With this release, you can configure the proxy on the Konnectivity sidecar of the OAUTH server so that traffic is routed appropriately, honoring your no_proxy settings. As a result, the OAUTH server can communicate properly with identity providers when a proxy is configured for the hosted cluster. (OCPBUGS-36932)

  • Previously, the Hosted Cluster Config Operator (HCCO) did not delete the ImageDigestMirrorSet CR (IDMS) after you removed the ImageContentSources field from the HostedCluster object. As a consequence, the IDMS persisted in the HostedCluster object when it should not. With this release, the HCCO manages the deletion of IDMS resources from the HostedCluster object. (OCPBUGS-34820)
  • Previously, deploying a hostedCluster in a disconnected environment required setting the hypershift.openshift.io/control-plane-operator-image annotation. With this update, the annotation is no longer needed. Additionally, the metadata inspector works as expected during the hosted Operator reconciliation, and OverrideImages is populated as expected. (OCPBUGS-34734)
  • Previously, hosted clusters on AWS leveraged their VPC’s primary CIDR range to generate security group rules on the data plane. As a consequence, if you installed a hosted cluster into an AWS VPC with multiple CIDR ranges, the generated security group rules could be insufficient. With this update, security group rules are generated based on the provided machine CIDR range to resolve this issue. (OCPBUGS-34274)
  • Previously, the OpenShift Cluster Manager container did not have the right TLS certificates. As a consequence, you could not use image streams in disconnected deployments. With this release, the TLS certificates are added as projected volumes to resolve this issue. (OCPBUGS-31446)
  • Previously, the bulk destroy option in the multicluster engine for Kubernetes Operator console for OpenShift Virtualization did not destroy a hosted cluster. With this release, this issue is resolved. (ACM-10165)

1.1.3. Known issues

  • If the annotation and the ManagedCluster resource name do not match, the multicluster engine for Kubernetes Operator console displays the cluster as Pending import. The cluster cannot be used by the multicluster engine Operator. The same issue happens when there is no annotation and the ManagedCluster name does not match the Infra-ID value of the HostedCluster resource.
  • When you use the multicluster engine for Kubernetes Operator console to add a new node pool to an existing hosted cluster, the same version of OpenShift Container Platform might appear more than once in the list of options. You can select any instance in the list for the version that you want.
  • When a node pool is scaled down to 0 workers, the list of hosts in the console still shows nodes in a Ready state. You can verify the number of nodes in two ways:

    • In the console, go to the node pool and verify that it has 0 nodes.
    • On the command-rline interface, run the following commands:

      • Verify that 0 nodes are in the node pool by running the following command:

        $ oc get nodepool -A
      • Verify that 0 nodes are in the cluster by running the following command:

        $ oc get nodes --kubeconfig
      • Verify that 0 agents are reported as bound to the cluster by running the following command:

        $ oc get agents -A
  • When you create a hosted cluster in an environment that uses the dual-stack network, you might encounter the following DNS-related issues:

    • CrashLoopBackOff state in the service-ca-operator pod: When the pod tries to reach the Kubernetes API server through the hosted control plane, the pod cannot reach the server because the data plane proxy in the kube-system namespace cannot resolve the request. This issue occurs because in the HAProxy setup, the front end uses an IP address and the back end uses a DNS name that the pod cannot resolve.
    • Pods stuck in ContainerCreating state: This issue occurs because the openshift-service-ca-operator cannot generate the metrics-tls secret that the DNS pods need for DNS resolution. As a result, the pods cannot resolve the Kubernetes API server. To resolve these issues, configure the DNS server settings for a dual stack network.
  • On the Agent platform, the hosted control planes feature periodically rotates the token that the Agent uses to pull ignition. As a result, if you have an Agent resource that was created some time ago, it might fail to pull ignition. As a workaround, in the Agent specification, delete the secret of the IgnitionEndpointTokenReference property then add or modify any label on the Agent resource. The system re-creates the secret with the new token.
  • If you created a hosted cluster in the same namespace as its managed cluster, detaching the managed hosted cluster deletes everything in the managed cluster namespace including the hosted cluster. The following situations can create a hosted cluster in the same namespace as its managed cluster:

    • You created a hosted cluster on the Agent platform through the multicluster engine for Kubernetes Operator console by using the default hosted cluster cluster namespace.
    • You created a hosted cluster through the command-line interface or API by specifying the hosted cluster namespace to be the same as the hosted cluster name.

1.1.4. Generally Available and Technology Preview features

Features which are Generally Available (GA) are fully supported and are suitable for production use. Technology Preview (TP) features are experimental features and are not intended for production use. For more information about TP features, see the Technology Preview scope of support on the Red Hat Customer Portal.

See the following table to know about hosted control planes GA and TP features:

Table 1.1. Hosted control planes GA and TP tracker
Feature4.154.164.17

Hosted control planes for OpenShift Container Platform on Amazon Web Services (AWS)

Technology Preview

Generally Available

Generally Available

Hosted control planes for OpenShift Container Platform on bare metal

General Availability

General Availability

General Availability

Hosted control planes for OpenShift Container Platform on OpenShift Virtualization

Generally Available

Generally Available

Generally Available

Hosted control planes for OpenShift Container Platform using non-bare-metal agent machines

Technology Preview

Technology Preview

Technology Preview

Hosted control planes for an ARM64 OpenShift Container Platform cluster on Amazon Web Services

Technology Preview

Technology Preview

Generally Available

Hosted control planes for OpenShift Container Platform on IBM Power

Technology Preview

Technology Preview

Generally Available

Hosted control planes for OpenShift Container Platform on IBM Z

Technology Preview

Technology Preview

Generally Available

Hosted control planes for OpenShift Container Platform on RHOSP

Not Available

Not Available

Developer Preview

Chapter 2. Hosted control planes overview

You can deploy OpenShift Container Platform clusters by using two different control plane configurations: standalone or hosted control planes. The standalone configuration uses dedicated virtual machines or physical machines to host the control plane. With hosted control planes for OpenShift Container Platform, you create control planes as pods on a management cluster without the need for dedicated virtual or physical machines for each control plane.

2.1. Introduction to hosted control planes

Hosted control planes is available by using a supported version of multicluster engine for Kubernetes Operator on the following platforms:

  • Bare metal by using the Agent provider
  • Non-bare-metal Agent machines, as a Technology Preview feature
  • OpenShift Virtualization
  • Amazon Web Services (AWS)
  • IBM Z
  • IBM Power

The hosted control planes feature is enabled by default.

2.1.1. Architecture of hosted control planes

OpenShift Container Platform is often deployed in a coupled, or standalone, model, where a cluster consists of a control plane and a data plane. The control plane includes an API endpoint, a storage endpoint, a workload scheduler, and an actuator that ensures state. The data plane includes compute, storage, and networking where workloads and applications run.

The standalone control plane is hosted by a dedicated group of nodes, which can be physical or virtual, with a minimum number to ensure quorum. The network stack is shared. Administrator access to a cluster offers visibility into the cluster’s control plane, machine management APIs, and other components that contribute to the state of a cluster.

Although the standalone model works well, some situations require an architecture where the control plane and data plane are decoupled. In those cases, the data plane is on a separate network domain with a dedicated physical hosting environment. The control plane is hosted by using high-level primitives such as deployments and stateful sets that are native to Kubernetes. The control plane is treated as any other workload.

Diagram that compares the hosted control plane model against OpenShift with a coupled control plane and workers

2.1.2. Benefits of hosted control planes

With hosted control planes, you can pave the way for a true hybrid-cloud approach and enjoy several other benefits.

  • The security boundaries between management and workloads are stronger because the control plane is decoupled and hosted on a dedicated hosting service cluster. As a result, you are less likely to leak credentials for clusters to other users. Because infrastructure secret account management is also decoupled, cluster infrastructure administrators cannot accidentally delete control plane infrastructure.
  • With hosted control planes, you can run many control planes on fewer nodes. As a result, clusters are more affordable.
  • Because the control planes consist of pods that are launched on OpenShift Container Platform, control planes start quickly. The same principles apply to control planes and workloads, such as monitoring, logging, and auto-scaling.
  • From an infrastructure perspective, you can push registries, HAProxy, cluster monitoring, storage nodes, and other infrastructure components to the tenant’s cloud provider account, isolating usage to the tenant.
  • From an operational perspective, multicluster management is more centralized, which results in fewer external factors that affect the cluster status and consistency. Site reliability engineers have a central place to debug issues and navigate to the cluster data plane, which can lead to shorter Time to Resolution (TTR) and greater productivity.

2.2. Relationship between hosted control planes, multicluster engine Operator, and RHACM

You can configure hosted control planes by using the multicluster engine for Kubernetes Operator. The multicluster engine is an integral part of Red Hat Advanced Cluster Management (RHACM) and is enabled by default with RHACM. The multicluster engine Operator cluster lifecycle defines the process of creating, importing, managing, and destroying Kubernetes clusters across various infrastructure cloud providers, private clouds, and on-premises data centers.

The multicluster engine Operator is the cluster lifecycle Operator that provides cluster management capabilities for OpenShift Container Platform and RHACM hub clusters. The multicluster engine Operator enhances cluster fleet management and supports OpenShift Container Platform cluster lifecycle management across clouds and data centers.

Figure 2.1. Cluster life cycle and foundation

Cluster life cycle and foundation

You can use the multicluster engine Operator with OpenShift Container Platform as a standalone cluster manager or as part of a RHACM hub cluster.

Tip

A management cluster is also known as the hosting cluster.

You can deploy OpenShift Container Platform clusters by using two different control plane configurations: standalone or hosted control planes. The standalone configuration uses dedicated virtual machines or physical machines to host the control plane. With hosted control planes for OpenShift Container Platform, you create control planes as pods on a management cluster without the need for dedicated virtual or physical machines for each control plane.

Figure 2.2. RHACM and the multicluster engine Operator introduction diagram

RHACM and the multicluster engine Operator introduction diagram

2.3. Versioning for hosted control planes

The hosted control planes feature includes the following components, which might require independent versioning and support levels:

  • Management cluster
  • HyperShift Operator
  • Hosted control planes (hcp) command-line interface (CLI)
  • hypershift.openshift.io API
  • Control Plane Operator

2.3.1. Management cluster

In management clusters for production use, you need multicluster engine for Kubernetes Operator, which is available through OperatorHub. The multicluster engine Operator bundles a supported build of the HyperShift Operator. For your management clusters to remain supported, you must use the version of OpenShift Container Platform that multicluster engine Operator runs on. In general, a new release of multicluster engine Operator runs on the following versions of OpenShift Container Platform:

  • The latest General Availability version of OpenShift Container Platform
  • Two versions before the latest General Availability version of OpenShift Container Platform

The full list of OpenShift Container Platform versions that you can install through the HyperShift Operator on a management cluster depends on the version of your HyperShift Operator. However, the list always includes at least the same OpenShift Container Platform version as the management cluster and two previous minor versions relative to the management cluster. For example, if the management cluster is running 4.17 and a supported version of multicluster engine Operator, the HyperShift Operator can install 4.17, 4.16, 4.15, and 4.14 hosted clusters.

With each major, minor, or patch version release of OpenShift Container Platform, two components of hosted control planes are released:

  • The HyperShift Operator
  • The hcp command-line interface (CLI)

2.3.2. HyperShift Operator

The HyperShift Operator manages the lifecycle of hosted clusters that are represented by the HostedCluster API resources. The HyperShift Operator is released with each OpenShift Container Platform release. The HyperShift Operator creates the supported-versions config map in the hypershift namespace. The config map contains the supported hosted cluster versions.

You can host different versions of control planes on the same management cluster.

Example supported-versions config map object

    apiVersion: v1
    data:
      supported-versions: '{"versions":["4.17"]}'
    kind: ConfigMap
    metadata:
      labels:
        hypershift.openshift.io/supported-versions: "true"
      name: supported-versions
      namespace: hypershift

2.3.3. hosted control planes CLI

You can use the hcp CLI to create hosted clusters. You can download the CLI from multicluster engine Operator. When you run the hcp version command, the output shows the latest OpenShift Container Platform that the CLI supports against your kubeconfig file.

2.3.4. hypershift.openshift.io API

You can use the hypershift.openshift.io API resources, such as, HostedCluster and NodePool, to create and manage OpenShift Container Platform clusters at scale. A HostedCluster resource contains the control plane and common data plane configuration. When you create a HostedCluster resource, you have a fully functional control plane with no attached nodes. A NodePool resource is a scalable set of worker nodes that is attached to a HostedCluster resource.

The API version policy generally aligns with the policy for Kubernetes API versioning.

Updates for hosted control planes involve updating the hosted cluster and the node pools. For more information, see "Updates for hosted control planes".

2.3.5. Control Plane Operator

The Control Plane Operator is released as part of each OpenShift Container Platform payload release image for the following architectures:

  • amd64
  • arm64
  • multi-arch

2.4. Glossary of common concepts and personas for hosted control planes

When you use hosted control planes for OpenShift Container Platform, it is important to understand its key concepts and the personas that are involved.

2.4.1. Concepts

hosted cluster
An OpenShift Container Platform cluster with its control plane and API endpoint hosted on a management cluster. The hosted cluster includes the control plane and its corresponding data plane.
hosted cluster infrastructure
Network, compute, and storage resources that exist in the tenant or end-user cloud account.
hosted control plane
An OpenShift Container Platform control plane that runs on the management cluster, which is exposed by the API endpoint of a hosted cluster. The components of a control plane include etcd, the Kubernetes API server, the Kubernetes controller manager, and a VPN.
hosting cluster
See management cluster.
managed cluster
A cluster that the hub cluster manages. This term is specific to the cluster lifecycle that the multicluster engine for Kubernetes Operator manages in Red Hat Advanced Cluster Management. A managed cluster is not the same thing as a management cluster. For more information, see Managed cluster.
management cluster
An OpenShift Container Platform cluster where the HyperShift Operator is deployed and where the control planes for hosted clusters are hosted. The management cluster is synonymous with the hosting cluster.
management cluster infrastructure
Network, compute, and storage resources of the management cluster.
node pool
A resource that contains the compute nodes. The control plane contains node pools. The compute nodes run applications and workloads.

2.4.2. Personas

cluster instance administrator
Users who assume this role are the equivalent of administrators in standalone OpenShift Container Platform. This user has the cluster-admin role in the provisioned cluster, but might not have power over when or how the cluster is updated or configured. This user might have read-only access to see some configuration projected into the cluster.
cluster instance user
Users who assume this role are the equivalent of developers in standalone OpenShift Container Platform. This user does not have a view into OperatorHub or machines.
cluster service consumer
Users who assume this role can request control planes and worker nodes, drive updates, or modify externalized configurations. Typically, this user does not manage or access cloud credentials or infrastructure encryption keys. The cluster service consumer persona can request hosted clusters and interact with node pools. Users who assume this role have RBAC to create, read, update, or delete hosted clusters and node pools within a logical boundary.
cluster service provider

Users who assume this role typically have the cluster-admin role on the management cluster and have RBAC to monitor and own the availability of the HyperShift Operator as well as the control planes for the tenant’s hosted clusters. The cluster service provider persona is responsible for several activities, including the following examples:

  • Owning service-level objects for control plane availability, uptime, and stability
  • Configuring the cloud account for the management cluster to host control planes
  • Configuring the user-provisioned infrastructure, which includes the host awareness of available compute resources

Chapter 3. Preparing to deploy hosted control planes

3.1. Requirements for hosted control planes

In the context of hosted control planes, a management cluster is an OpenShift Container Platform cluster where the HyperShift Operator is deployed and where the control planes for hosted clusters are hosted. The management cluster and workers must run on the same infrastructure. For example, you cannot run your management cluster on bare metal and your workers on the cloud. However, the management cluster and workers do not need to run on the same platform. For example, you might run your management cluster on bare metal and workers on OpenShift Virtualization.

The control plane is associated with a hosted cluster and runs as pods in a single namespace. When the cluster service consumer creates a hosted cluster, it creates a worker node that is independent of the control plane.

3.1.1. Support matrix for hosted control planes

Because multicluster engine for Kubernetes Operator includes the HyperShift Operator, releases of hosted control planes align with releases of multicluster engine Operator. For more information, see OpenShift Operator Life Cycles.

3.1.1.1. Management cluster support

Any supported standalone OpenShift Container Platform cluster can be a management cluster. The following table maps multicluster engine Operator versions to the management cluster versions that support them:

Table 3.1. Supported multicluster engine Operator versions for OpenShift Container Platform management clusters
Management cluster versionSupported multicluster engine Operator version

4.14 - 4.15

2.4

4.14 - 4.16

2.5

4.14 - 4.17

2.6

4.15 - 4.17

2.7

3.1.1.2. Hosted cluster support

For hosted clusters, no direct relationship exists between the management cluster version and the hosted cluster version. The hosted cluster version depends on the HyperShift Operator that is included with your multicluster engine Operator version.

Note

Ensure a minimum latency of 200 ms between the management cluster and hosted clusters.

The following table maps multicluster engine Operator versions to the hosted cluster versions that you can create by using the HyperShift Operator that is associated with that version of multicluster engine Operator:

Table 3.2. Hosted cluster versions that can be created by multicluster engine Operator versions
Hosted cluster versionmulticluster engine Operator 2.4multicluster engine Operator 2.5multicluster engine Operator 2.6multicluster engine Operator 2.7

4.14

Yes

Yes

Yes

Yes

4.15

No

Yes

Yes

Yes

4.16

No

No

Yes

Yes

4.17

No

No

No

Yes

3.1.1.3. Hosted cluster platform support

The following table indicates which OpenShift Container Platform versions are supported for each platform of hosted control planes. In the table, Management cluster version refers to the OpenShift Container Platform version where the multicluster engine Operator is enabled:

Table 3.3. Required OpenShift Container Platform versions for platforms
Hosted cluster platformManagement cluster versionHosted cluster version

Amazon Web Services

4.16 - 4.17

4.16 - 4.17

IBM Power

4.17

4.17

IBM Z

4.17

4.17

OpenShift Virtualization

4.14 - 4.17

4.14 - 4.17

Bare metal

4.14 - 4.17

4.14 - 4.17

Non-bare-metal agent machines (Technology Preview)

4.16 - 4.17

4.16 - 4.17

3.1.1.4. Updates of multicluster engine Operator

When you update to another version of the multicluster engine Operator, your hosted cluster can continue to run if the HyperShift Operator that is included in the version of multicluster engine Operator supports the hosted cluster version. The following table shows which hosted cluster versions are supported on which updated multicluster engine Operator versions:

Table 3.4. Updated multicluster engine Operator version support for hosted clusters
Updated multicluster engine Operator versionSupported hosted cluster version

Updated from 2.4 to 2.5

OpenShift Container Platform 4.14

Updated from 2.5 to 2.6

OpenShift Container Platform 4.14 - 4.15

Updated from 2.6 to 2.7

OpenShift Container Platform 4.14 - 4.16

For example, if you have an OpenShift Container Platform 4.14 hosted cluster on the management cluster and you update from multicluster engine Operator 2.4 to 2.5, the hosted cluster can continue to run.

3.1.1.5. Technology Preview features

The following list indicates Technology Preview features for this release:

  • Hosted control planes on IBM Z in a disconnected environment
  • Custom taints and tolerations for hosted control planes on OpenShift Virtualization
  • NVIDIA GPU devices on hosted control planes for OpenShift Virtualization

3.2. Sizing guidance for hosted control planes

Many factors, including hosted cluster workload and worker node count, affect how many hosted clusters can fit within a certain number of control-plane nodes. Use this sizing guide to help with hosted cluster capacity planning. This guidance assumes a highly available hosted control planes topology. The load-based sizing examples were measured on a bare-metal cluster. Cloud-based instances might have different limiting factors, such as memory size.

You can override the following resource utilization sizing measurements and disable the metric service monitoring.

See the following highly available hosted control planes requirements, which were tested with OpenShift Container Platform version 4.12.9 and later:

  • 78 pods
  • Three 8 GiB PVs for etcd
  • Minimum vCPU: approximately 5.5 cores
  • Minimum memory: approximately 19 GiB

Additional resources

3.2.1. Pod limits

The maxPods setting for each node affects how many hosted clusters can fit in a control-plane node. It is important to note the maxPods value on all control-plane nodes. Plan for about 75 pods for each highly available hosted control plane.

For bare-metal nodes, the default maxPods setting of 250 is likely to be a limiting factor because roughly three hosted control planes fit for each node given the pod requirements, even if the machine has plenty of resources to spare. Setting the maxPods value to 500 by configuring the KubeletConfig value allows for greater hosted control plane density, which can help you take advantage of additional compute resources.

Additional resources

3.2.2. Request-based resource limit

The maximum number of hosted control planes that the cluster can host is calculated based on the hosted control plane CPU and memory requests from the pods.

A highly available hosted control plane consists of 78 pods that request 5 vCPUs and 18 GB memory. These baseline numbers are compared to the cluster worker node resource capacities to estimate the maximum number of hosted control planes.

3.2.3. Load-based limit

The maximum number of hosted control planes that the cluster can host is calculated based on the hosted control plane pods CPU and memory utilizations when some workload is put on the hosted control plane Kubernetes API server.

The following method is used to measure the hosted control plane resource utilizations as the workload increases:

  • A hosted cluster with 9 workers that use 8 vCPU and 32 GiB each, while using the KubeVirt platform
  • The workload test profile that is configured to focus on API control-plane stress, based on the following definition:

    • Created objects for each namespace, scaling up to 100 namespaces total
    • Additional API stress with continuous object deletion and creation
    • Workload queries-per-second (QPS) and Burst settings set high to remove any client-side throttling

As the load increases by 1000 QPS, the hosted control plane resource utilization increases by 9 vCPUs and 2.5 GB memory.

For general sizing purposes, consider the 1000 QPS API rate that is a medium hosted cluster load, and a 2000 QPS API that is a heavy hosted cluster load.

Note

This test provides an estimation factor to increase the compute resource utilization based on the expected API load. Exact utilization rates can vary based on the type and pace of the cluster workload.

Table 3.5. Load table
Hosted control plane resource utilization scalingvCPUsMemory (GiB)

Resource utilization with no load

2.9

11.1

Resource utilization with 1000 QPS

9.0

2.5

As the load increases by 1000 QPS, the hosted control plane resource utilization increases by 9 vCPUs and 2.5 GB memory.

For general sizing purposes, consider a 1000 QPS API rate to be a medium hosted cluster load and a 2000 QPS API to be a heavy hosted cluster load.

The following example shows hosted control plane resource scaling for the workload and API rate definitions:

Table 3.6. API rate table
QPS (API rate)vCPU usageMemory usage (GiB)

Low load (Less than 50 QPS)

2.9

11.1

Medium load (1000 QPS)

11.9

13.6

High load (2000 QPS)

20.9

16.1

The hosted control plane sizing is about control-plane load and workloads that cause heavy API activity, etcd activity, or both. Hosted pod workloads that focus on data-plane loads, such as running a database, might not result in high API rates.

3.2.4. Sizing calculation example

This example provides sizing guidance for the following scenario:

  • Three bare-metal workers that are labeled as hypershift.openshift.io/control-plane nodes
  • maxPods value set to 500
  • The expected API rate is medium or about 1000, according to the load-based limits
Table 3.7. Limit inputs
Limit descriptionServer 1Server 2

Number of vCPUs on worker node

64

128

Memory on worker node (GiB)

128

256

Maximum pods per worker

500

500

Number of workers used to host control planes

3

3

Maximum QPS target rate (API requests per second)

1000

1000

Table 3.8. Sizing calculation example

Calculated values based on worker node size and API rate

Server 1

Server 2

Calculation notes

Maximum hosted control planes per worker based on vCPU requests

12.8

25.6

Number of worker vCPUs ÷ 5 total vCPU requests per hosted control plane

Maximum hosted control planes per worker based on vCPU usage

5.4

10.7

Number of vCPUS ÷ (2.9 measured idle vCPU usage + (QPS target rate ÷ 1000) × 9.0 measured vCPU usage per 1000 QPS increase)

Maximum hosted control planes per worker based on memory requests

7.1

14.2

Worker memory GiB ÷ 18 GiB total memory request per hosted control plane

Maximum hosted control planes per worker based on memory usage

9.4

18.8

Worker memory GiB ÷ (11.1 measured idle memory usage + (QPS target rate ÷ 1000) × 2.5 measured memory usage per 1000 QPS increase)

Maximum hosted control planes per worker based on per node pod limit

6.7

6.7

500 maxPods ÷ 75 pods per hosted control plane

Minimum of previously mentioned maximums

5.4

6.7

 
 

vCPU limiting factor

maxPods limiting factor

 

Maximum number of hosted control planes within a management cluster

16

20

Minimum of previously mentioned maximums × 3 control-plane workers

Table 3.9. Hosted control planes capacity metrics

Name

Description

mce_hs_addon_request_based_hcp_capacity_gauge

Estimated maximum number of hosted control planes the cluster can host based on a highly available hosted control planes resource request.

mce_hs_addon_low_qps_based_hcp_capacity_gauge

Estimated maximum number of hosted control planes the cluster can host if all hosted control planes make around 50 QPS to the clusters Kube API server.

mce_hs_addon_medium_qps_based_hcp_capacity_gauge

Estimated maximum number of hosted control planes the cluster can host if all hosted control planes make around 1000 QPS to the clusters Kube API server.

mce_hs_addon_high_qps_based_hcp_capacity_gauge

Estimated maximum number of hosted control planes the cluster can host if all hosted control planes make around 2000 QPS to the clusters Kube API server.

mce_hs_addon_average_qps_based_hcp_capacity_gauge

Estimated maximum number of hosted control planes the cluster can host based on the existing average QPS of hosted control planes. If you do not have an active hosted control planes, you can expect low QPS.

3.3. Overriding resource utilization measurements

The set of baseline measurements for resource utilization can vary in each hosted cluster.

3.3.1. Overriding resource utilization measurements for a hosted cluster

You can override resource utilization measurements based on the type and pace of your cluster workload.

Procedure

  1. Create the ConfigMap resource by running the following command:

    $ oc create -f <your-config-map-file.yaml>

    Replace <your-config-map-file.yaml> with the name of your YAML file that contains your hcp-sizing-baseline config map.

  2. Create the hcp-sizing-baseline config map in the local-cluster namespace to specify the measurements you want to override. Your config map might resemble the following YAML file:

    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: hcp-sizing-baseline
      namespace: local-cluster
    data:
      incrementalCPUUsagePer1KQPS: "9.0"
      memoryRequestPerHCP: "18"
      minimumQPSPerHCP: "50.0"
  3. Delete the hypershift-addon-agent deployment to restart the hypershift-addon-agent pod by running the following command:

    $ oc delete deployment hypershift-addon-agent -n open-cluster-management-agent-addon

Verification

  • Observe the hypershift-addon-agent pod logs. Verify that the overridden measurements are updated in the config map by running the following command:

    $ oc logs hypershift-addon-agent -n open-cluster-management-agent-addon

    Your logs might resemble the following output:

    Example output

    2024-01-05T19:41:05.392Z	INFO	agent.agent-reconciler	agent/agent.go:793	setting cpuRequestPerHCP to 5
    2024-01-05T19:41:05.392Z	INFO	agent.agent-reconciler	agent/agent.go:802	setting memoryRequestPerHCP to 18
    2024-01-05T19:53:54.070Z	INFO	agent.agent-reconciler	agent/hcp_capacity_calculation.go:141	The worker nodes have 12.000000 vCPUs
    2024-01-05T19:53:54.070Z	INFO	agent.agent-reconciler	agent/hcp_capacity_calculation.go:142	The worker nodes have 49.173369 GB memory

    If the overridden measurements are not updated properly in the hcp-sizing-baseline config map, you might see the following error message in the hypershift-addon-agent pod logs:

    Example error

    2024-01-05T19:53:54.052Z	ERROR	agent.agent-reconciler	agent/agent.go:788	failed to get configmap from the hub. Setting the HCP sizing baseline with default values.	{"error": "configmaps \"hcp-sizing-baseline\" not found"}

3.3.2. Disabling the metric service monitoring

After you enable the hypershift-addon managed cluster add-on, metric service monitoring is configured by default so that OpenShift Container Platform monitoring can gather metrics from hypershift-addon.

Procedure

You can disable metric service monitoring by completing the following steps:

  1. Log in to your hub cluster by running the following command:

    $ oc login
  2. Edit the hypershift-addon-deploy-config add-on deployment configuration specification by running the following command:

    $ oc edit addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine
  3. Add the disableMetrics=true customized variable to the specification, as shown in the following example:

    apiVersion: addon.open-cluster-management.io/v1alpha1
    kind: AddOnDeploymentConfig
    metadata:
      name: hypershift-addon-deploy-config
      namespace: multicluster-engine
    spec:
      customizedVariables:
      - name: hcMaxNumber
        value: "80"
      - name: hcThresholdNumber
        value: "60"
      - name: disableMetrics 1
        value: "true"
    1
    The disableMetrics=true customized variable disables metric service monitoring for both new and existing hypershift-addon managed cluster add-ons.
  4. Apply the changes to the configuration specification by running the following command:

    $ oc apply -f <filename>.yaml

3.4. Installing the hosted control planes command-line interface

The hosted control planes command-line interface, hcp, is a tool that you can use to get started with hosted control planes. For Day 2 operations, such as management and configuration, use GitOps or your own automation tool.

3.4.1. Installing the hosted control planes command-line interface from the terminal

You can install the hosted control planes command-line interface (CLI), hcp, from the terminal.

Procedure

  1. Get the URL to download the hcp binary by running the following command:

    $ oc get ConsoleCLIDownload hcp-cli-download -o json | jq -r ".spec"
  2. Download the hcp binary by running the following command:

    $ wget <hcp_cli_download_url> 1
    1
    Replace hcp_cli_download_url with the URL that you obtained from the previous step.
  3. Unpack the downloaded archive by running the following command:

    $ tar xvzf hcp.tar.gz
  4. Make the hcp binary file executable by running the following command:

    $ chmod +x hcp
  5. Move the hcp binary file to a directory in your path by running the following command:

    $ sudo mv hcp /usr/local/bin/.
Note

If you download the CLI on a Mac computer, you might see a warning about the hcp binary file. You need to adjust your security settings to allow the binary file to be run.

Verification

  • Verify that you see the list of available parameters by running the following command:

    $ hcp create cluster <platform> --help 1
    1
    You can use the hcp create cluster command to create and manage hosted clusters. The supported platforms are aws, agent, and kubevirt.

3.4.2. Installing the hosted control planes command-line interface by using the web console

You can install the hosted control planes command-line interface (CLI), hcp, by using the OpenShift Container Platform web console.

Procedure

  1. From the OpenShift Container Platform web console, click the Help iconCommand Line Tools.
  2. Click Download hcp CLI for your platform.
  3. Unpack the downloaded archive by running the following command:

    $ tar xvzf hcp.tar.gz
  4. Run the following command to make the binary file executable:

    $ chmod +x hcp
  5. Run the following command to move the binary file to a directory in your path:

    $ sudo mv hcp /usr/local/bin/.
Note

If you download the CLI on a Mac computer, you might see a warning about the hcp binary file. You need to adjust your security settings to allow the binary file to be run.

Verification

  • Verify that you see the list of available parameters by running the following command:

    $ hcp create cluster <platform> --help 1
    1
    You can use the hcp create cluster command to create and manage hosted clusters. The supported platforms are aws, agent, and kubevirt.

3.4.3. Installing the hosted control planes command-line interface by using the content gateway

You can install the hosted control planes command-line interface (CLI), hcp, by using the content gateway.

Procedure

  1. Navigate to the content gateway and download the hcp binary.
  2. Unpack the downloaded archive by running the following command:

    $ tar xvzf hcp.tar.gz
  3. Make the hcp binary file executable by running the following command:

    $ chmod +x hcp
  4. Move the hcp binary file to a directory in your path by running the following command:

    $ sudo mv hcp /usr/local/bin/.
Note

If you download the CLI on a Mac computer, you might see a warning about the hcp binary file. You need to adjust your security settings to allow the binary file to be run.

Verification

  • Verify that you see the list of available parameters by running the following command:

    $ hcp create cluster <platform> --help 1
    1
    You can use the hcp create cluster command to create and manage hosted clusters. The supported platforms are aws, agent, and kubevirt.

3.5. Distributing hosted cluster workloads

Before you get started with hosted control planes for OpenShift Container Platform, you must properly label nodes so that the pods of hosted clusters can be scheduled into infrastructure nodes. Node labeling is also important for the following reasons:

  • To ensure high availability and proper workload deployment. For example, you can set the node-role.kubernetes.io/infra label to avoid having the control-plane workload count toward your OpenShift Container Platform subscription.
  • To ensure that control plane workloads are separate from other workloads in the management cluster.
Important

Do not use the management cluster for your workload. Workloads must not run on nodes where control planes run.

3.5.1. Labeling management cluster nodes

Proper node labeling is a prerequisite to deploying hosted control planes.

As a management cluster administrator, you use the following labels and taints in management cluster nodes to schedule a control plane workload:

  • hypershift.openshift.io/control-plane: true: Use this label and taint to dedicate a node to running hosted control plane workloads. By setting a value of true, you avoid sharing the control plane nodes with other components, for example, the infrastructure components of the management cluster or any other mistakenly deployed workload.
  • hypershift.openshift.io/cluster: ${HostedControlPlane Namespace}: Use this label and taint when you want to dedicate a node to a single hosted cluster.

Apply the following labels on the nodes that host control-plane pods:

  • node-role.kubernetes.io/infra: Use this label to avoid having the control-plane workload count toward your subscription.
  • topology.kubernetes.io/zone: Use this label on the management cluster nodes to deploy highly available clusters across failure domains. The zone might be a location, rack name, or the hostname of the node where the zone is set. For example, a management cluster has the following nodes: worker-1a, worker-1b, worker-2a, and worker-2b. The worker-1a and worker-1b nodes are in rack1, and the worker-2a and worker-2b nodes are in rack2. To use each rack as an availability zone, enter the following commands:

    $ oc label node/worker-1a node/worker-1b topology.kubernetes.io/zone=rack1
    $ oc label node/worker-2a node/worker-2b topology.kubernetes.io/zone=rack2

Pods for a hosted cluster have tolerations, and the scheduler uses affinity rules to schedule them. Pods tolerate taints for control-plane and the cluster for the pods. The scheduler prioritizes the scheduling of pods into nodes that are labeled with hypershift.openshift.io/control-plane and hypershift.openshift.io/cluster: ${HostedControlPlane Namespace}.

For the ControllerAvailabilityPolicy option, use HighlyAvailable, which is the default value that the hosted control planes command line interface, hcp, deploys. When you use that option, you can schedule pods for each deployment within a hosted cluster across different failure domains by setting topology.kubernetes.io/zone as the topology key. Control planes that are not highly available are not supported.

Procedure

To enable a hosted cluster to require its pods to be scheduled into infrastructure nodes, set HostedCluster.spec.nodeSelector, as shown in the following example:

  spec:
    nodeSelector:
      role.kubernetes.io/infra: ""

This way, hosted control planes for each hosted cluster are eligible infrastructure node workloads, and you do not need to entitle the underlying OpenShift Container Platform nodes.

3.5.2. Priority classes

Four built-in priority classes influence the priority and preemption of the hosted cluster pods. You can create the pods in the management cluster in the following order from highest to lowest:

  • hypershift-operator: HyperShift Operator pods.
  • hypershift-etcd: Pods for etcd.
  • hypershift-api-critical: Pods that are required for API calls and resource admission to succeed. These pods include pods such as kube-apiserver, aggregated API servers, and web hooks.
  • hypershift-control-plane: Pods in the control plane that are not API-critical but still need elevated priority, such as the cluster version Operator.

3.5.3. Custom taints and tolerations

For hosted control planes on OpenShift Virtualization, by default, pods for a hosted cluster tolerate the control-plane and cluster taints. However, you can also use custom taints on nodes so that hosted clusters can tolerate those taints on a per-hosted-cluster basis by setting HostedCluster.spec.tolerations.

Important

Passing tolerations for a hosted cluster is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Example configuration

  spec:
    tolerations:
    - effect: NoSchedule
      key: kubernetes.io/custom
      operator: Exists

You can also set tolerations on the hosted cluster while you create a cluster by using the --tolerations hcp CLI argument.

Example CLI argument

--toleration="key=kubernetes.io/custom,operator=Exists,effect=NoSchedule"

For fine granular control of hosted cluster pod placement on a per-hosted-cluster basis, use custom tolerations with nodeSelectors. You can co-locate groups of hosted clusters and isolate them from other hosted clusters. You can also place hosted clusters in infra and control plane nodes.

Tolerations on the hosted cluster spread only to the pods of the control plane. To configure other pods that run on the management cluster and infrastructure-related pods, such as the pods to run virtual machines, you need to use a different process.

3.6. Enabling or disabling the hosted control planes feature

The hosted control planes feature, as well as the hypershift-addon managed cluster add-on, are enabled by default. If you want to disable the feature, or if you disabled it and want to manually enable it, see the following procedures.

3.6.1. Manually enabling the hosted control planes feature

If you need to manually enable hosted control planes, complete the following steps.

Procedure

  1. Run the following command to enable the feature:

    $ oc patch mce multiclusterengine --type=merge -p '{"spec":{"overrides":{"components":[{"name":"hypershift","enabled": true}]}}}' 1
    1
    The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.
  2. Run the following command to verify that the hypershift and hypershift-local-hosting features are enabled in the MultiClusterEngine custom resource:

    $ oc get mce multiclusterengine -o yaml 1
    1
    The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.

    Example output

    apiVersion: multicluster.openshift.io/v1
    kind: MultiClusterEngine
    metadata:
      name: multiclusterengine
    spec:
      overrides:
        components:
        - name: hypershift
          enabled: true
        - name: hypershift-local-hosting
          enabled: true

3.6.1.1. Manually enabling the hypershift-addon managed cluster add-on for local-cluster

Enabling the hosted control planes feature automatically enables the hypershift-addon managed cluster add-on. If you need to enable the hypershift-addon managed cluster add-on manually, complete the following steps to use the hypershift-addon to install the HyperShift Operator on local-cluster.

Procedure

  1. Create the ManagedClusterAddon HyperShift add-on by creating a file that resembles the following example:

    apiVersion: addon.open-cluster-management.io/v1alpha1
    kind: ManagedClusterAddOn
    metadata:
      name: hypershift-addon
      namespace: local-cluster
    spec:
      installNamespace: open-cluster-management-agent-addon
  2. Apply the file by running the following command:

    $ oc apply -f <filename>

    Replace filename with the name of the file that you created.

  3. Confirm that the hypershift-addon is installed by running the following command:

    $ oc get managedclusteraddons -n local-cluster hypershift-addon

    If the add-on is installed, the output resembles the following example:

    NAME               AVAILABLE   DEGRADED   PROGRESSING
    hypershift-addon   True

Your HyperShift add-on is installed and the hosting cluster is available to create and manage hosted clusters.

3.6.2. Disabling the hosted control planes feature

You can uninstall the HyperShift Operator and disable the hosted control planes feature. When you disable the hosted control planes feature, you must destroy the hosted cluster and the managed cluster resource on multicluster engine Operator, as described in the Managing hosted clusters topics.

3.6.2.1. Uninstalling the HyperShift Operator

To uninstall the HyperShift Operator and disable the hypershift-addon from the local-cluster, complete the following steps:

Procedure

  1. Run the following command to ensure that there is no hosted cluster running:

    $ oc get hostedcluster -A
    Important

    If a hosted cluster is running, the HyperShift Operator does not uninstall, even if the hypershift-addon is disabled.

  2. Disable the hypershift-addon by running the following command:

    $ oc patch mce multiclusterengine --type=merge -p '{"spec":{"overrides":{"components":[{"name":"hypershift-local-hosting","enabled": false}]}}}' 1
    1
    The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.
    Note

    You can also disable the hypershift-addon for the local-cluster from the multicluster engine Operator console after disabling the hypershift-addon.

3.6.2.2. Disabling the hosted control planes feature

To disable the hosted control planes feature, complete the following steps.

Prerequisites

  • You uninstalled the HyperShift Operator. For more information, see "Uninstalling the HyperShift Operator".

Procedure

  1. Run the following command to disable the hosted control planes feature:

    $ oc patch mce multiclusterengine --type=merge -p '{"spec":{"overrides":{"components":[{"name":"hypershift","enabled": false}]}}}' 1
    1
    The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.
  2. You can verify that the hypershift and hypershift-local-hosting features are disabled in the MultiClusterEngine custom resource by running the following command:

    $ oc get mce multiclusterengine -o yaml 1
    1
    The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.

    See the following example where hypershift and hypershift-local-hosting have their enabled: flags set to false:

    apiVersion: multicluster.openshift.io/v1
    kind: MultiClusterEngine
    metadata:
      name: multiclusterengine
    spec:
      overrides:
        components:
        - name: hypershift
          enabled: false
        - name: hypershift-local-hosting
          enabled: false

Chapter 4. Deploying hosted control planes

4.1. Deploying hosted control planes on AWS

A hosted cluster is an OpenShift Container Platform cluster with its API endpoint and control plane that are hosted on the management cluster. The hosted cluster includes the control plane and its corresponding data plane. To configure hosted control planes on premises, you must install multicluster engine for Kubernetes Operator in a management cluster. By deploying the HyperShift Operator on an existing managed cluster by using the hypershift-addon managed cluster add-on, you can enable that cluster as a management cluster and start to create the hosted cluster. The hypershift-addon managed cluster add-on is enabled by default for the local-cluster managed cluster.

You can use the multicluster engine Operator console or the hosted control plane command-line interface (CLI), hcp, to create a hosted cluster. The hosted cluster is automatically imported as a managed cluster. However, you can disable this automatic import feature into multicluster engine Operator.

4.1.1. Preparing to deploy hosted control planes on AWS

As you prepare to deploy hosted control planes on Amazon Web Services (AWS), consider the following information:

  • Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it.
  • Do not use clusters as a hosted cluster name.
  • Run the hub cluster and workers on the same platform for hosted control planes.
  • A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster.
4.1.1.1. Prerequisites to configure a management cluster

You must have the following prerequisites to configure the management cluster:

  • You have installed the multicluster engine for Kubernetes Operator 2.5 and later on an OpenShift Container Platform cluster. The multicluster engine Operator is automatically installed when you install Red Hat Advanced Cluster Management (RHACM). The multicluster engine Operator can also be installed without RHACM as an Operator from the OpenShift Container Platform OperatorHub.
  • You have at least one managed OpenShift Container Platform cluster for the multicluster engine Operator. The local-cluster is automatically imported in the multicluster engine Operator version 2.5 and later. You can check the status of your hub cluster by running the following command:

    $ oc get managedclusters local-cluster
  • You have installed the aws command-line interface (CLI).
  • You have installed the hosted control plane CLI, hcp.

4.1.2. Creating the Amazon Web Services S3 bucket and S3 OIDC secret

Before you can create and manage hosted clusters on Amazon Web Services (AWS), you must create the S3 bucket and S3 OIDC secret.

Procedure

  1. Create an S3 bucket that has public access to host OIDC discovery documents for your clusters by running the following commands:

    $ aws s3api create-bucket --bucket <bucket_name> \1
      --create-bucket-configuration LocationConstraint=<region> \2
      --region <region> 3
    1
    Replace <bucket_name> with the name of the S3 bucket you are creating.
    2 3
    To create the bucket in a region other than the us-east-1 region, include this line and replace <region> with the region you want to use. To create a bucket in the us-east-1 region, omit this line.
    $ aws s3api delete-public-access-block --bucket <bucket_name> 1
    1
    Replace <bucket_name> with the name of the S3 bucket you are creating.
    $ echo '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": "*",
                "Action": "s3:GetObject",
                "Resource": "arn:aws:s3:::<bucket_name>/*" 1
            }
        ]
    }' | envsubst > policy.json
    1
    Replace <bucket_name> with the name of the S3 bucket you are creating.
    $ aws s3api put-bucket-policy --bucket <bucket_name> --policy file://policy.json 1
    1
    Replace <bucket_name> with the name of the S3 bucket you are creating.
    Note

    If you are using a Mac computer, you must export the bucket name in order for the policy to work.

  2. Create an OIDC S3 secret named hypershift-operator-oidc-provider-s3-credentials for the HyperShift Operator.
  3. Save the secret in the local-cluster namespace.
  4. See the following table to verify that the secret contains the following fields:

    Table 4.1. Required fields for the AWS secret
    Field nameDescription

    bucket

    Contains an S3 bucket with public access to host OIDC discovery documents for your hosted clusters.

    credentials

    A reference to a file that contains the credentials of the default profile that can access the bucket. By default, HyperShift only uses the default profile to operate the bucket.

    region

    Specifies the region of the S3 bucket.

  5. To create an AWS secret, run the following command:

    $ oc create secret generic <secret_name> --from-file=credentials=<path>/.aws/credentials --from-literal=bucket=<s3_bucket> --from-literal=region=<region> -n local-cluster
    Note

    Disaster recovery backup for the secret is not automatically enabled. To add the label that enables the hypershift-operator-oidc-provider-s3-credentials secret to be backed up for disaster recovery, run the following command:

    $ oc label secret hypershift-operator-oidc-provider-s3-credentials -n local-cluster cluster.open-cluster-management.io/backup=true

4.1.3. Creating a routable public zone for hosted clusters

To access applications in your hosted clusters, you must configure the routable public zone. If the public zone exists, skip this step. Otherwise, the public zone affects the existing functions.

Procedure

  • To create a routable public zone for DNS records, enter the following command:

    $ aws route53 create-hosted-zone --name <basedomain> --caller-reference $(whoami)-$(date --rfc-3339=date) 1
    1
    Replace <basedomain> with your base domain, for example, www.example.com.

4.1.4. Creating an AWS IAM role and STS credentials

Before creating a hosted cluster on Amazon Web Services (AWS), you must create an AWS IAM role and STS credentials.

Procedure

  1. Get the Amazon Resource Name (ARN) of your user by running the following command:

    $ aws sts get-caller-identity --query "Arn" --output text

    Example output

    arn:aws:iam::1234567890:user/<aws_username>

    Use this output as the value for <arn> in the next step.

  2. Create a JSON file that contains the trust relationship configuration for your role. See the following example:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<arn>" 1
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }
    1
    Replace <arn> with the ARN of your user that you noted in the previous step.
  3. Create the Identity and Access Management (IAM) role by running the following command:

    $ aws iam create-role \
    --role-name <name> \1
    --assume-role-policy-document file://<file_name>.json \2
    --query "Role.Arn"
    1
    Replace <name> with the role name, for example, hcp-cli-role.
    2
    Replace <file_name> with the name of the JSON file you created in the previous step.

    Example output

    arn:aws:iam::820196288204:role/myrole

  4. Create a JSON file named policy.json that contains the following permission policies for your role:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "EC2",
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateDhcpOptions",
                    "ec2:DeleteSubnet",
                    "ec2:ReplaceRouteTableAssociation",
                    "ec2:DescribeAddresses",
                    "ec2:DescribeInstances",
                    "ec2:DeleteVpcEndpoints",
                    "ec2:CreateNatGateway",
                    "ec2:CreateVpc",
                    "ec2:DescribeDhcpOptions",
                    "ec2:AttachInternetGateway",
                    "ec2:DeleteVpcEndpointServiceConfigurations",
                    "ec2:DeleteRouteTable",
                    "ec2:AssociateRouteTable",
                    "ec2:DescribeInternetGateways",
                    "ec2:DescribeAvailabilityZones",
                    "ec2:CreateRoute",
                    "ec2:CreateInternetGateway",
                    "ec2:RevokeSecurityGroupEgress",
                    "ec2:ModifyVpcAttribute",
                    "ec2:DeleteInternetGateway",
                    "ec2:DescribeVpcEndpointConnections",
                    "ec2:RejectVpcEndpointConnections",
                    "ec2:DescribeRouteTables",
                    "ec2:ReleaseAddress",
                    "ec2:AssociateDhcpOptions",
                    "ec2:TerminateInstances",
                    "ec2:CreateTags",
                    "ec2:DeleteRoute",
                    "ec2:CreateRouteTable",
                    "ec2:DetachInternetGateway",
                    "ec2:DescribeVpcEndpointServiceConfigurations",
                    "ec2:DescribeNatGateways",
                    "ec2:DisassociateRouteTable",
                    "ec2:AllocateAddress",
                    "ec2:DescribeSecurityGroups",
                    "ec2:RevokeSecurityGroupIngress",
                    "ec2:CreateVpcEndpoint",
                    "ec2:DescribeVpcs",
                    "ec2:DeleteSecurityGroup",
                    "ec2:DeleteDhcpOptions",
                    "ec2:DeleteNatGateway",
                    "ec2:DescribeVpcEndpoints",
                    "ec2:DeleteVpc",
                    "ec2:CreateSubnet",
                    "ec2:DescribeSubnets"
                ],
                "Resource": "*"
            },
            {
                "Sid": "ELB",
                "Effect": "Allow",
                "Action": [
                    "elasticloadbalancing:DeleteLoadBalancer",
                    "elasticloadbalancing:DescribeLoadBalancers",
                    "elasticloadbalancing:DescribeTargetGroups",
                    "elasticloadbalancing:DeleteTargetGroup"
                ],
                "Resource": "*"
            },
            {
                "Sid": "IAMPassRole",
                "Effect": "Allow",
                "Action": "iam:PassRole",
                "Resource": "arn:*:iam::*:role/*-worker-role",
                "Condition": {
                    "ForAnyValue:StringEqualsIfExists": {
                        "iam:PassedToService": "ec2.amazonaws.com"
                    }
                }
            },
            {
                "Sid": "IAM",
                "Effect": "Allow",
                "Action": [
                    "iam:CreateInstanceProfile",
                    "iam:DeleteInstanceProfile",
                    "iam:GetRole",
                    "iam:UpdateAssumeRolePolicy",
                    "iam:GetInstanceProfile",
                    "iam:TagRole",
                    "iam:RemoveRoleFromInstanceProfile",
                    "iam:CreateRole",
                    "iam:DeleteRole",
                    "iam:PutRolePolicy",
                    "iam:AddRoleToInstanceProfile",
                    "iam:CreateOpenIDConnectProvider",
                    "iam:ListOpenIDConnectProviders",
                    "iam:DeleteRolePolicy",
                    "iam:UpdateRole",
                    "iam:DeleteOpenIDConnectProvider",
                    "iam:GetRolePolicy"
                ],
                "Resource": "*"
            },
            {
                "Sid": "Route53",
                "Effect": "Allow",
                "Action": [
                    "route53:ListHostedZonesByVPC",
                    "route53:CreateHostedZone",
                    "route53:ListHostedZones",
                    "route53:ChangeResourceRecordSets",
                    "route53:ListResourceRecordSets",
                    "route53:DeleteHostedZone",
                    "route53:AssociateVPCWithHostedZone",
                    "route53:ListHostedZonesByName"
                ],
                "Resource": "*"
            },
            {
                "Sid": "S3",
                "Effect": "Allow",
                "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:ListBucket",
                    "s3:DeleteObject",
                    "s3:DeleteBucket"
                ],
                "Resource": "*"
            }
        ]
    }
  5. Attach the policy.json file to your role by running the following command:

    $ aws iam put-role-policy \
      --role-name <role_name> \1
      --policy-name <policy_name> \2
      --policy-document file://policy.json 3
    1
    Replace <role_name> with the name of your role.
    2
    Replace <policy_name> with your policy name.
    3
    The policy.json file contains the permission policies for your role.
  6. Retrieve STS credentials in a JSON file named sts-creds.json by running the following command:

    $ aws sts get-session-token --output json > sts-creds.json

    Example sts-creds.json file

    {
                  "Credentials": {
                      "AccessKeyId": "ASIA1443CE0GN2ATHWJU",
                      "SecretAccessKey": "XFLN7cZ5AP0d66KhyI4gd8Mu0UCQEDN9cfelW1”,
                      "SessionToken": "IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMiJHMEUCIDyipkM7oPKBHiGeI0pMnXst1gDLfs/TvfskXseKCbshAiEAnl1l/Html7Iq9AEIqf////KQburfkq4A3TuppHMr/9j1TgCj1z83SO261bHqlJUazKoy7vBFR/a6LHt55iMBqtKPEsIWjBgj/jSdRJI3j4Gyk1//luKDytcfF/tb9YrxDTPLrACS1lqAxSIFZ82I/jDhbDs=",
                      "Expiration": "2025-05-16T04:19:32+00:00"
                  }
              }

4.1.5. Enabling external DNS for hosted control planes on AWS

The control plane and the data plane are separate in hosted control planes. You can configure DNS in two independent areas:

  • Ingress for workloads within the hosted cluster, such as the following domain: *.apps.service-consumer-domain.com.
  • Ingress for service endpoints within the management cluster, such as API or OAuth endpoints through the service provider domain: *.service-provider-domain.com.

The input for hostedCluster.spec.dns manages the ingress for workloads within the hosted cluster. The input for hostedCluster.spec.services.servicePublishingStrategy.route.hostname manages the ingress for service endpoints within the management cluster.

External DNS creates name records for hosted cluster Services that specify a publishing type of LoadBalancer or Route and provide a hostname for that publishing type. For hosted clusters with Private or PublicAndPrivate endpoint access types, only the APIServer and OAuth services support hostnames. For Private hosted clusters, the DNS record resolves to a private IP address of a Virtual Private Cloud (VPC) endpoint in your VPC.

A hosted control plane exposes the following services:

  • APIServer
  • OIDC

You can expose these services by using the servicePublishingStrategy field in the HostedCluster specification. By default, for the LoadBalancer and Route types of servicePublishingStrategy, you can publish the service in one of the following ways:

  • By using the hostname of the load balancer that is in the status of the Service with the LoadBalancer type.
  • By using the status.host field of the Route resource.

However, when you deploy hosted control planes in a managed service context, those methods can expose the ingress subdomain of the underlying management cluster and limit options for the management cluster lifecycle and disaster recovery.

When a DNS indirection is layered on the LoadBalancer and Route publishing types, a managed service operator can publish all public hosted cluster services by using a service-level domain. This architecture allows remapping on the DNS name to a new LoadBalancer or Route and does not expose the ingress domain of the management cluster. Hosted control planes uses external DNS to achieve that indirection layer.

You can deploy external-dns alongside the HyperShift Operator in the hypershift namespace of the management cluster. External DNS watches for Services or Routes that have the external-dns.alpha.kubernetes.io/hostname annotation. That annotation is used to create a DNS record that points to the Service, such as a record, or the Route, such as a CNAME record.

You can use external DNS on cloud environments only. For the other environments, you need to manually configure DNS and services.

For more information about external DNS, see external DNS.

4.1.5.1. Prerequisites

Before you can set up external DNS for hosted control planes on Amazon Web Services (AWS), you must meet the following prerequisites:

  • You created an external public domain.
  • You have access to the AWS Route53 Management console.
4.1.5.2. Setting up external DNS for hosted control planes

You can provision hosted control planes with external DNS or service-level DNS.

  1. Create an Amazon Web Services (AWS) credential secret for the HyperShift Operator and name it hypershift-operator-external-dns-credentials in the local-cluster namespace.
  2. See the following table to verify that the secret has the required fields:

    Table 4.2. Required fields for the AWS secret
    Field nameDescription

    Optional or required

    provider

    The DNS provider that manages the service-level DNS zone.

    Required

    domain-filter

    The service-level domain.

    Required

    credentials

    The credential file that supports all external DNS types.

    Optional when you use AWS keys

    aws-access-key-id

    The credential access key id.

    Optional when you use the AWS DNS service

    aws-secret-access-key

    The credential access key secret.

    Optional when you use the AWS DNS service

  3. To create an AWS secret, run the following command:

    $ oc create secret generic <secret_name> --from-literal=provider=aws --from-literal=domain-filter=<domain_name> --from-file=credentials=<path_to_aws_credentials_file> -n local-cluster
    Note

    Disaster recovery backup for the secret is not automatically enabled. To back up the secret for disaster recovery, add the hypershift-operator-external-dns-credentials by entering the following command:

    $ oc label secret hypershift-operator-external-dns-credentials -n local-cluster cluster.open-cluster-management.io/backup=""
4.1.5.3. Creating the public DNS hosted zone

The External DNS Operator uses the public DNS hosted zone to create your public hosted cluster.

You can create the public DNS hosted zone to use as the external DNS domain-filter. Complete the following steps in the AWS Route 53 management console.

Procedure

  1. In the Route 53 management console, click Create hosted zone.
  2. On the Hosted zone configuration page, type a domain name, verify that Publish hosted zone is selected as the type, and click Create hosted zone.
  3. After the zone is created, on the Records tab, note the values in the Value/Route traffic to column.
  4. In the main domain, create an NS record to redirect the DNS requests to the delegated zone. In the Value field, enter the values that you noted in the previous step.
  5. Click Create records.
  6. Verify that the DNS hosted zone is working by creating a test entry in the new subzone and testing it with a dig command, such as in the following example:

    $ dig +short test.user-dest-public.aws.kerberos.com

    Example output

    192.168.1.1

  7. To create a hosted cluster that sets the hostname for the LoadBalancer and Route services, enter the following command:

    $ hcp create cluster aws --name=<hosted_cluster_name> --endpoint-access=PublicAndPrivate --external-dns-domain=<public_hosted_zone> ... 1
    1
    Replace <public_hosted_zone> with the public hosted zone that you created.

    Example services block for the hosted cluster

      platform:
        aws:
          endpointAccess: PublicAndPrivate
    ...
      services:
      - service: APIServer
        servicePublishingStrategy:
          route:
            hostname: api-example.service-provider-domain.com
          type: Route
      - service: OAuthServer
        servicePublishingStrategy:
          route:
            hostname: oauth-example.service-provider-domain.com
          type: Route
      - service: Konnectivity
        servicePublishingStrategy:
          type: Route
      - service: Ignition
        servicePublishingStrategy:
          type: Route

The Control Plane Operator creates the Services and Routes resources and annotates them with the external-dns.alpha.kubernetes.io/hostname annotation. For Services and Routes, the Control Plane Operator uses a value of the hostname parameter in the servicePublishingStrategy field for the service endpoints. To create the DNS records, you can use a mechanism, such as the external-dns deployment.

You can configure service-level DNS indirection for public services only. You cannot set hostname for private services because they use the hypershift.local private zone.

The following table shows when it is valid to set hostname for a service and endpoint combinations:

Table 4.3. Service and endpoint combinations to set hostname
ServicePublic

PublicAndPrivate

Private

APIServer

Y

Y

N

OAuthServer

Y

Y

N

Konnectivity

Y

N

N

Ignition

Y

N

N

4.1.5.4. Creating a hosted cluster by using the external DNS on AWS

To create a hosted cluster by using the PublicAndPrivate or Public publishing strategy on Amazon Web Services (AWS), you must have the following artifacts configured in your management cluster:

  • The public DNS hosted zone
  • The External DNS Operator
  • The HyperShift Operator

You can deploy a hosted cluster, by using the hcp command-line interface (CLI).

Procedure

  1. To access your management cluster, enter the following command:

    $ export KUBECONFIG=<path_to_management_cluster_kubeconfig>
  2. Verify that the External DNS Operator is running by entering the following command:

    $ oc get pod -n hypershift -lapp=external-dns

    Example output

    NAME                            READY   STATUS    RESTARTS   AGE
    external-dns-7c89788c69-rn8gp   1/1     Running   0          40s

  3. To create a hosted cluster by using external DNS, enter the following command:

    $ hcp create cluster aws \
        --role-arn <arn_role> \ 1
        --instance-type <instance_type> \ 2
        --region <region> \ 3
        --auto-repair \
        --generate-ssh \
        --name <hosted_cluster_name> \ 4
        --namespace clusters \
        --base-domain <service_consumer_domain> \ 5
        --node-pool-replicas <node_replica_count> \ 6
        --pull-secret <path_to_your_pull_secret> \ 7
        --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 8
        --external-dns-domain=<service_provider_domain> \ 9
        --endpoint-access=PublicAndPrivate 10
        --sts-creds <path_to_sts_credential_file> 11
    1
    Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.
    2
    Specify the instance type, for example, m6i.xlarge.
    3
    Specify the AWS region, for example, us-east-1.
    4
    Specify your hosted cluster name, for example, my-external-aws.
    5
    Specify the public hosted zone that the service consumer owns, for example, service-consumer-domain.com.
    6
    Specify the node replica count, for example, 2.
    7
    Specify the path to your pull secret file.
    8
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.
    9
    Specify the public hosted zone that the service provider owns, for example, service-provider-domain.com.
    10
    Set as PublicAndPrivate. You can use external DNS with Public or PublicAndPrivate configurations only.
    11
    Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

4.1.7. Creating a hosted cluster on AWS

You can create a hosted cluster on Amazon Web Services (AWS) by using the hcp command-line interface (CLI).

By default for hosted control planes on Amazon Web Services (AWS), you use an AMD64 hosted cluster. However, you can enable hosted control planes to run on an ARM64 hosted cluster. For more information, see "Running hosted clusters on an ARM64 architecture".

For compatible combinations of node pools and hosted clusters, see the following table:

Table 4.5. Compatible architectures for node pools and hosted clusters
Hosted clusterNode pools

AMD64

AMD64 or ARM64

ARM64

ARM64 or AMD64

Prerequisites

  • You have set up the hosted control plane CLI, hcp.
  • You have enabled the local-cluster managed cluster as the management cluster.
  • You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials.

Procedure

  1. To create a hosted cluster on AWS, run the following command:

    $ hcp create cluster aws \
        --name <hosted_cluster_name> \1
        --infra-id <infra_id> \2
        --base-domain <basedomain> \3
        --sts-creds <path_to_sts_credential_file> \4
        --pull-secret <path_to_pull_secret> \5
        --region <region> \6
        --generate-ssh \
        --node-pool-replicas <node_pool_replica_count> \7
        --namespace <hosted_cluster_namespace> \8
        --role-arn <role_name> \9
        --render-into <file_name>.yaml 10
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify your infrastructure name. You must provide the same value for <hosted_cluster_name> and <infra_id>. Otherwise the cluster might not appear correctly in the multicluster engine for Kubernetes Operator console.
    3
    Specify your base domain, for example, example.com.
    4
    Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.
    5
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    6
    Specify the AWS region name, for example, us-east-1.
    7
    Specify the node pool replica count, for example, 3.
    8
    By default, all HostedCluster and NodePool custom resources are created in the clusters namespace. You can use the --namespace <namespace> parameter, to create the HostedCluster and NodePool custom resources in a specific namespace.
    9
    Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.
    10
    If you want to indicate whether the EC2 instance runs on shared or single tenant hardware, include this field. The --render-into flag renders Kubernetes resources into the YAML file that you specify in this field. Then, continue to the next step to edit the YAML file.
  2. If you included the --render-into flag in the previous command, edit the specified YAML file. Edit the NodePool specification in the YAML file to indicate whether the EC2 instance should run on shared or single-tenant hardware, similar to the following example:

    Example YAML file

    apiVersion: hypershift.openshift.io/v1beta1
    kind: NodePool
    metadata:
      name: <nodepool_name> 1
    spec:
      platform:
        aws:
          placement:
            tenancy: "default" 2

    1
    Specify the name of the NodePool resource.
    2
    Specify a valid value for tenancy: "default", "dedicated", or "host". Use "default" when node pool instances run on shared hardware. Use "dedicated" when each node pool instance runs on single-tenant hardware. Use "host" when node pool instances run on your pre-allocated dedicated hosts.

Verification

  1. Verify the status of your hosted cluster to check that the value of AVAILABLE is True. Run the following command:

    $ oc get hostedclusters -n <hosted_cluster_namespace>
  2. Get a list of your node pools by running the following command:

    $ oc get nodepools --namespace <hosted_cluster_namespace>
4.1.7.1. Accessing a hosted cluster on AWS by using the kubeadmin credentials

After creating a hosted cluster on Amazon Web Services (AWS), you can access a hosted cluster by getting the kubeconfig file, access secrets, and the kubeadmin credentials.

The hosted cluster namespace contains hosted cluster resources and the access secrets. The hosted control plane runs in the hosted control plane namespace.

The secret name formats are as follows:

  • The kubeconfig secret: <hosted_cluster_namespace>-<name>-admin-kubeconfig. For example, clusters-hypershift-demo-admin-kubeconfig.
  • The kubeadmin password secret: <hosted_cluster_namespace>-<name>-kubeadmin-password. For example, clusters-hypershift-demo-kubeadmin-password.
Note

The kubeadmin password secret is Base64-encoded and the kubeconfig secret contains a Base64-encoded kubeconfig configuration. You must decode the Base64-encoded kubeconfig configuration and save it into a <hosted_cluster_name>.kubeconfig file.

Procedure

  • Use your <hosted_cluster_name>.kubeconfig file that contains the decoded kubeconfig configuration to access the hosted cluster. Enter the following command:

    $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

    You must decode the kubeadmin password secret to log in to the API server or the console of the hosted cluster.

4.1.7.2. Accessing a hosted cluster on AWS by using the hcp CLI

You can access the hosted cluster by using the hcp command-line interface (CLI).

Procedure

  1. Generate the kubeconfig file by entering the following command:

    $ hcp create kubeconfig --namespace <hosted_cluster_namespace> --name <hosted_cluster_name> > <hosted_cluster_name>.kubeconfig
  2. After you save the kubeconfig file, access the hosted cluster by entering the following command:

    $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

4.1.8. Creating a hosted cluster in multiple zones on AWS

You can create a hosted cluster in multiple zones on Amazon Web Services (AWS) by using the hcp command-line interface (CLI).

Prerequisites

  • You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials.

Procedure

  • Create a hosted cluster in multiple zones on AWS by running the following command:

    $ hcp create cluster aws \
      --name <hosted_cluster_name> \1
      --node-pool-replicas=<node_pool_replica_count> \2
      --base-domain <basedomain> \3
      --pull-secret <path_to_pull_secret> \4
      --role-arn <arn_role> \5
      --region <region> \6
      --zones <zones> \7
      --sts-creds <path_to_sts_credential_file> 8
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the node pool replica count, for example, 2.
    3
    Specify your base domain, for example, example.com.
    4
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    5
    Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.
    6
    Specify the AWS region name, for example, us-east-1.
    7
    Specify availability zones within your AWS region, for example, us-east-1a, and us-east-1b.
    8
    Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

For each specified zone, the following infrastructure is created:

  • Public subnet
  • Private subnet
  • NAT gateway
  • Private route table

A public route table is shared across public subnets.

One NodePool resource is created for each zone. The node pool name is suffixed by the zone name. The private subnet for zone is set in spec.platform.aws.subnet.id.

4.1.8.1. Creating a hosted cluster by providing AWS STS credentials

When you create a hosted cluster by using the hcp create cluster aws command, you must provide an Amazon Web Services (AWS) account credentials that have permissions to create infrastructure resources for your hosted cluster.

Infrastructure resources include the following examples:

  • Virtual Private Cloud (VPC)
  • Subnets
  • Network address translation (NAT) gateways

You can provide the AWS credentials by using the either of the following ways:

  • The AWS Security Token Service (STS) credentials
  • The AWS cloud provider secret from multicluster engine Operator

Procedure

  • To create a hosted cluster on AWS by providing AWS STS credentials, enter the following command:

    $ hcp create cluster aws \
      --name <hosted_cluster_name> \1
      --node-pool-replicas <node_pool_replica_count> \2
      --base-domain <basedomain> \3
      --pull-secret <path_to_pull_secret> \4
      --sts-creds <path_to_sts_credential_file> \5
      --region <region> \6
      --role-arn <arn_role>  7
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the node pool replica count, for example, 2.
    3
    Specify your base domain, for example, example.com.
    4
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    5
    Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.
    6
    Specify the AWS region name, for example, us-east-1.
    7
    Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.

4.1.9. Running hosted clusters on an ARM64 architecture

By default for hosted control planes on Amazon Web Services (AWS), you use an AMD64 hosted cluster. However, you can enable hosted control planes to run on an ARM64 hosted cluster.

For compatible combinations of node pools and hosted clusters, see the following table:

Table 4.6. Compatible architectures for node pools and hosted clusters
Hosted clusterNode pools

AMD64

AMD64 or ARM64

ARM64

ARM64 or AMD64

4.1.9.1. Creating a hosted cluster on an ARM64 OpenShift Container Platform cluster

You can run a hosted cluster on an ARM64 OpenShift Container Platform cluster for Amazon Web Services (AWS) by overriding the default release image with a multi-architecture release image.

If you do not use a multi-architecture release image, the compute nodes in the node pool are not created and reconciliation of the node pool stops until you either use a multi-architecture release image in the hosted cluster or update the NodePool custom resource based on the release image.

Prerequisites

  • You must have an OpenShift Container Platform cluster with a 64-bit ARM infrastructure that is installed on AWS. For more information, see Create an OpenShift Container Platform Cluster: AWS (ARM).
  • You must create an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials. For more information, see "Creating an AWS IAM role and STS credentials".

Procedure

  1. Create a hosted cluster on an ARM64 OpenShift Container Platform cluster by entering the following command:

    $ hcp create cluster aws \
      --name <hosted_cluster_name> \1
      --node-pool-replicas <node_pool_replica_count> \2
      --base-domain <basedomain> \3
      --pull-secret <path_to_pull_secret> \4
      --sts-creds <path_to_sts_credential_file> \5
      --region <region> \6
      --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \7
      --role-arn <role_name> 8
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the node pool replica count, for example, 3.
    3
    Specify your base domain, for example, example.com.
    4
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    5
    Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.
    6
    Specify the AWS region name, for example, us-east-1.
    7
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see "Extracting the OpenShift Container Platform release image digest".
    8
    Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.
  2. Add a NodePool object to the hosted cluster by running the following command:

    $ hcp create nodepool aws \
      --cluster-name <hosted_cluster_name> \1
      --name <nodepool_name> \2
      --node-count <node_pool_replica_count> 3
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the node pool name.
    3
    Specify the node pool replica count, for example, 3.
4.1.9.2. Creating an ARM or AMD NodePool object on AWS hosted clusters

You can schedule application workloads that is the NodePool objects on 64-bit ARM and AMD from the same hosted control plane. You can define the arch field in the NodePool specification to set the required processor architecture for the NodePool object. The valid values for the arch field are as follows:

  • arm64
  • amd64

Prerequisites

Procedure

  • Add an ARM or AMD NodePool object to the hosted cluster on AWS by running the following command:

    $ hcp create nodepool aws \
      --cluster-name <hosted_cluster_name> \1
      --name <node_pool_name> \2
      --node-count <node_pool_replica_count> \3
      --arch <architecture> 4
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the node pool name.
    3
    Specify the node pool replica count, for example, 3.
    4
    Specify the architecture type, such as arm64 or amd64. If you do not specify a value for the --arch flag, the amd64 value is used by default.

4.1.10. Creating a private hosted cluster on AWS

After you enable the local-cluster as the hosting cluster, you can deploy a hosted cluster or a private hosted cluster on Amazon Web Services (AWS).

By default, hosted clusters are publicly accessible through public DNS and the default router for the management cluster.

For private clusters on AWS, all communication with the hosted cluster occurs over AWS PrivateLink.

Prerequisites

  • You enabled AWS PrivateLink. For more information, see "Enabling AWS PrivateLink".
  • You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials. For more information, see "Creating an AWS IAM role and STS credentials" and "Identity and Access Management (IAM) permissions".
  • You configured a bastion instance on AWS.

Procedure

  • Create a private hosted cluster on AWS by entering the following command:

    $ hcp create cluster aws \
      --name <hosted_cluster_name> \1
      --node-pool-replicas=<node_pool_replica_count> \2
      --base-domain <basedomain> \3
      --pull-secret <path_to_pull_secret> \4
      --sts-creds <path_to_sts_credential_file> \5
      --region <region> \6
      --endpoint-access Private \7
      --role-arn <role_name> 8
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the node pool replica count, for example, 3.
    3
    Specify your base domain, for example, example.com.
    4
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    5
    Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.
    6
    Specify the AWS region name, for example, us-east-1.
    7
    Defines whether a cluster is public or private.
    8
    Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole. For more information about ARN roles, see "Identity and Access Management (IAM) permissions".

    The following API endpoints for the hosted cluster are accessible through a private DNS zone:

  • api.<hosted_cluster_name>.hypershift.local
  • *.apps.<hosted_cluster_name>.hypershift.local
4.1.10.1. Accessing a private management cluster on AWS

Additional resources

You can access your private management cluster by using the command-line interface (CLI).

Procedure

  1. Find the private IPs of nodes by entering the following command:

    $ aws ec2 describe-instances --filter="Name=tag:kubernetes.io/cluster/<infra_id>,Values=owned" | jq '.Reservations[] | .Instances[] | select(.PublicDnsName=="") | .PrivateIpAddress'
  2. Create a kubeconfig file for the hosted cluster that you can copy to a node by entering the following command:

    $ hcp create kubeconfig > <hosted_cluster_kubeconfig>
  3. To SSH into one of the nodes through the bastion, enter the following command:

    $ ssh -o ProxyCommand="ssh ec2-user@<bastion_ip> -W %h:%p" core@<node_ip>
  4. From the SSH shell, copy the kubeconfig file contents to a file on the node by entering the following command:

    $ mv <path_to_kubeconfig_file> <new_file_name>
  5. Export the kubeconfig file by entering the following command:

    $ export KUBECONFIG=<path_to_kubeconfig_file>
  6. Observe the hosted cluster status by entering the following command:

    $ oc get clusteroperators clusterversion

4.2. Deploying hosted control planes on bare metal

You can deploy hosted control planes by configuring a cluster to function as a management cluster. The management cluster is the OpenShift Container Platform cluster where the control planes are hosted. In some contexts, the management cluster is also known as the hosting cluster.

Note

The management cluster is not the same thing as the managed cluster. A managed cluster is a cluster that the hub cluster manages.

The hosted control planes feature is enabled by default.

The multicluster engine Operator supports only the default local-cluster, which is a hub cluster that is managed, and the hub cluster as the management cluster. If you have Red Hat Advanced Cluster Management installed, you can use the managed hub cluster, also known as the local-cluster, as the management cluster.

A hosted cluster is an OpenShift Container Platform cluster with its API endpoint and control plane that are hosted on the management cluster. The hosted cluster includes the control plane and its corresponding data plane. You can use the multicluster engine Operator console or the hosted control plane command line interface, hcp, to create a hosted cluster.

The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see Disabling the automatic import of hosted clusters into multicluster engine Operator.

4.2.1. Preparing to deploy hosted control planes on bare metal

As you prepare to deploy hosted control planes on bare metal, consider the following information:

  • Run the hub cluster and workers on the same platform for hosted control planes.
  • All bare metal hosts require a manual start with a Discovery Image ISO that the central infrastructure management provides. You can start the hosts manually or through automation by using Cluster-Baremetal-Operator. After each host starts, it runs an Agent process to discover the host details and complete the installation. An Agent custom resource represents each host.
  • When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control plane etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see Recommended etcd practices and Persistent storage using logical volume manager storage.
4.2.1.1. Prerequisites to configure a management cluster
  • You need the multicluster engine for Kubernetes Operator 2.2 and later installed on an OpenShift Container Platform cluster. You can install multicluster engine Operator as an Operator from the OpenShift Container Platform OperatorHub.
  • The multicluster engine Operator must have at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported in multicluster engine Operator 2.2 and later. For more information about the local-cluster, see Advanced configuration in the Red Hat Advanced Cluster Management documentation. You can check the status of your hub cluster by running the following command:

    $ oc get managedclusters local-cluster
  • You must add the topology.kubernetes.io/zone label to your bare metal hosts on your management cluster. Otherwise, all of the hosted control plane pods are scheduled on a single node, causing single point of failure.
  • To provision hosted control planes on bare metal, you can use the Agent platform. The Agent platform uses the central infrastructure management service to add worker nodes to a hosted cluster. For more information, see Enabling the central infrastructure management service.
  • You need to install the hosted control plane command line interface.
4.2.1.2. Bare metal firewall, port, and service requirements

You must meet the firewall, port, and service requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters.

Note

Services run on their default ports. However, if you use the NodePort publishing strategy, services run on the port that is assigned by the NodePort service.

Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary. For production deployments, use a load balancer to simplify access through a single IP address.

If your hub cluster has a proxy configuration, ensure that it can reach the hosted cluster API endpoint by adding all hosted cluster API endpoints to the noProxy field on the Proxy object. For more information, see "Configuring the cluster-wide proxy".

A hosted control plane exposes the following services on bare metal:

  • APIServer

    • The APIServer service runs on port 6443 by default and requires ingress access for communication between the control plane components.
    • If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses.
  • OAuthServer

    • The OAuthServer service runs on port 443 by default when you use the route and ingress to expose the service.
    • If you use the NodePort publishing strategy, use a firewall rule for the OAuthServer service.
  • Konnectivity

    • The Konnectivity service runs on port 443 by default when you use the route and ingress to expose the service.
    • The Konnectivity agent establishes a reverse tunnel to allow the control plane to access the network for the hosted cluster. The agent uses egress to connect to the Konnectivity server. The server is exposed by using either a route on port 443 or a manually assigned NodePort.
    • If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443.
    • If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes.
  • Ignition

    • The Ignition service runs on port 443 by default when you use the route and ingress to expose the service.
    • If you use the NodePort publishing strategy, use a firewall rule for the Ignition service.

You do not need the following services on bare metal:

  • OVNSbDb
  • OIDC
4.2.1.3. Bare metal infrastructure requirements

The Agent platform does not create any infrastructure, but it does have the following requirements for infrastructure:

  • Agents: An Agent represents a host that is booted with a discovery image and is ready to be provisioned as an OpenShift Container Platform node.
  • DNS: The API and ingress endpoints must be routable.

4.2.2. DNS configurations on bare metal

The API Server for the hosted cluster is exposed as a NodePort service. A DNS entry must exist for api.<hosted_cluster_name>.<base_domain> that points to destination where the API Server can be reached.

The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane. The entry can also point to a load balancer that is deployed to redirect incoming traffic to the ingress pods.

Example DNS configuration

api.example.krnl.es.    IN A 192.168.122.20
api.example.krnl.es.    IN A 192.168.122.21
api.example.krnl.es.    IN A 192.168.122.22
api-int.example.krnl.es.    IN A 192.168.122.20
api-int.example.krnl.es.    IN A 192.168.122.21
api-int.example.krnl.es.    IN A 192.168.122.22
`*`.apps.example.krnl.es. IN A 192.168.122.23

If you are configuring DNS for a disconnected environment on an IPv6 network, the configuration looks like the following example.

Example DNS configuration for an IPv6 network

api.example.krnl.es.    IN A 2620:52:0:1306::5
api.example.krnl.es.    IN A 2620:52:0:1306::6
api.example.krnl.es.    IN A 2620:52:0:1306::7
api-int.example.krnl.es.    IN A 2620:52:0:1306::5
api-int.example.krnl.es.    IN A 2620:52:0:1306::6
api-int.example.krnl.es.    IN A 2620:52:0:1306::7
`*`.apps.example.krnl.es. IN A 2620:52:0:1306::10

If you are configuring DNS for a disconnected environment on a dual stack network, be sure to include DNS entries for both IPv4 and IPv6.

Example DNS configuration for a dual stack network

host-record=api-int.hub-dual.dns.base.domain.name,192.168.126.10
host-record=api.hub-dual.dns.base.domain.name,192.168.126.10
address=/apps.hub-dual.dns.base.domain.name/192.168.126.11
dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,192.168.126.20
dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,192.168.126.21
dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,192.168.126.22
dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,192.168.126.25
dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,192.168.126.26

host-record=api-int.hub-dual.dns.base.domain.name,2620:52:0:1306::2
host-record=api.hub-dual.dns.base.domain.name,2620:52:0:1306::2
address=/apps.hub-dual.dns.base.domain.name/2620:52:0:1306::3
dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5]
dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,[2620:52:0:1306::6]
dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,[2620:52:0:1306::7]
dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,[2620:52:0:1306::8]
dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,[2620:52:0:1306::9]

4.2.3. Creating a hosted cluster on bare metal

When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one.

As you create a hosted cluster, keep the following guidelines in mind:

  • Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it.
  • Do not use clusters as a hosted cluster name.
  • A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster.

Procedure

  1. Create the hosted control plane namespace by entering the following command:

    $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

    Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

  2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

    $ hcp create cluster agent \
        --name=<hosted_cluster_name> \1
        --pull-secret=<path_to_pull_secret> \2
        --agent-namespace=<hosted_control_plane_namespace> \3
        --base-domain=<basedomain> \4
        --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
        --etcd-storage-class=<etcd_storage_class> \6
        --ssh-key  <path_to_ssh_public_key> \7
        --namespace <hosted_cluster_namespace> \8
        --control-plane-availability-policy HighlyAvailable \9
        --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    3
    Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.
    4
    Specify your base domain, for example, krnl.es.
    5
    The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.
    6
    Specify the etcd storage class name, for example, lvm-storageclass.
    7
    Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.
    8
    Specify your hosted cluster namespace.
    9
    The default value for the control plane availability policy is HighlyAvailable.
    10
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.
  3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

    $ oc -n <hosted_control_plane_namespace> get pods

    Example output

    NAME                                             READY   STATUS    RESTARTS   AGE
    capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
    catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
    certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
    cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

4.2.3.1. Creating a hosted cluster on bare metal by using the console

To create a hosted cluster by using the console, complete the following steps.

Procedure

  1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials. For instructions to open the console, see Accessing the web console.
  2. In the console header, ensure that All Clusters is selected.
  3. Click Infrastructure → Clusters.
  4. Click Create cluster → Host inventory → Hosted control plane.

    The Create cluster page is displayed.

  5. On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation.

    Note

    As you enter details about the cluster, you might find the following tips useful:

    • If you want to use predefined values to automatically populate fields in the console, you can create a host inventory credential. For more information, see Creating a credential for an on-premises environment.
    • On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a host inventory credential, the pull secret is automatically populated.
    • On the Node pools page, the namespace contains the hosts for the node pool. If you created a host inventory by using the console, the console creates a dedicated namespace.
    • On the Networking page, you select an API server publishing strategy. The API server for the hosted cluster can be exposed either by using an existing load balancer or as a service of the NodePort type. A DNS entry must exist for the api.<hosted_cluster_name>.<base_domain> setting that points to the destination where the API server can be reached. This entry can be a record that points to one of the nodes in the management cluster or a record that points to a load balancer that redirects incoming traffic to the Ingress pods.
  6. Review your entries and click Create.

    The Hosted cluster view is displayed.

  7. Monitor the deployment of the hosted cluster in the Hosted cluster view.
  8. If you do not see information about the hosted cluster, ensure that All Clusters is selected, then click the cluster name.
  9. Wait until the control plane components are ready. This process can take a few minutes.
  10. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Next steps

4.2.3.2. Creating a hosted cluster on bare metal by using a mirror registry

You can use a mirror registry to create a hosted cluster on bare metal by specifying the --image-content-sources flag in the hcp create cluster command.

Procedure

  1. Create a YAML file to define Image Content Source Policies (ICSP). See the following example:

    - mirrors:
      - brew.registry.redhat.io
      source: registry.redhat.io
    - mirrors:
      - brew.registry.redhat.io
      source: registry.stage.redhat.io
    - mirrors:
      - brew.registry.redhat.io
      source: registry-proxy.engineering.redhat.com
  2. Save the file as icsp.yaml. This file contains your mirror registries.
  3. To create a hosted cluster by using your mirror registries, run the following command:

    $ hcp create cluster agent \
        --name=<hosted_cluster_name> \1
        --pull-secret=<path_to_pull_secret> \2
        --agent-namespace=<hosted_control_plane_namespace> \3
        --base-domain=<basedomain> \4
        --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
        --image-content-sources icsp.yaml  \6
        --ssh-key  <path_to_ssh_key> \7
        --namespace <hosted_cluster_namespace> \8
        --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    3
    Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.
    4
    Specify your base domain, for example, krnl.es.
    5
    The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.
    6
    Specify the icsp.yaml file that defines ICSP and your mirror registries.
    7
    Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.
    8
    Specify your hosted cluster namespace.
    9
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

Next steps

4.2.4. Verifying hosted cluster creation

After the deployment process is complete, you can verify that the hosted cluster was created successfully. Follow these steps a few minutes after you create the hosted cluster.

Procedure

  1. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

    $ oc extract -n <hosted-control-plane-namespace> secret/admin-kubeconfig --to=- > kubeconfig-<hosted-cluster-name>
  2. Use the kubeconfig to view the cluster Operators of the hosted cluster. Enter the following command:

    $ oc get co --kubeconfig=kubeconfig-<hosted-cluster-name>

    Example output

    NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    console                                    4.10.26   True        False         False      2m38s
    dns                                        4.10.26   True        False         False      2m52s
    image-registry                             4.10.26   True        False         False      2m8s
    ingress                                    4.10.26   True        False         False      22m

  3. You can also view the running pods on your hosted cluster by entering the following command:

    $ oc get pods -A --kubeconfig=kubeconfig-<hosted-cluster-name>

    Example output

    NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
    kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
    openshift-cluster-node-tuning-operator             tuned-dhw5p                                               1/1     Running            0               109s
    openshift-cluster-storage-operator                 cluster-storage-operator-5f784969f5-vwzgz                 1/1     Running            1 (113s ago)    20m
    openshift-cluster-storage-operator                 csi-snapshot-controller-6b7687b7d9-7nrfw                  1/1     Running            0               3m8s
    openshift-console                                  console-5cbf6c7969-6gk6z                                  1/1     Running            0               119s
    openshift-console                                  downloads-7bcd756565-6wj5j                                1/1     Running            0               4m3s
    openshift-dns-operator                             dns-operator-77d755cd8c-xjfbn                             2/2     Running            0               21m
    openshift-dns                                      dns-default-kfqnh                                         2/2     Running            0               113s

4.3. Deploying hosted control planes on OpenShift Virtualization

With hosted control planes and OpenShift Virtualization, you can create OpenShift Container Platform clusters with worker nodes that are hosted by KubeVirt virtual machines. Hosted control planes on OpenShift Virtualization provides several benefits:

  • Enhances resource usage by packing hosted control planes and hosted clusters in the same underlying bare metal infrastructure
  • Separates hosted control planes and hosted clusters to provide strong isolation
  • Reduces cluster provision time by eliminating the bare metal node bootstrapping process
  • Manages many releases under the same base OpenShift Container Platform cluster

The hosted control planes feature is enabled by default.

You can use the hosted control plane command line interface, hcp, to create an OpenShift Container Platform hosted cluster. The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see "Disabling the automatic import of hosted clusters into multicluster engine operator".

4.3.1. Requirements to deploy hosted control planes on OpenShift Virtualization

As you prepare to deploy hosted control planes on OpenShift Virtualization, consider the following information:

  • Run the hub cluster and workers on the same platform for hosted control planes.
  • Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it.
  • Do not use clusters as a hosted cluster name.
  • A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster.
  • When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control plane etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using Logical Volume Manager storage".
4.3.1.1. Prerequisites

You must meet the following prerequisites to create an OpenShift Container Platform cluster on OpenShift Virtualization:

  • You have administrator access to an OpenShift Container Platform cluster, version 4.14 or later, specified in the KUBECONFIG environment variable.
  • The OpenShift Container Platform management cluster has wildcard DNS routes enabled, as shown in the following DNS:

    $ oc patch ingresscontroller -n openshift-ingress-operator default --type=json -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]'
  • The OpenShift Container Platform management cluster has OpenShift Virtualization, version 4.14 or later, installed on it. For more information, see "Installing OpenShift Virtualization using the web console".
  • The OpenShift Container Platform management cluster is on-premise bare metal.
  • The OpenShift Container Platform management cluster is configured with OVNKubernetes as the default pod network CNI.
  • The OpenShift Container Platform management cluster has a default storage class. For more information, see "Postinstallation storage configuration". The following example shows how to set a default storage class:

    $ oc patch storageclass ocs-storagecluster-ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
  • You have a valid pull secret file for the quay.io/openshift-release-dev repository. For more information, see "Install OpenShift on any x86_64 platform with user-provisioned infrastructure".
  • You have installed the hosted control plane command line interface.
  • You have configured a load balancer. For more information, see "Configuring MetalLB".
  • For optimal network performance, you are using a network maximum transmission unit (MTU) of 9000 or greater on the OpenShift Container Platform cluster that hosts the KubeVirt virtual machines. If you use a lower MTU setting, network latency and the throughput of the hosted pods are affected. Enable multiqueue on node pools only when the MTU is 9000 or greater.
  • The multicluster engine Operator has at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported. For more information about the local-cluster, see "Advanced configuration" in the multicluster engine Operator documentation. You can check the status of your hub cluster by running the following command:

    $ oc get managedclusters local-cluster
  • On the OpenShift Container Platform cluster that hosts the OpenShift Virtualization virtual machines, you are using a ReadWriteMany (RWX) storage class so that live migration can be enabled.
4.3.1.2. Firewall and port requirements

Ensure that you meet the firewall and port requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters:

  • The kube-apiserver service runs on port 6443 by default and requires ingress access for communication between the control plane components.

    • If you use the NodePort publishing strategy, ensure that the node port that is assigned to the kube-apiserver service is exposed.
    • If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses.
  • If you use the NodePort publishing strategy, use a firewall rule for the ignition-server and Oauth-server settings.
  • The konnectivity agent, which establishes a reverse tunnel to allow bi-directional communication on the hosted cluster, requires egress access to the cluster API server address on port 6443. With that egress access, the agent can reach the kube-apiserver service.

    • If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443.
    • If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes.
  • If you change the default port of 6443, adjust the rules to reflect that change.
  • Ensure that you open any ports that are required by the workloads that run in the clusters.
  • Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary.
  • For production deployments, use a load balancer to simplify access through a single IP address.

4.3.2. Live migration for compute nodes

While the management cluster for hosted cluster virtual machines (VMs) is undergoing updates or maintenance, the hosted cluster VMs can be automatically live migrated to prevent disrupting hosted cluster workloads. As a result, the management cluster can be updated without affecting the availability and operation of the KubeVirt platform hosted clusters.

Important

The live migration of KubeVirt VMs is enabled by default provided that the VMs use ReadWriteMany (RWX) storage for both the root volume and the storage classes that are mapped to the kubevirt-csi CSI provider.

You can verify that the VMs in a node pool are capable of live migration by checking the KubeVirtNodesLiveMigratable condition in the status section of a NodePool object.

In the following example, the VMs cannot be live migrated because RWX storage is not used.

Example configuration where VMs cannot be live migrated

    - lastTransitionTime: "2024-10-08T15:38:19Z"
      message: |
        3 of 3 machines are not live migratable
        Machine user-np-ngst4-gw2hz: DisksNotLiveMigratable: user-np-ngst4-gw2hz is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-gw2hz-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
        Machine user-np-ngst4-npq7x: DisksNotLiveMigratable: user-np-ngst4-npq7x is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-npq7x-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
        Machine user-np-ngst4-q5nkb: DisksNotLiveMigratable: user-np-ngst4-q5nkb is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-q5nkb-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
      observedGeneration: 1
      reason: DisksNotLiveMigratable
      status: "False"
      type: KubeVirtNodesLiveMigratable

In the next example, the VMs meet the requirements to be live migrated.

Example configuration where VMs can be live migrated

    - lastTransitionTime: "2024-10-08T15:38:19Z"
      message: "All is well"
      observedGeneration: 1
      reason: AsExpected
      status: "True"
      type: KubeVirtNodesLiveMigratable

While live migration can protect VMs from disruption in normal circumstances, events such as infrastructure node failure can result in a hard restart of any VMs that are hosted on the failed node. For live migration to be successful, the source node that a VM is hosted on must be working correctly.

When the VMs in a node pool cannot be live migrated, workload disruption might occur on the hosted cluster during maintenance on the management cluster. By default, the hosted control planes controllers try to drain the workloads that are hosted on KubeVirt VMs that cannot be live migrated before the VMs are stopped. Draining the hosted cluster nodes before stopping the VMs allows pod disruption budgets to protect workload availability within the hosted cluster.

4.3.3. Creating a hosted cluster with the KubeVirt platform

With OpenShift Container Platform 4.14 and later, you can create a cluster with KubeVirt, to include creating with an external infrastructure.

4.3.3.1. Creating a hosted cluster with the KubeVirt platform by using the CLI

To create a hosted cluster, you can use the hosted control plane command line interface, hcp.

Procedure

  1. Enter the following command:

    $ hcp create cluster kubevirt \
      --name <hosted-cluster-name> \ 1
      --node-pool-replicas <worker-count> \ 2
      --pull-secret <path-to-pull-secret> \ 3
      --memory <value-for-memory> \ 4
      --cores <value-for-cpu> \ 5
      --etcd-storage-class=<etcd-storage-class> 6
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the worker count, for example, 2.
    3
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    4
    Specify a value for memory, for example, 6Gi.
    5
    Specify a value for CPU, for example, 2.
    6
    Specify the etcd storage class name, for example, lvm-storageclass.
    Note

    You can use the --release-image flag to set up the hosted cluster with a specific OpenShift Container Platform release.

    A default node pool is created for the cluster with two virtual machine worker replicas according to the --node-pool-replicas flag.

  2. After a few moments, verify that the hosted control plane pods are running by entering the following command:

    $ oc -n clusters-<hosted-cluster-name> get pods

    Example output

    NAME                                                  READY   STATUS    RESTARTS   AGE
    capi-provider-5cc7b74f47-n5gkr                        1/1     Running   0          3m
    catalog-operator-5f799567b7-fd6jw                     2/2     Running   0          69s
    certified-operators-catalog-784b9899f9-mrp6p          1/1     Running   0          66s
    cluster-api-6bbc867966-l4dwl                          1/1     Running   0          66s
    .
    .
    .
    redhat-operators-catalog-9d5fd4d44-z8qqk              1/1     Running   0          66s

    A hosted cluster that has worker nodes that are backed by KubeVirt virtual machines typically takes 10-15 minutes to be fully provisioned.

  3. To check the status of the hosted cluster, see the corresponding HostedCluster resource by entering the following command:

    $ oc get --namespace clusters hostedclusters

    See the following example output, which illustrates a fully provisioned HostedCluster object:

    NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
    clusters    example   4.x.0     example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

    Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

4.3.3.2. Creating a hosted cluster with the KubeVirt platform by using external infrastructure

By default, the HyperShift Operator hosts both the control plane pods of the hosted cluster and the KubeVirt worker VMs within the same cluster. With the external infrastructure feature, you can place the worker node VMs on a separate cluster from the control plane pods.

  • The management cluster is the OpenShift Container Platform cluster that runs the HyperShift Operator and hosts the control plane pods for a hosted cluster.
  • The infrastructure cluster is the OpenShift Container Platform cluster that runs the KubeVirt worker VMs for a hosted cluster.
  • By default, the management cluster also acts as the infrastructure cluster that hosts VMs. However, for external infrastructure, the management and infrastructure clusters are different.

Prerequisites

  • You must have a namespace on the external infrastructure cluster for the KubeVirt nodes to be hosted in.
  • You must have a kubeconfig file for the external infrastructure cluster.

Procedure

You can create a hosted cluster by using the hcp command line interface.

  • To place the KubeVirt worker VMs on the infrastructure cluster, use the --infra-kubeconfig-file and --infra-namespace arguments, as shown in the following example:

    $ hcp create cluster kubevirt \
      --name <hosted-cluster-name> \ 1
      --node-pool-replicas <worker-count> \ 2
      --pull-secret <path-to-pull-secret> \ 3
      --memory <value-for-memory> \ 4
      --cores <value-for-cpu> \ 5
      --infra-namespace=<hosted-cluster-namespace>-<hosted-cluster-name> \ 6
      --infra-kubeconfig-file=<path-to-external-infra-kubeconfig> 7
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the worker count, for example, 2.
    3
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    4
    Specify a value for memory, for example, 6Gi.
    5
    Specify a value for CPU, for example, 2.
    6
    Specify the infrastructure namespace, for example, clusters-example.
    7
    Specify the path to your kubeconfig file for the infrastructure cluster, for example, /user/name/external-infra-kubeconfig.

    After you enter that command, the control plane pods are hosted on the management cluster that the HyperShift Operator runs on, and the KubeVirt VMs are hosted on a separate infrastructure cluster.

4.3.3.3. Creating a hosted cluster by using the console

To create a hosted cluster with the KubeVirt platform by using the console, complete the following steps.

Procedure

  1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials.
  2. In the console header, ensure that All Clusters is selected.
  3. Click Infrastructure > Clusters.
  4. Click Create cluster > Red Hat OpenShift Virtualization > Hosted.
  5. On the Create cluster page, follow the prompts to enter details about the cluster and node pools.

    Note
    • If you want to use predefined values to automatically populate fields in the console, you can create a OpenShift Virtualization credential. For more information, see Creating a credential for an on-premises environment.
    • On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a OpenShift Virtualization credential, the pull secret is automatically populated.
  6. Review your entries and click Create.

    The Hosted cluster view is displayed.

  7. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, and click the cluster name.
  8. Wait until the control plane components are ready. This process can take a few minutes.
  9. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Additional resources

4.3.4. Configuring the default ingress and DNS for hosted control planes on OpenShift Virtualization

Every OpenShift Container Platform cluster includes a default application Ingress Controller, which must have an wildcard DNS record associated with it. By default, hosted clusters that are created by using the HyperShift KubeVirt provider automatically become a subdomain of the OpenShift Container Platform cluster that the KubeVirt virtual machines run on.

For example, your OpenShift Container Platform cluster might have the following default ingress DNS entry:

*.apps.mgmt-cluster.example.com

As a result, a KubeVirt hosted cluster that is named guest and that runs on that underlying OpenShift Container Platform cluster has the following default ingress:

*.apps.guest.apps.mgmt-cluster.example.com

Procedure

For the default ingress DNS to work properly, the cluster that hosts the KubeVirt virtual machines must allow wildcard DNS routes.

  • You can configure this behavior by entering the following command:

    $ oc patch ingresscontroller -n openshift-ingress-operator default --type=json -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]'
Note

When you use the default hosted cluster ingress, connectivity is limited to HTTPS traffic over port 443. Plain HTTP traffic over port 80 is rejected. This limitation applies to only the default ingress behavior.

4.3.5. Customizing ingress and DNS behavior

If you do not want to use the default ingress and DNS behavior, you can configure a KubeVirt hosted cluster with a unique base domain at creation time. This option requires manual configuration steps during creation and involves three main steps: cluster creation, load balancer creation, and wildcard DNS configuration.

4.3.5.1. Deploying a hosted cluster that specifies the base domain

To create a hosted cluster that specifies a base domain, complete the following steps.

Procedure

  1. Enter the following command:

    $ hcp create cluster kubevirt \
      --name <hosted_cluster_name> \ 1
      --node-pool-replicas <worker_count> \ 2
      --pull-secret <path_to_pull_secret> \ 3
      --memory <value_for_memory> \ 4
      --cores <value_for_cpu> \ 5
      --base-domain <basedomain> 6
    1
    Specify the name of your hosted cluster.
    2
    Specify the worker count, for example, 2.
    3
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    4
    Specify a value for memory, for example, 6Gi.
    5
    Specify a value for CPU, for example, 2.
    6
    Specify the base domain, for example, hypershift.lab.

    As a result, the hosted cluster has an ingress wildcard that is configured for the cluster name and the base domain, for example, .apps.example.hypershift.lab. The hosted cluster remains in Partial status because after you create a hosted cluster with unique base domain, you must configure the required DNS records and load balancer.

  2. View the status of your hosted cluster by entering the following command:

    $ oc get --namespace clusters hostedclusters

    Example output

    NAME            VERSION   KUBECONFIG                       PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
    example                   example-admin-kubeconfig         Partial    True        False         The hosted control plane is available

  3. Access the cluster by entering the following commands:

    $ hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig
    $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get co

    Example output

    NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    console                                    4.x.0     False       False         False      30m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.example.hypershift.lab): Get "https://console-openshift-console.apps.example.hypershift.lab": dial tcp: lookup console-openshift-console.apps.example.hypershift.lab on 172.31.0.10:53: no such host
    ingress                                    4.x.0     True        False         True       28m     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

    Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

Next steps

To fix the errors in the output, complete the steps in "Setting up the load balancer" and "Setting up a wildcard DNS".

Note

If your hosted cluster is on bare metal, you might need MetalLB to set up load balancer services. For more information, see "Configuring MetalLB".

4.3.5.2. Setting up the load balancer

Set up the load balancer service that routes ingress traffic to the KubeVirt VMs and assigns a wildcard DNS entry to the load balancer IP address.

Procedure

  1. A NodePort service that exposes the hosted cluster ingress already exists. You can export the node ports and create the load balancer service that targets those ports.

    1. Get the HTTP node port by entering the following command:

      $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get services -n openshift-ingress router-nodeport-default -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}'

      Note the HTTP node port value to use in the next step.

    2. Get the HTTPS node port by entering the following command:

      $ oc --kubeconfig <hosted_cluster_name>-kubeconfig get services -n openshift-ingress router-nodeport-default -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}'

      Note the HTTPS node port value to use in the next step.

  2. Create the load balancer service by entering the following command:

    oc apply -f -
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: <hosted_cluster_name>
      name: <hosted_cluster_name>-apps
      namespace: clusters-<hosted_cluster_name>
    spec:
      ports:
      - name: https-443
        port: 443
        protocol: TCP
        targetPort: <https_node_port> 1
      - name: http-80
        port: 80
        protocol: TCP
        targetPort: <http-node-port> 2
      selector:
        kubevirt.io: virt-launcher
      type: LoadBalancer
    1
    Specify the HTTPS node port value that you noted in the previous step.
    2
    Specify the HTTP node port value that you noted in the previous step.
4.3.5.3. Setting up a wildcard DNS

Set up a wildcard DNS record or CNAME that references the external IP of the load balancer service.

Procedure

  1. Get the external IP address by entering the following command:

    $ oc -n clusters-<hosted_cluster_name> get service <hosted-cluster-name>-apps -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

    Example output

    192.168.20.30

  2. Configure a wildcard DNS entry that references the external IP address. View the following example DNS entry:

    *.apps.<hosted_cluster_name\>.<base_domain\>.

    The DNS entry must be able to route inside and outside of the cluster.

    DNS resolutions example

    dig +short test.apps.example.hypershift.lab
    
    192.168.20.30

  3. Check that hosted cluster status has moved from Partial to Completed by entering the following command:

    $ oc get --namespace clusters hostedclusters

    Example output

    NAME            VERSION   KUBECONFIG                       PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
    example         4.x.0     example-admin-kubeconfig         Completed   True        False         The hosted control plane is available

    Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

4.3.6. Configuring MetalLB

You must install the MetalLB Operator before you configure MetalLB.

Procedure

Complete the following steps to configure MetalLB on your hosted cluster:

  1. Create a MetalLB resource by saving the following sample YAML content in the configure-metallb.yaml file:

    apiVersion: metallb.io/v1beta1
    kind: MetalLB
    metadata:
      name: metallb
      namespace: metallb-system
  2. Apply the YAML content by entering the following command:

    $ oc apply -f configure-metallb.yaml

    Example output

    metallb.metallb.io/metallb created

  3. Create a IPAddressPool resource by saving the following sample YAML content in the create-ip-address-pool.yaml file:

    apiVersion: metallb.io/v1beta1
    kind: IPAddressPool
    metadata:
      name: metallb
      namespace: metallb-system
    spec:
      addresses:
      - 192.168.216.32-192.168.216.122 1
    1
    Create an address pool with an available range of IP addresses within the node network. Replace the IP address range with an unused pool of available IP addresses in your network.
  4. Apply the YAML content by entering the following command:

    $ oc apply -f create-ip-address-pool.yaml

    Example output

    ipaddresspool.metallb.io/metallb created

  5. Create a L2Advertisement resource by saving the following sample YAML content in the l2advertisement.yaml file:

    apiVersion: metallb.io/v1beta1
    kind: L2Advertisement
    metadata:
      name: l2advertisement
      namespace: metallb-system
    spec:
      ipAddressPools:
       - metallb
  6. Apply the YAML content by entering the following command:

    $ oc apply -f l2advertisement.yaml

    Example output

    l2advertisement.metallb.io/metallb created

Additional resources

4.3.7. Configuring additional networks, guaranteed CPUs, and VM scheduling for node pools

If you need to configure additional networks for node pools, request a guaranteed CPU access for Virtual Machines (VMs), or manage scheduling of KubeVirt VMs, see the following procedures.

4.3.7.1. Adding multiple networks to a node pool

By default, nodes generated by a node pool are attached to the pod network. You can attach additional networks to the nodes by using Multus and NetworkAttachmentDefinitions.

Procedure

To add multiple networks to nodes, use the --additional-network argument by running the following command:

$ hcp create cluster kubevirt \
  --name <hosted_cluster_name> \ 1
  --node-pool-replicas <worker_node_count> \ 2
  --pull-secret <path_to_pull_secret> \ 3
  --memory <memory> \ 4
  --cores <cpu> \ 5
  --additional-network name:<namespace/name> \ 6
  –-additional-network name:<namespace/name>
1
Specify the name of your hosted cluster, for instance, example.
2
Specify your worker node count, for example, 2.
3
Specify the path to your pull secret, for example, /user/name/pullsecret.
4
Specify the memory value, for example, 8Gi.
5
Specify the CPU value, for example, 2.
6
Set the value of the –additional-network argument to name:<namespace/name>. Replace <namespace/name> with a namespace and name of your NetworkAttachmentDefinitions.
4.3.7.1.1. Using an additional network as default

You can add your additional network as a default network for the nodes by disabling the default pod network.

Procedure

  • To add an additional network as default to your nodes, run the following command:

    $ hcp create cluster kubevirt \
      --name <hosted_cluster_name> \ 1
      --node-pool-replicas <worker_node_count> \ 2
      --pull-secret <path_to_pull_secret> \ 3
      --memory <memory> \ 4
      --cores <cpu> \ 5
      --attach-default-network false \ 6
      --additional-network name:<namespace>/<network_name> 7
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify your worker node count, for example, 2.
    3
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    4
    Specify the memory value, for example, 8Gi.
    5
    Specify the CPU value, for example, 2.
    6
    The --attach-default-network false argument disables the default pod network.
    7
    Specify the additional network that you want to add to your nodes, for example, name:my-namespace/my-network.
4.3.7.2. Requesting guaranteed CPU resources

By default, KubeVirt VMs might share its CPUs with other workloads on a node. This might impact performance of a VM. To avoid the performance impact, you can request a guaranteed CPU access for VMs.

Procedure

  • To request guaranteed CPU resources, set the --qos-class argument to Guaranteed by running the following command:

    $ hcp create cluster kubevirt \
      --name <hosted_cluster_name> \ 1
      --node-pool-replicas <worker_node_count> \ 2
      --pull-secret <path_to_pull_secret> \ 3
      --memory <memory> \ 4
      --cores <cpu> \ 5
      --qos-class Guaranteed 6
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify your worker node count, for example, 2.
    3
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    4
    Specify the memory value, for example, 8Gi.
    5
    Specify the CPU value, for example, 2.
    6
    The --qos-class Guaranteed argument guarantees that the specified number of CPU resources are assigned to VMs.
4.3.7.3. Scheduling KubeVirt VMs on a set of nodes

By default, KubeVirt VMs created by a node pool are scheduled to any available nodes. You can schedule KubeVirt VMs on a specific set of nodes that has enough capacity to run the VM.

Procedure

  • To schedule KubeVirt VMs within a node pool on a specific set of nodes, use the --vm-node-selector argument by running the following command:

    $ hcp create cluster kubevirt \
      --name <hosted_cluster_name> \ 1
      --node-pool-replicas <worker_node_count> \ 2
      --pull-secret <path_to_pull_secret> \ 3
      --memory <memory> \ 4
      --cores <cpu> \ 5
      --vm-node-selector <label_key>=<label_value>,<label_key>=<label_value> 6
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify your worker node count, for example, 2.
    3
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    4
    Specify the memory value, for example, 8Gi.
    5
    Specify the CPU value, for example, 2.
    6
    The --vm-node-selector flag defines a specific set of nodes that contains the key-value pairs. Replace <label_key> and <label_value> with the key and value of your labels respectively.

4.3.8. Scaling a node pool

You can manually scale a node pool by using the oc scale command.

Procedure

  1. Run the following command:

    NODEPOOL_NAME=${CLUSTER_NAME}-work
    NODEPOOL_REPLICAS=5
    
    $ oc scale nodepool/$NODEPOOL_NAME --namespace clusters --replicas=$NODEPOOL_REPLICAS
  2. After a few moments, enter the following command to see the status of the node pool:

    $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

    Example output

    NAME                  STATUS   ROLES    AGE     VERSION
    example-9jvnf         Ready    worker   97s     v1.27.4+18eadca
    example-n6prw         Ready    worker   116m    v1.27.4+18eadca
    example-nc6g4         Ready    worker   117m    v1.27.4+18eadca
    example-thp29         Ready    worker   4m17s   v1.27.4+18eadca
    example-twxns         Ready    worker   88s     v1.27.4+18eadca

4.3.8.1. Adding node pools

You can create node pools for a hosted cluster by specifying a name, number of replicas, and any additional information, such as memory and CPU requirements.

Procedure

  1. To create a node pool, enter the following information. In this example, the node pool has more CPUs assigned to the VMs:

    export NODEPOOL_NAME=${CLUSTER_NAME}-extra-cpu
    export WORKER_COUNT="2"
    export MEM="6Gi"
    export CPU="4"
    export DISK="16"
    
    $ hcp create nodepool kubevirt \
      --cluster-name $CLUSTER_NAME \
      --name $NODEPOOL_NAME \
      --node-count $WORKER_COUNT \
      --memory $MEM \
      --cores $CPU \
      --root-volume-size $DISK
  2. Check the status of the node pool by listing nodepool resources in the clusters namespace:

    $ oc get nodepools --namespace clusters

    Example output

    NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
    example                   example         5               5               False         False        4.x.0
    example-extra-cpu         example         2                               False         False                  True              True             Minimum availability requires 2 replicas, current 0 available

    Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

  3. After some time, you can check the status of the node pool by entering the following command:

    $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

    Example output

    NAME                      STATUS   ROLES    AGE     VERSION
    example-9jvnf             Ready    worker   97s     v1.27.4+18eadca
    example-n6prw             Ready    worker   116m    v1.27.4+18eadca
    example-nc6g4             Ready    worker   117m    v1.27.4+18eadca
    example-thp29             Ready    worker   4m17s   v1.27.4+18eadca
    example-twxns             Ready    worker   88s     v1.27.4+18eadca
    example-extra-cpu-zh9l5   Ready    worker   2m6s    v1.27.4+18eadca
    example-extra-cpu-zr8mj   Ready    worker   102s    v1.27.4+18eadca

  4. Verify that the node pool is in the status that you expect by entering this command:

    $ oc get nodepools --namespace clusters

    Example output

    NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
    example                   example         5               5               False         False        4.x.0
    example-extra-cpu         example         2               2               False         False        4.x.0

    Replace 4.x.0 with the supported OpenShift Container Platform version that you want to use.

Additional resources

4.3.9. Verifying hosted cluster creation on OpenShift Virtualization

To verify that your hosted cluster was successfully created, complete the following steps.

Procedure

  1. Verify that the HostedCluster resource transitioned to the completed state by entering the following command:

    $ oc get --namespace clusters hostedclusters <hosted_cluster_name>

    Example output

    NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
    clusters    example   4.12.2    example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

  2. Verify that all the cluster operators in the hosted cluster are online by entering the following commands:

    $ hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig
    $ oc get co --kubeconfig=<hosted_cluster_name>-kubeconfig

    Example output

    NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    console                                    4.12.2   True        False         False      2m38s
    csi-snapshot-controller                    4.12.2   True        False         False      4m3s
    dns                                        4.12.2   True        False         False      2m52s
    image-registry                             4.12.2   True        False         False      2m8s
    ingress                                    4.12.2   True        False         False      22m
    kube-apiserver                             4.12.2   True        False         False      23m
    kube-controller-manager                    4.12.2   True        False         False      23m
    kube-scheduler                             4.12.2   True        False         False      23m
    kube-storage-version-migrator              4.12.2   True        False         False      4m52s
    monitoring                                 4.12.2   True        False         False      69s
    network                                    4.12.2   True        False         False      4m3s
    node-tuning                                4.12.2   True        False         False      2m22s
    openshift-apiserver                        4.12.2   True        False         False      23m
    openshift-controller-manager               4.12.2   True        False         False      23m
    openshift-samples                          4.12.2   True        False         False      2m15s
    operator-lifecycle-manager                 4.12.2   True        False         False      22m
    operator-lifecycle-manager-catalog         4.12.2   True        False         False      23m
    operator-lifecycle-manager-packageserver   4.12.2   True        False         False      23m
    service-ca                                 4.12.2   True        False         False      4m41s
    storage                                    4.12.2   True        False         False      4m43s

4.4. Deploying hosted control planes on non-bare metal agent machines

You can deploy hosted control planes by configuring a cluster to function as a hosting cluster. The hosting cluster is an OpenShift Container Platform cluster where the control planes are hosted. The hosting cluster is also known as the management cluster.

Important

Hosted control planes on non-bare-metal agent machines is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Note

The management cluster is not the same thing as the managed cluster. A managed cluster is a cluster that the hub cluster manages.

The hosted control planes feature is enabled by default.

The multicluster engine Operator supports only the default local-cluster managed hub cluster. On Red Hat Advanced Cluster Management (RHACM) 2.10, you can use the local-cluster managed hub cluster as the hosting cluster.

A hosted cluster is an OpenShift Container Platform cluster with its API endpoint and control plane that are hosted on the hosting cluster. The hosted cluster includes the control plane and its corresponding data plane. You can use the multicluster engine Operator console or the hcp command-line interface (CLI) to create a hosted cluster.

The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see "Disabling the automatic import of hosted clusters into multicluster engine Operator".

4.4.1. Preparing to deploy hosted control planes on non-bare metal agent machines

As you prepare to deploy hosted control planes on bare metal, consider the following information:

  • You can add agent machines as a worker node to a hosted cluster by using the Agent platform. Agent machine represents a host booted with a Discovery Image and ready to be provisioned as an OpenShift Container Platform node. The Agent platform is part of the central infrastructure management service. For more information, see Enabling the central infrastructure management service.
  • All hosts that are not bare metal require a manual boot with a Discovery Image ISO that the central infrastructure management provides.
  • When you scale up the node pool, a machine is created for every replica. For every machine, the Cluster API provider finds and installs an Agent that is approved, is passing validations, is not currently in use, and meets the requirements that are specified in the node pool specification. You can monitor the installation of an Agent by checking its status and conditions.
  • When you scale down a node pool, Agents are unbound from the corresponding cluster. Before you can reuse the Agents, you must restart them by using the Discovery image.
  • When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control planes etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using logical volume manager storage" in the OpenShift Container Platform documentation.
4.4.1.1. Prerequisites for deploying hosted control planes on non-bare metal agent machines

You must review the following prerequisites before deploying hosted control planes on non-bare metal agent machines:

  • You need the multicluster engine for Kubernetes Operator 2.5 and later installed on an OpenShift Container Platform cluster. The multicluster engine Operator is automatically installed when you install Red Hat Advanced Cluster Management (RHACM). You can also install the multicluster engine Operator without RHACM as an Operator from the OpenShift Container Platform OperatorHub.
  • You have at least one managed OpenShift Container Platform cluster for the multicluster engine Operator. The local-cluster managed hub cluster is automatically imported. See Advanced configuration for more information about the local-cluster. You can check the status of your hub cluster by running the following command:

    $ oc get managedclusters local-cluster
  • You enabled central infrastructure management. For more information, see Enabling the central infrastructure management service.
  • You installed the hcp command-line interface.
  • Your hosted cluster has a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for the multicluster engine Operator to manage it.
  • You run the hub cluster and workers on the same platform for hosted control planes.
4.4.1.2. Firewall, port, and service requirements for non-bare metal agent machines

You must meet the firewall and port requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters.

Note

Services run on their default ports. However, if you use the NodePort publishing strategy, services run on the port that is assigned by the NodePort service.

Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary. For production deployments, use a load balancer to simplify access through a single IP address.

A hosted control plane exposes the following services on non-bare metal agent machines:

  • APIServer

    • The APIServer service runs on port 6443 by default and requires ingress access for communication between the control plane components.
    • If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses.
  • OAuthServer

    • The OAuthServer service runs on port 443 by default when you use the route and ingress to expose the service.
    • If you use the NodePort publishing strategy, use a firewall rule for the OAuthServer service.
  • Konnectivity

    • The Konnectivity service runs on port 443 by default when you use the route and ingress to expose the service.
    • The Konnectivity agent establishes a reverse tunnel to allow the control plane to access the network for the hosted cluster. The agent uses egress to connect to the Konnectivity server. The server is exposed by using either a route on port 443 or a manually assigned NodePort.
    • If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443.
    • If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes.
  • Ignition

    • The Ignition service runs on port 443 by default when you use the route and ingress to expose the service.
    • If you use the NodePort publishing strategy, use a firewall rule for the Ignition service.

You do not need the following services on non-bare metal agent machines:

  • OVNSbDb
  • OIDC
4.4.1.3. Infrastructure requirements for non-bare metal agent machines

The Agent platform does not create any infrastructure, but it has the following infrastructure requirements:

  • Agents: An Agent represents a host that is booted with a discovery image and is ready to be provisioned as an OpenShift Container Platform node.
  • DNS: The API and ingress endpoints must be routable.

4.4.2. Configuring DNS on non-bare metal agent machines

The API Server for the hosted cluster is exposed as a NodePort service. A DNS entry must exist for api.<hosted_cluster_name>.<basedomain> that points to destination where the API Server can be reached.

The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane. The entry can also point to a load balancer that is deployed to redirect incoming traffic to the ingress pods.

  • If you are configuring DNS for a connected environment on an IPv4 network, see the following example DNS configuration:

    api-int.example.krnl.es.    IN A 192.168.122.22
    `*`.apps.example.krnl.es.   IN A 192.168.122.23
  • If you are configuring DNS for a disconnected environment on an IPv6 network, see the following example DNS configuration:

    api-int.example.krnl.es.    IN A 2620:52:0:1306::7
    `*`.apps.example.krnl.es.   IN A 2620:52:0:1306::10
  • If you are configuring DNS for a disconnected environment on a dual stack network, be sure to include DNS entries for both IPv4 and IPv6. See the following example DNS configuration:

    host-record=api-int.hub-dual.dns.base.domain.name,2620:52:0:1306::2
    address=/apps.hub-dual.dns.base.domain.name/2620:52:0:1306::3
    dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5]

4.4.3. Creating a hosted cluster on non-bare metal agent machines by using the CLI

When you create a hosted cluster with the Agent platform, the HyperShift Operator installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one.

As you create a hosted cluster, review the following guidelines:

  • Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it.
  • Do not use clusters as a hosted cluster name.
  • A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster.

Procedure

  1. Create the hosted control plane namespace by entering the following command:

    $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> 1
    1
    Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.
  2. Create a hosted cluster by entering the following command:

    $ hcp create cluster agent \
      --name=<hosted_cluster_name> \1
      --pull-secret=<path_to_pull_secret> \2
      --agent-namespace=<hosted_control_plane_namespace> \3
      --base-domain=<basedomain> \4
      --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
      --etcd-storage-class=<etcd_storage_class> \6
      --ssh-key  <path_to_ssh_key> \7
      --namespace <hosted_cluster_namespace> \8
      --control-plane-availability-policy HighlyAvailable \9
      --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release> 10
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    3
    Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.
    4
    Specify your base domain, for example, krnl.es.
    5
    The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.
    6
    Verify that you have a default storage class configured for your cluster. Otherwise, you might end up with pending PVCs. Specify the etcd storage class name, for example, lvm-storageclass.
    7
    Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.
    8
    Specify your hosted cluster namespace.
    9
    The default value for the control plane availability policy is HighlyAvailable.
    10
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.

Verification

  • After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

    $ oc -n <hosted_control_plane_namespace> get pods

    Example output

    NAME                                             READY   STATUS    RESTARTS   AGE
    catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
    control-plane-operator-f6b4c8465-4k5dh           1/1     Running   0          4m32s

4.4.3.1. Creating a hosted cluster on non-bare metal agent machines by using the web console

You can create a hosted cluster on non-bare metal agent machines by using the OpenShift Container Platform web console.

Prerequisites

  • You have access to the cluster with cluster-admin privileges.
  • You have access to the OpenShift Container Platform web console.

Procedure

  1. Open the OpenShift Container Platform web console and log in by entering your administrator credentials.
  2. In the console header, select All Clusters.
  3. Click Infrastructure → Clusters.
  4. Click Create cluster Host inventory → Hosted control plane.

    The Create cluster page is displayed.

  5. On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation.

As you enter details about the cluster, you might find the following tips useful:

  • If you want to use predefined values to automatically populate fields in the console, you can create a host inventory credential. For more information, see Creating a credential for an on-premises environment.
  • On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a host inventory credential, the pull secret is automatically populated.
  • On the Node pools page, the namespace contains the hosts for the node pool. If you created a host inventory by using the console, the console creates a dedicated namespace.
  • On the Networking page, you select an API server publishing strategy. The API server for the hosted cluster can be exposed either by using an existing load balancer or as a service of the NodePort type. A DNS entry must exist for the api.<hosted_cluster_name>.<basedomain> setting that points to the destination where the API server can be reached. This entry can be a record that points to one of the nodes in the management cluster or a record that points to a load balancer that redirects incoming traffic to the Ingress pods.

    1. Review your entries and click Create.

    The Hosted cluster view is displayed.

    1. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, and click the cluster name. Wait until the control plane components are ready. This process can take a few minutes.
    2. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster.

Next steps

4.4.3.2. Creating a hosted cluster on bare metal by using a mirror registry

You can use a mirror registry to create a hosted cluster on bare metal by specifying the --image-content-sources flag in the hcp create cluster command.

Procedure

  1. Create a YAML file to define Image Content Source Policies (ICSP). See the following example:

    - mirrors:
      - brew.registry.redhat.io
      source: registry.redhat.io
    - mirrors:
      - brew.registry.redhat.io
      source: registry.stage.redhat.io
    - mirrors:
      - brew.registry.redhat.io
      source: registry-proxy.engineering.redhat.com
  2. Save the file as icsp.yaml. This file contains your mirror registries.
  3. To create a hosted cluster by using your mirror registries, run the following command:

    $ hcp create cluster agent \
        --name=<hosted_cluster_name> \1
        --pull-secret=<path_to_pull_secret> \2
        --agent-namespace=<hosted_control_plane_namespace> \3
        --base-domain=<basedomain> \4
        --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
        --image-content-sources icsp.yaml  \6
        --ssh-key  <path_to_ssh_key> \7
        --namespace <hosted_cluster_namespace> \8
        --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    3
    Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.
    4
    Specify your base domain, for example, krnl.es.
    5
    The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.
    6
    Specify the icsp.yaml file that defines ICSP and your mirror registries.
    7
    Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.
    8
    Specify your hosted cluster namespace.
    9
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

Next steps

4.4.4. Verifying hosted cluster creation on non-bare metal agent machines

After the deployment process is complete, you can verify that the hosted cluster was created successfully. Follow these steps a few minutes after you create the hosted cluster.

Procedure

  1. Obtain the kubeconfig file for your new hosted cluster by entering the following command:

    $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>
  2. Use the kubeconfig file to view the cluster Operators of the hosted cluster. Enter the following command:

    $ oc get co --kubeconfig=kubeconfig-<hosted_cluster_name>

    Example output

    NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    console                                    4.10.26   True        False         False      2m38s
    csi-snapshot-controller                    4.10.26   True        False         False      4m3s
    dns                                        4.10.26   True        False         False      2m52s

  3. View the running pods on your hosted cluster by entering the following command:

    $ oc get pods -A --kubeconfig=kubeconfig-<hosted_cluster_name>

    Example output

    NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
    kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
    openshift-cluster-samples-operator                 cluster-samples-operator-6b5bcb9dff-kpnbc                 2/2     Running            0               20m
    openshift-monitoring                               alertmanager-main-0                                       6/6     Running            0               100s
    openshift-monitoring                               openshift-state-metrics-677b9fb74f-qqp6g                  3/3     Running            0               104s

4.5. Deploying hosted control planes on IBM Z

You can deploy hosted control planes by configuring a cluster to function as a management cluster. The management cluster is the OpenShift Container Platform cluster where the control planes are hosted. The management cluster is also known as the hosting cluster.

Note

The management cluster is not the managed cluster. A managed cluster is a cluster that the hub cluster manages.

You can convert a managed cluster to a management cluster by using the hypershift add-on to deploy the HyperShift Operator on that cluster. Then, you can start to create the hosted cluster.

The multicluster engine Operator supports only the default local-cluster, which is a hub cluster that is managed, and the hub cluster as the management cluster.

To provision hosted control planes on bare metal, you can use the Agent platform. The Agent platform uses the central infrastructure management service to add worker nodes to a hosted cluster. For more information, see "Enabling the central infrastructure management service".

Each IBM Z system host must be started with the PXE images provided by the central infrastructure management. After each host starts, it runs an Agent process to discover the details of the host and completes the installation. An Agent custom resource represents each host.

When you create a hosted cluster with the Agent platform, HyperShift Operator installs the Agent Cluster API provider in the hosted control plane namespace.

4.5.1. Prerequisites to configure hosted control planes on IBM Z

  • The multicluster engine for Kubernetes Operator version 2.5 or later must be installed on an OpenShift Container Platform cluster. You can install multicluster engine Operator as an Operator from the OpenShift Container Platform OperatorHub.
  • The multicluster engine Operator must have at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported in multicluster engine Operator 2.5 and later. For more information about the local-cluster, see Advanced configuration in the Red Hat Advanced Cluster Management documentation. You can check the status of your hub cluster by running the following command:

    $ oc get managedclusters local-cluster
  • You need a hosting cluster with at least three worker nodes to run the HyperShift Operator.
  • You need to enable the central infrastructure management service. For more information, see Enabling the central infrastructure management service.
  • You need to install the hosted control plane command line interface. For more information, see Installing the hosted control plane command line interface.

4.5.2. IBM Z infrastructure requirements

The Agent platform does not create any infrastructure, but requires the following resources for infrastructure:

  • Agents: An Agent represents a host that is booted with a discovery image, or PXE image and is ready to be provisioned as an OpenShift Container Platform node.
  • DNS: The API and Ingress endpoints must be routable.

The hosted control planes feature is enabled by default. If you disabled the feature and want to manually enable it, or if you need to disable the feature, see Enabling or disabling the hosted control planes feature.

4.5.3. DNS configuration for hosted control planes on IBM Z

The API server for the hosted cluster is exposed as a NodePort service. A DNS entry must exist for the api.<hosted_cluster_name>.<base_domain> that points to the destination where the API server is reachable.

The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane.

The entry can also point to a load balancer deployed to redirect incoming traffic to the Ingress pods.

See the following example of a DNS configuration:

$ cat /var/named/<example.krnl.es.zone>

Example output

$ TTL 900
@ IN  SOA bastion.example.krnl.es.com. hostmaster.example.krnl.es.com. (
      2019062002
      1D 1H 1W 3H )
  IN NS bastion.example.krnl.es.com.
;
;
api                   IN A 1xx.2x.2xx.1xx 1
api-int               IN A 1xx.2x.2xx.1xx
;
;
*.apps        IN A 1xx.2x.2xx.1xx
;
;EOF

1
The record refers to the IP address of the API load balancer that handles ingress and egress traffic for hosted control planes.

For IBM z/VM, add IP addresses that correspond to the IP address of the agent.

compute-0              IN A 1xx.2x.2xx.1yy
compute-1              IN A 1xx.2x.2xx.1yy

4.5.4. Creating a hosted cluster on bare metal

When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one.

As you create a hosted cluster, keep the following guidelines in mind:

  • Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it.
  • Do not use clusters as a hosted cluster name.
  • A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster.

Procedure

  1. Create the hosted control plane namespace by entering the following command:

    $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

    Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

  2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

    $ hcp create cluster agent \
        --name=<hosted_cluster_name> \1
        --pull-secret=<path_to_pull_secret> \2
        --agent-namespace=<hosted_control_plane_namespace> \3
        --base-domain=<basedomain> \4
        --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
        --etcd-storage-class=<etcd_storage_class> \6
        --ssh-key  <path_to_ssh_public_key> \7
        --namespace <hosted_cluster_namespace> \8
        --control-plane-availability-policy HighlyAvailable \9
        --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    3
    Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.
    4
    Specify your base domain, for example, krnl.es.
    5
    The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.
    6
    Specify the etcd storage class name, for example, lvm-storageclass.
    7
    Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.
    8
    Specify your hosted cluster namespace.
    9
    The default value for the control plane availability policy is HighlyAvailable.
    10
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.
  3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

    $ oc -n <hosted_control_plane_namespace> get pods

    Example output

    NAME                                             READY   STATUS    RESTARTS   AGE
    capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
    catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
    certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
    cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

4.5.5. Creating an InfraEnv resource for hosted control planes on IBM Z

An InfraEnv is an environment where hosts that are booted with PXE images can join as agents. In this case, the agents are created in the same namespace as your hosted control plane.

Procedure

  1. Create a YAML file to contain the configuration. See the following example:

    apiVersion: agent-install.openshift.io/v1beta1
    kind: InfraEnv
    metadata:
      name: <hosted_cluster_name>
      namespace: <hosted_control_plane_namespace>
    spec:
      cpuArchitecture: s390x
      pullSecretRef:
        name: pull-secret
      sshAuthorizedKey: <ssh_public_key>
  2. Save the file as infraenv-config.yaml.
  3. Apply the configuration by entering the following command:

    $ oc apply -f infraenv-config.yaml
  4. To fetch the URL to download the PXE images, such as, initrd.img, kernel.img, or rootfs.img, which allows IBM Z machines to join as agents, enter the following command:

    $ oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json

4.5.6. Adding IBM Z agents to the InfraEnv resource

To attach compute nodes to a hosted control plane, create agents that help you to scale the node pool. Adding agents in an IBM Z environment requires additional steps, which are described in detail in this section.

Unless stated otherwise, these procedures apply to both z/VM and RHEL KVM installations on IBM Z and IBM LinuxONE.

4.5.6.1. Adding IBM Z KVM as agents

For IBM Z with KVM, run the following command to start your IBM Z environment with the downloaded PXE images from the InfraEnv resource. After the Agents are created, the host communicates with the Assisted Service and registers in the same namespace as the InfraEnv resource on the management cluster.

Procedure

  1. Run the following command:

    virt-install \
       --name "<vm_name>" \ 1
       --autostart \
       --ram=16384 \
       --cpu host \
       --vcpus=4 \
       --location "<path_to_kernel_initrd_image>,kernel=kernel.img,initrd=initrd.img" \ 2
       --disk <qcow_image_path> \ 3
       --network network:macvtap-net,mac=<mac_address> \ 4
       --graphics none \
       --noautoconsole \
       --wait=-1
       --extra-args "rd.neednet=1 nameserver=<nameserver>   coreos.live.rootfs_url=http://<http_server>/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs=console=tty1 console=ttyS1,115200n8" 5
    1
    Specify the name of the virtual machine.
    2
    Specify the location of the kernel_initrd_image file.
    3
    Specify the disk image path.
    4
    Specify the Mac address.
    5
    Specify the server name of the agents.
  2. For ISO boot, download ISO from the InfraEnv resource and boot the nodes by running the following command:

    virt-install \
      --name "<vm_name>" \ 1
      --autostart \
      --memory=16384 \
      --cpu host \
      --vcpus=4 \
      --network network:macvtap-net,mac=<mac_address> \ 2
      --cdrom "<path_to_image.iso>" \ 3
      --disk <qcow_image_path> \
      --graphics none \
      --noautoconsole \
      --os-variant <os_version> \ 4
      --wait=-1
    1
    Specify the name of the virtual machine.
    2
    Specify the Mac address.
    3
    Specify the location of the image.iso file.
    4
    Specify the operating system version that you are using.
4.5.6.2. Adding IBM Z LPAR as agents

You can add the Logical Partition (LPAR) on IBM Z or IBM LinuxONE as a compute node to a hosted control plane.

Procedure

  1. Create a boot parameter file for the agents:

    Example parameter file

    rd.neednet=1 cio_ignore=all,!condev \
    console=ttysclp0 \
    ignition.firstboot ignition.platform.id=metal
    coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \1
    coreos.inst.persistent-kargs=console=ttysclp0
    ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \2
    rd.znet=qeth,<network_adaptor_range>,layer2=1
    rd.<disk_type>=<adapter> \3
    zfcp.allow_lun_scan=0
    ai.ip_cfg_override=1 \4
    random.trust_cpu=on rd.luks.options=discard

    1
    For the coreos.live.rootfs_url artifact, specify the matching rootfs artifact for the kernel and initramfs that you are starting. Only HTTP and HTTPS protocols are supported.
    2
    For the ip parameter, manually assign the IP address, as described in Installing a cluster with z/VM on IBM Z and IBM LinuxONE.
    3
    For installations on DASD-type disks, use rd.dasd to specify the DASD where Red Hat Enterprise Linux CoreOS (RHCOS) is to be installed. For installations on FCP-type disks, use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed.
    4
    Specify this parameter when you use an Open Systems Adapter (OSA) or HiperSockets.
  2. Download the .ins and initrd.img.addrsize files from the InfraEnv resource.

    By default, the URL for the .ins and initrd.img.addrsize files is not available in the InfraEnv resource. You must edit the URL to fetch those artifacts.

    1. Update the kernel URL endpoint to include ins-file by running the following command:

      $ curl -k -L -o generic.ins "< url for ins-file >"

      Example URL

      https://…/boot-artifacts/ins-file?arch=s390x&version=4.17.0

    2. Update the initrd URL endpoint to include s390x-initrd-addrsize:

      Example URL

      https://…./s390x-initrd-addrsize?api_key=<api-key>&arch=s390x&version=4.17.0

  3. Transfer the initrd, kernel, generic.ins, and initrd.img.addrsize parameter files to the file server. For more information about how to transfer the files with FTP and boot, see "Installing in an LPAR".
  4. Start the machine.
  5. Repeat the procedure for all other machines in the cluster.

Additional resources

4.5.6.3. Adding IBM z/VM as agents

If you want to use a static IP for z/VM guest, you must configure the NMStateConfig attribute for the z/VM agent so that the IP parameter persists in the second start.

Complete the following steps to start your IBM Z environment with the downloaded PXE images from the InfraEnv resource. After the Agents are created, the host communicates with the Assisted Service and registers in the same namespace as the InfraEnv resource on the management cluster.

Procedure

  1. Update the parameter file to add the rootfs_url, network_adaptor and disk_type values.

    Example parameter file

    rd.neednet=1 cio_ignore=all,!condev \
    console=ttysclp0  \
    ignition.firstboot ignition.platform.id=metal \
    coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \1
    coreos.inst.persistent-kargs=console=ttysclp0
    ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \2
    rd.znet=qeth,<network_adaptor_range>,layer2=1
    rd.<disk_type>=<adapter> \3
    zfcp.allow_lun_scan=0
    ai.ip_cfg_override=1 \4

    1
    For the coreos.live.rootfs_url artifact, specify the matching rootfs artifact for the kernel and initramfs that you are starting. Only HTTP and HTTPS protocols are supported.
    2
    For the ip parameter, manually assign the IP address, as described in Installing a cluster with z/VM on IBM Z and IBM LinuxONE.
    3
    For installations on DASD-type disks, use rd.dasd to specify the DASD where Red Hat Enterprise Linux CoreOS (RHCOS) is to be installed. For installations on FCP-type disks, use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed.
    4
    Specify this parameter when you use an Open Systems Adapter (OSA) or HiperSockets.
  2. Move initrd, kernel images, and the parameter file to the guest VM by running the following commands:

    vmur pun -r -u -N kernel.img $INSTALLERKERNELLOCATION/<image name>
    vmur pun -r -u -N generic.parm $PARMFILELOCATION/paramfilename
    vmur pun -r -u -N initrd.img $INSTALLERINITRAMFSLOCATION/<image name>
  3. Run the following command from the guest VM console:

    cp ipl c
  4. To list the agents and their properties, enter the following command:

    $ oc -n <hosted_control_plane_namespace> get agents

    Example output

    NAME    CLUSTER APPROVED    ROLE    STAGE
    50c23cda-cedc-9bbd-bcf1-9b3a5c75804d    auto-assign
    5e498cd3-542c-e54f-0c58-ed43e28b568a    auto-assign

  5. Run the following command to approve the agent.

    $ oc -n <hosted_control_plane_namespace> patch agent \
      50c23cda-cedc-9bbd-bcf1-9b3a5c75804d -p \
      '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-zvm-0.hostedn.example.com"}}' \1
      --type merge
    1
    Optionally, you can set the agent ID <installation_disk_id> and <hostname> in the specification.
  6. Run the following command to verify that the agents are approved:

    $ oc -n <hosted_control_plane_namespace> get agents

    Example output

    NAME                                            CLUSTER     APPROVED   ROLE          STAGE
    50c23cda-cedc-9bbd-bcf1-9b3a5c75804d             true       auto-assign
    5e498cd3-542c-e54f-0c58-ed43e28b568a             true       auto-assign

4.5.7. Scaling the NodePool object for a hosted cluster on IBM Z

The NodePool object is created when you create a hosted cluster. By scaling the NodePool object, you can add more compute nodes to the hosted control plane.

When you scale up a node pool, a machine is created. The Cluster API provider finds an Agent that is approved, is passing validations, is not currently in use, and meets the requirements that are specified in the node pool specification. You can monitor the installation of an Agent by checking its status and conditions.

When you scale down a node pool, Agents are unbound from the corresponding cluster. Before you reuse the clusters, you must boot the clusters by using the PXE image to update the number of nodes.

Procedure

  1. Run the following command to scale the NodePool object to two nodes:

    $ oc -n <clusters_namespace> scale nodepool <nodepool_name> --replicas 2

    The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through the transition phases in the following order:

    • binding
    • discovering
    • insufficient
    • installing
    • installing-in-progress
    • added-to-existing-cluster
  2. Run the following command to see the status of a specific scaled agent:

    $ oc -n <hosted_control_plane_namespace> get agent -o \
      jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} \
      Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

    Example output

    BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound
    BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient

  3. Run the following command to see the transition phases:

    $ oc -n <hosted_control_plane_namespace> get agent

    Example output

    NAME                                   CLUSTER           APPROVED       ROLE        STAGE
    50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   hosted-forwarder   true          auto-assign
    5e498cd3-542c-e54f-0c58-ed43e28b568a                      true          auto-assign
    da503cf1-a347-44f2-875c-4960ddb04091   hosted-forwarder   true          auto-assign

  4. Run the following command to generate the kubeconfig file to access the hosted cluster:

    $ hcp create kubeconfig --namespace <clusters_namespace> --name <hosted_cluster_namespace> > <hosted_cluster_name>.kubeconfig
  5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes by entering the following command:

    $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

    Example output

    NAME                             STATUS   ROLES    AGE      VERSION
    worker-zvm-0.hostedn.example.com Ready    worker   5m41s    v1.24.0+3882f8f
    worker-zvm-1.hostedn.example.com Ready    worker   6m3s     v1.24.0+3882f8f

    Cluster Operators start to reconcile by adding workloads to the nodes.

  6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

    $ oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io

    Example output

    NAME                                CLUSTER  NODENAME PROVIDERID     PHASE     AGE   VERSION
    hosted-forwarder-79558597ff-5tbqp   hosted-forwarder-crqq5   worker-zvm-0.hostedn.example.com   agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   Running   41h   4.15.0
    hosted-forwarder-79558597ff-lfjfk   hosted-forwarder-crqq5   worker-zvm-1.hostedn.example.com   agent://5e498cd3-542c-e54f-0c58-ed43e28b568a   Running   41h   4.15.0

  7. Run the following command to check the cluster version:

    $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

    Example output

    NAME                                         VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
    clusterversion.config.openshift.io/version   4.15.0-ec.2   True        False         40h     Cluster version is 4.15.0-ec.2

  8. Run the following command to check the cluster operator status:

    $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators

For each component of your cluster, the output shows the following cluster operator statuses: NAME, VERSION, AVAILABLE, PROGRESSING, DEGRADED, SINCE, and MESSAGE.

For an output example, see Initial Operator configuration.

Additional resources

4.6. Deploying hosted control planes on IBM Power

You can deploy hosted control planes by configuring a cluster to function as a hosting cluster. The hosting cluster is an OpenShift Container Platform cluster where the control planes are hosted. The hosting cluster is also known as the management cluster.

Note

The management cluster is not the managed cluster. A managed cluster is a cluster that the hub cluster manages.

The multicluster engine Operator supports only the default local-cluster, which is a hub cluster that is managed, and the hub cluster as the hosting cluster.

To provision hosted control planes on bare metal, you can use the Agent platform. The Agent platform uses the central infrastructure management service to add worker nodes to a hosted cluster. For more information, see "Enabling the central infrastructure management service".

Each IBM Power host must be started with a Discovery Image that the central infrastructure management provides. After each host starts, it runs an Agent process to discover the details of the host and completes the installation. An Agent custom resource represents each host.

When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace.

4.6.1. Prerequisites to configure hosted control planes on IBM Power

  • The multicluster engine for Kubernetes Operator version 2.7 and later installed on an OpenShift Container Platform cluster. The multicluster engine Operator is automatically installed when you install Red Hat Advanced Cluster Management (RHACM). You can also install the multicluster engine Operator without RHACM as an Operator from the OpenShift Container Platform OperatorHub.
  • The multicluster engine Operator must have at least one managed OpenShift Container Platform cluster. The local-cluster managed hub cluster is automatically imported in the multicluster engine Operator version 2.7 and later. For more information about local-cluster, see Advanced configuration in the RHACM documentation. You can check the status of your hub cluster by running the following command:

    $ oc get managedclusters local-cluster
  • You need a hosting cluster with at least 3 worker nodes to run the HyperShift Operator.
  • You need to enable the central infrastructure management service. For more information, see "Enabling the central infrastructure management service".
  • You need to install the hosted control plane command-line interface. For more information, see "Installing the hosted control plane command-line interface".

The hosted control planes feature is enabled by default. If you disabled the feature and want to manually enable it, see "Manually enabling the hosted control planes feature". If you need to disable the feature, see "Disabling the hosted control planes feature".

4.6.2. IBM Power infrastructure requirements

The Agent platform does not create any infrastructure, but requires the following resources for infrastructure:

  • Agents: An Agent represents a host that is booted with a discovery image and is ready to be provisioned as an OpenShift Container Platform node.
  • DNS: The API and Ingress endpoints must be routable.

4.6.3. DNS configuration for hosted control planes on IBM Power

The API server for the hosted cluster is exposed. A DNS entry must exist for the api.<hosted_cluster_name>.<basedomain> entry that points to the destination where the API server is reachable.

The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane.

The entry can also point to a load balancer that is deployed to redirect incoming traffic to the ingress pods.

See the following example of a DNS configuration:

$ cat /var/named/<example.krnl.es.zone>

Example output

$ TTL 900
@ IN  SOA bastion.example.krnl.es.com. hostmaster.example.krnl.es.com. (
      2019062002
      1D 1H 1W 3H )
  IN NS bastion.example.krnl.es.com.
;
;
api                   IN A 1xx.2x.2xx.1xx 1
api-int               IN A 1xx.2x.2xx.1xx
;
;
*.apps.<hosted-cluster-name>.<basedomain>           IN A 1xx.2x.2xx.1xx
;
;EOF

1
The record refers to the IP address of the API load balancer that handles ingress and egress traffic for hosted control planes.

For IBM Power, add IP addresses that correspond to the IP address of the agent.

Example configuration

compute-0              IN A 1xx.2x.2xx.1yy
compute-1              IN A 1xx.2x.2xx.1yy

4.6.4. Creating a hosted cluster on bare metal

When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one.

As you create a hosted cluster, keep the following guidelines in mind:

  • Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it.
  • Do not use clusters as a hosted cluster name.
  • A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster.

Procedure

  1. Create the hosted control plane namespace by entering the following command:

    $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

    Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

  2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

    $ hcp create cluster agent \
        --name=<hosted_cluster_name> \1
        --pull-secret=<path_to_pull_secret> \2
        --agent-namespace=<hosted_control_plane_namespace> \3
        --base-domain=<basedomain> \4
        --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
        --etcd-storage-class=<etcd_storage_class> \6
        --ssh-key  <path_to_ssh_public_key> \7
        --namespace <hosted_cluster_namespace> \8
        --control-plane-availability-policy HighlyAvailable \9
        --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10
    1
    Specify the name of your hosted cluster, for instance, example.
    2
    Specify the path to your pull secret, for example, /user/name/pullsecret.
    3
    Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.
    4
    Specify your base domain, for example, krnl.es.
    5
    The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.
    6
    Specify the etcd storage class name, for example, lvm-storageclass.
    7
    Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.
    8
    Specify your hosted cluster namespace.
    9
    The default value for the control plane availability policy is HighlyAvailable.
    10
    Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.
  3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

    $ oc -n <hosted_control_plane_namespace> get pods

    Example output

    NAME                                             READY   STATUS    RESTARTS   AGE
    capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
    catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
    certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
    cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Chapter 5. Managing hosted control planes

5.1. Managing hosted control planes on AWS

When you use hosted control planes for OpenShift Container Platform on Amazon Web Services (AWS), the infrastructure requirements vary based on your setup.

5.1.1. Prerequisites to manage AWS infrastructure and IAM permissions

To configure hosted control planes for OpenShift Container Platform on Amazon Web Services (AWS), you must meet the following the infrastructure requirements:

  • You configured hosted control planes before you can create hosted clusters.
  • You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials.
5.1.1.1. Infrastructure requirements for AWS

When you use hosted control planes on Amazon Web Services (AWS), the infrastructure requirements fit in the following categories:

  • Prerequired and unmanaged infrastructure for the HyperShift Operator in an arbitrary AWS account
  • Prerequired and unmanaged infrastructure in a hosted cluster AWS account
  • Hosted control planes-managed infrastructure in a management AWS account
  • Hosted control planes-managed infrastructure in a hosted cluster AWS account
  • Kubernetes-managed infrastructure in a hosted cluster AWS account

Prerequired means that hosted control planes requires AWS infrastructure to properly work. Unmanaged means that no Operator or controller creates the infrastructure for you.

5.1.1.2. Unmanaged infrastructure for the HyperShift Operator in an AWS account

An arbitrary Amazon Web Services (AWS) account depends on the provider of the hosted control planes service.

In self-managed hosted control planes, the cluster service provider controls the AWS account. The cluster service provider is the administrator who hosts cluster control planes and is responsible for uptime. In managed hosted control planes, the AWS account belongs to Red Hat.

In a prerequired and unmanaged infrastructure for the HyperShift Operator, the following infrastructure requirements apply for a management cluster AWS account:

  • One S3 Bucket

    • OpenID Connect (OIDC)
  • Route 53 hosted zones

    • A domain to host private and public entries for hosted clusters
5.1.1.3. Unmanaged infrastructure requirements for a management AWS account

When your infrastructure is prerequired and unmanaged in a hosted cluster Amazon Web Services (AWS) account, the infrastructure requirements for all access modes are as follows:

  • One VPC
  • One DHCP Option
  • Two subnets

    • A private subnet that is an internal data plane subnet
    • A public subnet that enables access to the internet from the data plane
  • One internet gateway
  • One elastic IP
  • One NAT gateway
  • One security group (worker nodes)
  • Two route tables (one private and one public)
  • Two Route 53 hosted zones
  • Enough quota for the following items:

    • One Ingress service load balancer for public hosted clusters
    • One private link endpoint for private hosted clusters
Note

For private link networking to work, the endpoint zone in the hosted cluster AWS account must match the zone of the instance that is resolved by the service endpoint in the management cluster AWS account. In AWS, the zone names are aliases, such as us-east-2b, which do not necessarily map to the same zone in different accounts. As a result, for private link to work, the management cluster must have subnets or workers in all zones of its region.

5.1.1.4. Infrastructure requirements for a management AWS account

When your infrastructure is managed by hosted control planes in a management AWS account, the infrastructure requirements differ depending on whether your clusters are public, private, or a combination.

For accounts with public clusters, the infrastructure requirements are as follows:

  • Network load balancer: a load balancer Kube API server

    • Kubernetes creates a security group
  • Volumes

    • For etcd (one or three depending on high availability)
    • For OVN-Kube

For accounts with private clusters, the infrastructure requirements are as follows:

  • Network load balancer: a load balancer private router
  • Endpoint service (private link)

For accounts with public and private clusters, the infrastructure requirements are as follows:

  • Network load balancer: a load balancer public router
  • Network load balancer: a load balancer private router
  • Endpoint service (private link)
  • Volumes

    • For etcd (one or three depending on high availability)
    • For OVN-Kube
5.1.1.5. Infrastructure requirements for an AWS account in a hosted cluster

When your infrastructure is managed by hosted control planes in a hosted cluster Amazon Web Services (AWS) account, the infrastructure requirements differ depending on whether your clusters are public, private, or a combination.

For accounts with public clusters, the infrastructure requirements are as follows:

  • Node pools must have EC2 instances that have Role and RolePolicy defined.

For accounts with private clusters, the infrastructure requirements are as follows:

  • One private link endpoint for each availability zone
  • EC2 instances for node pools

For accounts with public and private clusters, the infrastructure requirements are as follows:

  • One private link endpoint for each availability zone
  • EC2 instances for node pools
5.1.1.6. Kubernetes-managed infrastructure in a hosted cluster AWS account

When Kubernetes manages your infrastructure in a hosted cluster Amazon Web Services (AWS) account, the infrastructure requirements are as follows:

  • A network load balancer for default Ingress
  • An S3 bucket for registry

5.1.2. Identity and Access Management (IAM) permissions

In the context of hosted control planes, the consumer is responsible to create the Amazon Resource Name (ARN) roles. The consumer is an automated process to generate the permissions files. The consumer might be the CLI or OpenShift Cluster Manager. Hosted control planes can enable granularity to honor the principle of least-privilege components, which means that every component uses its own role to operate or create Amazon Web Services (AWS) objects, and the roles are limited to what is required for the product to function normally.

The hosted cluster receives the ARN roles as input and the consumer creates an AWS permission configuration for each component. As a result, the component can authenticate through STS and preconfigured OIDC IDP.

The following roles are consumed by some of the components from hosted control planes that run on the control plane and operate on the data plane:

  • controlPlaneOperatorARN
  • imageRegistryARN
  • ingressARN
  • kubeCloudControllerARN
  • nodePoolManagementARN
  • storageARN
  • networkARN

The following example shows a reference to the IAM roles from the hosted cluster:

...
endpointAccess: Public
  region: us-east-2
  resourceTags:
  - key: kubernetes.io/cluster/example-cluster-bz4j5
    value: owned
rolesRef:
    controlPlaneOperatorARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-control-plane-operator
    imageRegistryARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-openshift-image-registry
    ingressARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-openshift-ingress
    kubeCloudControllerARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-cloud-controller
    networkARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-cloud-network-config-controller
    nodePoolManagementARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-node-pool
    storageARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-aws-ebs-csi-driver-controller
type: AWS
...

The roles that hosted control planes uses are shown in the following examples:

  • ingressARN

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "elasticloadbalancing:DescribeLoadBalancers",
                    "tag:GetResources",
                    "route53:ListHostedZones"
                ],
                "Resource": "\*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "route53:ChangeResourceRecordSets"
                ],
                "Resource": [
                    "arn:aws:route53:::PUBLIC_ZONE_ID",
                    "arn:aws:route53:::PRIVATE_ZONE_ID"
                ]
            }
        ]
    }
  • imageRegistryARN

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:CreateBucket",
                    "s3:DeleteBucket",
                    "s3:PutBucketTagging",
                    "s3:GetBucketTagging",
                    "s3:PutBucketPublicAccessBlock",
                    "s3:GetBucketPublicAccessBlock",
                    "s3:PutEncryptionConfiguration",
                    "s3:GetEncryptionConfiguration",
                    "s3:PutLifecycleConfiguration",
                    "s3:GetLifecycleConfiguration",
                    "s3:GetBucketLocation",
                    "s3:ListBucket",
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject",
                    "s3:ListBucketMultipartUploads",
                    "s3:AbortMultipartUpload",
                    "s3:ListMultipartUploadParts"
                ],
                "Resource": "\*"
            }
        ]
    }
  • storageARN

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:AttachVolume",
                    "ec2:CreateSnapshot",
                    "ec2:CreateTags",
                    "ec2:CreateVolume",
                    "ec2:DeleteSnapshot",
                    "ec2:DeleteTags",
                    "ec2:DeleteVolume",
                    "ec2:DescribeInstances",
                    "ec2:DescribeSnapshots",
                    "ec2:DescribeTags",
                    "ec2:DescribeVolumes",
                    "ec2:DescribeVolumesModifications",
                    "ec2:DetachVolume",
                    "ec2:ModifyVolume"
                ],
                "Resource": "\*"
            }
        ]
    }
  • networkARN

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:DescribeInstances",
                    "ec2:DescribeInstanceStatus",
                    "ec2:DescribeInstanceTypes",
                    "ec2:UnassignPrivateIpAddresses",
                    "ec2:AssignPrivateIpAddresses",
                    "ec2:UnassignIpv6Addresses",
                    "ec2:AssignIpv6Addresses",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeNetworkInterfaces"
                ],
                "Resource": "\*"
            }
        ]
    }
  • kubeCloudControllerARN

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "ec2:DescribeInstances",
                    "ec2:DescribeImages",
                    "ec2:DescribeRegions",
                    "ec2:DescribeRouteTables",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeVolumes",
                    "ec2:CreateSecurityGroup",
                    "ec2:CreateTags",
                    "ec2:CreateVolume",
                    "ec2:ModifyInstanceAttribute",
                    "ec2:ModifyVolume",
                    "ec2:AttachVolume",
                    "ec2:AuthorizeSecurityGroupIngress",
                    "ec2:CreateRoute",
                    "ec2:DeleteRoute",
                    "ec2:DeleteSecurityGroup",
                    "ec2:DeleteVolume",
                    "ec2:DetachVolume",
                    "ec2:RevokeSecurityGroupIngress",
                    "ec2:DescribeVpcs",
                    "elasticloadbalancing:AddTags",
                    "elasticloadbalancing:AttachLoadBalancerToSubnets",
                    "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
                    "elasticloadbalancing:CreateLoadBalancer",
                    "elasticloadbalancing:CreateLoadBalancerPolicy",
                    "elasticloadbalancing:CreateLoadBalancerListeners",
                    "elasticloadbalancing:ConfigureHealthCheck",
                    "elasticloadbalancing:DeleteLoadBalancer",
                    "elasticloadbalancing:DeleteLoadBalancerListeners",
                    "elasticloadbalancing:DescribeLoadBalancers",
                    "elasticloadbalancing:DescribeLoadBalancerAttributes",
                    "elasticloadbalancing:DetachLoadBalancerFromSubnets",
                    "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                    "elasticloadbalancing:ModifyLoadBalancerAttributes",
                    "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                    "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
                    "elasticloadbalancing:AddTags",
                    "elasticloadbalancing:CreateListener",
                    "elasticloadbalancing:CreateTargetGroup",
                    "elasticloadbalancing:DeleteListener",
                    "elasticloadbalancing:DeleteTargetGroup",
                    "elasticloadbalancing:DescribeListeners",
                    "elasticloadbalancing:DescribeLoadBalancerPolicies",
                    "elasticloadbalancing:DescribeTargetGroups",
                    "elasticloadbalancing:DescribeTargetHealth",
                    "elasticloadbalancing:ModifyListener",
                    "elasticloadbalancing:ModifyTargetGroup",
                    "elasticloadbalancing:RegisterTargets",
                    "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
                    "iam:CreateServiceLinkedRole",
                    "kms:DescribeKey"
                ],
                "Resource": [
                    "\*"
                ],
                "Effect": "Allow"
            }
        ]
    }
  • nodePoolManagementARN

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "ec2:AllocateAddress",
                    "ec2:AssociateRouteTable",
                    "ec2:AttachInternetGateway",
                    "ec2:AuthorizeSecurityGroupIngress",
                    "ec2:CreateInternetGateway",
                    "ec2:CreateNatGateway",
                    "ec2:CreateRoute",
                    "ec2:CreateRouteTable",
                    "ec2:CreateSecurityGroup",
                    "ec2:CreateSubnet",
                    "ec2:CreateTags",
                    "ec2:DeleteInternetGateway",
                    "ec2:DeleteNatGateway",
                    "ec2:DeleteRouteTable",
                    "ec2:DeleteSecurityGroup",
                    "ec2:DeleteSubnet",
                    "ec2:DeleteTags",
                    "ec2:DescribeAccountAttributes",
                    "ec2:DescribeAddresses",
                    "ec2:DescribeAvailabilityZones",
                    "ec2:DescribeImages",
                    "ec2:DescribeInstances",
                    "ec2:DescribeInternetGateways",
                    "ec2:DescribeNatGateways",
                    "ec2:DescribeNetworkInterfaces",
                    "ec2:DescribeNetworkInterfaceAttribute",
                    "ec2:DescribeRouteTables",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeVpcs",
                    "ec2:DescribeVpcAttribute",
                    "ec2:DescribeVolumes",
                    "ec2:DetachInternetGateway",
                    "ec2:DisassociateRouteTable",
                    "ec2:DisassociateAddress",
                    "ec2:ModifyInstanceAttribute",
                    "ec2:ModifyNetworkInterfaceAttribute",
                    "ec2:ModifySubnetAttribute",
                    "ec2:ReleaseAddress",
                    "ec2:RevokeSecurityGroupIngress",
                    "ec2:RunInstances",
                    "ec2:TerminateInstances",
                    "tag:GetResources",
                    "ec2:CreateLaunchTemplate",
                    "ec2:CreateLaunchTemplateVersion",
                    "ec2:DescribeLaunchTemplates",
                    "ec2:DescribeLaunchTemplateVersions",
                    "ec2:DeleteLaunchTemplate",
                    "ec2:DeleteLaunchTemplateVersions"
                ],
                "Resource": [
                    "\*"
                ],
                "Effect": "Allow"
            },
            {
                "Condition": {
                    "StringLike": {
                        "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                    }
                },
                "Action": [
                    "iam:CreateServiceLinkedRole"
                ],
                "Resource": [
                    "arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"
                ],
                "Effect": "Allow"
            },
            {
                "Action": [
                    "iam:PassRole"
                ],
                "Resource": [
                    "arn:*:iam::*:role/*-worker-role"
                ],
                "Effect": "Allow"
            }
        ]
    }
  • controlPlaneOperatorARN

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateVpcEndpoint",
                    "ec2:DescribeVpcEndpoints",
                    "ec2:ModifyVpcEndpoint",
                    "ec2:DeleteVpcEndpoints",
                    "ec2:CreateTags",
                    "route53:ListHostedZones"
                ],
                "Resource": "\*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "route53:ChangeResourceRecordSets",
                    "route53:ListResourceRecordSets"
                ],
                "Resource": "arn:aws:route53:::%s"
            }
        ]
    }

5.1.3. Creating AWS infrastructure and IAM resources separate

By default, the hcp create cluster aws command creates cloud infrastructure with the hosted cluster and applies it. You can create the cloud infrastructure portion separately so that you can use the hcp create cluster aws command only to create the cluster, or render it to modify it before you apply it.

To create the cloud infrastructure portion separately, you need to create the Amazon Web Services (AWS) infrastructure, create the AWS Identity and Access (IAM) resources, and create the cluster.

5.1.3.1. Creating the AWS infrastructure separately

To create the Amazon Web Services (AWS) infrastructure, you need to create a Virtual Private Cloud (VPC) and other resources for your cluster. You can use the AWS console or an infrastructure automation and provisioning tool. For instructions to use the AWS console, see Create a VPC plus other VPC resources in the AWS Documentation.

The VPC must include private and public subnets and resources for external access, such as a network address translation (NAT) gateway and an internet gateway. In addition to the VPC, you need a private hosted zone for the ingress of your cluster. If you are creating clusters that use PrivateLink (Private or PublicAndPrivate access modes), you need an additional hosted zone for PrivateLink.

Create the AWS infrastructure for your hosted cluster by using the following example configuration:

---
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: clusters
spec: {}
status: {}
---
apiVersion: v1
data:
  .dockerconfigjson: xxxxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    hypershift.openshift.io/safe-to-delete-with-cluster: "true"
  name: <pull_secret_name> 1
  namespace: clusters
---
apiVersion: v1
data:
  key: xxxxxxxxxxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    hypershift.openshift.io/safe-to-delete-with-cluster: "true"
  name: <etcd_encryption_key_name> 2
  namespace: clusters
type: Opaque
---
apiVersion: v1
data:
  id_rsa: xxxxxxxxx
  id_rsa.pub: xxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    hypershift.openshift.io/safe-to-delete-with-cluster: "true"
  name: <ssh-key-name> 3
  namespace: clusters
---
apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  creationTimestamp: null
  name: <hosted_cluster_name> 4
  namespace: clusters
spec:
  autoscaling: {}
  configuration: {}
  controllerAvailabilityPolicy: SingleReplica
  dns:
    baseDomain: <dns_domain> 5
    privateZoneID: xxxxxxxx
    publicZoneID: xxxxxxxx
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
          storageClassName: gp3-csi
        type: PersistentVolume
    managementType: Managed
  fips: false
  infraID: <infra_id> 6
  issuerURL: <issuer_url> 7
  networking:
    clusterNetwork:
    - cidr: 10.132.0.0/14
    machineNetwork:
    - cidr: 10.0.0.0/16
    networkType: OVNKubernetes
    serviceNetwork:
    - cidr: 172.31.0.0/16
  olmCatalogPlacement: management
  platform:
    aws:
      cloudProviderConfig:
        subnet:
          id: <subnet_xxx> 8
        vpc: <vpc_xxx> 9
        zone: us-west-1b
      endpointAccess: Public
      multiArch: false
      region: us-west-1
      rolesRef:
        controlPlaneOperatorARN: arn:aws:iam::820196288204:role/<infra_id>-control-plane-operator
        imageRegistryARN: arn:aws:iam::820196288204:role/<infra_id>-openshift-image-registry
        ingressARN: arn:aws:iam::820196288204:role/<infra_id>-openshift-ingress
        kubeCloudControllerARN: arn:aws:iam::820196288204:role/<infra_id>-cloud-controller
        networkARN: arn:aws:iam::820196288204:role/<infra_id>-cloud-network-config-controller
        nodePoolManagementARN: arn:aws:iam::820196288204:role/<infra_id>-node-pool
        storageARN: arn:aws:iam::820196288204:role/<infra_id>-aws-ebs-csi-driver-controller
    type: AWS
  pullSecret:
    name: <pull_secret_name>
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.16-x86_64
  secretEncryption:
    aescbc:
      activeKey:
        name: <etcd_encryption_key_name>
    type: aescbc
  services:
  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  - service: OVNSbDb
    servicePublishingStrategy:
      type: Route
  sshKey:
    name: <ssh_key_name>
status:
  controlPlaneEndpoint:
    host: ""
    port: 0
---
apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  creationTimestamp: null
  name: <node_pool_name> 10
  namespace: clusters
spec:
  arch: amd64
  clusterName: <hosted_cluster_name>
  management:
    autoRepair: true
    upgradeType: Replace
  nodeDrainTimeout: 0s
  platform:
    aws:
      instanceProfile: <instance_profile_name> 11
      instanceType: m6i.xlarge
      rootVolume:
        size: 120
        type: gp3
      subnet:
        id: <subnet_xxx>
    type: AWS
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.16-x86_64
  replicas: 2
status:
  replicas: 0
1
Replace <pull_secret_name> with the name of your pull secret.
2
Replace <etcd_encryption_key_name> with the name of your etcd encryption key.
3
Replace <ssh_key_name> with the name of your SSH key.
4
Replace <hosted_cluster_name> with the name of your hosted cluster.
5
Replace <dns_domain> with your base DNS domain, such as example.com.
6
Replace <infra_id> with the value that identifies the IAM resources that are associated with the hosted cluster.
7
Replace <issuer_url> with your issuer URL, which ends with your infra_id value. For example, https://example-hosted-us-west-1.s3.us-west-1.amazonaws.com/example-hosted-infra-id.
8
Replace <subnet_xxx> with your subnet ID. Both private and public subnets need to be tagged. For public subnets, use kubernetes.io/role/elb=1. For private subnets, use kubernetes.io/role/internal-elb=1.
9
Replace <vpc_xxx> with your VPC ID.
10
Replace <node_pool_name> with the name of your NodePool resource.
11
Replace <instance_profile_name> with the name of your AWS instance.
5.1.3.2. Creating the AWS IAM resources

In Amazon Web Services (AWS), you must create the following IAM resources:

5.1.3.3. Creating a hosted cluster separately

You can create a hosted cluster separately on Amazon Web Services (AWS).

To create a hosted cluster separately, enter the following command:

$ hcp create cluster aws \
    --infra-id <infra_id> \1
    --name <hosted_cluster_name> \2
    --sts-creds <path_to_sts_credential_file> \3
    --pull-secret <path_to_pull_secret> \4
    --generate-ssh \5
    --node-pool-replicas 3
    --role-arn <role_name> 6
1
Replace <infra_id> with the same ID that you specified in the create infra aws command. This value identifies the IAM resources that are associated with the hosted cluster.
2
Replace <hosted_cluster_name> with the name of your hosted cluster.
3
Replace <path_to_sts_credential_file> with the same name that you specified in the create infra aws command.
4
Replace <path_to_pull_secret> with the name of the file that contains a valid OpenShift Container Platform pull secret.
5
The --generate-ssh flag is optional, but is good to include in case you need to SSH to your workers. An SSH key is generated for you and is stored as a secret in the same namespace as the hosted cluster.
6
Replace <role_name> with the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole. Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole. For more information about ARN roles, see "Identity and Access Management (IAM) permissions".

You can also add the --render flag to the command and redirect output to a file where you can edit the resources before you apply them to the cluster.

After you run the command, the following resources are applied to your cluster:

  • A namespace
  • A secret with your pull secret
  • A HostedCluster
  • A NodePool
  • Three AWS STS secrets for control plane components
  • One SSH key secret if you specified the --generate-ssh flag.

5.2. Managing hosted control planes on bare metal

After you deploy hosted control planes on bare metal, you can manage a hosted cluster by completing the following tasks.

5.2.1. Accessing the hosted cluster

You can access the hosted cluster by either getting the kubeconfig file and kubeadmin credential directly from resources, or by using the hcp command line interface to generate a kubeconfig file.

Prerequisites

To access the hosted cluster by getting the kubeconfig file and credentials directly from resources, you must be familiar with the access secrets for hosted clusters. The hosted cluster (hosting) namespace contains hosted cluster resources and the access secrets. The hosted control plane namespace is where the hosted control plane runs.

The secret name formats are as follows:

  • kubeconfig secret: <hosted_cluster_namespace>-<name>-admin-kubeconfig. For example, clusters-hypershift-demo-admin-kubeconfig.
  • kubeadmin password secret: <hosted_cluster_namespace>-<name>-kubeadmin-password. For example, clusters-hypershift-demo-kubeadmin-password.

The kubeconfig secret contains a Base64-encoded kubeconfig field, which you can decode and save into a file to use with the following command:

$ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

The kubeadmin password secret is also Base64-encoded. You can decode it and use the password to log in to the API server or console of the hosted cluster.

Procedure

  • To access the hosted cluster by using the hcp CLI to generate the kubeconfig file, take the following steps:

    1. Generate the kubeconfig file by entering the following command:

      $ hcp create kubeconfig --namespace <hosted_cluster_namespace> --name <hosted_cluster_name> > <hosted_cluster_name>.kubeconfig
    2. After you save the kubeconfig file, you can access the hosted cluster by entering the following example command:

      $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

5.2.2. Scaling the NodePool object for a hosted cluster

You can scale up the NodePool object by adding nodes to your hosted cluster. When you scale a node pool, consider the following information:

  • When you scale a replica by the node pool, a machine is created. For every machine, the Cluster API provider finds and installs an Agent that meets the requirements that are specified in the node pool specification. You can monitor the installation of an Agent by checking its status and conditions.
  • When you scale down a node pool, Agents are unbound from the corresponding cluster. Before you can reuse the Agents, you must restart them by using the Discovery image.

Procedure

  1. Scale the NodePool object to two nodes:

    $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

    The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through states in the following order:

    • binding
    • discovering
    • insufficient
    • installing
    • installing-in-progress
    • added-to-existing-cluster
  2. Enter the following command:

    $ oc -n <hosted_control_plane_namespace> get agent

    Example output

    NAME                                   CLUSTER         APPROVED   ROLE          STAGE
    4dac1ab2-7dd5-4894-a220-6a3473b67ee6   hypercluster1   true       auto-assign
    d9198891-39f4-4930-a679-65fb142b108b                   true       auto-assign
    da503cf1-a347-44f2-875c-4960ddb04091   hypercluster1   true       auto-assign

  3. Enter the following command:

    $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

    Example output

    BMH: ocp-worker-2 Agent: 4dac1ab2-7dd5-4894-a220-6a3473b67ee6 State: binding
    BMH: ocp-worker-0 Agent: d9198891-39f4-4930-a679-65fb142b108b State: known-unbound
    BMH: ocp-worker-1 Agent: da503cf1-a347-44f2-875c-4960ddb04091 State: insufficient

  4. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

    $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>
  5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes in the hosted cluster by entering the following command:

    $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get nodes

    Example output

    NAME           STATUS   ROLES    AGE     VERSION
    ocp-worker-1   Ready    worker   5m41s   v1.24.0+3882f8f
    ocp-worker-2   Ready    worker   6m3s    v1.24.0+3882f8f

    Cluster Operators start to reconcile by adding workloads to the nodes.

  6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

    $ oc -n <hosted_control_plane_namespace> get machines

    Example output

    NAME                            CLUSTER               NODENAME       PROVIDERID                                     PHASE     AGE   VERSION
    hypercluster1-c96b6f675-m5vch   hypercluster1-b2qhl   ocp-worker-1   agent://da503cf1-a347-44f2-875c-4960ddb04091   Running   15m   4.x.z
    hypercluster1-c96b6f675-tl42p   hypercluster1-b2qhl   ocp-worker-2   agent://4dac1ab2-7dd5-4894-a220-6a3473b67ee6   Running   15m   4.x.z

    The clusterversion reconcile process eventually reaches a point where only Ingress and Console cluster operators are missing.

  7. Enter the following command:

    $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get clusterversion,co

    Example output

    NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    clusterversion.config.openshift.io/version             False       True          40m     Unable to apply 4.x.z: the cluster operator console has not yet successfully rolled out
    
    NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    clusteroperator.config.openshift.io/console                                      4.12z     False       False         False      11m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.hypercluster1.domain.com): Get "https://console-openshift-console.apps.hypercluster1.domain.com": dial tcp 10.19.3.29:443: connect: connection refused
    clusteroperator.config.openshift.io/csi-snapshot-controller                      4.12z     True        False         False      10m
    clusteroperator.config.openshift.io/dns                                          4.12z     True        False         False      9m16s

5.2.2.1. Adding node pools

You can create node pools for a hosted cluster by specifying a name, number of replicas, and any additional information, such as an agent label selector.

Procedure

  1. To create a node pool, enter the following information:

    $ hcp create nodepool agent \
      --cluster-name <hosted_cluster_name> \1
      --name <nodepool_name> \2
      --node-count <worker_node_count> \3
      --agentLabelSelector '{"matchLabels": {"size": "medium"}}' 4
    1
    Replace <hosted_cluster_name> with your hosted cluster name.
    2
    Replace <nodepool_name> with the name of your node pool, for example, <hosted_cluster_name>-extra-cpu.
    3
    Replace <worker_node_count> with the worker node count, for example, 2.
    4
    The --agentLabelSelector flag is optional. The node pool uses agents with the "size" : "medium" label.
  2. Check the status of the node pool by listing nodepool resources in the clusters namespace:

    $ oc get nodepools --namespace clusters
  3. Extract the admin-kubeconfig secret by entering the following command:

    $ oc extract -n <hosted_control_plane_namespace> secret/admin-kubeconfig --to=./hostedcluster-secrets --confirm

    Example output

    hostedcluster-secrets/kubeconfig

  4. After some time, you can check the status of the node pool by entering the following command:

    $ oc --kubeconfig ./hostedcluster-secrets get nodes

Verification

  • Verify that the number of available node pools match the number of expected node pools by entering this command:

    $ oc get nodepools --namespace clusters
5.2.2.2. Enabling node auto-scaling for the hosted cluster

When you need more capacity in your hosted cluster and spare agents are available, you can enable auto-scaling to install new worker nodes.

Procedure

  1. To enable auto-scaling, enter the following command:

    $ oc -n <hosted_cluster_namespace> patch nodepool <hosted_cluster_name> --type=json -p '[{"op": "remove", "path": "/spec/replicas"},{"op":"add", "path": "/spec/autoScaling", "value": { "max": 5, "min": 2 }}]'
    Note

    In the example, the minimum number of nodes is 2, and the maximum is 5. The maximum number of nodes that you can add might be bound by your platform. For example, if you use the Agent platform, the maximum number of nodes is bound by the number of available agents.

  2. Create a workload that requires a new node.

    1. Create a YAML file that contains the workload configuration, by using the following example:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        creationTimestamp: null
        labels:
          app: reversewords
        name: reversewords
        namespace: default
      spec:
        replicas: 40
        selector:
          matchLabels:
            app: reversewords
        strategy: {}
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: reversewords
        spec:
          containers:
          - image: quay.io/mavazque/reversewords:latest
            name: reversewords
            resources:
              requests:
                memory: 2Gi
      status: {}
    2. Save the file as workload-config.yaml.
    3. Apply the YAML by entering the following command:

      $ oc apply -f workload-config.yaml
  3. Extract the admin-kubeconfig secret by entering the following command:

    $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=./hostedcluster-secrets --confirm

    Example output

    hostedcluster-secrets/kubeconfig

  4. You can check if new nodes are in the Ready status by entering the following command:

    $ oc --kubeconfig ./hostedcluster-secrets get nodes
  5. To remove the node, delete the workload by entering the following command:

    $ oc --kubeconfig ./hostedcluster-secrets -n <namespace> delete deployment <deployment_name>
  6. Wait for several minutes to pass without requiring the additional capacity. On the Agent platform, the agent is decommissioned and can be reused. You can confirm that the node was removed by entering the following command:

    $ oc --kubeconfig ./hostedcluster-secrets get nodes
Note

For IBM Z agents, compute nodes are detached from the cluster only for IBM Z with KVM agents. For z/VM and LPAR, you must delete the compute nodes manually.

Agents can be reused only for IBM Z with KVM. For z/VM and LPAR, re-create the agents to use them as compute nodes.

5.2.2.3. Disabling node auto-scaling for the hosted cluster

To disable node auto-scaling, complete the following procedure.

Procedure

  • Enter the following command to disable node auto-scaling for the hosted cluster:

    $ oc -n <hosted_cluster_namespace> patch nodepool <hosted_cluster_name> --type=json -p '[\{"op":"remove", "path": "/spec/autoScaling"}, \{"op": "add", "path": "/spec/replicas", "value": <specify_value_to_scale_replicas>]'

    The command removes "spec.autoScaling" from the YAML file, adds "spec.replicas", and sets "spec.replicas" to the integer value that you specify.

5.2.3. Handling ingress in a hosted cluster on bare metal

Every OpenShift Container Platform cluster has a default application Ingress Controller that typically has an external DNS record associated with it. For example, if you create a hosted cluster named example with the base domain krnl.es, you can expect the wildcard domain *.apps.example.krnl.es to be routable.

Procedure

To set up a load balancer and wildcard DNS record for the *.apps domain, perform the following actions on your guest cluster:

  1. Deploy MetalLB by creating a YAML file that contains the configuration for the MetalLB Operator:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: metallb
      labels:
        openshift.io/cluster-monitoring: "true"
      annotations:
        workload.openshift.io/allowed: management
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: metallb-operator-operatorgroup
      namespace: metallb
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: metallb-operator
      namespace: metallb
    spec:
      channel: "stable"
      name: metallb-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
  2. Save the file as metallb-operator-config.yaml.
  3. Enter the following command to apply the configuration:

    $ oc apply -f metallb-operator-config.yaml
  4. After the Operator is running, create the MetalLB instance:

    1. Create a YAML file that contains the configuration for the MetalLB instance:

      apiVersion: metallb.io/v1beta1
      kind: MetalLB
      metadata:
        name: metallb
        namespace: metallb
    2. Save the file as metallb-instance-config.yaml.
    3. Create the MetalLB instance by entering this command:

      $ oc apply -f metallb-instance-config.yaml
  5. Create an IPAddressPool resource with a single IP address. This IP address must be on the same subnet as the network that the cluster nodes use.

    1. Create a file, such as ipaddresspool.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        namespace: metallb
        name: <ip_address_pool_name> 1
      spec:
        addresses:
          - <ingress_ip>-<ingress_ip> 2
        autoAssign: false
      1
      Specify the IPAddressPool resource name.
      2
      Specify the IP address for your environment. For example, 192.168.122.23.
    2. Apply the configuration for the IP address pool by entering the following command:

      $ oc apply -f ipaddresspool.yaml
  6. Create a L2 advertisement.

    1. Create a file, such as l2advertisement.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: L2Advertisement
      metadata:
        name: <l2_advertisement_name> 1
        namespace: metallb
      spec:
        ipAddressPools:
         - <ip_address_pool_name> 2
      1
      Specify the L2Advertisement resource name.
      2
      Specify the IPAddressPool resource name.
    2. Apply the configuration by entering the following command:

      $ oc apply -f l2advertisement.yaml
  7. After creating a service of the LoadBalancer type, MetalLB adds an external IP address for the service.

    1. Configure a new load balancer service that routes ingress traffic to the ingress deployment by creating a YAML file named metallb-loadbalancer-service.yaml:

      kind: Service
      apiVersion: v1
      metadata:
        annotations:
          metallb.universe.tf/address-pool: ingress-public-ip
        name: metallb-ingress
        namespace: openshift-ingress
      spec:
        ports:
          - name: http
            protocol: TCP
            port: 80
            targetPort: 80
          - name: https
            protocol: TCP
            port: 443
            targetPort: 443
        selector:
          ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
        type: LoadBalancer
    2. Save the metallb-loadbalancer-service.yaml file.
    3. Enter the following command to apply the YAML configuration:

      $ oc apply -f metallb-loadbalancer-service.yaml
    4. Enter the following command to reach the OpenShift Container Platform console:

      $ curl -kI https://console-openshift-console.apps.example.krnl.es

      Example output

      HTTP/1.1 200 OK

    5. Check the clusterversion and clusteroperator values to verify that everything is running. Enter the following command:

      $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

      Example output

      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version   4.x.y      True        False        3m32s   Cluster version is 4.x.y
      
      NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      clusteroperator.config.openshift.io/console                                      4.x.y     True        False         False      3m50s
      clusteroperator.config.openshift.io/ingress                                      4.x.y     True        False         False      53m

      Replace <4.x.y> with the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.

5.2.4. Enabling machine health checks on bare metal

You can enable machine health checks on bare metal to repair and replace unhealthy managed cluster nodes automatically. You must have additional agent machines that are ready to install in the managed cluster.

Consider the following limitations before enabling machine health checks: