Installing OpenShift Container Platform with the Assisted Installer
User Guide
Abstract
Preface Copy linkLink copied to clipboard!
Providing feedback on the Assisted Installer documentation
You can give feedback or report an error in our documentation by creating a Jira issue. You must have a Red Hat Jira account.
Procedure
- Log in to Jira.
- Click Create.
Set the following fields as specified:
- Space: Hybrid Cloud Infrastructure Documentation
- Work type: Bug
- Component: Assisted Installer SaaS
- Complete the Summary, Description, Reporter, and Priority fields.
Click Create to submit the form.
The form creates an issue in the Red Hat Hybrid Cloud Infrastructure (HCIDOCS) Jira project.
Chapter 1. About the Assisted Installer Copy linkLink copied to clipboard!
You can install OpenShift Container Platform on on-premise hardware or on-premise virtual machines by using the Assisted Installer.
1.1. Using the Assisted Installer Copy linkLink copied to clipboard!
The Assisted Installer for Red Hat OpenShift Container Platform is a user-friendly installation solution offered on the Red Hat Hybrid Cloud Console. The Assisted Installer supports various deployment platforms with a focus on bare metal, Nutanix, vSphere, and Oracle Cloud Infrastructure. The Assisted Installer also supports various CPU architectures, including x86_64, s390x (IBM Z®), arm64, and ppc64le (IBM Power®).
You can install OpenShift Container Platform on premises in a connected environment, with an optional HTTP/S proxy, for the following platforms:
- Highly available OpenShift Container Platform or single-node OpenShift cluster
- OpenShift Container Platform on bare metal or vSphere with full platform integration, or other virtualization platforms without integration
- Optionally, OpenShift Virtualization and Red Hat OpenShift Data Foundation
For access, see the following links:
1.2. Features Copy linkLink copied to clipboard!
The Assisted Installer provides installation functionality as a service. This software-as-a-service (SaaS) approach has the following features:
- Web interface
- You can install your cluster by using the Assisted Installer in the Hybrid Cloud Console instead of creating installation configuration files manually.
- No bootstrap node
- You do not need a bootstrap node because the bootstrapping process runs on a node within the cluster.
- Streamlined installation workflow
- You do not need in-depth knowledge of OpenShift Container Platform to deploy a cluster. The Assisted Installer provides reasonable default configurations.
- You do not need to run the OpenShift Container Platform installer locally.
- You have access to the latest Assisted Installer for the latest tested z-stream releases.
- Advanced networking options
- The Assisted Installer supports IPv4 and dual stack networking with OVN only, NMState-based static IP addressing, and an HTTP/S proxy.
- OVN is the default Container Network Interface (CNI) for OpenShift Container Platform 4.12 and later.
- SDN is supported up to OpenShift Container Platform 4.14. SDN supports IPv4 only.
- Preinstallation validation
Before installing, the Assisted Installer checks the following configurations:
- Network connectivity
- Network bandwidth
- Connectivity to the registry
- Upstream DNS resolution of the domain name
- Time synchronization between cluster nodes
- Cluster node hardware
- Installation configuration parameters
- REST API
- You can automate the installation process by using the Assisted Installer REST API.
1.3. Host topology Copy linkLink copied to clipboard!
The OpenShift Container Platform architecture allows you to select a standard Kubernetes role for each of the discovered hosts. These roles define the function of the host within the cluster.
1.3.1. Supported host roles Copy linkLink copied to clipboard!
During the installation process, you can select a role for a host or configure the Assisted Installer to assign it for you.
The host must meet the minimum requirements for the role you selected. You can find the hardware requirements by referring to the Prerequisites section of this document or using the preflight requirement API.
If you do not select a role, the system selects one for you. You can change the role at any time before installation starts.
Each host can have any of the following roles:
- Control plane
- Compute
- Arbiter
- Auto-assign
1.3.1.1. Control plane (master) role Copy linkLink copied to clipboard!
The control plane nodes run the services that are required to control the cluster, including the API server. The control plane schedules workloads, maintains cluster state, and ensures stability.
1.3.1.2. Compute (worker) role Copy linkLink copied to clipboard!
The compute nodes are responsible for executing workloads for cluster users. Compute nodes advertise their capacity, so that the control plane scheduler can identify suitable compute nodes for running pods and containers.
1.3.1.3. Arbiter role Copy linkLink copied to clipboard!
Arbiter nodes are a more cost-effective alternative to control plane nodes. They function similarly but run only the essential components required to maintain the etcd quorum and prevent a split-brain condition. Because they do not host the full control plane or any workloads, arbiter nodes can use less powerful hardware.
The Assisted Installer provides arbiter nodes for Two-Node OpenShift with Arbiter (TNA) clusters. Support for Two-Node OpenShift with Arbiter clusters begins with OpenShift Container Platform version 4.19 and later. For more details, see "Two-Node OpenShift with Arbiter (TNA) resource requirements" in Additional resources.
To install a Two-Node OpenShift with Arbiter cluster, assign the arbiter or auto-assign role to at least one of the nodes, and set the control plane node count for the cluster to 2.
Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
1.3.1.4. Auto-assign role Copy linkLink copied to clipboard!
The Assisted Installer sets each host to an auto-assign role by default. Auto-assign allows the Assisted Installer to automatically determine whether the host should function as a control plane, arbiter, or compute (worker) role, based on detected hardware and network latency.
To determine the most suitable role, the Assisted Installer evaluates each host’s memory, CPU, disk space, and network performance. It assigns an internal suggested_role value to each host, which drives the auto-assignment process when auto-assigned is enabled. Pre-installation validations ensure the resulting role allocation is valid.
The logic for auto-assigning roles is as follows:
Sort hosts by capability
The Assisted Installer sorts the hosts by their hardware capabilities, from weakest to strongest. All hosts must meet the minimum requirements.
Assign control plane roles
The Assisted Installer assigns the control plane role to the weakest hosts first, until it reaches the number of control plane nodes specified by the
control_plane_countfield. A host is assigned a control plane role only if passes the necessary control plane role validations. For details on specifying the control plane count, see Additional resources.Assign arbiter role
The Assisted Installer assigns the arbiter role to a host only when the following conditions are met:
- The control plane count is 2.
- The host meets the minimum hardware requirements for the cluster.
One of the following applies:
- The cluster already contains two control plane nodes, either manually assigned or auto-assigned; or
- The host does not meet the minimum hardware requirements for a control plane node.
Handle GPU hosts
By default, the Assisted Installer designates non-GPU hosts as control plane nodes and GPU hosts as worker nodes.
The Assisted Installer designates a GPU host as a control plane node in either of the following scenarios:
- When no other available hosts meet the minimum requirements.
- When the number of required control plane nodes exceeds the number of available non-GPU hosts.
Assign remaining hosts
The Assisted Installer designates all remaining hosts as worker (compute) nodes. This approach ensures that the Assisted Installer prioritizes the most capable hosts for worker roles, while still maintaining the necessary number of valid control plane and arbiter nodes.
To assign a role to a host using the web console or the API, or to troubleshoot pre-installation validation errors for hosts with an auto-assign role, see Additional resources.
1.3.2. Control plane count Copy linkLink copied to clipboard!
The control plane count is the number of control plane (master) nodes in the cluster. Using a higher number of control plane nodes boosts fault tolerance and availability, minimizing downtime during failures. The number of control plane nodes that the Assisted Installer supports varies according to OpenShift Container Platform version:
- All versions of OpenShift Container Platform support one or three control plane nodes, where one control plane node is a Single-node OpenShift cluster.
- From OpenShift Container Platform version 4.18 and later, the Assisted Installer also supports four or five control plane nodes on a bare metal or user-managed networking platform with an x86_64 architecture. An implementation can support any number of compute nodes.
- From OpenShift Container Platform version 4.19 and later, the Assisted Installer also supports two control plane nodes for a Two-Node OpenShift with Arbiter (TNA) cluster topology. A cluster with only two control plane nodes must have at least one host with an arbiter role. For details, see "Supported host roles" in Additional resources.
Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
To specify the required number of control plane nodes for your cluster in either the web console or API, see Additional resources.
1.3.3. Control plane workload scheduling Copy linkLink copied to clipboard!
For smaller clusters, scheduling workloads to run on control plane nodes improves efficiency and maximizes resource usage. You can enable this option during installation setup or as a postinstallation step.
Use the following guidelines to determine when to use this feature:
- Single-node OpenShift clusters, Two-Node OpenShift with Arbiter clusters, or clusters with up to one worker (compute) node: The system schedules workloads on control plane nodes by default. This setting cannot be changed.
- Clusters of between two to seven worker nodes: This configuration supports the manual scheduling of workloads on both control plane (master) and compute (worker) nodes.
- Clusters with more than seven worker nodes: Scheduling workloads on control plane nodes is not recommended.
Schedulable control plane nodes have the role Control plane, Worker.
When you configure control plane nodes to be schedulable for workloads, an additional subscription is required for each control plane node that function as a compute (worker) node.
For guidance on configuring control plane nodes as schedulable in the Assisted Installer during installation, and for post-installation steps in OpenShift Container Platform, see Additional resources.
1.4. API support policy Copy linkLink copied to clipboard!
Assisted Installer APIs are supported for a minimum of three months from the announcement of deprecation.
1.4.1. API deprecation notice Copy linkLink copied to clipboard!
The following table presents the deprecated and modified APIs in the Assisted Installer.
assisted_service API
Expand Affected models Description of change -
cluster -
cluster-create-params
The
high_availability_modefield is deprecated starting from April 2025, and is planned to be removed in three months. Red Hat will provide bug fixes and support for this feature during the current release lifecycle, but this feature will no longer receive enhancements and will be removed.The alternative is to use
control_plane_countinstead. This change enables support for clusters with 4 or 5 control plane nodes, in addition to the previously supported configurations with 1 or 3 control plane nodes. The Assisted Installer supports 4 or 5 control plane nodes starting from OpenShift Container Platform version 4.18 and later.-
Chapter 2. Prerequisites Copy linkLink copied to clipboard!
The Assisted Installer validates the following prerequisites to ensure successful installation. If you use a firewall, you must configure it so that the Assisted Installer can access the resources it requires to function.
2.1. Supported CPU architectures Copy linkLink copied to clipboard!
The Assisted Installer is supported on the following CPU architectures:
- x86_64
- arm64
- ppc64le (IBM Power®)
- s390x (IBM Z®)
2.2. Supported drive types Copy linkLink copied to clipboard!
Install Red Hat OpenShift Container Platform with the Assisted Installer using the appropriate drive types.
- Supported drive types
This table shows the installation drive types supported for the different OpenShift Container Platform versions and CPU architectures.
Expand Drive types RHOCP Version Supported CPU Architectures Comments HDD
All
All
A hard disk drive.
SSD
All
All
An SSD or NVMe drive.
Multipath
All
All
A Linux multipath device that can aggregate paths for various protocols. Using multipath enhances availablity and performance. Currently, the Assisted Installer supports multipathing for Fibre Channel and iSCSI protocals.
FC (Fibre Channel)
All
s390x, x86_64
Indicates a single path Fibre Channel (FC) drive. For a multipath Fibre Channel configuration, see 'Multipath' in this table.
iSCSI
4.15 and later
x86_64
- You can install a cluster on a single or multipath iSCSI boot device.
- The Assisted Installer supports iSCSI boot volumes through iPXE boot.
- A minimal ISO image is mandatory on iSCSI boot volumes. Using a full ISO image will result in an error.
- iSCSI boot requires two machine network interfaces; one for the iSCSI traffic and the other for the OpenShift Container Platform cluster installation.
- A static IP address is not supported when using iSCSI boot volumes.
RAID
4.14 and later
All
A software RAID drive. The RAID should be configured via BIOS/UEFI. If this option is unavailable, you can configure OpenShift Container Platform to mirror the drives. For details, see Encrypting and mirroring disks during installation.
ECK
All
s390x
IBM drive.
ECKD (ESE)
All
s390x
IBM drive.
FBA
All
s390x
IBM drive.
- Unsupported drive types
This table shows the installation drive types that are not supported.
Expand Drive types Comments Unknown
The system could not detect the drive type.
FDD
A floppy disk drive.
ODD
An optical disk drive (e.g., CD-ROM).
Virtual
A loopback device.
LVM
A Linux Logical Volume Management drive.
2.3. Resource requirements Copy linkLink copied to clipboard!
This section describes the resource requirements for different clusters and installation options.
The multicluster engine for Kubernetes requires additional resources.
If you deploy the multicluster engine with storage, such as OpenShift Data Foundation or LVM Storage, you must also assign additional resources to each node.
2.3.1. Multi-node cluster resource requirements Copy linkLink copied to clipboard!
The resource requirements of a multi-node (high-availability) cluster depend on the installation options.
- Description
- A standard OpenShift Container Platform cluster configuration consists of three to five control plane nodes and two or more worker nodes. This configuration ensures full high availability for control plane services.
- Multi-node cluster basic installation
Control plane nodes:
- 4 CPU cores
- 16 GB RAM
100 GB storage
NoteThe disks must be reasonably fast, with an etcd
wal_fsync_duration_secondsp99 duration that is less than 10 ms. For more information, see the Red Hat Knowledgebase solution How to Use 'fio' to Check Etcd Disk Performance in OCP.
Compute nodes:
- 2 CPU cores
- 8 GB RAM
- 100 GB storage
- Multi-node cluster + multicluster engine
- Additional 4 CPU cores
Additional 16 GB RAM
NoteIf you deploy multicluster engine without OpenShift Data Foundation, no storage is configured. You configure the storage after the installation.
- Multi-node cluster + multicluster engine + OpenShift Data Foundation
- Additional 75 GB storage
2.3.2. Two-Node OpenShift with Arbiter (TNA) cluster resource requirements Copy linkLink copied to clipboard!
The resource requirements of a Two-Node OpenShift with Arbiter (TNA) cluster depend on the installation options.
- Description
A Two-Node OpenShift with Arbiter (TNA) cluster is a compact, cost-effective OpenShift Container Platform topology. It consists of two control plane nodes and a lightweight arbiter node. The arbiter node stores the full etcd data, maintaining an etcd quorum and preventing split brain. It does not run the additional control plane components
kube-apiserverandkube-controller-manager, nor does it run workloads. For details, see Overview of etcd.To install a Two-Node OpenShift with Arbiter cluster, assign an arbiter role to at least one of the nodes and set the control plane node count for the cluster to
2. Although OpenShift Container Platform does not currently impose a limit on the number of arbiter nodes, the typical deployment includes only one to minimize the use of hardware resources.Following installation, you can add additional arbiter nodes to a Two-Node OpenShift with Arbiter cluster but not to a standard multi-node cluster. It is also not possible to convert between a Two-Node OpenShift with Arbiter and standard topology.
Support for a Two-Node OpenShift with Arbiter cluster begins with OpenShift Container Platform version 4.19 and later. This configuration is available only for bare-metal installations.
Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
- Two-Node OpenShift with Arbiter basic installation
Control plane nodes:
- 4 CPU cores
- 16 GB RAM
100 GB storage
NoteThe disks must be reasonably fast, with an etcd
wal_fsync_duration_secondsp99 duration that is less than 10 ms. For more information, see the Red Hat Knowledgebase solution How to Use 'fio' to Check Etcd Disk Performance in OCP.
Arbiter node:
- 2 CPU cores
- 8 GB RAM
- 50 GB storage
- Two-Node OpenShift with Arbiter + multicluster engine
- Additional 4 CPU cores
Additional 16 GB RAM
NoteIf you deploy multicluster engine without OpenShift Data Foundation, no storage is configured. You configure the storage after the installation.
- Two-Node OpenShift with Arbiter + multicluster engine + OpenShift Data Foundation
- Additional 75 GB storage
2.3.3. Single-node OpenShift cluster resource requirements Copy linkLink copied to clipboard!
The resource requirements for single-node OpenShift depend on the installation options.
- Description
- A single-node OpenShift cluster is an OpenShift Container Platform deployment that runs entirely on a single node. Single-node OpenShift includes the control plane and worker functionality on one physical or virtual machine.
- Single-node OpenShift basic installation
- 8 CPU cores
- 16 GB RAM
- 100 GB storage
- Single-node OpenShift + multicluster engine
- Additional 8 CPU cores
Additional 32 GB RAM
NoteIf you deploy multicluster engine without OpenShift Data Foundation, LVM Storage is enabled.
- Single-node OpenShift + multicluster engine + OpenShift Data Foundation
- Additional 95 GB storage
2.4. Networking requirements Copy linkLink copied to clipboard!
For hosts of type VMware, set clusterSet disk.EnableUUID to TRUE, even when the platform is not vSphere.
2.4.1. General networking requirements Copy linkLink copied to clipboard!
The network must meet the following requirements:
- You have configured either dynamic (DHCP) or static IP addressing.
You have selected the correct route configuration for your IP addressing method:
- For dynamic IP addressing, ensure that you have configured your network routes dynamically via DHCP.
- For static IP addressing, ensure that you have configured the network routes manually via the static networking configurations.
ImportantYou cannot combine dynamic IP addresses with static route configurations. When the Assisted Installer receives a dynamic IP address (with a
/128prefix), it specifically looks for network routes that were also configured dynamically, such as those advertised via Router Advertisement (RA). If a network route is configured manually (with a/64prefix, for example), the Assisted Installer ignores it.-
You have opened port 6443 to allow the API URL access to the cluster using the
ocCLI tool when outside the firewall. - You have opened port 22624 in all firewalls. The Machine Config Operator (MCO) and new worker nodes use port 22624 to get the ignition data from the cluster API.
- You have opened port 443 to allow console access outside the firewall. Port 443 is also used for all ingress traffic.
- You have configured DNS to connect to the cluster API or ingress endpoints from outside the cluster.
- Optional: You have created a DNS Pointer record (PTR) for each node in the cluster if using static IP addressing.
You must create a DNS PTR record to boot with a preset hostname if the hostname will not come from another source (/etc/hosts or DHCP). Otherwise, the Assisted Installer’s automatic node renaming feature will rename the nodes to their network interface MAC address.
2.4.2. External DNS Copy linkLink copied to clipboard!
Installing a multi-node cluster with user-managed networking requires external DNS records. The Assisted Installer does not require external DNS records to complete the installation of multi-node clusters with cluster-managed networking or single-node OpenShift clusters. Configure the external DNS records after installation to connect to the cluster from an external source.
External DNS requires the following records:
- A/AAAA record for api.<cluster_name>.<base_domain>.
- A/AAAA record for api-int.<cluster_name>.<base_domain>.
- A/AAAA record with a wildcard for *.apps.<cluster_name>.<base_domain>.
- A/AAAA record for each node in the cluster.
- Do not create a wildcard, such as *.<cluster_name>.<base_domain>, or the installation will not proceed.
- A/AAAA record settings at top-level domain registrars can take significant time to update. Ensure the newly created DNS Records are resolving before installation to prevent installation delays.
- For DNS record examples, see Example DNS configuration.
The OpenShift Container Platform cluster’s network must also meet the following requirements:
- Connectivity between all cluster nodes
- Connectivity for each node to the internet
- Access to an NTP server for time synchronization between the cluster nodes
The following DNS configuration provides A and PTR record configuration examples that meet the DNS requirements for deploying OpenShift Container Platform using the Assisted Installer. The examples are not meant to recommend one DNS solution over another.
In the examples, the cluster name is ocp4 and the base domain is example.com.
2.4.2.1. Example DNS A record configuration Copy linkLink copied to clipboard!
The following example DNS zone database is a BIND zone file that shows sample A records for name resolution in a cluster installed using the Assisted Installer:
$TTL 1W
@ IN SOA ns1.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.example.com.
IN MX 10 smtp.example.com.
;
;
ns1.example.com. IN A 192.168.1.1
smtp.example.com. IN A 192.168.1.5
;
helper.example.com. IN A 192.168.1.5
;
api.ocp4.example.com. IN A 192.168.1.5
api-int.ocp4.example.com. IN A 192.168.1.5
;
*.apps.ocp4.example.com. IN A 192.168.1.5
;
control-plane0.ocp4.example.com. IN A 192.168.1.97
control-plane1.ocp4.example.com. IN A 192.168.1.98
control-plane2.ocp4.example.com. IN A 192.168.1.99
;
worker0.ocp4.example.com. IN A 192.168.1.11
worker1.ocp4.example.com. IN A 192.168.1.7
;
;EOF
where:
-
api.ocp4.example.com.provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer. -
api-int.ocp4.example.com.provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer and is used for internal cluster communications. *.apps.ocp4.example.com.provides name resolution for the wildcard routes. The record refers to the IP address of the application ingress load balancer. The application ingress load balancer targets the machines that run the Ingress Controller pods. The Ingress Controller pods run on the worker machines by default.NoteIn the example, the same load balancer is used for the Kubernetes API and application ingress traffic. In production scenarios, you can deploy the API and application ingress load balancers separately so that you can scale the load balancer infrastructure for each in isolation.
-
control-plane0.ocp4.example.com.and adjacent records provide name resolution for the control plane machines. -
worker0.ocp4.example.com.and adjacent record provide name resolution for the worker machines.
2.4.2.2. Example DNS PTR record configuration Copy linkLink copied to clipboard!
The following example DNS zone database for reverse records is a BIND zone file that shows sample PTR records for reverse name resolution in a cluster installed using the Assisted Installer:
$TTL 1W
@ IN SOA ns1.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.example.com.
;
5.1.168.192.in-addr.arpa. IN PTR api.ocp4.example.com.
5.1.168.192.in-addr.arpa. IN PTR api-int.ocp4.example.com.
;
97.1.168.192.in-addr.arpa. IN PTR control-plane0.ocp4.example.com.
98.1.168.192.in-addr.arpa. IN PTR control-plane1.ocp4.example.com.
99.1.168.192.in-addr.arpa. IN PTR control-plane2.ocp4.example.com.
;
11.1.168.192.in-addr.arpa. IN PTR worker0.ocp4.example.com.
7.1.168.192.in-addr.arpa. IN PTR worker1.ocp4.example.com.
;
;EOF
where:
-
api.ocp4.example.comprovides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer. -
api-int.ocp4.example.comprovides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer and is used for internal cluster communications. -
control-plane0.ocp4.example.comand adjacent records provide reverse DNS resolution for the control plane machines. -
worker0.ocp4.example.comand adjacent record provide reverse DNS resolution for the worker machines.
A PTR record is not required for the OpenShift Container Platform application wildcard.
2.4.3. Networking requirements for IBM Z Copy linkLink copied to clipboard!
In IBM Z® environments, advanced networking technologies like Open Systems Adapter (OSA), HiperSockets, and Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) require specific configurations that deviate from the standard settings used in Assisted Installer deployments. These overrides are necessary to accommodate their unique requirements and ensure a successful and efficient deployment on IBM Z®.
The following table lists the network devices that are supported for the network configuration override functionality:
| Network device | z/VM | KVM | LPAR Classic | LPAR Dynamic Partition Manager (DPM) |
|---|---|---|---|---|
| OSA virtual switch | Supported | Not applicable | Not applicable | Not applicable |
| Direct attached OSA | Supported | Only through a Linux bridge | Supported | Supported |
| RDMA over Converged Ethernet (RoCE) | Supported | Only through a Linux bridge | Supported | Supported |
| HiperSockets | Supported | Only through a Linux bridge | Supported | Supported |
| Linux bridge | Not applicable | Supported | Not applicable | Not applicable |
2.4.3.1. Configuring network overrides in IBM Z Copy linkLink copied to clipboard!
You can specify a static IP address on IBM Z® machines that uses Logical Partition (LPAR) and z/VM. This is specially useful when the network devices do not have a static MAC address assigned to them.
Procedure
If you have an existing
.parmfile, edit it to include the following entry:ai.ip_cfg_override=1This parameter allows the file to add the network settings to the CoreOS installer.
The following is an example of the
.parmfile:rd.neednet=1 cio_ignore=all,!condev console=ttysclp0 coreos.live.rootfs_url=<coreos_url> ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> rd.znet=qeth,<network_adaptor_range>,layer2=1 rd.<disk_type>=<adapter> rd.zfcp=<adapter>,<wwpn>,<lun> random.trust_cpu=on zfcp.allow_lun_scan=0 ai.ip_cfg_override=1 ignition.firstboot ignition.platform.id=metal random.trust_cpu=onReplace the parameters as follows:
-
For the
coreos.live.rootfs_urlartifact, specify the matchingrootfsartifact for thekernelandinitramfsthat you are booting. Only HTTP and HTTPS protocols are supported. -
Regarding
rd.<disk_type>, for installations on direct access storage devices (DASD) type disks, userd.to specify the DASD where Red Hat Enterprise Linux (RHEL) is to be installed. For installations on Fibre Channel Protocol (FCP) disks, userd.zfcp=<adapter>,<wwpn>,<lun>to specify the FCP disk where RHEL is to be installed. -
For
rd.zfcp, specify values foradapter,wwpn, andlunas in the following example:rd.zfcp=0.0.8002,0x500507630400d1e3,0x4000404600000000. Regarding
ai.ip_cfg_override=1, specify this parameter when using an OSA network adapter or HiperSockets.NoteThe
overrideparameter overrides the host’s network configuration settings.
-
For the
2.5. Preflight validations Copy linkLink copied to clipboard!
The Assisted Installer ensures the cluster meets the prerequisites before installation, because it eliminates complex postinstallation troubleshooting, thereby saving significant amounts of time and effort. Before installing software on the nodes, the Assisted Installer conducts the following validations:
- Ensures network connectivity
- Ensures sufficient network bandwidth
- Ensures connectivity to the registry
- Ensures that any upstream DNS can resolve the required domain name
- Ensures time synchronization between cluster nodes
- Verifies that the cluster nodes meet the minimum hardware requirements
- Validates the installation configuration parameters
If the Assisted Installer does not successfully validate the foregoing requirements, installation will not proceed.
Chapter 3. Customizing your environment using Operators and Operator Bundles Copy linkLink copied to clipboard!
You can customize the OpenShift Container Platform deployment by selecting one or more Operators or Operator bundles during the installation.
3.1. Customizing with Operators Copy linkLink copied to clipboard!
The Assisted Installer Operators are used to package, deploy, and manage services and applications in OpenShift Container Platform. Through the Assisted Installer, you can install the following Operators, categorized according to their functionality.
Before starting the installation, familiarize yourself with the Assisted Installer Operators, including their prerequisites and limitations.
The additional requirements apply to each Operator individually. If you select more than one Operator, or if the Assisted Installer automatically selects an Operator due to dependencies, the total required resources is the sum of the resource requirements for each Operator.
If you require advanced options, install the Operators after you have installed the cluster.
3.1.1. Storage Operators Copy linkLink copied to clipboard!
The Storage category contains the following Operators:
- Local Storage Operator
- Logical Volume Manager Storage Operator
- OpenShift Data Foundation Operator
- OpenShift API for Data Protection (OADP) Operator
3.1.1.1. Installing the Local Storage Operator Copy linkLink copied to clipboard!
The Local Storage Operator (LSO) enables the provisioning of persistent storage through local volumes. Local persistent volumes provide access to local storage devices, such as drives or partitions, by using the standard persistent volume claim interface.
You can perform the following actions using Local Storage Operator (LSO):
- Assign the storage devices to the storage classes without modifying the device configuration.
- Statically provision PVs and storage classes by configuring the LocalVolume custom resource (CR).
- Create workloads and PVCs while being aware of the underlying storage topology.
Selecting the OpenShift Virtualization Operator, either independently or as part of the Virtualization bundle, automatically activates Local Storage Operator (LSO) in the background.
While installing the Local Storage Operator (LSO) through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.1.2. Installing the Logical Volume Manager Storage Operator Copy linkLink copied to clipboard!
You can use LVM Storage to dynamically provision block storage on a limited resources cluster.
While installing the Logical Volume Manager Storage Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
Prerequisites
- Requires at least 1 non-boot drive per host.
- Requires 100 MiB of additional RAM.
- Requires 1 additional CPU core for each non-boot drive.
3.1.1.3. Installing the Red Hat OpenShift Data Foundation Operator Copy linkLink copied to clipboard!
You can use OpenShift Data Foundation for file, block, and object storage. This storage option is recommended for all OpenShift Container Platform clusters. OpenShift Data Foundation requires a separate subscription.
While installing the Red Hat OpenShift Data Foundation Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
Prerequisites
- There are at least 3 compute (workers) nodes, each with 19 additional GiB of memory and 8 additional CPU cores.
- There are at least 2 drives per compute node. For each drive, there is an additional 5 GB of RAM.
- You comply to the additional requirements specified here: Planning your deployment.
You cannot install the OpenShift Data Foundation Operator on Oracle third-party platforms such as Oracle® Cloud Infrastructure or Oracle® Compute Cloud@Customer.
3.1.1.4. Installing the OpenShift API for Data Protection (OADP) Operator Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) product safeguards customer applications on OpenShift Container Platform. It offers comprehensive disaster recovery protection, covering OpenShift Container Platform applications, application-related cluster resources, persistent volumes, and internal images. OADP is also capable of backing up both containerized applications and virtual machines (VMs). OADP does not serve as a disaster recovery solution for etcd or OpenShift Operators.
Using the Assisted Installer, you can install the OADP Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.2. Virtualization Operators Copy linkLink copied to clipboard!
The Virtualization category contains the following Operators:
- OpenShift Virtualization Operator
- Migration Toolkit for Virtualization Operator
- OpenShift sandboxed containers Operator
3.1.2.1. Installing the OpenShift Virtualization Operator Copy linkLink copied to clipboard!
You can deploy OpenShift Virtualization to perform the following tasks:
- Create and manage Linux and Windows virtual machines (VMs).
- Run pod and VM workloads alongside each other in a cluster.
- Connect to VMs through a variety of consoles and CLI tools.
- Import and clone existing VMs.
- Manage network interface controllers and storage drives attached to VMs.
- Live migrate VMs between nodes.
The OpenShift Virtualization Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:
- None - If the CPU architecture is ARM64, no storage Operator is activated.
- LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
- Local Storage Operator (LSO) - For all other deployments.
Prerequisites
- Requires enabled CPU virtualization support in the firmware on all nodes.
- Requires an additional 360 MiB of memory and 2 CPU cores for each compute (worker) node.
- Requires an additional 150 MiB of memory and 4 CPU cores for each control plane node.
Requires Red Hat OpenShift Data Foundation (recommended for creating additional on-premise clusters), Logical Volume Manager Storage, or another persistent storage service.
ImportantDeploying OpenShift Virtualization without Red Hat OpenShift Data Foundation results in the following scenarios:
- Multi-node cluster: No storage is configured. You must configure storage after the OpenShift Data Foundation configuration.
- Single-node OpenShift: Logical Volume Manager Storage (LVM Storage) is installed.
You must review the prerequisites to ensure that your environment has sufficient additional resources for OpenShift Virtualization.
- OpenShift Virtualization is not supported on the following platforms: Nutanix, vSphere.
- OpenShift Virtualization is not compatible with the following CPU architectures: S390X, PPC64LE.
- OpenShift Virtualization is supported from OpenShift Container Platform 4:14 and later.
3.1.2.2. Installing the Migration Toolkit for Virtualization Operator Copy linkLink copied to clipboard!
The Migration Toolkit for Virtualization Operator allows you to migrate virtual machines at scale to a local or remote Red Hat OpenShift Virtualization cluster. You can perform the migration from any of the following source providers:
- VMware vSphere
- Red Hat Virtualization (RHV)
- Red Hat OpenShift Virtualization
- OpenStack
When you select the Migration Toolkit for Virtualization Operator, the Assisted Installer automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.
You can install the Migration Toolkit for Virtualization Operator on OpenShift Container Platform using the Assisted Installer, either independently or as part of the OpenShift Virtualization Operator bundle.
Prerequisites
- Requires OpenShift Container Platform version 4.14 or later.
- Requires an x86_64 CPU architecture.
- Requires an additional 1024 MiB of memory and 1 CPU core for each control plane node and worker node.
- Requires the additional resources specified for the OpenShift Virtualization Operator, installed together with OpenShift Virtualization. For details, see the prerequisites in the OpenShift Virtualization Operator section.
Next steps
After completing the installation, the Migration menu appears in the navigation pane of the Red Hat OpenShift web console.
The Migration menu provides access to the Migration Toolkit for Virtualization. Use the toolkit to create and execute a migration plan with the relevant source and destination providers.
For details, see either of the following chapters in the Migration Toolkit for Virtualization Guide:
3.1.2.3. Installing the OpenShift sandboxed containers Operator Copy linkLink copied to clipboard!
The OpenShift sandboxed containers Operator provides an additional virtual machine (VM) isolation layer for pods, which manages the installation, configuration, and updating of sandboxed containers runtime (Kata containers) on Red Hat OpenShift clusters. You can install the sandboxed containers runtime in an Red Hat OpenShift cluster by using the Assisted Installer.
Prerequisites
The required functionality for the OpenShift Container Platform is supported by two main components:
- OpenShift Container Platform: Use OpenShift Container Platform version 4.17 or later to install OpenShift sandboxed containers on an Red Hat OpenShift cluster using the Assisted Installer. To learn more about the requirements for OpenShift sandboxed containers, see OpenShift sandboxed containers.
Kata runtime: This includes Red Hat Enterprise Linux CoreOS (RHCOS) and updates with every OpenShift Container Platform release. The Operator depends on the features that come with the RHCOS host and the environment that it runs in.
NoteYou must install Red Hat Enterprise Linux CoreOS (RHCOS) on the worker nodes. RHEL nodes are not supported.
3.1.3. Artificial Intelligence (AI) Operators Copy linkLink copied to clipboard!
The AI category contains the following Operators:
- Red Hat® OpenShift® Artificial Intelligence (AI) Operator
- AMD GPU Operator
- NVIDIA GPU Operator
3.1.3.1. Installing the OpenShift Artificial Intelligence (AI) Operator Copy linkLink copied to clipboard!
Red Hat® OpenShift® Artificial Intelligence (AI) is a flexible, scalable artificial intelligence (AI) and machine learning (ML) platform that enables enterprises to create and deliver AI-enabled applications at scale across hybrid cloud environments. Red Hat® OpenShift® AI enables the following functionality:
- Data acquisition and preparation.
- Model training and fine-tuning.
- Model serving and model monitoring.
- Hardware acceleration.
The OpenShift AI Operator enables you to install Red Hat® OpenShift® AI on your OpenShift Container Platform cluster. From OpenShift Container Platform version 4.17 and later, you can use the Assisted Installer to deploy the OpenShift AI Operator to your cluster during the installation.
You can install the OpenShift Artificial Intelligence (AI) Operator either separately or as part of the OpenShift AI Operator bundle.
The integration of the OpenShift AI Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
The prerequisites for installing the OpenShift AI Operator separately are as follows:
- You are installing OpenShift Container Platform version 4.17 or later.
You meet the following miminum requirements for the OpenShift AI Operator:
- Requires at least 2 compute (worker) nodes, each with 32 additional GiB of memory and 8 additional CPU cores.
- Requires at least 1 supported GPU. Both AMD and NVIDIA GPUs are supported.
- You meet the additional minimum requirements specified for the dependent Red Hat OpenShift Data Foundation Operator.
- You meet the additional requirements specified here: Requirements for OpenShift AI.
- See the additional prerequisites for the OpenShift AI Operator bundle, if you are installing the Operator as part of the bundle.
You cannot install the OpenShift AI Operator on Oracle third-party platforms such as Oracle® Cloud Infrastructure or Oracle® Compute Cloud@Customer.
3.1.3.2. Installing the AMD GPU Operator Copy linkLink copied to clipboard!
The Advanced Micro Devices (AMD) Graphics Processing Unit (GPU) Operator simplifies the deployment and management of AMD Instinct™ GPUs within a Red Hat OpenShift Container Platform cluster. The hardware acceleration capabilities of the Operator automate several key tasks, making it easier to create artificial intelligence and machine learning (AI/ML) applications. Accelerating specific areas of GPU functions can minimize CPU processing and memory usage, improving overall application speed, memory consumption, and bandwidth restrictions.
In the Assisted Installer, you can install the AMD GPU Operator separately or as part of the OpenShift AI Operator bundle. Selecting the AMD GPU Operator automatically activates the Kernel Module Management Operator.
While installing the AMD GPU Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
The integration of the AMD GPU Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- Requires at least 1 supported AMD GPU.
- See the additional prerequisites for the OpenShift AI Operator bundle if you are installing the Operator as part of the bundle.
3.1.3.3. Installing the NVIDIA GPU Operator Copy linkLink copied to clipboard!
The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision Graphical Processing Units (GPUs).
Some of these software components are as follows:
- NVIDIA drivers to enable Compute Unified Device Architecture (CUDA).
- The Kubernetes device plugin for GPUs.
- The NVIDIA Container Toolkit.
- Automatic node labelling using GPU Feature Discovery (GFD)
- GPU monitoring through the Data Center GPU Manager (DCGM).
In OpenShift Container Platform, the Operator provides a consistent, automated, and cloud-native way to leverage the power of NVIDIA GPUs for artificial intelligence, machine learning, high-performance computing, and other GPU-accelerated workloads.
You can install the NVIDIA GPU Operator either separately or as part of the OpenShift AI Operator bundle. Selecting the NVIDIA GPU Operator automatically activates the Node Feature Discovery Operator.
The integration of the NVIDIA GPU Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- Requires at least 1 supported NVIDIA GPU.
- See the additional prerequisites for the OpenShift AI Operator bundle if you are installing the Operator as part of the bundle.
3.1.4. Network Operators Copy linkLink copied to clipboard!
The Network category contains the following Operators:
- Kubernetes NMState Operator
- OpenShift Service Mesh Operator
- MetalLB Operator
3.1.4.1. Installing the Kubernetes NMState Operator Copy linkLink copied to clipboard!
NMState is a declarative NetworkManager API designed for configuring network settings using YAML or JSON-based instructions. The Kubernetes NMState Operator allows you to configure network interface types, DNS, and routing on the cluster nodes using NMState.
You can install the Kubernetes NMState Operator on OpenShift Container Platform using the Assisted Installer, either separately or as part of the OpenShift Virtualization Operator bundle. Installing the Kubernetes NMState Operator with the Assisted Installer automatically creates a kubernetes-nmstate instance, which deploys the NMState State Controller as a daemon set across all of the cluster nodes. The daemons on the cluster nodes periodically report on the state of each node’s network interfaces to the API server.
Prerequisites
- Supports OpenShift Container Platform 4.12 or later.
- Requires an x86_64 CPU architecture.
- Cannot be installed on the Nutanix and Oracle Cloud Infrastructure platforms.
3.1.4.2. Installing the OpenShift Service Mesh Operator Copy linkLink copied to clipboard!
Red Hat OpenShift Service Mesh addresses a variety of problems in a microservice architecture by creating a centralized point of control in an application. It adds a transparent layer on existing distributed applications without requiring any changes to the application code.
Microservice architectures split the work of enterprise applications into modular services, which can make scaling and maintenance easier. However, as an enterprise application built on a microservice architecture grows in size and complexity, it becomes difficult to understand and manage. Service Mesh can address those architecture problems by capturing or intercepting traffic between services and can modify, redirect, or create new requests to other services.
Service Mesh provides an easy way to create a network of deployed services that provides discovery, load balancing, service-to-service authentication, failure recovery, metrics, and monitoring. A service mesh also provides more complex operational functionality, including A/B testing, canary releases, access control, and end-to-end authentication
Red Hat OpenShift Service Mesh requires the use of the Red Hat OpenShift Service Mesh Operator which allows you to connect, secure, control, and observe the microservices that comprise your applications. You can also install other Operators to enhance your service mesh experience. Service mesh is based on the open source Istio project.
Using the Assisted Installer, you can install the OpenShift Service Mesh Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.
The integration of the OpenShift Service Mesh Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
- See Preparing to install service mesh in the OpenShift Container Platform documentation.
3.1.4.3. Installing the MetalLB Operator Copy linkLink copied to clipboard!
You can install the MetalLB Operator to enable the use of LoadBalancer services in environments that do not have a built-in cloud load balancer, such as bare-metal clusters.
When you create a LoadBalancer service, MetalLB assigns an external IP address from a predefined pool. MetalLB advertises this IP address on the host network, making the service reachable from outside the cluster. When external traffic enters your OpenShift Container Platform cluster through the MetalLB LoadBalancer service, the return traffic to the client has the external IP address of the load balancer as the source IP.
Using the Assisted Installer, you can install the MetalLB Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.5. Remediation Operators Copy linkLink copied to clipboard!
The Remediation category contains the following Operators:
- Node Health Check Operator
- Fence Agents Remediation Operator
- Self Node Remediation Operator
3.1.5.1. Installing the Node Health Check Operator Copy linkLink copied to clipboard!
The Node Health Check Operator monitors node conditions based on a defined set of criteria to assess their health status. When detecting an issue, the Operator delegates remediation tasks to the appropriate remediation provider to remediate the unhealthy nodes. The Assisted Installer supports the following remediation providers:
- Self Node Remediation Operator - An internal solution for rebooting unhealthy nodes.
- Fence Agents Remediation Operator - Leverages external management capabilities to forcefully isolate and reboot nodes.
Using the Assisted Installer, you can install the Node Health Check Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.5.2. Installing the Fence Agents Remediation Operator Copy linkLink copied to clipboard!
You can use Fence Agents Remediation Operator to automatically recover unhealthy nodes on environments with a traditional API end-point. When a node in the OpenShift Container Platform cluster becomes unhealthy or unresponsive, the Fence Agents Remediation Operator utilizes an external set of fencing agents to isolate it from the rest of the cluster. A fencing agent then resets the unhealthy node in an attempt to resolve transient hardware or software issues. Before or during the reboot process, the Fence Agents Remediation Operator safely moves workloads (pods) running on the unhealthy node to other healthy nodes in the cluster.
Using the Assisted Installer, you can install the Fence Agents Remediation Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Next steps
-
Create the
FenceAgentsRemediationTemplatecustom resource to define the required fencing agents and remediation parameters. For details, see Configuring the Fence Agents Remediation Operator. -
Configure the
NodeHealthCheckcustom resource by either replacing the defaultSelfNodeRemediationprovider withFenceAgentsRemediationor by addingFenceAgentsRemediationas an additional remediation provider.
3.1.5.3. Installing the Self Node Remediation Operator Copy linkLink copied to clipboard!
The Self Node Remediation Operator automatically reboots unhealthy nodes. This remediation strategy minimizes downtime for stateful applications and ReadWriteOnce (RWO) volumes, and restores compute capacity if transient failures occur.
You can use the Self Node Remediation Operator as a remediation provider for the Node Health Check Operator. Currently, it is only possible to install the Self Node Remediation Operator through the API.
The integration of the Self Node Remediation Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Procedure
3.1.6. Security and Access Control Operators Copy linkLink copied to clipboard!
The Security & Access Control category contains the following Operators:
- Authorino Operator
- Kernel Module Management Operator
3.1.6.1. Installing the Authorino Operator Copy linkLink copied to clipboard!
The Authorino Operator provides an easy way to install Authorino, providing configurability options at the time of installation.
Authorino is a Kubernetes-native, external authorization service designed to secure APIs and applications. It intercepts requests to services and determines whether to allow or deny access based on configured authentication and authorization policies. Authorino provides a centralized and declarative way to manage access control for your Kubernetes-based applications without requiring code changes.
Using the Assisted Installer, you can install the Authorino Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.
The integration of the Authorino Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
3.1.6.2. Installing the Kernel Module Management Operator Copy linkLink copied to clipboard!
The Kernel Module Management (KMM) Operator manages, builds, signs, and deploys out-of-tree kernel modules and device plugins on OpenShift Container Platform clusters.
KMM adds a new Module CRD which describes an out-of-tree kernel module and its associated device plugin. You can use Module resources to configure how to load the module, define ModuleLoader images for kernel versions, and include instructions for building and signing modules for specific kernel versions.
KMM is designed to accommodate multiple kernel versions at once for any kernel module, allowing for seamless node upgrades and reduced application downtime.
You can install the Kernel Module Management Operator either independently or as part of the OpenShift AI Operator bundle.
The integration of the Kernel Module Management Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- If you are installing the Operator as part of the OpenShift AI Operator bundle, see the bundle prerequisites.
- If you are installing the Operator separately, there are no additional prerequisites.
3.1.7. CI/CD and Development Productivity Operators Copy linkLink copied to clipboard!
The CI/CD & Dev Productivity category contains the following Operators:
- OpenShift Pipelines Operator
- OpenShift Serverless Operator
3.1.7.1. Installing the OpenShift Pipelines Operator Copy linkLink copied to clipboard!
Red Hat OpenShift Pipelines is a cloud-native, continuous integration and continuous delivery (CI/CD) solution based on Kubernetes resources. It uses Tekton building blocks to automate deployments across multiple platforms by abstracting away the underlying implementation details. Tekton introduces various standard custom resource definitions (CRDs) for defining CI/CD pipelines that are portable across Kubernetes distributions.
The Red Hat OpenShift Pipelines Operator handles the installation and management of OpenShift Pipelines. The Operator supports the following use cases:
- Continuous Integration (CI) - Automating code compilation, testing, and static analysis.
- Continuous Delivery/Deployment (CD) - Automating the deployment of applications to various environments (development, staging, production).
- Microservices Development - Supporting decentralized teams working on microservice-based architectures.
- Building Container Images - Efficiently building and pushing container images to registries.
- Orchestrating Complex Workflows - Defining multi-step processes for building, testing, and deploying applications across different platforms.
Using the Assisted Installer, you can install the OpenShift Pipelines Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.
The integration of the OpenShift Pipelines Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
3.1.7.2. Installing the OpenShift Serverless Operator Copy linkLink copied to clipboard!
The Red Hat OpenShift Serverless Operator enables you to install and use the following components on your OpenShift Container Platform cluster:
- Knative Serving - Deploys and automatically scales stateless, containerized applications according to demand. It simplifies code deployment, and handles web requests and background processes.
- Knative Eventing - Provides the building blocks for an event-driven architecture on Kubernetes. It enables loose coupling between services by allowing them to communicate asynchronously through events, rather than through direct calls.
- Knative Broker for Apache Kafka - This is a specific implementation of a Knative Broker. It provides a robust, scalable, and high-performance mechanism for routing events within Knative Eventing, in environments where Apache Kafka is the preferred message broker.
The OpenShift Serverless Operator manages Knative custom resource definitions (CRDs) for your cluster and enables you to configure them without directly modifying individual config maps for each component.
Using the Assisted Installer, you can install the OpenShift Serverless Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.
The integration of the OpenShift Serverless Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
3.1.8. Platform Operations and Lifecycle Operators Copy linkLink copied to clipboard!
The Platform Operations & Lifecycle category contains the following Operators:
- Multicluster engine Operator
- Node Maintenance Operator
- Cluster Observability Operator
- OpenShift Loki Operator
- OpenShift Logging Operator
3.1.8.1. Installing the Multicluster engine for Kubernetes Operator Copy linkLink copied to clipboard!
You can deploy the multicluster engine for Kubernetes to perform the following tasks in a large, multi-cluster environment:
- Provision and manage additional Kubernetes clusters from your initial cluster.
- Use hosted control planes to reduce management costs and optimize cluster deployment by decoupling the control and data planes.
- Use GitOps Zero Touch Provisioning to manage remote edge sites at scale.
You can deploy the multicluster engine with OpenShift Data Foundation on all OpenShift Container Platform clusters.
Prerequisites
- Requires an additional 16384 MiB of memory and 4 CPU cores for each compute (worker) node.
- Requires an additional 16384 MiB of memory and 4 CPU cores for each control plane node.
Requires OpenShift Data Foundation (recommended for creating additional on-premise clusters), LVM Storage, or another persistent storage service.
ImportantDeploying multicluster engine without OpenShift Data Foundation results in the following scenarios:
- Multi-node cluster: No storage is configured. You must configure storage after the installation process.
- Single-node OpenShift: LVM Storage is installed.
You must review the prerequisites to ensure that your environment has enough additional resources for the multicluster engine.
3.1.8.2. Installing the Node Maintenance Operator Copy linkLink copied to clipboard!
The Node Maintenance Operator facilitates planned maintenance by placing nodes into maintenance mode.
The Node Maintenance Operator watches for new or deleted NodeMaintenance custom resources (CRs). When it detects a new NodeMaintenance CR, it prevents new workloads from being scheduled on that node, and cordons off the node from the rest of the cluster. The Operator then evicts all pods that can be evicted from the node. When the administrator deletes the NodeMaintenance CR associated with the node, maintenance ends and the Operator makes the node available for new workloads.
Using the Assisted Installer, you can install the Node Maintenance Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.8.3. Installing the Cluster Observability Operator Copy linkLink copied to clipboard!
The Cluster Observability Operator (COO) is an optional component of the OpenShift Container Platform designed for creating and managing highly customizable monitoring stacks. It enables cluster administrators to automate configuration and management of monitoring needs extensively, offering a more tailored and detailed view of each namespace compared to the default OpenShift Container Platform monitoring system.
The Cluster Observability Operator deploys the following monitoring components:
- Prometheus - A highly available Prometheus instance capable of sending metrics to an external endpoint by using remote write.
- Thanos Querier (optional) - Enables querying of Prometheus instances from a central location.
- Alertmanager (optional) - Provides alert configuration capabilities for different services.
- UI plugins (optional) - Enhances the observability capabilities with plugins for monitoring, logging, distributed tracing and troubleshooting.
- Korrel8r (optional) - Provides observability signal correlation, powered by the open source Korrel8r project.
Using the Assisted Installer, you can install the Cluster Observability Operator Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. Install the Cluster Observability Operator Operator together with OpenShift Logging and OpenShift Loki Operators, to support Red Hat OpenShift Virtualization Engine deployments. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".
While installing the Cluster Observability Operator Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.8.4. Installing the OpenShift Loki Operator Copy linkLink copied to clipboard!
The OpenShift Loki Operator provides a log aggregation system designed to store and query logs from all applications and infrastructure components. The Operator implements this functionality through the LokiStack custom resource (CR).
The LokiStack CR manages Loki, which is a scalable, highly-available, multi-tenant log aggregation system. The resource also includes a web proxy with OpenShift Container Platform authentication, which enforces multi-tenancy and facilitates the saving and indexing of data in Loki log stores. You can configure Loki as the backend to store all collected flows with a maximal level of detail.
With the Assisted Installer, you can install the OpenShift Loki Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. Install the OpenShift Loki Operator together with the Cluster Observability Operator and OpenShift Logging Operators, to support Red Hat OpenShift Virtualization Engine deployments.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.8.5. Installing the OpenShift Logging Operator Copy linkLink copied to clipboard!
The OpenShift Logging Operator installs and manages the entire logging stack for your cluster. It works automatically with the OpenShift Container Platform security settings, ensuring that the system collects and stores logs without compromising multi-tenant isolation.
The Operator deploys collector agents, such as Vector or Fluentd, as pods on every node. These collectors use specific cluster roles to securely gather data from three distinct sources:
-
collect-audit-logs: Gathers Kubernetes and OpenShift API logs and security-related events. -
collect-application-logs: Gathers logs from user-created projects and containers. -
collect-infrastructure-logs: Gathers logs from the platform itself, including nodes and control plane services.
Once collected, the Operator manages the pipelines that filter and route this data. You can send your logs to internal storage such as Loki or Elasticsearch, or forward them to an external logging service. Throughout this process, the Operator ensures that all data handling follows the cluster’s built-in security and access control policies.
Using the Assisted Installer, you can install the OpenShift Logging Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. Install the OpenShift Logging Operator together with the Cluster Observability Operator and OpenShift Loki Operators, to support Red Hat OpenShift Virtualization Engine deployments.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.9. Scheduling Operators Copy linkLink copied to clipboard!
The Scheduling category contains the following Operators:
- Node Feature Discovery (NFD) Operator
- Kube Descheduler Operator
- NUMA Resources Operator
3.1.9.1. Installing the Node Feature Discovery Operator Copy linkLink copied to clipboard!
The Node Feature Discovery (NFD) Operator automates the deployment and management of the Node Feature Discovery (NFD) add-on. The Node Feature Discovery add-on detects the configurations and hardware features of each node in an OpenShift Container Platform cluster. The add-on labels each node with hardware-specific information such as vendor, kernel configuration, or operating system version, making the cluster aware of the underlying hardware and software capabilities of the nodes.
With the Node Feature Discovery (NFD) Operator, administrators can easily gather information about the nodes to use for scheduling, resource management, and more by controlling the life-cycle of Node Feature Discovery (NFD).
You can install the Node Feature Discovery Operator separately or as part of the OpenShift AI Operator bundle.
While installing the Node Feature Discovery Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
The integration of the Node Feature Discovery Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- If you are installing the Operator as part of the OpenShift AI Operator bundle, see the bundle prerequisites.
- If you are installing the Operator separately, there are no additional prerequisites.
3.1.9.2. Installing the Kube Descheduler Operator Copy linkLink copied to clipboard!
The Kube Descheduler Operator is a Kubernetes Operator that automates the deployment, configuration, and management of the Kubernetes Descheduler within a cluster. You can use the Kube Descheduler Operator to evict pods (workloads) based on specific strategies, so that the pods can be rescheduled onto more appropriate nodes.
You can benefit from descheduling running pods in situations such as the following:
- Nodes are underutilized or overutilized.
- Pods and node affinity requirements, such as taints or labels, have changed and the original scheduling decisions are no longer appropriate for certain nodes.
- Node failure requires pods to be moved.
- New nodes are added to clusters.
- Pods have been restarted excessively.
Using the Assisted Installer, you can install the Kube Descheduler Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.
While installing the Kube Descheduler Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.9.3. Installing the NUMA Resources Operator Copy linkLink copied to clipboard!
Non-Uniform Memory Access (NUMA) is a compute platform architecture that allows different CPUs to access different regions of memory at different speeds. NUMA resource topology refers to the locations of CPUs, memory, and PCI devices relative to each other in the compute node. Colocated resources are said to be in the same NUMA zone. For high-performance applications, the cluster needs to process pod workloads in a single NUMA zone.
The NUMA Resources Operator allows you to schedule high-performance workloads in the same NUMA zone. It deploys a node resources exporting agent that reports on available cluster node NUMA resources, and a secondary scheduler that manages the workloads.
Using the Assisted Installer, you can install the NUMA Resources Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. For more information about the use of this Operator in OpenShift Container Platform, see Additional resources.
While installing the NUMA Resources Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Next steps
Create the NUMAResourcesOperator custom resource and deploy the NUMA-aware secondary pod scheduler. For details, see Scheduling NUMA-aware workloads (RHOCP).
3.2. Customizing with Operator Bundles Copy linkLink copied to clipboard!
An Operator bundle is a recommended packaging format that combines related Operators to deliver a comprehensive set of capabilities. By selecting a bundle, administrators can extend functionality beyond that of a single Operator.
This approach makes the Assisted Installer more opinionated, offering an optimized platform for each selected bundle. It reduces the adoption barrier and minimizes the expertise required for customers to quickly access essential features. Additionally, it establishes a single, well-tested, and widely recognized path for platform deployments.
Meanwhile, individual Operators remain independent and free of unnecessary dependencies, ensuring a lightweight and flexible solution for small or specialized deployments, such as on single-node OpenShift.
When an administrator specifies an Operator bundle, the Assisted Installer automatically provisions the associated Operators included in the bundle. These Operators are predefined and cannot be deselected, ensuring consistency. Administrators can modify the selection after the installation has completed.
3.2.1. Installing the Virtualization Operator bundle Copy linkLink copied to clipboard!
Virtualization lets you create multiple simulated environments or resources from a single, physical hardware system. The Virtualization Operator bundle provides a recommended and proven path for virtualization platform deployments, minimizing obstacles. The solution supports the addition of nodes and Day-2 administrative operations.
The Virtualization Operator bundle prompts the Assisted Installer to install the following Operators together:
- Fence Agents Remediation Operator - Externally fences failed nodes using power controllers.
- Kube Descheduler Operator - Evicts pods to reschedule them on more suitable nodes.
- Local Storage Operator - Allows provisioning of persistent storage by using local volumes.
- Migration Toolkit for Virtualization Operator - Enables you to migrate virtual machines from VMware vSphere, Red Hat Virtualization, or OpenStack to OpenShift Virtualization running on Red Hat OpenShift Container Platform.
- Kubernetes NMState Operator - Enables you to configure various network interface types, DNS, and routing on cluster nodes.
- Node Health Check Operator - Identifies unhealthy nodes.
- Node Maintenance Operator - Places nodes into maintenance mode.
- OpenShift Virtualization Operator - Runs virtual machines alongside containers on one platform.
- Cluster Observability Operator - Provides observability and monitoring capabilities for your OpenShift cluster.
- MetalLB Operator - Provides load balancer services for bare metal OpenShift clusters.
- NUMA Resources Operator - Provides NUMA-aware scheduling to improve workload performance on NUMA systems.
- OpenShift API for Data Protection (OADP) Operator - Enables the backup and restoral of OpenShift Container Platform cluster resources and persistent volumes.
Prerequisites
- You are installing OpenShift Container Platform version 4.14 or later.
- There is enabled CPU virtualization support in BIOS (Intel-VT/AMD-V) on all nodes.
- Each control plane (master) node has an an additional 1024 MiB of memory and 3 CPU cores.
- Each compute (worker) node has an additional 1024 MiB of memory and 5 CPU cores.
- You have included the additional resources required to support the selected storage Operator.
- You are installing a cluster of three or more nodes. The Virtualization Operator bundle is not available on single-node OpenShift.
3.2.2. Installing the OpenShift AI Operator bundle Copy linkLink copied to clipboard!
The OpenShift AI Operator bundle enables the training, serving, monitoring, and management of Artificial Intelligence (AI) and Machine Learning (ML) models and applications. It simplifies the deployment of AI and ML components on your OpenShift cluster.
The OpenShift AI Operator bundle bundle prompts the Assisted Installer to install the following Operators together:
- AMD GPU Operator - Automates the management of AMD software components needed to provision and monitor Graphics Processing Units (GPUs).
- Authorino Operator - Provides a lightweight external authorization service for tailor-made Zero Trust API security.
- Kernel Module Management Operator - Manages kernel modules and associated device plugins.
- Local Storage Operator - Allows provisioning of persistent storage by using local volumes.
- Node Feature Discovery Operator - Manages the detection of hardware features and configuration by labeling nodes with hardware-specific information.
- NVIDIA GPU Operator - Automates the management of NVIDIA software components needed to provision and monitor GPUs.
- OpenShift AI Operator - Trains, serves, monitors and manages AI/ML models and applications.
- Red Hat OpenShift Data Foundation Operator - Provides persistent software-defined storage for hybrid applications.
- OpenShift Pipelines Operator - Provides a cloud-native continuous integration and delivery (CI/CD) solution for building pipelines using Tekton.
- OpenShift Serverless Operator - Deploys workflow applications based on the CNCF (Cloud Native Computing Foundation) Serverless Workflow specification.
- OpenShift Service Mesh Operator - Provides behavioral insight and operational control over a service mesh.
The OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- The installation of the NVIDIA GPU, AMD GPU, and Kernel Module Management Operators depends on the Graphics Processing Unit (GPU) detected on your hosts following host discovery.
Chapter 4. Installing with the Assisted Installer web console Copy linkLink copied to clipboard!
After you meet the cluster nodes and network requirements, you can begin installing the cluster.
4.1. Preinstallation considerations Copy linkLink copied to clipboard!
Before installing OpenShift Container Platform with the Assisted Installer, you must consider the following configuration choices:
- Which base domain to use
- Which OpenShift Container Platform product version to install
- Whether to install a full cluster or single-node OpenShift
- Whether to use a DHCP server or a static network configuration
- Whether to use IPv4 or dual-stack networking
- Whether to install OpenShift Virtualization
- Whether to install Red Hat OpenShift Data Foundation
- Whether to install multicluster engine for Kubernetes
- Whether to integrate with the platform when installing on vSphere or Nutanix
- Whether to install a multi-architecture compute cluster
4.2. Setting the cluster details Copy linkLink copied to clipboard!
To create a cluster with the Assisted Installer web user interface, use the following procedure.
Procedure
- Log in to the Red Hat Hybrid Cloud Console.
- On the Red Hat OpenShift tile, click OpenShift.
- On the Red Hat OpenShift Container Platform tile, click Create cluster.
- Click the Datacenter tab.
- Under Assisted Installer, click Create cluster.
Enable the I’m installing on a disconnected/air-gapped/secured environment toggle to deploy an OpenShift Container Platform cluster by using a self-contained ISO image. The image includes a simplified user interface together with all necessary artifacts. This method supports installations in both connected and disconnected environments without requiring an external or local registry. It currently addresses Red Hat OpenShift Virtualization Engine use cases.
This installation method is outside the scope of this guide. For details and installation instructions, see Installing a cluster without an external registry in the OpenShift Container Platform documentation.
ImportantInstalling a Red Hat OpenShift Virtualization Engine cluster using this method is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
- Enter a name for the cluster in the Cluster name field.
Enter a base domain for the cluster in the Base domain field. All subdomains for the cluster will use this base domain.
NoteThe base domain must be a valid DNS name. You must not have a wildcard domain set up for the base domain.
From the OpenShift version dropdown list, select the version that you want to install and click Select. By default, the dropdown lists the latest OpenShift version. If you need an older version that is not displayed, click Show all available versions at the bottom of the list, and use the search box to find it.
Important-
For a multi-architecture compute cluster installation, select OpenShift Container Platform 4.12 or later, and use the
-multioption. For instructions on installing a multi-architecture compute cluster, see Installing multi-architecture compute clusters. - For IBM Power® and IBM Z® platforms, only OpenShift Container Platform 4.13 and later is supported.
- If you are booting from an iSCSI drive, select OpenShift Container Platform version 4.15 or later.
-
For a multi-architecture compute cluster installation, select OpenShift Container Platform 4.12 or later, and use the
Optional: The Assisted Installer defaults to using x86_64 CPU architecture. If you are installing OpenShift Container Platform on a different architecture, select the architecture to use. Valid values are arm64, ppc64le, and s390x. Remember that some features are not available with arm64, ppc64le, and s390x CPU architectures.
ImportantFor a multi-architecture compute cluster installation, you can use
x86_64or 64-bit ARM CPU architecture for the control plane nodes. Automatic conversion fromx86_64to 64-bit ARM is only supported on Amazon Web Services (AWS). For instructions on installing a multi-architecture compute cluster, see Installing multi-architecture compute clusters.- Optional: The Assisted Installer already has the pull secret associated to your account. If you want to use a different pull secret, select Edit pull secret.
Optional: If you are installing OpenShift Container Platform on a third-party platform, select the platform from the Integrate with external partner platforms list. Valid values are
Nutanix,vSphereorOracle Cloud Infrastructure. The Assisted Installer defaults to having no platform integration.Note- The Assisted Installer supports Oracle Cloud Infrastructure (OCI) integration from OpenShift Container Platform 4.14 and later.
- For details on external partner integrations, see Additional resources.
From the Number of control plane nodes field, optionally change the default value of three control plane nodes for your installation. The possible options are
1(single-node OpenShift),2,3,4, or5.ImportantThe Assisted Installer supports the following control plane node configurations:
-
4or5control plane nodes from OpenShift Container Platform 4.18 and later, on a bare metal or user-managed networking platform with an x86_64 CPU architecture. -
2control plane nodes from OpenShift Container Platform 4.19 and later, for a Two-Node OpenShift with Arbiter cluster topology. If the number of control plane nodes for a cluster is2, then it must have at least one additional arbiter host. - Currently, single-node OpenShift is not supported on IBM Power® platform.
For details, see Control plane count.
-
Optional: Select Include custom manifests if you have at least one custom manifest to include in the installation. A custom manifest has additional configurations not currently supported in the Assisted Installer. Selecting the checkbox adds the Custom manifests step to the wizard, where you upload the manifests.
Important- If you are installing OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) third-party platform, it is mandatory to add the custom manifests provided by Oracle.
- If you have already added custom manifests, clearing the Include custom manifests checkbox automatically deletes them all. You must confirm the deletion.
Optional: The Assisted Installer defaults to DHCP networking. If you are using a static IP configuration, bridges or bonds for the cluster nodes instead of DHCP reservations, select Static IP, bridges, and bonds. Selecting the checkbox adds the Static network configurations step to the wizard. For details, see Configuring static networks.
ImportantA static IP configuration is not supported in the following scenarios:
- OpenShift Container Platform installations on Oracle Cloud Infrastructure.
- OpenShift Container Platform installations on iSCSI boot volumes.
Optional: If you want to enable encryption of the installation disks, under Enable encryption of installation disks you can select one of the following:
- For single-node OpenShift, select Control plane node, worker.
For multi-node clusters, select Control plane nodes to encrypt the control plane node installation disks. Select Workers to encrypt worker node installation disks. Select Arbiter to encrypt the arbiter node installation disks.
ImportantYou cannot change the base domain, the single-node OpenShift checkbox, the CPU architecture, the host’s network configuration, or the disk-encryption after installation begins.
4.3. Configuring static networks Copy linkLink copied to clipboard!
The Assisted Installer supports the following static network configurations:
- Network-wide configurations - These settings apply to all hosts across the cluster and include configurations such as networking stack type, VLANs, and DNS.
- Host-specific configurations - These settings enable the configuration of individual host interfaces, and support features such as bonds, bridges, and VLANs for each host.
You define these settings in the NMState YAML file, which uses the NMState library for declarative network management. The Static networking step of the Assisted Installer web console provides two options for configuring the NMState YAML file:
- Form view - The form view provides fields for entering configuration parameters and supports basic network configurations. The network-wide configurations and host-specific configurations are two separate steps. The form creates the YAML file in the background when saved.
- YAML view - In the YAML view, you can upload an NMState YAML file that you created locally. This method supports advanced configurations, and includes both network-wide and host-specific settings within a single file.
- For installations on IBM Z® with z/VM, ensure that the z/VM nodes and vSwitches are properly configured for static networks and NMState. Also, the z/VM nodes must have a fixed MAC address assigned as the pool MAC addresses might cause issues with NMState. For more information about NMState, see NMState Declarative Network API.
A static IP configuration is not supported in the following scenarios:
- OpenShift Container Platform installations on Oracle Cloud Infrastructure.
- OpenShift Container Platform installations on iSCSI boot volumes.
4.3.1. Configuring static networks by using the form view Copy linkLink copied to clipboard!
Select the form view for basic configurations that use a single network interface.
To add new hosts that will use the new or edited configurations, you will need to regenerate the Discovery ISO in the 'Host discovery' step and boot your new hosts from it.
Prerequisites
- You have selected the Static IP, bridges and bonds option under Hosts' network configuration on the Cluster details page. Selecting this option adds the Static network configurations step to the wizard.
Procedure
- Go to the Static network configurations page.
- From the Configure via options, select Form view.
Enter the network-wide configurations:
Select the Networking stack type. Valid options are IPv4 and Dual stack.
Important- IPv6 is not currently supported for single stack.
- IPv4 networking with SDN is supported up to OpenShift Container Platform 4.14.
- IPv4 and dual-stack IPv4/IPv6 networking with OVN only are supported from OpenShift Container Platform 4.15 and later.
- If the cluster hosts are on a shared VLAN, select the Use VLAN checkbox and enter the VLAN ID.
Enter the network-wide IP addresses. If you selected Dual stack networking, you must enter both IPv4 and IPv6 addresses.
- Enter the DNS server IP address.
- Enter the cluster subnet’s IP address range in CIDR notation.
- Enter the default gateway IP address.
- Click Next.
Enter the host-specific configurations:
- Enter the IP address and the MAC address for the host.
Optional: You can use bonds to combine network interfaces for increased bandwidth and ensure redundancy. Creating a bond with a static IP address aggregates two network interfaces per host.
- Select the Use bond checkbox.
-
From the Bond type dropdown list, select the bond type. The default bond type is
Active-Backup (1). For a description of the bond types, see Bonding modes. - Enter the MAC address for Port 1.
- Enter the MAC address for Port 2.
- Enter the IP address for the bond.
- Click Add another host configuration to configure additional hosts.
- Click Next.
4.3.2. Configuring static networks by using the YAML view Copy linkLink copied to clipboard!
If you are configuring multiple interfaces or other advanced networking features, select the YAML view to enter the network state for each host that uses NMState syntax. For more information about NMState, see NMState Declarative Network API.
Prerequisites
- You have selected the Static IP, bridges and bonds option under Hosts' network configuration on the Cluster details page. Selecting this option adds the Static network configurations step to the wizard.
Procedure
- Go to the Static network configurations page.
- From the Configure via options, select YAML view.
- Upload, drag and drop, or copy and paste a YAML file containing NMState into the editor for network configurations. For guidance, see NMState configuration examples.
- In the MAC to interface name mapping fields, enter the MAC address and interface name for each host interface used in your network configuration. For guidance, see MAC to interface mapping.
- Select the Copy the YAML content checkbox to copy the YAML content between multiple hosts.
- Click Add another host configuration to configure additional hosts.
- Click Next.
4.4. Installing Operators and Operator bundles Copy linkLink copied to clipboard!
You can customize your deployment by selecting Operators and Operator bundles during the installation. If you require advanced options, add the Operators or bundles after you have installed the cluster.
When installing Operators and Operator bundles through the web console, the following conditions apply:
- Some Operators are only available as part of a bundle. You cannot install them individually.
- Some Operators are available either individually or as part of a bundle. If you install them as part of a bundle, you can only remove them by deselecting the bundle.
- Some Operators are only available for installing individually. They are not included in either of the bundles.
This step is optional.
4.4.1. Installing standalone Operators Copy linkLink copied to clipboard!
You can select more than one standalone Operator and add Operator bundles as needed. Operators that appear greyed out are only available for installation as part of a bundle.
For instructions on installing Operator bundles, see Installing Operator bundles.
Prerequisites
- You have reviewed the overview of each Operator that you intend to install, together with its prerequisites and dependencies. For details, see Customizing your installation using Operators.
Procedure
- On the Operators page, expand the Single Operators arrow to display the full list of Operators. You can use the Find Bundles or Operators field to filter the list and display matching items.
Select from the following Operators:
Expand Category Operator Comments Storage
Logical Volume Manager Storage
OpenShift Data Foundation
Virtualization
OpenShift Virtualization
The OpenShift Virtualization Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:
- None - If the CPU architecture is ARM64, no storage Operator is activated.
- LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
- Local Storage Operator (LSO) - For all other deployments.
Migration Toolkit for Virtualization
Selecting the Migration Toolkit for Virtualization Operator automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.
OpenShift sandboxed containers
Artificial Intelligence (AI)
OpenShift AI
Developer Preview feature.
AMD GPU
- Developer Preview feature.
- Selecting the AMD GPU Operator automatically activates the Kernel Module Management Operator.
NVIDIA GPU
- Developer Preview feature.
- Selecting the NVIDIA GPU Operator automatically activates the Node Feature Discovery Operator.
Network
NMState
Currently, you cannot install the Kubernetes NMState Operator on the Nutanix or Oracle Cloud Infrastructure (OCI) third-party platforms.
Security & Access Control
Kernel Module Management
Developer Preview feature.
Platform Operations & Lifecycle
Multicluster engine
You can deploy the multicluster engine with OpenShift Data Foundation on all OpenShift Container Platform clusters. Deploying the multicluster engine without OpenShift Data Foundation results in the following storage configurations:
- Multi-node cluster: No storage is configured. You must configure storage after the installation.
- Single-node OpenShift: LVM Storage is installed.
Cluster Observability
Selecting the Cluster Observability Operator automatically activates the OpenShift Loki and OpenShift Logging Operators.
Scheduling
Node Feature Discovery
Developer Preview feature.
ImportantThe integration of the AMD GPU, Kernel Module Management, Node Feature Discovery, NVIDIA GPU, and OpenShift AI Operators into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
- To install the Self Node Remediation Operator, see Installing Operators by using the API. It is not currently possible to install this Operator using the web console.
- Click Next.
4.4.2. Installing Operator bundles Copy linkLink copied to clipboard!
You can select more than one Operator bundle together with additional Operators as needed.
For instructions on installing individual Operators, see Installing Operators.
Prerequisites
- You have reviewed the overview of each Operator bundle that you intend to install, together with its prerequisites and associated Operators. For details, see Customizing your installation using Operator bundles.
Procedure
On the Operators page, select from the following Operator bundles:
Virtualization - Contains the following Operators:
- OpenShift Virtualization
- Kube Descheduler
- Node Maintenance
- Migration Toolkit for Virtualization
- Kubernetes NMState
- Fence Agents Remediation
- Node Health Check
- Local Storage Operator (LSO)
- Cluster Observability
- Loki
- OpenShift Logging
- MetalLB
- NUMA Resources
- OADP
OpenShift AI - Contains the following Operators:
- Kubernetes Authorino
- OpenShift Data Foundation
- OpenShift AI
- AMD GPU
- Node Feature Discovery
- NVIDIA GPU
- OpenShift Pipelines
- OpenShift Service Mesh
- OpenShift Serverless
- Kernel Module Management
ImportantThe OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
- Expand the Single Operators arrow to view the Operators selected for the bundle. You can use the Find Bundles or Operators field to filter the list and display matching items.
- Click Next.
4.5. Adding hosts to the cluster Copy linkLink copied to clipboard!
You must add one or more hosts to the cluster. Adding a host to the cluster involves generating a discovery ISO. The discovery ISO runs Red Hat Enterprise Linux CoreOS (RHCOS) in-memory with an agent.
If you are installing the IBM Z® architecture, use the following table to identify the image file type:
Expand Architecture Boot method Image type Logical Partition-Classic
iPXE
Full image file: Download a self-contained ISO image
Logical Partition-Data Protection Manager
ISO or iPXE
Minimal image file: Download an ISO image that fetches content when booting up
- ISO images are not supported for installations on IBM Z (s390x) with z/VM or logical partitioning (LPAR) nodes; use the "Booting hosts with iPXE" procedure. ISO images and iPXE are supported for installations on RHEL KVM.
Perform the following procedure for each host on the cluster.
Procedure
In the Host discovery step, click the Add hosts button and select the provisioning type.
Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot. The nodes must have virtual media capability. This is the recommended method for
x86_64andarm64architectures.ImportantThis option is mandatory in the following scenarios:
- If you are installing OpenShift Container Platform on Oracle Cloud Infrastructure.
- If you are installing OpenShift Container Platform on iSCSI boot volumes.
-
Select Full image file: Provision with physical media to download the larger full image. This is the recommended method for the
ppc64learchitecture and for thes390xarchitecture when installing with RHEL KVM. Select iPXE: Provision from your network server to boot the hosts using iPXE. This is the recommended method on IBM Z® with z/VM nodes and LPAR (both static and DPM). ISO boot is the recommended method on the RHEL KVM installation.
NoteIf you are installing OpenShift Container Platform on RHEL KVM, in some circumstances, the VMs on the KVM host are not rebooted on first boot and need to be restarted manually.
Optional: Activate the Run workloads on control plane nodes switch to schedule workloads to run on control plane nodes, in addition to the default worker nodes.
NoteThis option is available for clusters of five or more nodes. For clusters of under five nodes, the system runs workloads on the control plane nodes only, by default. For more details, see Control plane workload scheduling.
Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. For details, see Configuring your firewall for OpenShift Container Platform.
NoteThe proxy username and password must be URL-encoded.
Optional: Add an SSH public key so that you can connect to the cluster nodes as the
coreuser. Having a login to the cluster nodes can provide you with debugging information during the installation.ImportantDo not skip this procedure in production environments, where disaster recovery and debugging is required.
- If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
-
In the SSH public key field, click Browse to upload the
id_rsa.pubfile containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
- Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy, or if the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates. Add additional certificates in X.509 format.
- Configure the discovery image if needed. For details, see Configuring the discovery image.
- Optional: If you are installing on a platform and want to integrate with the platform, select Integrate with your virtualization platform. You must boot all hosts and ensure they appear in the host inventory. All the hosts must be on the same platform. For more details, see the relevant platform in Additional resources.
- Click Generate Discovery ISO or Generate Script File.
- Download the discovery ISO or iPXE script.
- Boot the host(s) with the discovery image or iPXE script. For details, see Booting hosts with the discovery image.
4.6. Configuring hosts Copy linkLink copied to clipboard!
After booting the hosts with the discovery ISO, the hosts will appear in a table on the Host Discovery page. You can optionally configure the hostname and role for each host. You can also delete a host if necessary.
Procedure
- Go to the Host Discovery tab.
In multi-host clusters, you can select a role for each host after host discovery:
- In the Role column, expand the Auto-assign arrow for the host.
Choose one of the following options:
-
Auto-assign - Automatically determines whether the host is a
control plane,workerorarbiternode. This is the default setting. - Control plane node - Assigns the control plane (master) role to the host, allowing the host to manage and coordinate the cluster.
- Worker - Assigns the compute (worker) role to the host, enabling the host to run application workloads.
- Arbiter - Assigns the arbiter role to a host, providing a cost-effective solution for components that require a quorum.
The minimum hardware requirements for control plane nodes exceed that of worker nodes. If you assign a role to a host, ensure that you assign the control plane role to hosts that meet the minimum hardware requirements.
For more details about the different host roles, see Supported host roles.
-
Auto-assign - Automatically determines whether the host is a
- Click the Status link to view hardware, network, and operator validations for the host.
- Optionally select or deselect all hosts by clicking the <number> selected checkbox or alternatively by selecting Select all or Select none from the dropdown list.
Optionally change one or more hostnames:
- To rename a single host, from the Options (⋮) menu for the host, select Change hostname. If necessary, enter a new name for the host and click Change. You must ensure that each host has a valid and unique hostname.
To rename multiple selected hosts, from the Actions list, select Change hostname. In the Change Hostname dialog, type the new name and include
{{n}}to make each hostname unique. Then click Change.NoteYou can see the new names appearing in the Preview pane as you type. The name will be identical for all selected hosts, with the exception of a single-digit increment per host.
Optionally remove one or more hosts:
- To remove a single host, from the Options (⋮) menu for the host, select Remove host. Click Remove host to confirm the deletion.
- To remove multiple selected hosts at the same time, from the Actions list, select Remove. Click Remove hosts to confirm the deletion.
NoteIn a regular deployment, a cluster can have three or more hosts, and at least three of these must be control plane nodes. If you delete a host that is also a control plane node, or if there are only two hosts, you will get a message saying that the system is not ready. To restore a host, you must reboot it from the discovery ISO.
- From the Options (⋮) menu for the host, optionally select View host events. The events in the list are presented chronologically.
Click the arrow to the left of a host name to expand the host details.
Once all cluster hosts appear with a status of Ready, proceed to the next step.
4.7. Configuring storage disks Copy linkLink copied to clipboard!
Each of the hosts retrieved during host discovery can have multiple storage disks. The storage disks are listed for the host on the Storage page of the Assisted Installer wizard.
You can optionally modify the default configurations for each disk.
- Starting from OpenShift Container Platform 4.14, you can configure nodes with Intel® Virtual RAID on CPU (VROC) to manage NVMe RAIDs. For details, see Configuring an Intel® Virtual RAID on CPU (VROC) data volume.
- Starting from OpenShift Container Platform 4.15, you can install a cluster on a single or multipath iSCSI boot device using the Assisted Installer.
4.7.1. Changing the installation disk Copy linkLink copied to clipboard!
The Assisted Installer randomly assigns an installation disk by default. If there are multiple storage disks for a host, you can select a different disk to be the installation disk. This automatically unassigns the previous disk.
Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing over Fibre Channel on the installation disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with an /etc/multipath.conf configuration. For details, see Modifying the DM Multipath configuration file.
Procedure
- Navigate to the Storage page of the wizard.
- Expand a host to display the associated storage disks.
Select Installation disk from the Role list.
NoteMultipath devices are automatically discovered and listed in the host’s inventory. To assign a multipath Fibre Channel disk as the installation disk, choose a disk with Drive type set to
Multipath, rather than toFCwhich indicates a single path.- When all storage disks return to Ready status, proceed to the next step.
4.7.2. Disabling disk formatting Copy linkLink copied to clipboard!
The Assisted Installer marks all bootable disks for formatting during the installation process by default, regardless of whether or not they have been defined as the installation disk. Formatting causes data loss.
You can choose to disable the formatting of a specific disk. Perform this with caution, as bootable disks can interfere with the installation process, mainly in terms of boot order.
You cannot disable formatting for the installation disk.
Procedure
- Navigate to the Storage page of the wizard.
- Expand a host to display the associated storage disks.
- Clear Format for a disk.
- When all storage disks return to Ready status, proceed to the next step.
4.8. Configuring networking Copy linkLink copied to clipboard!
Before installing OpenShift Container Platform, you must configure the cluster network.
Procedure
In the Networking step, select one of the following network management types if it is not already selected for you:
Cluster-Managed Networking: Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology. This configuration includes an integrated load balancer and virtual routing for managing the API and Ingress VIP addresses. For details, see Cluster-managed networking.
Note- Currently, cluster-managed networking is not supported on IBM Z® and IBM Power®.
- Cluster-managed networking is not supported on single-node OpenShift.
User-Managed Networking: Selecting user-managed networking deploys OpenShift Container Platform with a non-standard network topology. Select user-managed networking if you want to deploy with an external load balancer and DNS, or if you intend to deploy the cluster nodes across many distinct subnets. For details, see User-managed networking.
NoteOracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.
ImportantThe Assisted Installer supports a third network management type called Cluster-Managed Networking with a User-Managed Load Balancer. This network management type provides automated cluster networking with an external load balancer. Currently you can configure this network management type through the API only. For details, see Installing cluster-managed networking with a user-managed load balancer.
For cluster-managed networking, configure the following settings:
Select your Networking stack type:
- IPv4: Select this type when your hosts are only using IPv4.
Dual-stack: Select dual-stack when your hosts use IPv4 with IPv6, with either stack set as primary.
Note- IPv6 is not currently supported for single stack.
- IPv4 networking with SDN is supported up to OpenShift Container Platform 4.14.
- IPv4 and dual-stack IPv4/IPv6 networking with OVN only are supported in OpenShift Container Platform 4.15 and later.
ImportantSupport for IPv6 as the primary stack in a dual-stack configuration is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
Define the Machine network. You can use the default network or select a subnet from the dropdown list.
For a dual-stack implementation, specify both a primary and a secondary machine network, ensuring that one is IPv4 and the other IPv6. When you select IPv6 as the primary network, the secondary network dropdown list shows only IPv4 options, and vice versa.
ImportantFor iSCSI boot volumes, the hosts connect over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. Ensure you select the OpenShift Container Platform network from the dropdown list. The iSCSI host IP address should not be on the machine network. Choosing the iSCSI network will result in an Insufficient status for the host in the Networking step.
Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with and configure the platform.
For a dual-stack implementation, define the API virtual IP for both the primary and secondary stack. The IP family (IPv4 or IPv6) must match the corresponding machine network family.
Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.
For a dual-stack implementation, define the Ingress virtual IP for both the primary and secondary stack. The IP family (IPv4 or IPv6) must match the corresponding machine network family.
Configure the advanced networking properties:
For an IPv4 implementation, optionally select Use advanced networking to configure the following properties:
- Cluster network CIDR: Define an IP address block to assign pod IP addresses.
- Cluster network host prefix: Define a subnet prefix length to assign to each node.
- Service network CIDR: Define an IP address block to assign service IP addresses.
- For a dual-stack implementation, these fields are mandatory and populated automatically for both primary and secondary stacks, according to the machine networks selected. You can change these values as required.
- Optional: Select Host SSH Public Key for troubleshooting after installation to connect to hosts using a public SSH key for troubleshooting after installation.
- Wait for the validations to complete, and then click Next. The Review and create page presents a summary of your configurations in the Networking section.
4.9. Adding manifests and patches Copy linkLink copied to clipboard!
You can upload custom manifests and patches for system manifests in the Assisted Installer web console. You can also replace and remove these files.
For information about adding and modifying custom manifests by using the Assisted Installer API, see Additional resources.
4.9.1. Preparing custom manifests and manifest patches Copy linkLink copied to clipboard!
This section provides an overview of custom manifests and system manifest patches, including formatting considerations and the required naming conventions for uploading the files.
Follow these guidelines to ensure that the files you upload comply with the system requirements.
4.9.1.1. Custom manifests Copy linkLink copied to clipboard!
A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer user interface. You can create a custom manifest or use one provided by a third party.
You can upload a custom manifest from your file system to either the openshift folder or the manifests folder. The number of custom manifest files permitted is unlimited.
You can upload only one file at a time. However, each uploaded YAML file can contain multiple custom manifests. Uploading a multi-document YAML manifest is faster than adding the YAML files individually.
For a file containing a single custom manifest, accepted file extensions include .yaml, .yml, or .json. For a file containing multiple custom manifests, accepted file types include .yaml or .yml.
Single custom manifest example:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: master
name: 99-openshift-machineconfig-master-kargs
spec:
kernelArguments:
- loglevel=7
Multiple custom manifest example:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: master
name: 99-openshift-machineconfig-master-kargs
spec:
kernelArguments:
- loglevel=7
---
apiVersion: machineconfiguration.openshift.io/v2
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 98-openshift-machineconfig-worker-kargs
spec:
kernelArguments:
- loglevel=5
When you install OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) external platform, you must add the custom manifests provided by Oracle. For additional external partner integrations such as vSphere or Nutanix, this step is optional.
4.9.1.2. Patches for system manifests Copy linkLink copied to clipboard!
A manifest patch file conforms to the syntax of a YAML patch. Its purpose is to modify a system manifest that is automatically created by the Assisted Installer during installation preparation. Manifest patches are used to adjust onfigurations, manage updates, or apply changes in a structured and automated way. This approach ensures consistency and helps avoid errors when altering complex YAML documents.
4.9.1.2.1. General YAML syntax for system manifest patches Copy linkLink copied to clipboard!
The yaml-patch package is an implementation of JavaScript Object Notation (JSON) Patch, directly transposed to YAML. The general syntax of a system manifest YAML patch is the following:
- op: <add | remove | replace | move | copy | test>
from: <source-path>
path: <target-path>
value: <any-yaml-structure>
where:
-
opspecifies the operation, which can be one of the following:add,remove,replace,move,copy, ortest. See JavaScript Object Notation (JSON) Patch for an explanation of each operation. -
fromspecifies the source path and is used with themoveandcopyoperations. -
pathspecifies the target path and is always required. -
valueaccepts any valid YAML structure and is used with theadd,replaceandtestoperations.
4.9.1.2.2. Creating a manifest patch Copy linkLink copied to clipboard!
When creating a new patch for a system manifest, use the following naming convention: <file to be patched>.patch_<suffix>. The name ensures that the correct manifest is overwritten, and the suffix allows for the application of many patches to the same manifest.
For example, if the original file has the name 50-masters-chrony-configuration.yaml, call the new patch 50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp or similar.
Procedure
Verify that the Assisted Installer has automatically added the original YAML file to the cluster manifests at the start of the installation, as in the example below:
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 50-masters-chrony-configuration spec: config: ignition: version: 3.1.0 storage: files: - contents: source: data:text/plain;charset=utf-8;base64,CnBvb2wgMC5yaGVsLnBvb2wubnRwLm9yZyBpYnVyc3QKZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc2VydmVyIHN0YXRpYy4xOTAuMTExLjE2MS41LmNsaWVudHMueW91ci1zZXJ2ZXIuZGUgaWJ1cnN0CnNlcnZlciAyMDguNjcuNzIuNDMgaWJ1cnN0CnNlcnZlciBoMTM1LTEzNC0xMTEtMTIyLm1kc253aS5icm9hZGJhbmQuZHluYW1pYy50ZHMubmV0IGlidXJzdApzZXJ2ZXIgdGltZS5yaWNoaWVtY2ludG9zaC5jb20gaWJ1cnN0CnNlcnZlciBhcm0xLm1heGhvc3QuaW8gaWJ1cnN0CnNlcnZlciBzcGlkZXkucmVsbGltLmNvbSBpYnVyc3QKc2VydmVyIHVzLWVhc3QtMi5jbGVhcm5ldC5wdyBpYnVyc3QKc2VydmVyIDEwOC42MS43My4yNDMgaWJ1cnN0CnNlcnZlciBudHAxLm50cC0wMDEucHJvZC5pYWQyLmRjLnJlZGhhdC5jb20gaWJ1cnN0CnNlcnZlciBjbG9jay5yZWRoYXQuY29tIGlidXJzdApzZXJ2ZXIgaXAyMjkuaXAtNTEtODEtMjI2LnVzIGlidXJzdApzZXJ2ZXIgNjYuMjA1LjI0OS4yOCBpYnVyc3QKc2VydmVyIDEwLjExLjE2MC4yMzggaWJ1cnN0CnNlcnZlciBjLTczLTE5My02Mi01NC5oc2QxLndhLmNvbWNhc3QubmV0IGlidXJzdApzZXJ2ZXIgbnRwMi53aWt0ZWwuY29tIGlidXJzdA== mode: 420 path: /etc/chrony.conf overwrite: truewhere:
-
Directory name is
OpenShift. -
Filename is
50-masters-chrony-configuration.yaml.
-
Directory name is
To patch this YAML file with different content, generate a new
base64representation of the content and create a patch file:Generate
base64file content for/etc/chrony.conf:$ cat << EOF | base64 --wrap 0of driftfile /var/lib/chrony/drift makestep 1.0 3 rtcsync logdir /var/log/chrony sourcedir /run/chrony-dhcp EOF echo ZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc291cmNlZGlyIC9ydW4vY2hyb255LWRoY3AKCreate a patch file using this
base64string:--- - op: replace path: /spec/config/storage/files/0/contents value: data:text/plain;charset=utf-8;base64,ZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc291cmNlZGlyIC9ydW4vY2hyb255LWRoY3AKwhere:
-
Directory name is
OpenShift. -
Filename is
50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp
-
Directory name is
- Submit the patch file through the Assisted Installer web console or API.
4.9.2. Uploading custom manifests and manifest patches Copy linkLink copied to clipboard!
When uploading a custom manifest or patch, enter the filename and select a destination folder. The filename must be unique across both folders; you cannot use the same file name in both folders.
Prerequisites
- You have saved a custom manifest file to a local directory using an appropriate file name and extension.
Procedure
- On the Cluster details page of the wizard, select the Include custom manifests checkbox.
On the Custom manifest page, in the folder field, select the Assisted Installer folder where you want to save the manifest or patch.
NoteYou can upload a file to either the openShift or manifest folder. For a manifest patch, the system will look in both folders for the target file that it needs to patch.
In the Filename field, enter a name for the manifest file, including the extension:
-
For custom manifests, examples include
manifest1.jsonormultiple1.yaml. - For manifest patches, an example is 50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp.
-
For custom manifests, examples include
- Under Content, click the Upload icon or Browse button to upload a file. Alternatively, drag the file into the Content field from your file system.
- To upload another file, click Add another manifest and repeat the process. This saves any previously uploaded files.
- Click Next to save all files and proceed to the Review and create page. Custom manifests displays a list of the uploaded custom manifests and patches.
4.9.3. Modifying custom manifests and manifest patches Copy linkLink copied to clipboard!
You can rename uploaded custom manifest or patch files, and save custom manifest files to a different folder. Additionally, you can copy the contents of an existing file, or download it to the folder specified in your Chrome download settings.
It is not possible to edit the content of an uploaded manifest or patch file. Instead, you can overwrite the existing file.
Prerequisites
- You have uploaded at least one custom manifest or patch file.
Procedure
- To change the location of a custom manifest file, select a different folder from the Folder list.
- To change the file name, type the new name for the manifest or patch in the File name field. Patch files should respect the patch naming conventions discussed earlier in this section.
To overwrite a manifest or patch file, save a new file with the same file name in either the openshift or manifest folder.
NoteThe system will automatically detect and replace the original file, regardless of which folder it is in.
- To download a manifest or patch to your file system, click the Download icon.
- To copy a manifest or patch, click the Copy to clipboard icon.
- To apply the changes, click either Add another manifest or Next.
4.9.4. Removing custom manifests and manifest patches Copy linkLink copied to clipboard!
You can remove uploaded custom manifests or patches before installation in one of two ways:
- Removing a single custom manifest or patch.
- Removing all manifests and patches at the same time.
Once you have removed a manifest or patch file you cannot undo the action. The workaround is to upload the file again.
4.9.4.1. Removing all custom manifests and patches Copy linkLink copied to clipboard!
You can remove all custom manifests and patches at the same time. This also hides the Custom manifest page.
Prerequisites
- You have uploaded at least one custom manifest or patch file.
Procedure
- Browse to the Cluster details page of the wizard.
- Clear the Include custom manifests checkbox.
- In the Remove custom manifests dialog box, click Remove.
4.9.4.2. Removing a single custom manifest or patch Copy linkLink copied to clipboard!
You can delete one file at a time. This option does not allow deletion of the last remaining manifest or patch.
Prerequisites
- You have uploaded at least two custom manifest or patch files.
Procedure
- Browse to the Custom manifests page.
- Hover over the manifest name to display the Delete (minus) icon.
- Click the icon and then click Delete in the dialog box.
4.10. Installing the cluster Copy linkLink copied to clipboard!
After completing the configuration and ensuring all the nodes are Ready, begin the installation. The installation process takes time, and you can monitor progress in the Assisted Installer web console. Nodes will reboot during the installation, and initialize after installation.
Prerequisites
- Before installing the cluster, ensure the cluster and each host have passed the preinstallation validations. For details, see Preinstallation validations.
Procedure
- In the Review and create step, click Install cluster.
- Click the link in the Status column of the Host Inventory list to see the installation status of a particular host.
4.11. Completing the installation Copy linkLink copied to clipboard!
After the cluster is installed and initialized, the Assisted Installer indicates that the installation is finished. The Assisted Installer provides the console URL, the kubeadmin username and password, and the kubeconfig file. Additionally, the Assisted Installer provides cluster details including the OpenShift Container Platform version, base domain, CPU architecture, API and Ingress IP addresses, and the cluster and service network IP addresses.
Prerequisites
-
You have installed the
ocCLI tool.
Procedure
-
Make a copy of the
kubeadminusername and password. Download the
kubeconfigfile and copy it to theauthdirectory under your working directory:$ mkdir -p <working_directory>/auth$ cp kubeconfig <working_directory>/authNoteThe
kubeconfigfile is available for download for 20 days after completing the installation.Add the
kubeconfigfile to your environment:$ export KUBECONFIG=<your working directory>/auth/kubeconfigLogin with the
ocCLI tool:$ oc login -u kubeadmin -p <password>Replace
<password>with the password of thekubeadminuser.- Click the web console URL or click Launch OpenShift Console to open the console.
-
Enter the
kubeadminusername and password. Follow the instructions in the OpenShift Container Platform console to configure an identity provider and configure alert receivers. - Add a bookmark of the OpenShift Container Platform console.
- Complete any postinstallation platform integration steps.
Chapter 5. Installing with the Assisted Installer API Copy linkLink copied to clipboard!
After you meet the cluster nodes and network requirements, you can begin installing the cluster by using the Assisted Installer API. To use the API, you must perform the following procedures:
- Set up the API authentication.
- Configure the pull secret.
- Register a new cluster definition.
- Create an infrastructure environment for the cluster.
Once you perform these steps, you can modify the cluster definition, create discovery ISOs, add hosts to the cluster, and install the cluster.
5.1. Generating the offline token Copy linkLink copied to clipboard!
Download the offline token from the Assisted Installer web console. You will use the offline token to set the API token.
Prerequisites
-
Install
jq. - Log in to the OpenShift Cluster Manager as a user with cluster creation privileges.
Procedure
- In the menu, click Downloads.
- In the Tokens section under OpenShift Cluster Manager API Token, click View API Token.
Click Load Token.
ImportantDisable pop-up blockers.
- In the Your API token section, copy the offline token.
In your terminal, set the offline token to the
OFFLINE_TOKENvariable:$ export OFFLINE_TOKEN=<copied_token>TipTo make the offline token permanent, add it to your profile.
(Optional) Confirm the
OFFLINE_TOKENvariable definition.$ echo ${OFFLINE_TOKEN}
5.2. Authenticating with the REST API Copy linkLink copied to clipboard!
API calls require authentication with the API token. Assuming you use API_TOKEN as a variable name, add -H "Authorization: Bearer ${API_TOKEN}" to API calls to authenticate with the REST API.
The API token expires after 15 minutes.
Prerequisites
-
You have generated the
OFFLINE_TOKENvariable.
Procedure
On the command line terminal, set the
API_TOKENvariable using theOFFLINE_TOKENto validate the user.$ export API_TOKEN=$( \ curl \ --silent \ --header "Accept: application/json" \ --header "Content-Type: application/x-www-form-urlencoded" \ --data-urlencode "grant_type=refresh_token" \ --data-urlencode "client_id=cloud-services" \ --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \ "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \ | jq --raw-output ".access_token" \ )Confirm the
API_TOKENvariable definition:$ echo ${API_TOKEN}Create a script in your path for one of the token generating methods. For example:
$ vim ~/.local/bin/refresh-tokenexport API_TOKEN=$( \ curl \ --silent \ --header "Accept: application/json" \ --header "Content-Type: application/x-www-form-urlencoded" \ --data-urlencode "grant_type=refresh_token" \ --data-urlencode "client_id=cloud-services" \ --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \ "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \ | jq --raw-output ".access_token" \ )Then, save the file.
Change the file mode to make it executable:
$ chmod +x ~/.local/bin/refresh-tokenRefresh the API token:
$ source refresh-tokenVerify that you can access the API by running the following command:
$ curl -s https://api.openshift.com/api/assisted-install/v2/component-versions -H "Authorization: Bearer ${API_TOKEN}" | jqResult
{ "release_tag": "v2.11.3", "versions": { "assisted-installer": "registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:v1.0.0-211", "assisted-installer-controller": "registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:v1.0.0-266", "assisted-installer-service": "quay.io/app-sre/assisted-service:78d113a", "discovery-agent": "registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:v1.0.0-195" } }
5.3. Configuring the pull secret Copy linkLink copied to clipboard!
Many of the Assisted Installer API calls require the pull secret. Download the pull secret to a file so that you can reference it in API calls. The pull secret is a JSON object that will be included as a value within the request’s JSON object. The pull secret JSON must be formatted to escape the quotes.
For example, convert the pull secret JSON from its original format here:
{"auths":{"cloud.openshift.com": ...
To the following format with escaped quotes for use in an API request:
{\"auths\":{\"cloud.openshift.com\": ...
Procedure
- In the menu, click OpenShift.
- In the submenu, click Downloads.
- In the Tokens section under Pull secret, click Download.
To use the pull secret from a shell variable, execute the following command:
$ export PULL_SECRET=$(cat ~/Downloads/pull-secret.txt | jq -R .)To slurp the pull secret file using
jq, reference it in thepull_secretvariable, piping the value totojsonto ensure that it is properly formatted as escaped JSON. For example:$ curl https://api.openshift.com/api/assisted-install/v2/clusters \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' { "name": "testcluster", "control_plane_count": "3", "openshift_version": "4.11", "pull_secret": $pull_secret[0] | tojson, "base_dns_domain": "example.com" } ')"where:
-
--slurpfile pull_secret ~/Downloads/pull-secret.txtslurps the pull secret file. -
"pull_secret": $pull_secret[0] | tojsonformats the pull secret to escaped JSON format.
-
Confirm the
PULL_SECRETvariable definition:$ echo ${PULL_SECRET}
5.4. Generating the SSH public key Copy linkLink copied to clipboard!
During the installation of OpenShift Container Platform, you can optionally provide an SSH public key to the installation program. This is useful for initiating an SSH connection to a remote node when troubleshooting an installation error.
If you do not have an existing SSH key pair on your local machine to use for the authentication, create one now.
For more information, see Generating a key pair for cluster node SSH access.
Prerequisites
-
Generate the
OFFLINE_TOKENandAPI_TOKENvariables.
Procedure
From the root user in your terminal, get the SSH public key:
$ cat /root/.ssh/id_rsa.pubSet the SSH public key to the
CLUSTER_SSHKEYvariable:$ CLUSTER_SSHKEY=<downloaded_ssh_key>Confirm the
CLUSTER_SSHKEYvariable definition:$ echo ${CLUSTER_SSHKEY}
5.5. Registering a new cluster Copy linkLink copied to clipboard!
To register a new cluster definition with the API, use the /v2/clusters endpoint.
The following parameters are mandatory:
-
name -
openshift-version -
pull_secret -
cpu_architecture
See the cluster-create-params model in the API viewer for details on the fields you can set when registering a new cluster.
This document does not cover every endpoint of the Assisted Installer API, but you can review all of the endpoints in the API viewer or the swagger.yaml file.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. - You have downloaded the pull secret.
-
Optional: You have assigned the pull secret to the
$PULL_SECRETvariable.
Procedure
Refresh the API token:
$ source refresh-tokenRegister a new cluster by using one of the following methods:
Reference the pull secret file in the request:
$ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' \ { \ "name": "testcluster", \ "openshift_version": "<version>", \ "control_plane_count": "<number>", \ "cpu_architecture" : "<architecture_name>", \ "base_dns_domain": "example.com", \ "pull_secret": $pull_secret[0] | tojson \ } \ ')" | jq '.id'Write the configuration to a JSON file and then reference it in the request:
$ cat << EOF > cluster.json { "name": "testcluster", "openshift_version": "<version>", "control_plane_count": "<number>", "base_dns_domain": "example.com", "network_type": "examplenetwork", "cluster_network_cidr":"11.111.1.0/14" "cluster_network_host_prefix": 11, "service_network_cidr": "111.11.1.0/16", "api_vips":[{"ip": ""}], "ingress_vips": [{"ip": ""}], "vip_dhcp_allocation": false, "additional_ntp_source": "clock.redhat.com,clock2.redhat.com", "ssh_public_key": "$CLUSTER_SSHKEY", "pull_secret": $PULL_SECRET } EOF$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters" \ -d @./cluster.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'
Set the values as follows:
Replace
<version>with the OpenShift Container Platform version:-
To install the latest OpenShift version, use the
x.yformat, such as4.16for version4.16.10. To install a specific OpenShift version, use thex.y.zformat, such as4.16.3. -
To install a multi-architecture compute cluster, add the
-multiextension, such as4.16-multifor the latest version or4.16.3-multifor a specific version. - If you are booting from an iSCSI drive, enter OpenShift Container Platform version 4.15 or later.
-
To install the latest OpenShift version, use the
Replace
<number>with the number of control planes in your cluster. Possible values are as follows:-
1for a single-node OpenShift cluster. Currently, single-node OpenShift is not supported on IBM Power® platform. -
2or more for a Two-Node OpenShift with Arbiter cluster. This option is supported from OpenShift Container Platform 4.19 and later on a Two-Node OpenShift with Arbiter cluster topology. A cluster with2control plane nodes must have at least one additional arbiter host. For details, see Two-Node OpenShift with Arbiter resource requirements. -
3,4, or5for a multi-node OpenShift Container Platform cluster. Ifcontrol_plane_countis omitted, the default setting is3. The Assisted Installer supports4or5control plane nodes from OpenShift Container Platform 4.18 and later, on a bare metal or user-managed networking platform with an x86_64 CPU architecture. For details, see Control plane count.
-
If you are referencing the pull secret, replace
<architecture_name>with the CPU architecture. Options includex86_64,arm64,ppc64le,s390x, ormulti. Specifymultifor a multi-architecture compute cluster.NoteThe
control_plane_countfield replaces thehigh_availability_modefield, which is deprecated. For details, see API deprecation notice.
Assign the returned
cluster_idto theCLUSTER_IDvariable and export it:$ export CLUSTER_ID=<cluster_id>NoteIf you close your terminal session, you need to export the
CLUSTER_IDvariable again in a new terminal session.Check the status of the new cluster:
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jqOnce you register a new cluster definition, create the infrastructure environment for the cluster.
NoteYou cannot see the cluster configuration settings in the Assisted Installer user interface until you create the infrastructure environment.
5.5.1. Installing Operators Copy linkLink copied to clipboard!
You can customize your deployment by adding Operators to the cluster during installation. You can install one or more Operators individually or add a group of Operators that form a bundle. If you require advanced options, add the Operators after you have installed the cluster.
This step is optional.
5.5.1.1. Installing standalone Operators Copy linkLink copied to clipboard!
Before selecting Operators for installation, you can verify which Operators are available in the Assisted Installer. You can also check whether an Operator is supported for a specific OCP version, CPU architecture, or platform.
You set the required Operator definitions by using the POST method for the assisted-service/v2/clusters/{cluster_id} endpoint and by setting the olm_operators parameter.
The Assisted Installer allows you to install the following Operators:
| Category | Operator | API value | Comments |
|---|---|---|---|
| Storage | |||
| Local Storage Operator (LSO) |
| Also part of the OpenShift AI Operator bundle and Virtualization bundle. | |
| Logical Volume Manager Storage |
| ||
| OpenShift Data Foundation |
| Also part of the OpenShift AI Operator bundle. | |
| OpenShift API for Data Protection |
| Also part of the Virtualization bundle. | |
| Virtualization | |||
| OpenShift Virtualization |
| This Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:
Currently, OpenShift Virtualization is not supported on IBM Z® and IBM Power®. This Operator is also part of the Virtualization bundle. | |
| Migration Toolkit for Virtualization |
|
| |
| OpenShift sandboxed containers |
| ||
| Artificial Intelligence (AI) | |||
| OpenShift AI |
| Developer Preview feature. | |
| AMD GPU |
|
| |
| NVIDIA GPU |
|
| |
| Network | |||
| Kubernetes NMState |
|
| |
| OpenShift Service Mesh |
|
| |
| MetalLB |
|
| |
| Remediation | |||
| Node Health Check |
| Also part of the Virtualization bundle. | |
| Fence Agents Remediation |
| Also part of the Virtualization bundle. | |
| Self Node Remediation |
| You can only install this Operator through the API. | |
| Security & Access Control | |||
| Authorino |
|
| |
| Kernel Module Management |
| Developer Preview feature. | |
| CI/CD & Dev Productivity | |||
| OpenShift Pipelines |
|
| |
| OpenShift Serverless |
( |
| |
| Platform Operations & Lifecycle | |||
| Multicluster engine |
| Deploying the multicluster engine without OpenShift Data Foundation results in the following storage configurations:
| |
| Node Maintenance |
| Also part of the Virtualization Operator bundle. | |
| Cluster Observability Operator |
|
| |
| OpenShift Loki |
|
| |
| OpenShift Logging |
|
| |
| Scheduling | |||
| Node Feature Discovery |
| Developer Preview feature. | |
| Kube Descheduler |
| Also part of the Virtualization Operator bundle. | |
| NUMA Resources |
| Also part of the Virtualization Operator bundle. | |
The integration of some of these Operators into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
For instructions on installing Operator bundles, see Installing bundle Operators.
Prerequisites
- You have reviewed Customizing your installation using Operators for an overview of each Operator that you intend to install, together with its prerequisites and dependencies.
Procedure
Optional: Verify which Operators are available in the Assisted Installer, by running the following command:
$ curl -s "https://api.openshift.com/api/assisted-install/v2/supported-operators" -H "Authorization: Bearer ${API_TOKEN}" | jq .Optional: Verify whether an Operator is supported for a specific OpenShift Container Platform version by running the following command:
$ curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=<version-number> -H "Authorization: Bearer ${API_TOKEN}" | jq .features.<OPERATOR-NAME>This command accepts the following optional parameters:
-
cpu_architecture: Possible values arex86_64,aarch64,arm64,ppc64le,s390x, ormulti. -
platform_type: Possible values arebaremetal,none,nutanix,vsphere, orexternal. -
external_platform_name: A third-party platform such asoci,vSphere,nutanix, orovirt. feature_ids: An array representing specific features. Currently, the only value is "SNO".Example input:
$ curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=4.13&cpu_architecture=x86_64&platform_type=baremetal&external_platform_name=oci&feature_ids=SNO" -H "Authorization: Bearer ${API_TOKEN}" | jq .features.OSCSpecify the name of the Operator in upper case, for example,
.NODE-FEATURE-DISCOVERYfor Node Feature Discovery or.OSCfor OpenShift sandboxed containers.Example output:
"supported"Possible output statuses are
supported,dev-preview,tech-preview, andunavailable.
-
Optional: Remove the
.featuressuffix from the previous command to retrieve a list of supported features for the specified parameters.Example input:
$ curl -X GET "http://api.openshift.com/api/assisted-install/v2/support-levels/features/?openshift_version=4.18.1&cpu_architecture=x86_64&platform_type=none" -s | jqOptional: Add the
/detailedparameter to the previous command to retrieve a detailed response for the specified parameters. When a feature is not supported, the output includes the reason. Operator dependencies appear underdependencies.Example input:
$ curl -X GET "http://api.openshift.com/api/assisted-install/v2/support-levels/features/detailed?openshift_version=4.18.1&cpu_architecture=x86_64&platform_type=none" -s | jqSpecify the Operators to install by running the following command:
$ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' { "name": "testcluster", "openshift_version": "4.15", "cpu_architecture" : "x86_64", "base_dns_domain": "example.com", "olm_operators": [ { "name": "mce" } , { "name": "odf" } , { "name": "amd-gpu" } ] "pull_secret": $pull_secret[0] | tojson } ')" | jq '.id'In the
olm_operatorsarray, list the Operators in the bundle that you want to install, for examplecnvfor OpenShift Virtualization ormcefor multicluster engine. Installing an Operator automatically activates any dependent Operators.
5.5.1.2. Installing Operator bundles Copy linkLink copied to clipboard!
The Assisted Installer supports the following Operator bundles through the web console:
- Virtualization Operator bundle
- OpenShift AI Operator bundle
Although the API does not provide an endpoint for installing Operator bundles, you can identify the Operators included in a bundle and install those Operators together.
The OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- You have reviewed Customizing your installation using Operator bundles for an overview of the Operator bundles, together with their prerequisites and associated Operators.
Procedure
Optional: Check which Operator bundles are available in the Assisted Installer by running the following command:
$ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles" -H "Authorization: Bearer ${API_TOKEN}" | jq .This command accepts the following optional parameters:
-
openshift_version: The OpenShift Container Platform version number. -
cpu_architecture: Possible values arex86_64,aarch64,arm64,ppc64le,s390x, ormulti. -
platform_type: Possible values arebaremetal,none,nutanix,vsphere, orexternal. -
external_platform_name: A third-party platform such asoci,vSphere,nutanix, orovirt. -
feature_ids: An array representing specific features. Currently, the only value is "SNO", which supports the OpenShift AI Operator bundle, but not the Virtualization bundle.
Example input:
$ curl -X GET "http://127.0.0.1:8090/api/assisted-install/v2/operators/bundles?openshift_version=4.18.1&cpu_architecture=x86_64&platform_type=none&feature_ids=SNO" -s | jq .Example output:
[ { "description": "Train, serve, monitor and manage AI/ML models and applications using GPUs.", "id": "openshift-ai", "operators": [ "lvm", "amd-gpu", "node-feature-discovery", "nvidia-gpu", "openshift-ai" ], "title": "OpenShift AI" } ]-
Optional: Check which Operators belong to a specific bundle by running the following command:
$ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles/<bundle-name>" -H "Authorization: Bearer ${API_TOKEN}" | jq .For
<bundle-name>, specifyvirtualizationoropenshift-ai.Example input:
$ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles/virtualization" -H "Authorization: Bearer ${API_TOKEN}" | jq .Example output:
{ "description": "Run virtual machines alongside containers on one platform.", "id": "virtualization", "operators": [ "kube-descheduler", "node-maintenance", "mtv", "nmstate", "fence-agents-remediation", "cnv", "node-healthcheck", "cluster-observability" "metallb" "numaresources" "oadp" "loki" "openshift-logging" ], "title": "Virtualization" }Install the Operators associated with the bundle by running the command displayed in the following example:
$ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' { "name": "testcluster", "openshift_version": "4.15", "cpu_architecture" : "x86_64", "base_dns_domain": "example.com", "olm_operators": [ { "name": "node-healthcheck" } { "name": "fence-agents-remediation" } { "name": "kube-descheduler" } { "name": "mtv" } { "name": "nmstate" } { "name": "node-maintenance" } { "name": "cnv" } { "name": "cluster-observability" } { "name": "metallb" } { "name": "numaresources" } { "name": "oadp" } ] "pull_secret": $pull_secret[0] | tojson } ')" | jq '.id'where:
-
The
olm_operatorsarray specifies the Operators in the bundle you are installing. The example lists the Operators for the Virtualization bundle. -
In the Virtualization bundle, specifying
cnvautomatically installslsoin the background. In the OpenShift AI Operator bundle:
-
Specifying
nvidia-gpuautomatically installsnode-feature-discovery. -
Specifying
amd-gpuautomatically installskmm.
-
Specifying
-
The
5.5.2. Scheduling workloads to run on control plane nodes Copy linkLink copied to clipboard!
Use the schedulable_masters attribute to enable workloads to run on control plane nodes.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. -
You have created a
$PULL_SECRETvariable. - You are installing OpenShift Container Platform 4.14 or later.
Procedure
- Follow the instructions for installing Assisted Installer using the Assisted Installer API.
When you reach the step for registering a new cluster, set the
schedulable_mastersvalue totrue, as follows:$ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "schedulable_masters": true } ' | jq
5.5.3. Configuring the network management type Copy linkLink copied to clipboard!
You can install the following network management types through the Assisted Installer:
- Cluster-managed networking
- User-managed networking
- Cluster-managed networking with a user-managed load balancer
This step is optional. If you do not define a network management type, the Assisted Installer applies cluster-managed networking by default to all highly available clusters. For single-node OpenShift, the Assisted Installer applies user-managed networking by default.
You define the network management type by adding the user_managed_networking and load_balancer attributes to the cluster definition, as in the following example:
$ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
-H "Authorization: Bearer ${API_TOKEN}" \
-H "Content-Type: application/json" \
-d "$(jq --null-input \
--slurpfile pull_secret ~/Downloads/pull-secret.txt '
{
"name": "testcluster",
"openshift_version": "4.18",
"cpu_architecture" : "x86_64",
"base_dns_domain": "example.com",
"user_managed_networking": "false",
"load_balancer": {
"type": "cluster-managed"
}
"pull_secret": $pull_secret[0] | tojson
}
')" | jq '.id'
where:
-
user_managed_networkingis eithertrueorfalse. -
load_balancercan have the typeuser-managedorcluster-managed.
You can review the user_managed_networking and load_balancer valid values in the swagger.yaml file. For details, see Additional resources.
5.5.3.1. Installing cluster-managed networking Copy linkLink copied to clipboard!
Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology. This configuration includes an integrated load balancer and virtual routing for managing the API and Ingress VIP addresses. For details, see Cluster-managed networking.
Prerequisites
You are installing an OpenShift Container Platform cluster of three or more control plane nodes.
NoteCurrently, cluster-managed networking is not supported on IBM Z® and IBM Power®.
Procedure
To define cluster-managed networking, add the following attributes and values to your cluster definition:
"user_managed_networking": false, "load_balancer": { "type": "cluster-managed" }Where the
load_balancerattribute is optional. If omitted for this configuration, thetypeis automatically set touser-managedfor single-node OpenShift or tocluster-managedfor all other implementations.
5.5.3.2. Installing user-managed networking Copy linkLink copied to clipboard!
Selecting user-managed networking deploys OpenShift Container Platform with a non-standard network topology. Select user-managed networking if you want to deploy a cluster with an external load balancer and DNS, or if you intend to deploy the cluster nodes across many distinct subnets.
For details, see Cluster-managed networking.
The Assisted Installer lets you deploy more than one external load balancer for user-managed networking.
Oracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.
Procedure
To define user-managed networking, add the following attributes to your cluster definition:
"user_managed_networking": true,NoteThe
load_balancerattribute is not required when user-managed networking is set totrue, because you will be provisioning your own load balancer.
When you enable user-managed networking, the following network validations change:
- The L3 connectivity check (ICMP) replaces the L2 check (ARP).
- The maximum transmission unit (MTU) validation verifies the MTU value for all interfaces and not only for the machine network.
5.5.3.3. Installing cluster-managed networking with a user-managed load balancer Copy linkLink copied to clipboard!
Cluster-managed networking with a user-managed load balancer is a hybrid network management type designed for scenarios that require automated cluster networking with external control over load balancing. This approach enables users to provide one or more external load balancers (for example, an API load balancer and an Ingress load balancer), while retaining the bare-metal features installed in cluster-managed networking.
For details, see Cluster-managed networking with a user-managed load balancer.
Cluster-managed networking with a user-managed load balancer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Use the Assisted Installer API to deploy cluster-managed networking with a user-managed load balancer on a bare-metal or vSphere platform.
Prerequisites
- You are installing OpenShift Container Platform version 4.16 or higher.
- You are installing on a bare-metal or vSphere platform.
- You are using IPv4 single-stack networking.
- You are installing an OpenShift Container Platform cluster of three or more control plane nodes.
- For a vSphere platform installation, you meet the additional requirements specified in vSphere installation requirements.
Procedure
Configure the load balancer to be accessible from all hosts and have access to the following services:
- OpenShift Machine Config Operator (MCO) - On control plane nodes.
- OpenShift API - On control plane nodes.
- Ingress Controller - On compute (worker) nodes.
For details, see Configuring a user-managed load balancer (steps 1 and 2).
Configure the DNS records for your cluster to target the front-end IP addresses of the user-managed load balancer. You must update records to your DNS server for the cluster API and applications over the load balancer:
Configure the DNS record to make your primary API accessible:
<load_balancer_ip_address> <record_name> api.<cluster_name>.<base_domain>Configure the DNS record to route external traffic to your applications via an ingress controller:
<load_balancer_ip_address> <record_name> apps.<cluster_name>.<base_domain>For vSphere only, configure the DNS record to support internal API access within your network:
<load_balancer_ip_address> <record_name> api-int.<cluster_name>.<base_domain>
For details, see Configuring a user-managed load balancer (step 3).
Add the following configurations to the Assisted Installer API cluster definitions:
Set the
user_managed_networkingandload_balancerfields to the following values:"user_managed_networking": false, "load_balancer": { "type": "user-managed" }For details, see Changing the network management type.
Specify the Ingress and API VIPs. These should correspond to the load balancer IP address:
"ingress_vips": [ { "cluster_id": "<cluster-id>", "ip": "<load-balancer-ip>" } ], "api_vips": [ { "cluster_id": "<cluster-id>", "ip": "<load-balancer-ip>" } ]Specify a list of machine networks to ensure the following:
- Each node has at least one network interface card (NIC) with an IP address in at least one machine network.
The load balancer IP, which is also the API VIP and Ingress VIP, is included in at least one of the machine networks.
Example:
"machine_networks": [ { "cidr": "<hosts-cidr-1>", "cluster_id": "<cluster-id>" }, { "cidr": "<hosts-cidr-2>", "cluster_id": "<cluster-id>" }, { "cidr": "<load-balancer-cidr>", "cluster_id": "<cluster-id>" } ]
For more details, see Machine network.
When you enable this network management type, the following network validations change:
- The L3 connectivity check (ICMP) replaces the L2 check (ARP).
- The maximum transmission unit (MTU) validation verifies the MTU value for all interfaces and not only for the machine network.
5.6. Modifying a cluster Copy linkLink copied to clipboard!
To modify a cluster definition with the API, use the /v2/clusters/{cluster_id} endpoint. Modifying a cluster resource is a common operation for adding settings such as changing the network type or enabling user-managed networking. See the v2-cluster-update-params model in the API viewer for details on the fields you can set when modifying a cluster definition.
You can add or remove Operators from a cluster resource that has already been registered.
To create partitions on nodes, see Configuring storage on nodes in the OpenShift Container Platform documentation.
Prerequisites
- You have created a new cluster resource.
Procedure
Refresh the API token:
$ source refresh-tokenModify the cluster. For example, change the SSH key:
$ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZrD4LMkAEeoU2vShhF8VM+cCZtVRgB7tqtsMxms2q3TOJZAgfuqReKYWm+OLOZTD+DO3Hn1pah/mU3u7uJfTUg4wEX0Le8zBu9xJVym0BVmSFkzHfIJVTn6SfZ81NqcalisGWkpmkKXVCdnVAX6RsbHfpGKk9YPQarmRCn5KzkelJK4hrSWpBPjdzkFXaIpf64JBZtew9XVYA3QeXkIcFuq7NBuUH9BonroPEmIXNOa41PUP1IWq3mERNgzHZiuU8Ks/pFuU5HCMvv4qbTOIhiig7vidImHPpqYT/TCkuVi5w0ZZgkkBeLnxWxH0ldrfzgFBYAxnpTU8Ih/4VhG538Ix1hxPaM6cXds2ic71mBbtbSrk+zjtNPaeYk1O7UpcCw4jjHspU/rVV/DY51D5gSiiuaFPBMucnYPgUxy4FMBFfGrmGLIzTKiLzcz0DiSz1jBeTQOX++1nz+KDLBD8CPdi5k4dq7lLkapRk85qdEvgaG5RlHMSPSS3wDrQ51fD8= user@hostname" } ' | jq
5.6.1. Modifying Operators by using the API Copy linkLink copied to clipboard!
You can add or remove Operators from a cluster resource that has already been registered as part of a previous installation. This is only possible before you start the OpenShift Container Platform installation.
You modify the required Operator definition by using the PATCH method for the assisted-service/v2/clusters/{cluster_id} endpoint and by setting the olm_operators parameter.
For an overview of each Operator that you intend to install, together with its prerequisites and dependencies, see Customizing your installation using Operators and Operator Bundles.
Prerequisites
- You have refreshed the API token.
-
You have exported the
CLUSTER_IDas an environment variable.
Procedure
Run the following command to modify the Operators:
$ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "olm_operators": [{"name": "mce"}, {"name": "cnv"}], } ' | jq '.id'-
In the
olm_operatorsarray, enter the complete list of Operators to install, rather than only the differences. The example above includesmcefor multicluster engine andcnvfor OpenShift Virtualization. For a full list of Operators and their API values, see Installing Operators. -
To remove all Operators, specify an empty array as follows:
"olm_operators": [].
-
In the
{
<various cluster properties>,
"monitored_operators": [
{
"cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
"name": "console",
"operator_type": "builtin",
"status_updated_at": "0001-01-01T00:00:00.000Z",
"timeout_seconds": 3600
},
{
"cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
"name": "cvo",
"operator_type": "builtin",
"status_updated_at": "0001-01-01T00:00:00.000Z",
"timeout_seconds": 3600
},
{
"cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
"name": "mce",
"namespace": "multicluster-engine",
"operator_type": "olm",
"status_updated_at": "0001-01-01T00:00:00.000Z",
"subscription_name": "multicluster-engine",
"timeout_seconds": 3600
},
{
"cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
"name": "cnv",
"namespace": "openshift-cnv",
"operator_type": "olm",
"status_updated_at": "0001-01-01T00:00:00.000Z",
"subscription_name": "hco-operatorhub",
"timeout_seconds": 3600
},
{
"cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
"name": "lvm",
"namespace": "openshift-local-storage",
"operator_type": "olm",
"status_updated_at": "0001-01-01T00:00:00.000Z",
"subscription_name": "local-storage-operator",
"timeout_seconds": 4200
}
],
<more cluster properties>
The output is the description of the new cluster state. The monitored_operators property in the output contains Operators of two types:
-
"operator_type": "builtin": Operators of this type are an integral part of OpenShift Container Platform. -
"operator_type": "olm": Operators of this type are added manually by a user or automatically, as a dependency. In this example, the LVM Storage Operator is added automatically as a dependency of OpenShift Virtualization.
5.6.2. Disabling the Machine API on a bare-metal cluster Copy linkLink copied to clipboard!
When you install an OpenShift Container Platform cluster on a bare-metal platform, the Machine API (MAPI) included with the platform can interfere with nodes managed externally by the Cluster API (CAPM3). This occurs because both the Machine API and the Cluster API continuously overwrite the providerID attribute, creating a reconciliation loop.
The Machine API configures
providerIDas follows:providerID: baremetalhost:///openshift-machine-api/dev-master-0/96ca2aa2-a42c-4312-bc58-b0280655e722The Cluster ID configures
providerIDas follows:providerID: metal3://test-capi/bmh-vm-01/test-sno-vnr4f
Because the Cluster API manages nodes externally, the workload cluster does not require Machine API for node lifecycle management. However, it still requires the networking components provided by the bare-metal platform, such as keepalived and CoreDNS.
To prevent a conflict, install OpenShift Container Platform on a bare-metal platform with the Machine API disabled. This is possible from OpenShift Container Platform 4.19 and later.
You can disable the Machine API at install time only. After the installation is complete, this option is no longer available.
Prerequisites
- You have registered a new cluster. For details, see Registering a new cluster.
Procedure
Refresh the API token by running the following command:
$ source refresh-tokenPatch the installation configuration for the bare-metal cluster by running the command in the example below. Update the list of capabilities as needed for your installation, but exclude
MachineAPIto disable the Machine API.For a list of supported capabilities, see capabilities in the OpenShift Container Platform documentation.
$ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID}/install-config \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "capabilities": { "baselineCapabilitySet": "None", "additionalEnabledCapabilities": [ "openshift-samples", "marketplace", "Console", "baremetal", "Insights", "Storage", "NodeTuning", "CSISnapshot", "OperatorLifecycleManager", "Ingress" ] } } ' | jqAfter applying this patch, the Assisted Installer generates an installation configuration that supports a bare-metal installation without deploying the Machine API.
5.7. Registering a new infrastructure environment Copy linkLink copied to clipboard!
Once you register a new cluster definition with the Assisted Installer API, create an infrastructure environment using the v2/infra-envs endpoint. Registering a new infrastructure environment requires the following settings:
-
name -
pull_secret -
cpu_architecture
See the infra-env-create-params model in the API viewer for details on the fields you can set when registering a new infrastructure environment. You can modify an infrastructure environment after you create it. As a best practice, consider including the cluster_id when creating a new infrastructure environment. The cluster_id will associate the infrastructure environment with a cluster definition. When creating the new infrastructure environment, the Assisted Installer will also generate a discovery ISO.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. - You have downloaded the pull secret.
-
Optional: You have registered a new cluster definition and exported the
cluster_id.
Procedure
Refresh the API token:
$ source refresh-tokenRegister a new infrastructure environment. Provide a name, preferably something including the cluster name. This example provides the cluster ID to associate the infrastructure environment with the cluster resource. The following example specifies the
image_type. You can specify eitherfull-isoorminimal-iso. The default value isminimal-iso.Optional: You can register a new infrastructure environment by slurping the pull secret file in the request:
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt \ --arg cluster_id ${CLUSTER_ID} ' { "name": "testcluster-infra-env", "image_type":"full-iso", "cluster_id": $cluster_id, "cpu_architecture" : "<architecture_name>", "pull_secret": $pull_secret[0] | tojson } ')" | jq '.id'Replace
<architecture_name>with one of the following CPU architectures:x86_64,arm64,ppc64le,s390x, ormulti.Optional: You can register a new infrastructure environment by writing the configuration to a JSON file and then referencing it in the request:
$ cat << EOF > infra-envs.json { "name": "testcluster", "pull_secret": $PULL_SECRET, "proxy": { "http_proxy": "", "https_proxy": "", "no_proxy": "" }, "ssh_authorized_key": "$CLUSTER_SSHKEY", "image_type": "full-iso", "cluster_id": "${CLUSTER_ID}", "openshift_version": "4.11" } EOF$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/infra-envs" -d @./infra-envs.json -H "Content-Type: application/json" -H "Authorization: Bearer $API_TOKEN" | jq '.id'
Assign the returned
idto theINFRA_ENV_IDvariable and export it:$ export INFRA_ENV_ID=<id>NoteOnce you create an infrastructure environment and associate it to a cluster definition via the
cluster_id, you can see the cluster settings in the Assisted Installer web user interface. If you close your terminal session, you need to re-export theidin a new terminal session.
5.8. Modifying an infrastructure environment Copy linkLink copied to clipboard!
You can modify an infrastructure environment using the /v2/infra-envs/{infra_env_id} endpoint. Modifying an infrastructure environment is a common operation for adding settings such as networking, SSH keys, or ignition configuration overrides.
See the infra-env-update-params model in the API viewer for details on the fields you can set when modifying an infrastructure environment. When modifying the new infrastructure environment, the Assisted Installer will also re-generate the discovery ISO.
Prerequisites
- You have created a new infrastructure environment.
Procedure
Refresh the API token:
$ source refresh-tokenModify the infrastructure environment:
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' { "image_type":"minimal-iso", "pull_secret": $pull_secret[0] | tojson } ')" | jq
5.8.1. Adding kernel arguments Copy linkLink copied to clipboard!
Providing kernel arguments to the Red Hat Enterprise Linux CoreOS (RHCOS) kernel via the Assisted Installer means passing specific parameters or options to the kernel at boot time, particularly when you cannot customize the kernel parameters of the discovery ISO. Kernel parameters can control various aspects of the kernel’s behavior and the operating system’s configuration, affecting hardware interaction, system performance, and functionality. Kernel arguments are used to customize or inform the node’s RHCOS kernel about the hardware configuration, debugging preferences, system services, and other low-level settings.
The RHCOS installer kargs modify command supports the append, delete, and replace options.
You can modify an infrastructure environment using the /v2/infra-envs/{infra_env_id} endpoint. When modifying the new infrastructure environment, the Assisted Installer will also re-generate the discovery ISO.
Procedure
Refresh the API token:
$ source refresh-tokenModify the kernel arguments:
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' { "kernel_arguments": [{ "operation": "append", "value": "<karg>=<value>" }], "image_type":"minimal-iso", "pull_secret": $pull_secret[0] | tojson } ')" | jq-
Replace
<karg>with the kernel argument and<value>with the kernel argument value. For example:rd.net.timeout.carrier=60. - You can specify multiple kernel arguments by adding a JSON object for each kernel argument.
-
Replace
5.9. Applying a static network configuration Copy linkLink copied to clipboard!
You can optionally apply a static network configuration by using the Assisted Installer API.
A static IP configuration is not supported in the following scenarios:
- OpenShift Container Platform installations on Oracle Cloud Infrastructure.
- OpenShift Container Platform installations on iSCSI boot volumes.
Prerequisites
- You have created an infrastructure environment using the API or have created a cluster using the web console.
-
You have your infrastructure environment ID exported in your shell as
$INFRA_ENV_ID. -
You have credentials to use when accessing the API and have exported a token as
$API_TOKENin your shell.
Procedure
-
Create YAML files with a static network configuration available as
server-a.yamlandserver-b.yaml. For guidance, see NMState configuration examples. Optional: Get the MAC address of the host by running the following command:
$ ip addrYou will need this information for the MAC to interface name mapping in the next step.
Create a temporary file
/tmp/request-body.txtwith the API request and the MAC to interface mapping, as in the example below:$ jq -n --arg NMSTATE_YAML1 "$(cat server-a.yaml)" --arg NMSTATE_YAML2 "$(cat server-b.yaml)" \ '{ "static_network_config": [ { "network_yaml": $NMSTATE_YAML1, "mac_interface_map": [{"mac_address": "02:00:00:2c:23:a5", "logical_nic_name": "eth0"}, {"mac_address": "02:00:00:68:73:dc", "logical_nic_name": "eth1"}] }, { "network_yaml": $NMSTATE_YAML2, "mac_interface_map": [{"mac_address": "02:00:00:9f:85:eb", "logical_nic_name": "eth1"}, {"mac_address": "02:00:00:c8:be:9b", "logical_nic_name": "eth0"}] } ] }' >> /tmp/request-body.txtFor more details, see MAC to interface mapping.
Refresh the API token:
$ source refresh-tokenSend the request to the Assisted Service API endpoint:
$ curl -H "Content-Type: application/json" \ -X PATCH -d @/tmp/request-body.txt \ -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID
5.10. Adding hosts Copy linkLink copied to clipboard!
After configuring the cluster resource and infrastructure environment, download the discovery ISO image. You can choose from two images:
-
Full ISO image: Use the full ISO image when booting must be self-contained. The image includes everything needed to boot and start the Assisted Installer agent. The ISO image is about 1GB in size. This is the recommended method for the
s390xarchitecture when installing with RHEL KVM. Minimal ISO image: Use the minimal ISO image when the virtual media connection has limited bandwidth. This is the default setting. The image includes only what the agent requires to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.
This option is mandatory in the following scenarios:
- If you are installing OpenShift Container Platform on Oracle Cloud Infrastructure.
- If you are installing OpenShift Container Platform on iSCSI boot volumes.
Currently, ISO images are supported on IBM Z® (s390x) with KVM, iPXE with z/VM, and LPAR (both static and DPM). For details, see Booting hosts using iPXE.
You can boot hosts with the discovery image using three methods. For details, see Booting hosts with the discovery image.
Prerequisites
- You have created a cluster.
- You have created an infrastructure environment.
- You have completed the configuration.
If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.
NoteThe proxy username and password must be URL-encoded.
-
You have selected an image type or will use the default
minimal-iso.
Procedure
- Configure the discovery image if needed. For details, see Configuring the discovery image.
Refresh the API token:
$ source refresh-tokenGet the download URL:
$ curl -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-urlExample output:
{ "expires_at": "2024-02-07T20:20:23.000Z", "url": "https://api.openshift.com/api/assisted-images/bytoken/<TOKEN>/<OCP_VERSION>/<CPU_ARCHITECTURE>/<FULL_OR_MINIMAL_IMAGE>.iso" }Download the discovery image:
$ wget -O discovery.iso <url>Replace
<url>with the download URL from the previous step.- Boot the host(s) with the discovery image.
- Assign a role to host(s).
5.10.1. Selecting a role Copy linkLink copied to clipboard!
You can select a role for the host by using the /v2/infra-envs/{infra_env_id}/hosts/{host_id} endpoint. A host can have one of the following roles:
-
master- Assigns the control plane role to the host, allowing the host to manage and coordinate the cluster. -
arbiter- Assigns the arbiter role to the host, providing a cost-effective solution for components that require a quorum. -
worker- Assigns the compute role to the host, enabling the host to run application workloads. -
auto-assign- Automatically determines whether the host is amaster,worker, orarbiter. This is the default setting.
Use this procedure to assign a role to the host. If the host_role setting is omitted, the host defaults to auto-assign.
Prerequisites
- You have added hosts to the cluster.
Procedure
Refresh the API token:
$ source refresh-tokenGet the host IDs:
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'Example output:
[ "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5" ]Add the
host_rolesetting:$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "host_role":"worker" } ' | jq-
Replace
<host_id>with the ID of the host. -
For
host_role, enter"master", `"arbiter", or"worker". For details, see Supported host roles.
-
Replace
5.10.2. Defining storage disk configuration Copy linkLink copied to clipboard!
Each host retrieved during host discovery can have multiple storage disks. You can optionally change the default configurations for each disk.
- Starting from OpenShift Container Platform 4.14, you can configure nodes with Intel® Virtual RAID on CPU (VROC) to manage NVMe RAIDs. For details, see Additional resources.
- Starting from OpenShift Container Platform 4.15, you can install a cluster on a single or multipath iSCSI boot device using the Assisted Installer.
5.10.2.1. Viewing the storage disks Copy linkLink copied to clipboard!
You can view the hosts in your cluster, and the disks on each host. You can then perform actions on a specific disk.
Prerequisites
- Configure the cluster and discover the hosts.
Procedure
Refresh the API token:
$ source refresh-tokenGet the host IDs for the cluster:
$ curl -s "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'Example output:
"1022623e-7689-8b2d-7fbd-e6f4d5bb28e5"NoteThis is the ID of a single host. Multiple host IDs are separated by commas.
Get the disks for a specific host:
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -H "Authorization: Bearer ${API_TOKEN}" \ | jq '.inventory | fromjson | .disks'Replace
<host_id>with the ID of the relevant host.Example output:
[ { "by_id": "/dev/disk/by-id/wwn-0x6c81f660f98afb002d3adc1a1460a506", "by_path": "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:0:0", "drive_type": "HDD", "has_uuid": true, "hctl": "1:2:0:0", "id": "/dev/disk/by-id/wwn-0x6c81f660f98afb002d3adc1a1460a506", "installation_eligibility": { "eligible": true, "not_eligible_reasons": null }, "model": "PERC_H710P", "name": "sda", "path": "/dev/sda", "serial": "0006a560141adc3a2d00fb8af960f681", "size_bytes": 6595056500736, "vendor": "DELL", "wwn": "0x6c81f660f98afb002d3adc1a1460a506" } ]NoteThis is the output for one disk. It has the
disk_idandinstallation_eligibilityproperties for the disk.
5.10.2.2. Changing the installation disk Copy linkLink copied to clipboard!
The Assisted Installer randomly assigns an installation disk by default. If there are multiple storage disks for a host, you can select a different disk to be the installation disk. This automatically unassigns the previous disk.
You can select any disk whose installation_eligibility property is eligible: true to be the installation disk.
Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing over Fibre Channel on the installation disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with an /etc/multipath.conf configuration. For details, see Modifying the DM Multipath configuration file.
Prerequisites
- Configure the cluster and discover the hosts.
Procedure
- Get the host and storage disk IDs. For details, see Viewing the storage disks.
Optional: Identify the current installation disk:
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -H "Authorization: Bearer ${API_TOKEN}" \ | jq '.installation_disk_id'Replace
<host_id>with the ID of the relevant host.Assign a new installation disk:
NoteMultipath devices are automatically discovered and listed in the host’s inventory. To assign a multipath Fibre Channel disk as the installation disk, choose a disk with
"drive_type"set to"Multipath", rather than to"FC"which indicates a single path.$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -X PATCH \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${API_TOKEN}" \ { "disks_selected_config": [ { "id": "<disk_id>", "role": "install" } ] }-
Replace
<host_id>with the ID of the host. -
Replace
<disk_id>with the ID of the new installation disk.
-
Replace
5.10.2.3. Disabling disk formatting Copy linkLink copied to clipboard!
The Assisted Installer marks all bootable disks for formatting during the installation process by default, regardless of whether or not they have been defined as the installation disk. Formatting causes data loss.
You can choose to disable the formatting of a specific disk. Disable formatting with caution, as bootable disks can interfere with the installation process, specifically the boot order.
You cannot disable formatting for the installation disk.
Prerequisites
- Configure the cluster and discover the hosts.
Procedure
- Get the host and storage disk IDs. For details, see Viewing the storage disks.
Run the following command:
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -X PATCH \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${API_TOKEN}" \ { "disks_skip_formatting": [ { "disk_id": "<disk_id>", "skip_formatting": true } ] }-
Replace
<host_id>with the ID of the host. -
Replace
<disk_id>with the ID of the disk. If there is more than one disk, separate the IDs with a comma.
-
Replace
-
To re-enable formatting, change the
skip_formattingvalue back tofalse.
5.11. Modifying hosts Copy linkLink copied to clipboard!
After adding hosts, modify the hosts as needed. The most common modifications are to the host_name and the host_role parameters.
You can modify a host by using the /v2/infra-envs/{infra_env_id}/hosts/{host_id} endpoint. See the host-update-params model in the API viewer for details on the fields you can set when modifying a host.
A host might be one of the following roles:
-
master- Assigns the control plane role to the host, allowing the host to manage and coordinate the cluster. -
arbiter- Assigns the arbiter role to the host, providing a cost-effective solution for components that require a quorum. -
worker- Assigns the compute role to the host, enabling the host to run application workloads. -
auto-assign- Automatically determines whether the host is amaster,worker, or `arbiter' node.
Use the following procedure to set the host’s role. If the host_role setting is omitted, the host defaults to auto-assign.
Prerequisites
- You have added hosts to the cluster.
Procedure
Refresh the API token:
$ source refresh-tokenGet the host IDs:
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'Modify the host settings by using the example below:
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "host_role":"worker" "host_name" : "worker-1" } ' | jqReplace
<host_id>with the ID of the host.
5.12. Adding manifests and patches Copy linkLink copied to clipboard!
The Assisted Installer supports the following manifest types:
- Custom manifests - A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer. You can create a custom manifest or use one provided by a third party.
- Manifest patches - A manifest patch is used to adjust configurations, manage updates, or apply changes in a structured and automated way. Its purpose is to update a system manifest that is automatically created by the Assisted Installer during installation preparation.
You can get and add custom manifests and manifest patches by using the Assisted Installer API.
5.12.1. Getting manifests Copy linkLink copied to clipboard!
You can get a list of manifests stored in the installation configuration to aid you when adding, removing, replacing, or patching manifests. To get a manifest with the API, use the /v2/clusters/$CLUSTER_ID/manifests endpoint.
Prerequisites
-
You have registered a new cluster definition and exported the
cluster_idto the$CLUSTER_IDBASH variable. -
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes.
Procedure
Refresh the API token:
$ source refresh-tokenGet the list of manifests by running the following command:
$ curl "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests?include_system_generated=true" \ -H "Authorization: Bearer $API_TOKEN" \ | jqExample output:
[ { "file_name": "manifest.json", "folder": "manifests", "manifest_source": "user" } ]If there are no manifests, you receive an HTTP 404 "Object not found" error.
Example output:
{ "code": "404", "href": "", "id": 404, "kind": "Error", "reason": "Object Not Found" }Get a specific manifest by running the following command:
$ curl -X 'GET' \ 'http://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_I/manifests/files?folder=<folder_name>&file_name=<file_name>' \ -H "Authorization: Bearer $API_TOKEN" \ -H 'accept: application/octet-stream'If the file does not exist, you receive an HTTP 404 "Folder/file doesn’t exist in cluster" error.
Example output:
{ "code": "404", "href": "", "id": 404, "kind": "Error", "reason": "Cluster manifest <folder_name>/<file_name>.json doesn't exist in cluster $CLUSTER_ID" }Delete a specific manifest by running the following command:
curl -X 'DELETE' \ 'http://api.openshift.com/api/assisted-install/v2/clusters/c07b16ec-7eed-45d1-9ccd-88d2e86c4087/manifests?folder=manifests&file_name=manifest.json' \ -H "Authorization: Bearer $API_TOKEN" \ -H 'accept: application/json'
5.12.2. Adding custom manifests Copy linkLink copied to clipboard!
A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer user interface. You can create a custom manifest or use one provided by a third party. To create a custom manifest with the API, use the /v2/clusters/$CLUSTER_ID/manifests endpoint.
You can upload a base64-encoded custom manifest to either the openshift folder or the manifests folder with the Assisted Installer API. There is no limit to the number of custom manifests permitted.
You can only upload one base64-encoded JSON manifest at a time. However, each uploaded base64-encoded YAML file can contain multiple custom manifests. Uploading a multi-document YAML manifest is faster than adding the YAML files individually.
For a file containing a single custom manifest, accepted file extensions include .yaml, .yml, or .json.
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"metadata": {
"labels": {
"machineconfiguration.openshift.io/role": "primary"
},
"name": "10_primary_storage_config"
},
"spec": {
"config": {
"ignition": {
"version": "3.2.0"
},
"storage": {
"disks": [
{
"device": "</dev/xxyN>",
"partitions": [
{
"label": "recovery",
"startMiB": 32768,
"sizeMiB": 16384
}
]
}
],
"filesystems": [
{
"device": "/dev/disk/by-partlabel/recovery",
"label": "recovery",
"format": "xfs"
}
]
}
}
}
}
For a file containing multiple custom manifests, accepted file types include .yaml or .yml.
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: master
name: 99-openshift-machineconfig-master-kargs
spec:
kernelArguments:
- loglevel=7
---
apiVersion: machineconfiguration.openshift.io/v2
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 98-openshift-machineconfig-worker-kargs
spec:
kernelArguments:
- loglevel=5
- When you install OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) external platform, you must add the custom manifests provided by Oracle. For additional external partner integrations such as vSphere or Nutanix, this step is optional.
- For more information about custom manifests, see Preparing custom manifests and manifest patches.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. -
You have registered a new cluster definition and exported the
cluster_idto the$CLUSTER_IDBASH variable.
Procedure
- Create a custom manifest file.
- Save the custom manifest file using the appropriate extension for the file format.
Refresh the API token:
$ source refresh-tokenAdd the custom manifest to the cluster by executing the following command:
$ curl -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests" \ -H "Authorization: Bearer $API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "file_name":"manifest.json", "folder":"manifests", "content":"'"$(base64 -w 0 ~/manifest.json)"'" }' | jqReplace
manifest.jsonwith the name of your manifest file. The second instance ofmanifest.jsonis the path to the file. Ensure the path is correct.Example output:
{ "file_name": "manifest.json", "folder": "manifests" }NoteThe
base64 -w 0command base64-encodes the manifest as a string and omits carriage returns. Encoding with carriage returns will generate an exception.Verify that the Assisted Installer added the manifest:
$ curl -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests/files?folder=manifests&file_name=manifest.json" -H "Authorization: Bearer $API_TOKEN"Replace
manifest.jsonwith the name of your manifest file.
5.12.3. Adding manifest patches Copy linkLink copied to clipboard!
A manifest patch modifies a system manifest that is automatically created by the Assisted Installer before installation. You can use a manifest patch to change the folder name, file name, or contents of the generated manifest.
Patches must follow the syntax of a YAML patch and use the .yml.patch or yaml.patch extension. The extension can include an additional suffix, consisting of an underscore followed by alphanumeric characters, for example .yml.patch_ocp_manifests or .yaml.patch_ocp_manifests.
For details, see Preparing custom manifests and manifest patches.
Prerequisites
-
You have registered a new cluster definition and exported the
cluster_idto the$CLUSTER_IDBASH variable. -
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes.
Procedure
Refresh the API token by running the following command:
$ source refresh-tokenCreate a patch file, such as
~/mypatch.yml.patch, with this content:- op: replace path: /status/infrastructureTopology value: HighlyAvailableExample:
$ cat <<EOF > ~/mypatch.yml.patch - op: replace path: /status/infrastructureTopology value: HighlyAvailable EOFRun the following command to upload the patch file, setting
file_nameto the name of the file you created:$ curl -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests" \ -H "Authorization: Bearer $API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "file_name":"mypatch.yml.patch", "folder":"manifests", "content":"'"$(base64 -w 0 ~/mypatch.yml.patch)"'" }' | jq
5.13. Installing the cluster Copy linkLink copied to clipboard!
Once the cluster hosts past validation, you can install the cluster.
Prerequisites
- You have created a cluster and infrastructure environment.
- You have added hosts to the infrastructure environment.
- You have ensured that the cluster and hosts have passed validation. For details, see Preinstallation validations.
Procedure
Refresh the API token:
$ source refresh-tokenInstall the cluster:
$ curl -H "Authorization: Bearer $API_TOKEN" \ -X POST \ https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/actions/install | jq- Complete any postinstallation platform integration steps.
Chapter 6. Enabling disk encryption Copy linkLink copied to clipboard!
You can enable encryption of installation disks by using either the TPM v2 or Tang encryption modes.
In some situations, when you enable TPM disk encryption in the firmware for a bare-metal host and then boot it from an ISO that you generate with the Assisted Installer, the cluster deployment can get stuck. This can happen if there are left-over TPM encryption keys from an earlier installation on the host. For more information, see "Bug BZ#2011634" in Additional resources. If you experience this problem, contact Red Hat Support.
6.1. Enabling TPM v2 encryption Copy linkLink copied to clipboard!
You can enable Trusted Platform Module (TPM) v2 encryption from the Assisted Installer web console or API.
Prerequisites
-
Ensure that TPM v2 encryption is enabled in the BIOS firmware on each host. Most Dell systems require this. Check the manual for your computer. The Assisted Installer will also validate that TPM is enabled in the firmware. See the
disk-encryptionmodel in the Assisted Installer API for additional details. - Verify that a TPM v2 encryption chip is installed on each node and enabled in the firmware.
Procedure
- Optional: Using the web console, in the Cluster details step, enable the encryption of installation disks for any of the following nodes: control plane nodes, workers, or arbiter.
Optional: Using the API, include the following settings in the "Modifying hosts" procedure to enable TPM v2 encryption:
Set the
disk_encryption.enable_onsetting to one of the following:-
"none" -
"all" -
"masters" -
"arbiters" -
"workers" -
"masters,arbiters" -
"masters,workers" -
"arbiters,workers" -
"masters,arbiters,workers"
-
Set the
disk_encryption.modesetting totpmv2.Example
$ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "disk_encryption": { "enable_on": "none", "mode": "tpmv2" } } ' | jq
6.2. Enabling Tang encryption Copy linkLink copied to clipboard!
You can enable Tang encryption from the Assisted Installer web console or API.
Prerequisites
- You have access to a Red Hat Enterprise Linux (RHEL) 8 machine that you can use to generate a thumbprint of the Tang exchange key.
Procedure
- Set up a Tang server or access an existing one. See Network-bound disk encryption for instructions. You can set multiple Tang servers, but the Assisted Installer must be able to connect to all of them during installation.
On the Tang server, retrieve the thumbprint for the Tang server using
tang-show-keys:$ tang-show-keys <port>Optional: Replace
<port>with the port number. The default port number is80.Example thumbprint
1gYTN_LpU9ZMB35yn5IbADY5OQ0Optional: Retrieve the thumbprint for the Tang server using
jose.Ensure
joseis installed on the Tang server:$ sudo dnf install joseOn the Tang server, retrieve the thumbprint using
jose:$ sudo jose jwk thp -i /var/db/tang/<public_key>.jwkReplace
<public_key>with the public exchange key for the Tang server.Example thumbprint
1gYTN_LpU9ZMB35yn5IbADY5OQ0
- Optional: Using the web console, in the Cluster details step, enable the encryption of installation disks for any of the following nodes: control plane nodes, workers, or arbiter. You will be required to enter URLs and thumbprints for the Tang servers.
Optional: Using the API, include the following settings in the "Modifying hosts" procedure to enable Tang encryption:
Set the
disk_encryption.enable_onsetting to one of the following:-
"none" -
"all" -
"masters" -
"arbiters" -
"workers" -
"masters,arbiters" -
"masters,workers" -
"arbiters,workers" -
"masters,arbiters,workers"
-
-
Set the
disk_encryption.modesetting totang. Set
disk_encyrption.tang_serversto provide the URL and thumbprint details about one or more Tang servers. Within thetang_serversvalue, comment out the quotes within the object(s).Example
$ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "disk_encryption": { "enable_on": "all", "mode": "tang", "tang_servers": "[{\"url\":\"http://tang.example.com:7500\",\"thumbprint\":\"PLjNyRdGw03zlRoGjQYMahSZGu9\"},{\"url\":\"http://tang2.example.com:7500\",\"thumbprint\":\"XYjNyRdGw03zlRoGjQYMahSZGu3\"}]" } } ' | jq
Chapter 7. Configuring the discovery image Copy linkLink copied to clipboard!
The Assisted Installer uses a preconfigured discovery image to run an agent that performs hardware and network validations before installing OpenShift Container Platform. You can customize this discovery image by using a utility called Ignition. Changes to the discovery image are not saved and apply only during the current validation run.
7.1. Creating an Ignition configuration file Copy linkLink copied to clipboard!
Ignition is a low-level system configuration utility, which is part of the temporary initial root filesystem, the initramfs. When Ignition runs on the first boot, it finds configuration data in the Ignition configuration file and applies it to the host before switch_root is called to pivot to the host’s root filesystem.
Ignition uses a JSON configuration specification file to represent the set of changes that occur on the first boot.
Ignition versions newer than 3.2 are not supported, and will raise an error.
Procedure
Create an Ignition file and specify the configuration specification version:
$ vim ~/ignition.conf{ "ignition": { "version": "3.1.0" } }Add configuration data to the Ignition file. For example, add a password to the
coreuser.Generate a password hash:
$ openssl passwd -6Add the generated password hash to the
coreuser:{ "ignition": { "version": "3.1.0" }, "passwd": { "users": [ { "name": "core", "passwordHash": "$6$spam$M5LGSMGyVD.9XOboxcwrsnwNdF4irpJdAWy.1Ry55syyUiUssIzIAHaOrUHr2zg6ruD8YNBPW9kW0H8EnKXyc1" } ] } }
Save the Ignition file and export it to the
IGNITION_FILEvariable:$ export IGNITION_FILE=~/ignition.conf
7.2. Modifying the discovery image with Ignition Copy linkLink copied to clipboard!
After you create an Ignition configuration file, you can modify the discovery image by patching the infrastructure environment through the Assisted Installer API.
Prerequisites
- If you used the web console to create the cluster, you have set up the API authentication.
-
You have an infrastructure environment and you have exported the infrastructure environment
idto theINFRA_ENV_IDvariable. -
You have a valid Ignition file and have exported the file name as
$IGNITION_FILE.
Procedure
Create an
ignition_config_overrideJSON object and redirect it to a file:$ jq -n \ --arg IGNITION "$(jq -c . $IGNITION_FILE)" \ '{ignition_config_override: $IGNITION}' \ > discovery_ignition.jsonRefresh the API token:
$ source refresh-tokenPatch the infrastructure environment:
$ curl \ --header "Authorization: Bearer $API_TOKEN" \ --header "Content-Type: application/json" \ -XPATCH \ -d @discovery_ignition.json \ https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID | jqThe
ignition_config_overrideobject references the Ignition file.- Download the updated discovery image.
Chapter 8. Booting hosts with the discovery image Copy linkLink copied to clipboard!
The Assisted Installer uses an initial image to run an agent that performs hardware and network validations before attempting to install OpenShift Container Platform. You can boot hosts with the discovery image by using a USB drive, Redfish virtual media, or iPXE.
8.1. Creating an ISO image on a USB drive Copy linkLink copied to clipboard!
You can install the Assisted Installer agent by using a USB drive that contains the discovery ISO image. Starting the host with the USB drive prepares the host for the software installation.
Procedure
- On the administration host, insert a USB drive into a USB port.
Copy the ISO image to the USB drive, for example:
# dd if=<path_to_iso> of=<path_to_usb> status=progress-
Replace
<path_to_iso>with the relative path to the downloaded discovery ISO file, for example,discovery.iso. -
Replace
<path_to_usb>with the location of the connected USB drive, for example,/dev/sdb.
After you copy the ISO to the USB drive, you can use the USB drive to install the Assisted Installer agent on the cluster host.
-
Replace
8.2. Booting with a USB drive Copy linkLink copied to clipboard!
To register nodes with the Assisted Installer using a bootable USB drive, use the following procedure.
Procedure
- Insert the RHCOS discovery ISO USB drive into the target host.
- Configure the boot drive order in the server firmware settings to boot from the attached discovery ISO, and then reboot the server.
Wait for the host to boot up.
- For web console installations, on the administration host, return to the browser. Wait for the host to appear in the list of discovered hosts.
For API installations, refresh the token, check the enabled host count, and gather the host IDs:
$ source refresh-token$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.enabled_host_count'$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'The command returns output similar to the following example:
[ "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5" ]
8.3. Booting from an HTTP-hosted ISO image using the Redfish API Copy linkLink copied to clipboard!
You can provision hosts in your network by using ISOs that you install using the Redfish Baseboard Management Controller (BMC) API.
Prerequisites
- Download the installation Red Hat Enterprise Linux CoreOS (RHCOS) ISO.
Procedure
- Copy the ISO file to an HTTP server accessible in your network.
Boot the host from the hosted ISO file, for example:
Call the redfish API to set the hosted ISO as the
VirtualMediaboot media by running the following command:$ curl -k -u <bmc_username>:<bmc_password> \ -d '{"Image":"<hosted_iso_file>", "Inserted": true}' \ -H "Content-Type: application/json" \ -X POST <host_bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia-
Replace
<bmc_username>:<bmc_password>with the username and password for the target host BMC. -
Replace
<hosted_iso_file>with the URL for the hosted installation ISO, for example:https://example.com/rhcos-live-minimal.iso. The ISO must be accessible from the target host machine. -
Replace
<host_bmc_address>with the BMC IP address of the target host machine.
-
Replace
Set the host to boot from the
VirtualMediadevice by running the following command:$ curl -k -u <bmc_username>:<bmc_password> \ -X PATCH -H 'Content-Type: application/json' \ -d '{"Boot": {"BootSourceOverrideTarget": "Cd", "BootSourceOverrideMode": "UEFI", "BootSourceOverrideEnabled": "Once"}}' \ <host_bmc_address>/redfish/v1/Systems/System.Embedded.1Reboot the host:
$ curl -k -u <bmc_username>:<bmc_password> \ -d '{"ResetType": "ForceRestart"}' \ -H 'Content-type: application/json' \ -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.ResetOptional: If the host is powered off, you can boot it using the
{"ResetType": "On"}switch. Run the following command:$ curl -k -u <bmc_username>:<bmc_password> \ -d '{"ResetType": "On"}' -H 'Content-type: application/json' \ -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset
8.4. Booting hosts using iPXE Copy linkLink copied to clipboard!
The Assisted Installer provides an iPXE script including all of the artifacts needed to boot the discovery image for an infrastructure environment. Due to the limitations of the current HTTPS implementation of iPXE, the recommendation is to download and expose the needed artifacts in an HTTP server. Currently, even if iPXE supports HTTPS protocol, the supported algorithms are old and not recommended.
The full list of supported ciphers is in https://ipxe.org/crypto.
Prerequisites
- You have created an infrastructure environment by using the API or you have created a cluster by using the web console.
-
You have your infrastructure environment ID exported in your shell as
$INFRA_ENV_ID. -
You have credentials to use when accessing the API and have exported a token as
$API_TOKENin your shell.
If you configure iPXE by using the web console, the $INFRA_ENV_ID and $API_TOKEN variables are preset.
- You have an HTTP server to host the images.
IBM Power® only supports PXE, which has the following requirements:
-
GRUB2 installed at
/var/lib/tftpboot - DHCP and TFTP for PXE
Procedure
Download the iPXE script directly from the web console, or get the iPXE script from the Assisted Installer by running the following command:
$ curl \ --silent \ --header "Authorization: Bearer $API_TOKEN" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script > ipxe-scriptThe following example shows the iPXE script format:
#!ipxe initrd --name initrd http://api.openshift.com/api/assisted-images/images/<infra_env_id>/pxe-initrd?arch=x86_64&image_token=<token_string>&version=4.10 kernel http://api.openshift.com/api/assisted-images/boot-artifacts/kernel?arch=x86_64&version=4.10 initrd=initrd coreos.live.rootfs_url=http://api.openshift.com/api/assisted-images/boot-artifacts/rootfs?arch=x86_64&version=4.10 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8" bootDownload the required artifacts by extracting URLs from the
ipxe-script:Download the initial RAM disk by running the following command:
$ awk '/^initrd /{print $NF}' ipxe-script \ | xargs curl -o initrd.img -LDownload the Linux kernel by running the following command:
$ awk '/^kernel /{print $2}' ipxe-script | xargs curl -o kernel -LDownload the root filesystem by running the following command:
$ grep ^kernel ipxe_script | xargs -n1 | grep ^coreos.live.rootfs_url | cut -d = -f 2,3,4 | xargs curl -o rootfs.img -L
Change the URLs to the different artifacts in the
ipxe-scriptto match your local HTTP server. For example:#!ipxe set webserver http://192.168.0.1 initrd --name initrd $webserver/initrd.img kernel $webserver/kernel initrd=initrd coreos.live.rootfs_url=$webserver/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8" bootOptional: When installing with RHEL KVM on IBM Z® you must boot the host by specifying additional kernel arguments:
random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8NoteWhen you install with iPXE on RHEL KVM, the VMs on the VM host might not start on the first boot. You must start them manually.
Optional: When installing on IBM Power® you must download the
initramfs,kernel, androotimages as follows:-
Copy the
initrd.imgandkernel.imgimages to the/var/lib/tftpboot/rhcosPXE directory. -
Copy the
rootfs.imgto the/var/www/html/installHTTPD directory. Add the following entry to the
/var/lib/tftpboot/boot/grub2/grub.cfgdirectory:if [ ${net_default_mac} == fa:1d:67:35:13:20 ]; then default=0 fallback=1 timeout=1 menuentry "CoreOS (BIOS)" { echo "Loading kernel" linux "/rhcos/kernel.img" ip=dhcp rd.neednet=1 ignition.platform.id=metal ignition.firstboot coreos.live.rootfs_url=http://9.114.98.8:8000/install/rootfs.img echo "Loading initrd" initrd "/rhcos/initrd.img" } fi
-
Copy the
Chapter 9. Preinstallation validations Copy linkLink copied to clipboard!
Before starting an installation, the Assisted Installer runs a series of checks to verify that the cluster configuration meets all requirements. You can review validation results and resolve any issues before proceeding with installation.
9.1. Overview of preinstallation validations Copy linkLink copied to clipboard!
The Assisted Installer aims to make cluster installation as simple, efficient, and error-free as possible. It performs validation checks on the configuration and the gathered telemetry before starting an installation.
The Assisted Installer uses the information provided before installation, such as control plane topology, network configuration and hostnames. It will also use real time telemetry from the hosts you are attempting to install.
When a host boots the discovery ISO, an agent will start on the host. The agent will send information about the state of the host to the Assisted Installer.
The Assisted Installer uses all of this information to compute real-time preinstallation validations.
9.1.1. Host and cluster validations Copy linkLink copied to clipboard!
The Assisted Installer performs validations at both the host and cluster levels.
- Host
- Host validations ensure that the configuration of a given host is valid for installation.
- Cluster
- Cluster validations ensure that the configuration of the whole cluster is valid for installation.
9.1.2. Blocking and non-blocking validations Copy linkLink copied to clipboard!
All validations are either blocking or non-blocking to the installation.
- Blocking
- A blocking validation prevents progress of the installation, meaning that you need to resolve the issue and pass the blocking validation before you can proceed.
- Non-blocking
- A non-blocking validation is a warning and will tell you of things that might cause you a problem.
9.2. Host validations Copy linkLink copied to clipboard!
You can retrieve and review validation results for each discovered host.
9.2.1. Getting host validations by using the REST API Copy linkLink copied to clipboard!
Run the following procedure to retrieve host validation results by using the REST API.
If you use the web console, many of these validations will not show up by name. To get a list of validations consistent with the labels, use the following procedure.
Prerequisites
-
You have installed the
jqutility. - You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
- You have hosts booted with the discovery ISO
-
You have your Cluster ID exported in your shell as
CLUSTER_ID. -
You have credentials to use when accessing the API and have exported a token as
API_TOKENin your shell.
Procedure
Refresh the API token:
$ source refresh-tokenGet all validations for all hosts:
$ curl \ --silent \ --header "Authorization: Bearer $API_TOKEN" \ https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/hosts \ | jq -r .[].validations_info \ | jq 'map(.[])'Get non-passing validations for all hosts:
$ curl \ --silent \ --header "Authorization: Bearer $API_TOKEN" \ https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/hosts \ | jq -r .[].validations_info \ | jq 'map(.[]) | map(select(.status=="failure" or .status=="pending")) | select(length>0)'
9.2.2. Host validations in detail Copy linkLink copied to clipboard!
The Assisted Installer performs the following host validations.
| Parameter | Validation type | Description |
|---|---|---|
|
| non-blocking | Checks that the host has recently communicated with the Assisted Installer. |
|
| non-blocking | Checks that the Assisted Installer received the inventory from the host. |
|
| non-blocking | Checks that the number of CPU cores meets the minimum requirements. |
|
| non-blocking | Checks that the amount of memory meets the minimum requirements. |
|
| non-blocking | Checks that at least one available disk meets the eligibility criteria. |
|
| blocking | Checks that the number of cores meets the minimum requirements for the host role. |
|
| blocking | Checks that the amount of memory meets the minimum requirements for the host role. |
|
| blocking | For Day 2 hosts, checks that the host can download ignition configuration from the Day 1 cluster. |
|
| blocking | The majority group is the largest full-mesh connectivity group on the cluster, where all members can communicate with all other members. [role="_abstract"] This validation checks that hosts in a multi-node, Day 1 cluster are in the majority group. |
|
| blocking | Checks that the platform is valid for the network settings. |
|
| non-blocking | Checks if an NTP server has been successfully used to synchronize time on the host. |
|
| non-blocking | Checks if container images have been successfully pulled from the image registry. |
|
| blocking | Checks that disk speed metrics from an earlier installation meet requirements, if they exist. |
|
| blocking | Checks that the average network latency between hosts in the cluster meets the requirements. |
|
| blocking | Checks that the network packet loss between hosts in the cluster meets the requirements. |
|
| blocking | Checks that the host has a default route configured. |
|
| blocking | For a multi node cluster with user managed networking. Checks that the host is able to resolve the API domain name for the cluster. |
|
| blocking | For a multi node cluster with user managed networking. Checks that the host is able to resolve the internal API domain name for the cluster. |
|
| blocking | For a multi node cluster with user managed networking. Checks that the host is able to resolve the internal apps domain name for the cluster. |
|
| non-blocking | Checks that the host is compatible with the cluster platform |
|
| blocking | Checks that the wildcard DNS *.<cluster_name>.<base_domain> is not configured, because this causes known problems for OpenShift |
|
| non-blocking | Checks that the type of host and disk encryption configured meet the requirements. |
|
| blocking | Checks that this host does not have any overlapping subnets. |
|
| blocking | Checks that the hostname is unique in the cluster. |
|
| blocking | Checks the validity of the hostname, meaning that it matches the general form of hostnames and is not forbidden.
|
|
| blocking | Checks that the host IP is in the address range of the machine CIDR. |
|
| blocking | Validates that the host meets the requirements of the Local Storage Operator. |
|
| blocking | Validates that the host meets the requirements of the OpenShift Data Foundation Operator.
|
|
| blocking | Validates that the host meets the requirements of Container Native Virtualization.
|
|
| blocking | Validates that the host meets the requirements of the Logical Volume Manager Storage Operator.
|
|
| non-blocking |
Verifies that each valid disk sets |
|
| blocking | Checks that the discovery agent version is compatible with the agent docker image version. |
|
| blocking | Checks that installation disk is not skipping disk formatting. |
|
| blocking | Checks that all disks marked to skip formatting are in the inventory. A disk ID can change on reboot, and this validation prevents issues caused by that. |
|
| blocking | Checks the connection of the installation media to the host. |
|
| non-blocking | Checks that the machine network definition exists for the cluster. |
|
| blocking | Checks that the platform is compatible with the network settings. Some platforms are only permitted when installing Single Node Openshift or when using User Managed Networking. |
|
| non-blocking | Checks the maximum transmission unit (MTU) of hosts and networking devices in the cluster environment to identify compatibility issues. For details, see Changing the MTU for the cluster network. |
9.3. Cluster validations Copy linkLink copied to clipboard!
You can retrieve and review validation results for the cluster.
9.3.1. Getting cluster validations by using the REST API Copy linkLink copied to clipboard!
Run the following procedure to retrieve cluster validation results by using the REST API.
If you use the web console, many of these validations will not show up by name. To obtain a list of validations consistent with the labels, use the following procedure.
Prerequisites
-
You have installed the
jqutility. - You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
-
You have your Cluster ID exported in your shell as
CLUSTER_ID. -
You have credentials to use when accessing the API and have exported a token as
API_TOKENin your shell.
Procedure
Refresh the API token:
$ source refresh-tokenGet all cluster validations:
$ curl \ --silent \ --header "Authorization: Bearer $API_TOKEN" \ https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID \ | jq -r .validations_info \ | jq 'map(.[])'Get non-passing cluster validations:
$ curl \ --silent \ --header "Authorization: Bearer $API_TOKEN" \ https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID \ | jq -r .validations_info \ | jq '. | map(.[] | select(.status=="failure" or .status=="pending")) | select(length>0)'
9.3.2. Cluster validations in detail Copy linkLink copied to clipboard!
The Assisted Installer performs the following cluster validations.
| Parameter | Validation type | Description |
|---|---|---|
|
| non-blocking | Checks that the machine network definition exists for the cluster. |
|
| non-blocking | Checks that the cluster network definition exists for the cluster. |
|
| non-blocking | Checks that the service network definition exists for the cluster. |
|
| blocking | Checks that the defined networks do not overlap. |
|
| blocking | Checks that the defined networks share the same address families (valid address families are IPv4, IPv6) |
|
| blocking | Checks the cluster network prefix to ensure that it is valid and allows enough address space for all hosts. |
|
| blocking |
For a non user managed networking cluster. Checks that |
|
| non-blocking |
For a non user managed networking cluster. Checks that |
|
| blocking |
For a non user managed networking cluster. Checks if the |
|
| blocking |
For a non user managed networking cluster. Checks that |
|
| non-blocking |
For a non user managed networking cluster. Checks if the |
|
| blocking | Checks that all hosts in the cluster are in the "ready to install" status. |
|
| blocking [role="_abstract"] |
|
|
| non-blocking | Checks that the base DNS domain exists for the cluster. |
|
| non-blocking | Checks that the pull secret exists. Does not check that the pull secret is valid or authorized. |
|
| blocking | Checks that each of the host clocks are no more than 4 minutes out of sync with each other. |
|
| blocking | Validates that the cluster meets the requirements of the Local Storage Operator. |
|
| blocking | Validates that the cluster meets the requirements of the OpenShift Data Foundation Operator.
|
|
| blocking | Validates that the cluster meets the requirements of Container Native Virtualization.
|
|
| blocking | Validates that the cluster meets the requirements of the Logical Volume Manager Storage Operator.
|
|
| blocking | Checks the validity of the network type if it exists.
|
Chapter 10. Network configuration Copy linkLink copied to clipboard!
The following sections describe the basics of network configuration with the Assisted Installer.
10.1. Cluster networking requirements Copy linkLink copied to clipboard!
OpenShift Container Platform uses the network types and addresses listed in the following table.
| Type | DNS | Description |
|---|---|---|
|
| The IP address pools from which pod IP addresses are allocated. | |
|
| The IP address pool for services. | |
|
|
The IP address blocks for machines that form the cluster. For a dual-stack configuration, specify both a primary and a secondary | |
|
|
| The VIPs to use for API communication. You must provide this setting or preconfigure the address in the DNS so that the default name resolves correctly. For dual-stack networking, specify one IPv4 address and one IPv6 address, with the primary address listed first. |
|
|
| The VIPs to use for ingress traffic. For dual-stack networking, specify one IPv4 address and one IPv6 address, with the primary address listed first. |
Currently, the Assisted Service can deploy OpenShift Container Platform clusters by using one of the following configurations:
- Single-stack IPv4. IPv6 is not currently supported in a single-stack configuration.
- Dual-stack IPv4 and IPv6, with either address family as primary.
Open Virtual Network (OVN) is the default Container Network Interface (CNI) in OpenShift Container Platform 4.12 and later releases. Software-Defined Networking (SDN) is supported up to OpenShift Container Platform 4.14 only.
Support for IPv6 as the primary stack in a dual-stack configuration is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
10.1.1. Networking Limitations Copy linkLink copied to clipboard!
Cluster networking has the following limitations.
- Software Defined Networking (SDN)
- The SDN controller is not supported with single-node OpenShift.
- The SDN controller does not support dual-stack networking.
- The SDN controller is not supported for OpenShift Container Platform 4.15 and later releases. For more information, see Deprecation of the OpenShift SDN network plugin in the OpenShift Container Platform release notes.
- OVN-Kubernetes
- For more information, see About the OVN-Kubernetes network plugin.
10.1.2. Cluster network Copy linkLink copied to clipboard!
The cluster network is a network from which every pod deployed in the cluster gets its IP address. Given that the workload might live across many nodes forming the cluster, it is important for the network provider to be able to easily find an individual node based on the pod’s IP address. To do this, clusterNetwork.cidr is further split into subnets of the size defined in clusterNetwork.hostPrefix.
The host prefix specifies a length of the subnet assigned to each individual node in the cluster. An example of how a cluster might assign addresses for the multi-node cluster:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
Creating a 3-node cluster by using this snippet might create the following network topology:
-
Pods scheduled in node #1 get IPs from
10.128.0.0/23 -
Pods scheduled in node #2 get IPs from
10.128.2.0/23 -
Pods scheduled in node #3 get IPs from
10.128.4.0/23
Explaining OVN-Kubernetes internals is out of scope for this document. However, the pattern described earlier provides a way to route Pod-to-Pod traffic between different nodes without keeping a big list of mapping between Pods and their corresponding nodes.
10.1.3. Machine network Copy linkLink copied to clipboard!
Machine networks are IP networks that connect all the cluster nodes within OpenShift Container Platform.
The Assisted Installer supports a single machine network for most cluster installations. In such cases, the Assisted Installer automatically determines the appropriate machine network based on the API and Ingress virtual IPs (VIPs) that you specify.
The Assisted Installer supports two machine networks in the following scenarios:
- For dual stack configurations, the Assisted Installer automatically allocates two machine networks, based on the IPv4 and IPv6 subnets and the API and Ingress VIPs that you specify.
- For iSCSI boot volumes, the hosts are automatically connected over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. During the installation process, ensure that you select the OpenShift Container Platform network. Using the iSCSI network will result in an error for the host.
The Assisted Installer supports multiple machine networks for the "cluster-managed networking with a user-managed load balancer" network management type. When installing this network management type, you must manually define the machine networks in the API cluster definitions, with the following conditions:
- Each node must have at least one network interface in at least one machine network.
- The load balancer IPs (VIPs) should be included in at least one of the machine networks.
Currently, you can install cluster-managed networking with a user-managed load balancer using the Assisted Installer API only.
10.1.4. Network requirements for single-node versus multi-node clusters Copy linkLink copied to clipboard!
Depending on whether you are deploying single-node OpenShift or a multi-node OpenShift Container Platform cluster, different values are mandatory. The following table explains this in more detail.
| Parameter | Single-node OpenShift | Multi-node cluster with DHCP mode | Multi-node cluster without DHCP mode |
|---|---|---|---|
|
| Required | Required | Required |
|
| Required | Required | Required |
|
| Auto-assign possible (*) | Auto-assign possible (*) | Auto-assign possible (*) |
|
| Forbidden | Forbidden | Required |
|
| Forbidden | Forbidden | Required in 4.12 and later releases |
|
| Forbidden | Forbidden | Required |
|
| Forbidden | Forbidden | Required in 4.12 and later releases |
(*) Auto assignment of the machine network CIDR happens if there is only a single host network. Otherwise you need to specify it explicitly.
10.1.5. Air-gapped environments Copy linkLink copied to clipboard!
The workflow for deploying a cluster without Internet access has some prerequisites, which are out of scope of this document. You can consult Zero Touch Provisioning the hard way Git repository for some insights.
10.2. Network management types Copy linkLink copied to clipboard!
The Assisted Installer supports the following network management types.
For details on changing the network management type, see either of the following sections:
10.2.1. Cluster-managed networking Copy linkLink copied to clipboard!
Cluster-managed networking is the default option for deploying OpenShift Container Platform clusters. It minimizes user intervention by automatically provisioning and managing key network components.
The main characteristics of cluster-managed networking are the following:
- Integrates automated load balancing and virtual routing for managing the Virtual IP (VIP) addresses to ensure redundancy.
- Automatically supports an extensive internal DNS (CoreDNS) for service discovery.
- Hosts all control plane nodes within a single, contiguous subnet, simplifying routing and connectivity within the cluster.
- Supports the installation of platform-specific features such as the Bare Metal Operator for bare metal.
- Available for clusters with three or more control plane nodes; not available for single-node OpenShift.
You can configure cluster-managed networking both the web console or API. If you do not define a network management type, the Assisted Installer applies cluster-managed networking automatically for highly available clusters.
10.2.2. User-managed networking Copy linkLink copied to clipboard!
User-managed networking allows customers with custom or non-standard network topologies to deploy OpenShift Container Platform clusters. It provides control and flexibility, allowing you to integrate OpenShift Container Platform with existing and complex network infrastructures.
The main characteristics of user-managed networking are the following:
- Allows users to configure one or more external load balancers for handling API and Ingress IP addresses.
- Enables control plane nodes to span multiple subnets.
- Can be deployed on both single-node OpenShift and high-availability clusters.
You can configure user-managed networking in both the Assisted Installer web console or API.
10.2.3. Cluster-managed networking with a user-managed load balancer Copy linkLink copied to clipboard!
Cluster-managed networking with a user-managed load balancer is a hybrid network management type designed for scenarios that require automated cluster networking with external control over load balancing.
This approach combines elements from both cluster-managed and user-managed networking. The main characteristics of this network management type are as follows:
- Allows users to configure one or more external load balancers for handling API and Ingress IP addresses.
- Automatically supports an extensive internal DNS (CoreDNS) for service discovery.
- Enables control plane nodes to span multiple subnets.
- Supports the installation of platform specific features such as the Bare Metal Operator for bare metal.
- Provides high fault tolerance and disaster recovery for the control plane nodes.
The Assisted Installer supports cluster-managed networking with a user-managed load balancer on a bare-metal or vSphere platform. Currently you can configure this network management type through the API only.
Cluster-managed networking with a user-managed load balancer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
10.3. VIP DHCP allocation Copy linkLink copied to clipboard!
The VIP DHCP allocation is a feature allowing users to skip the requirement of manually providing virtual IPs for API and Ingress by leveraging the ability of a service to automatically assign those IP addresses from the DHCP server.
If you enable the VIP DHCP allocation feature, the service will not use the api_vips and ingress_vips defined in the cluster configuration. Instead, it will request IP addresses from the DHCP server on the machine network and use the assigned VIPs accordingly.
VIP DHCP allocation is currently limited to the OpenShift Container Platform SDN network type. SDN is not supported from OpenShift Container Platform version 4.15 and later. Therefore, support for VIP DHCP allocation is also ending from OpenShift Container Platform 4.15 and later.
10.3.1. Enabling VIP DHCP allocation Copy linkLink copied to clipboard!
You can enable automatic VIP allocation through DHCP.
This is not an OpenShift Container Platform feature and it has been implemented in the Assisted Service to make the configuration easier. For a more detailed explanation of the syntax for the VIP addresses, see Installer-provisioned infrastructure for a bare-metal installation.
Procedure
- Follow the instructions for registering a new cluster by using the API. For details, see Registering a new cluster.
Add the following payload settings to the cluster configuration:
-
Set
vip_dhcp_allocationtotrue. -
Set
network_typetoOpenShiftSDN. -
Include your network configurations for
cluster_networks,service_networks, andmachine_networks.
Example payload to enable auto-allocation:
$ cat << EOF > payload.json { "vip_dhcp_allocation": true, "network_type": "OpenShiftSDN", "user_managed_networking": false, "cluster_networks": [ { "cidr": "10.128.0.0/14", "host_prefix": 23 } ], "service_networks": [ { "cidr": "172.30.0.0/16" } ], "machine_networks": [ { "cidr": "192.168.127.0/24" } ] } EOF-
Set
Submit the payload to the Assisted Service API to apply the configuration, by running the following command:
$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \ -d @./payload.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'
10.3.2. Disabling VIP DHCP allocation Copy linkLink copied to clipboard!
If you want to manually control your VIP assignments, you can disable VIP DHCP allocation.
Procedure
- Follow the instructions for registering a new cluster by using the API. For details, see Registering a new cluster.
Add the following payload settings to the cluster configuration:
-
Set
vip_dhcp_allocationtofalse. -
Specify the IP addresses for
api_vipsandingress_vips. You can take these IPs from yourmachine_networksconfiguration. -
Set
network_typetoOVNKubernetes,OpenShiftSDN, or another supported SDN type, if applicable. -
Include your network configurations for
cluster_networksandservice_networks.
Example payload to disable autoallocation:
$ cat << EOF > payload.json { "api_vips": [ { "ip": "192.168.127.100" } ], "ingress_vips": [ { "ip": "192.168.127.101" } ], "vip_dhcp_allocation": false, "network_type": "OVNKubernetes", "user_managed_networking": false, "cluster_networks": [ { "cidr": "10.128.0.0/14", "host_prefix": 23 } ], "service_networks": [ { "cidr": "172.30.0.0/16" } ] } EOF-
Set
Submit the payload to the Assisted Service API to apply the configuration, by running the following command:
$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \ -d @./payload.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'
10.4. Static network configuration Copy linkLink copied to clipboard!
You can define static network configurations for each host through either the Assisted Installer web console or API. The Assisted Installer applies the settings to the Discovery ISO when you create a new ISO or update an existing one.
When using the API or the YAML view in the web console, create one or more NMState YAML files and map each host MAC address to its corresponding network interface name.
The Form view in the web console does not require these steps.
10.4.1. NMState configuration examples Copy linkLink copied to clipboard!
The NMState YAML file specifies the required static network configuration for the host, including interface details, IP addresses, routes, and DNS settings. The Assisted Installer replaces the logical interface names (for example, eth0) with the actual names during host discovery.
The following examples show NMState YAML configurations that you can copy and adapt. For more examples, see the NMState documentation.
For details on applying the static networking configurations in the Assisted Installer, see Configuring static networks (web console) or Applying a static network configuration (API).
- Standard NMState configuration example
This example shows a standard static network configuration with a default route and DNS server.
dns-resolver: config: server: - 192.168.126.1 interfaces: - ipv4: address: - ip: 192.168.126.30 prefix-length: 24 dhcp: false enabled: true name: eth0 state: up type: ethernet - ipv4: address: - ip: 192.168.141.30 prefix-length: 24 dhcp: false enabled: true name: eth1 state: up type: ethernet routes: config: - destination: 0.0.0.0/0 next-hop-address: 192.168.126.1 next-hop-interface: eth0 table-id: 254- Tagged VLAN example
Replace the relevant section of the standard YAML as follows to define a tagged VLAN interface on top of a physical network interface (NIC).
ImportantThis example and the next one show part of the YAML file only and are not meant for use as-is. Using them incorrectly can cause your machines to lose network connectivity.
interfaces: - ipv4: address: - ip: 192.168.143.15 prefix-length: 24 dhcp: false enabled: true ipv6: enabled: false name: eth0.404 state: up type: vlan vlan: base-iface: eth0 id: 404 reorder-headers: true- Network bond example
Replace the relevant section of the standard YAML as follows to configure a network bond for redundancy by using the
active-backupmode.interfaces: - ipv4: address: - ip: 192.168.138.15 prefix-length: 24 dhcp: false enabled: true ipv6: enabled: false link-aggregation: mode: active-backup options: miimon: "140" port: - eth0 - eth1 name: bond0 state: up type: bond
10.4.2. MAC-to-NIC mapping examples Copy linkLink copied to clipboard!
Each host requires a mapping between its MAC addresses and corresponding network interface cards (NICs). This mapping serves two main purposes:
- To identify the correct node on which to apply YAML file.
-
To replace logical/temporary interface names, such as
eth0orens3, in cases where the YAML file does not already use physical network interface names oridentifier: mac-address.
You define the MAC-to-NIC mapping configurations in the NMState YAML file when using the Assisted Installer API for the installation.
If you are using the YAML view of the web console for the installation, this mapping is not required. Instead, you specify the mapping manually in the MAC to interface name mapping fields. For details, see Configuring static networks using YAML view (web console).
- Example of MAC interface mapping with logical interface names
In this example, the mapping identifies the node and replaces the temporary interface name.
YAML file:
dns-resolver: config: server: - 192.168.126.1 interfaces: - ipv4: address: - ip: 192.168.126.30 prefix-length: 24 dhcp: false enabled: true name: eth0 state: up type: ethernet - ipv4: address: - ip: 192.168.141.30 prefix-length: 24 dhcp: false enabled: true name: eth1 state: up type: ethernet routes: config: - destination: 0.0.0.0/0 next-hop-address: 192.168.126.1 next-hop-interface: eth0 table-id: 254MAC mapping:
mac_interface_map: [ { mac_address: 02:00:00:2c:23:a5, logical_nic_name: eth0 }, { mac_address: 02:00:00:68:73:dc, logical_nic_name: eth1 } ]
- Example of MAC interface mapping with
identifier: mac-addressinterface names In this example, the NMState YAML configuration contains
identifier: mac-address. This means the mapping only needs to specify a single MAC address to identify one node.YAML file:
dns-resolver: config: server: - 192.168.126.1 interfaces: - name: eth0 type: ethernet state: up identifier: mac-address mac-address: 1e:bd:23:e9:fb:94 ipv4: enabled: true dhcp: true ipv6: enabled: true dhcp: true autoconf: true routes: config: - destination: 0.0.0.0/0 next-hop-address: 192.168.126.1 next-hop-interface: eth0 table-id: 254MAC mapping:
mac_interface_map: [ { mac_address: 1e:bd:23:e9:fb:95, logical_nic_name: test }, ]
10.5. Converting to dual-stack networking Copy linkLink copied to clipboard!
A dual-stack configuration enables clusters to host pods across both IPv4 and IPv6 subnets.
You configure dual-stack by specifying both IPv4 and IPv6 network address families in the configuration file. The order in which you list the IPv4 and IPv6 values determines the primary and secondary stack. The order must remain consistent across all networking parameters, including the machine network, cluster network, service network, API VIP, and Ingress VIP.
In the examples, listing the IPv4 network first makes it the primary stack, with IPv6 as the secondary stack. To set IPv6 as the primary stack, reverse the IPv4 and IPv6 values.
Support for IPv6 as the primary stack in a dual-stack configuration is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
Before starting, ensure that you are familiar with Converting to IPv4/IPv6 dual stack networking.
- Example payload for a single-node OpenShift cluster
{
"network_type": "OVNKubernetes",
"user_managed_networking": false,
"cluster_networks": [
{
"cidr": "10.128.0.0/14",
"host_prefix": 23
},
{
"cidr": "fd01::/48",
"host_prefix": 64
}
],
"service_networks": [
{"cidr": "172.30.0.0/16"}, {"cidr": "fd02::/112"}
],
"machine_networks": [
{"cidr": "192.168.127.0/24"},{"cidr": "1001:db8::/120"}
]
}
- Example payload for a multi-node OpenShift cluster
{
"vip_dhcp_allocation": false,
"network_type": "OVNKubernetes",
"user_managed_networking": false,
"api_vips": [
{
"ip": "192.168.127.100"
},
{
"ip": "2001:0db8:85a3:0000:0000:8a2e:0370:7334"
}
],
"ingress_vips": [
{
"ip": "192.168.127.101"
},
{
"ip": "2001:0db8:85a3:0000:0000:8a2e:0370:7335"
}
],
"cluster_networks": [
{
"cidr": "10.128.0.0/14",
"host_prefix": 23
},
{
"cidr": "fd01::/48",
"host_prefix": 64
}
],
"service_networks": [
{"cidr": "172.30.0.0/16"}, {"cidr": "fd02::/112"}
],
"machine_networks": [
{"cidr": "192.168.127.0/24"},{"cidr": "1001:db8::/120"}
]
}
Chapter 11. Expanding the cluster Copy linkLink copied to clipboard!
You can expand a cluster installed with the Assisted Installer by adding hosts through the user interface or the API.
11.1. Checking for multi-architecture support Copy linkLink copied to clipboard!
You must check that your cluster can support multiple architectures before you add a node with a different architecture.
Procedure
- Log in to the cluster using the CLI.
Check that your cluster uses the architecture payload by running the following command:
$ oc adm release info -o json | jq .metadata.metadata
Verification
If you see the following output, your cluster supports multiple architectures:
{ "release.openshift.io/architecture": "multi" }
11.2. Installing multi-architecture compute clusters Copy linkLink copied to clipboard!
A cluster with an x86_64 or arm64 control plane can support worker nodes that have two different CPU architectures. Multi-architecture clusters combine the strengths of each architecture and support a variety of workloads.
For example, you can add arm64, IBM Power® (ppc64le), or IBM Z® (s390x) worker nodes to an existing OpenShift Container Platform cluster with an x86_64.
The main steps of the installation are as follows:
- Create and register a multi-architecture compute cluster.
-
Create an
x86_64orarm64infrastructure environment, download the ISO discovery image for the environment, and add the control plane. Anarm64infrastructure environment is available for Amazon Web Services (AWS) and Google Cloud (GC) only. -
Create an
arm64,ppc64le, ors390xinfrastructure environment, download the ISO discovery images forarm64,ppc64le, ors390x, and add the worker nodes.
For the supported platforms for each OpenShift Container Platform version, see About clusters with multi-architecture compute machines. Use the appropriate platforms for the version you are installing.
Procedure
- Start the procedure for installing OpenShift Container Platform using the API. For details, see Installing with the Assisted Installer API.
When you reach the Registering a new cluster step of the installation, register the cluster as a multi-architecture compute cluster:
$ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' { "name": "testcluster", "openshift_version": "<version-number>-multi", "cpu_architecture" : "multi" "control_plane_count": "<number>" "base_dns_domain": "example.com", "pull_secret": $pull_secret[0] | tojson } ')" | jq '.id'-
Use the
multi-option foropenshift-version; for example,"4.20-multi". -
Set
cpu_architectureto"multi". -
Set
control_plane_countto"3","4", or"5". The option of 4 or 5 control plane nodes is available from OpenShift Container Platform 4.18 and later. Single-node OpenShift is not supported for a multi-architecture compute cluster. Thecontrol_plane_countfield replaceshigh_availability_mode, which is deprecated.
-
Use the
When you reach the Registering a new infrastructure environment step of the installation, set
cpu_architectureto"x86_64":$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt \ --arg cluster_id ${CLUSTER_ID} ' { "name": "testcluster-infra-env", "image_type":"full-iso", "cluster_id": $cluster_id, "cpu_architecture" : "x86_64" "pull_secret": $pull_secret[0] | tojson } ')" | jq '.id'When you reach the Adding hosts step of the installation, set
host_roleto"master":NoteFor details, see Supported host roles.
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "host_role":"master" } ' | jq-
Download the discovery image for the
x86_64architecture. -
Boot the
x86_64architecture hosts using the generated discovery image. - Start the installation and wait for the cluster to be fully installed.
Repeat the Registering a new infrastructure environment step of the installation. This time, set
cpu_architectureto one of the following:"ppc64le"(for IBM Power®),"s390x"(for IBM Z®), or"arm64". For example:$ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d "$(jq --null-input \ --slurpfile pull_secret ~/Downloads/pull-secret.txt ' { "name": "testcluster", "openshift_version": "4.12", "cpu_architecture" : "arm64" "control_plane_count": "3" "base_dns_domain": "example.com", "pull_secret": $pull_secret[0] | tojson } ')" | jq '.id'Repeat the Adding hosts step of the installation. This time, set
host_roleto"worker":NoteFor details, see Supported host roles.
$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "host_role":"worker" } ' | jq- Download the discovery image for the arm64, ppc64le or s390x architecture.
- Boot the architecture hosts using the generated discovery image.
- Start the installation and wait for the cluster to be fully installed.
To verify the installation, view the arm64, ppc64le, or s390x worker nodes in the cluster by running the following command:
$ oc get nodes -o wide
11.3. Adding hosts with the web console Copy linkLink copied to clipboard!
You can add hosts to clusters that were created using the Assisted Installer.
When you add new worker nodes to an existing cluster on Day 2 using the Assisted Installer, these nodes do not become part of an existing machine set. As a result, the Machine API will not automatically scale or replace them. To remove these nodes, you must do so manually, rather than through the machine set.
For details on removing nodes in OpenShift Container Platform, see Manually scaling a compute machine set for nodes managed as part of a machine set and Deleting a machine for un-managed nodes that are not part of a machine set.
Prerequisites
- Ensure that you are installing OpenShift Container Platform version 4.11 and later.
-
Ensure that the new node shares the same subnet as the Day 1 network. The subnet is specified in the
machineNetworkfield of theinstall-config.yamlfile. This requirement applies to cluster-managed networks such as bare metal or vSphere, and not to user-managed networks.
Procedure
- Log in to OpenShift Cluster Manager and click the cluster that you want to expand.
- Click Add hosts and download the discovery ISO for the new host, adding an SSH public key and configuring cluster-wide proxy settings as needed.
- Optional: Modify ignition files as needed.
- Boot the target host using the discovery ISO, and wait for the host to be discovered in the console.
-
Select the host role. It can be either a
workeror acontrol planehost. - Start the installation.
As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host. When prompted, approve the pending CSRs to complete the installation.
When the host is successfully installed, it is listed as a host in the cluster web console.
ImportantNew hosts will be encrypted using the same method as the original cluster.
11.4. Adding hosts with the API Copy linkLink copied to clipboard!
You can add hosts to clusters using the Assisted Installer REST API.
When you add new worker nodes to an existing cluster on Day 2 using the Assisted Installer, these nodes do not become part of an existing machine set. As a result, the Machine API will not automatically scale or replace them. To remove these nodes, you must do so manually, rather than through the machine set.
For details on removing nodes in OpenShift Container Platform, see Manually scaling a compute machine set for nodes managed as part of a machine set and Deleting a machine for un-managed nodes that are not part of a machine set.
Prerequisites
-
Install the Red Hat OpenShift Cluster Manager CLI (
ocm). - Log in to Red Hat OpenShift Cluster Manager as a user with cluster creation privileges.
-
Install
jq. - Ensure that all the required DNS records exist for the cluster that you want to expand.
- Ensure that you are installing OpenShift Container Platform version 4.11 and later.
-
Ensure that the new node shares the same subnet as the Day 1 network. The subnet is specified in the
machineNetworkfield of theinstall-config.yamlfile. This requirement applies to cluster-managed networks such as bare metal or vSphere, and not to user-managed networks.
Procedure
- Authenticate against the Assisted Installer REST API and generate an API token for your session. The generated token is valid for 15 minutes only.
Set the
$API_URLvariable by running the following command:$ export API_URL=<api_url>Replace
<api_url>with the Assisted Installer API URL, for example,https://api.openshift.com.Import the cluster by running the following commands:
Set the
$CLUSTER_IDvariable:Log in to the cluster and run the following command:
$ export CLUSTER_ID=$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}')Display the
$CLUSTER_IDvariable output:$ echo ${CLUSTER_ID}
Set the
$CLUSTER_REQUESTvariable that is used to import the cluster:$ export CLUSTER_REQUEST=$(jq --null-input --arg openshift_cluster_id "$CLUSTER_ID" \ '{ "api_vip_dnsname": "<api_vip>", "openshift_cluster_id": "<cluster_id>", "name": "<openshift_cluster_name>" }')-
Replace
<api_vip>with the hostname for the cluster’s API server. This can be the DNS domain for the API server or the IP address of the single node which the host can reach. For example,api.compute-1.example.com. -
Replace
<cluster_id>with the$CLUSTER_IDoutput from the previous substep. -
Replace
<openshift_cluster_name>with the plain text name for the cluster. The cluster name should match the cluster name that was set during the Day 1 cluster installation.
-
Replace
Import the cluster and set the
$CLUSTER_IDvariable. Run the following command:$ CLUSTER_ID=$(curl "$API_URL/api/assisted-install/v2/clusters/import" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' \ -d "$CLUSTER_REQUEST" | tee /dev/stderr | jq -r '.id')
Generate the
InfraEnvresource for the cluster and set the$INFRA_ENV_IDvariable by running the following commands:- Download the pull secret file from Red Hat OpenShift Cluster Manager at console.redhat.com.
Set the
$INFRA_ENV_REQUESTvariable:export INFRA_ENV_REQUEST=$(jq --null-input \ --slurpfile pull_secret <path_to_pull_secret_file> \// --arg ssh_pub_key "$(cat <path_to_ssh_pub_key>)" \// --arg cluster_id "$CLUSTER_ID" '{ "name": "<infraenv_name>", "pull_secret": $pull_secret[0] | tojson, "cluster_id": $cluster_id, "ssh_authorized_key": $ssh_pub_key, "image_type": "<iso_image_type>", "cpu_architecture": "<architecture_name>" }')-
Replace
<path_to_pull_secret_file>with the path to the local file containing the downloaded pull secret from Red Hat OpenShift Cluster Manager at console.redhat.com. -
Replace
<path_to_ssh_pub_key>with the path to the public SSH key required to access the host. If you do not set this value, you cannot access the host while in discovery mode. -
Replace
<infraenv_name>with the plain text name for theInfraEnvresource. -
Replace
<iso_image_type>with the ISO image type, eitherfull-isoorminimal-iso.
-
Replace
Post the
$INFRA_ENV_REQUESTto the /v2/infra-envs API and set the$INFRA_ENV_IDvariable:$ INFRA_ENV_ID=$(curl "$API_URL/api/assisted-install/v2/infra-envs" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' -d "$INFRA_ENV_REQUEST" | tee /dev/stderr | jq -r '.id')
Get the URL of the discovery iPXE or ISO for the cluster host by running the following command:
$ curl -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.download_url'Example output
https://api.openshift.com/api/assisted-images/images/41b91e72-c33e-42ee-b80f-b5c5bbf6431a?arch=x86_64&image_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTYwMjYzNzEsInN1YiI6IjQxYjkxZTcyLWMzM2UtNDJlZS1iODBmLWI1YzViYmY2NDMxYSJ9.1EX_VGaMNejMhrAvVRBS7PDPIQtbOOc8LtG8OukE1a4&type=minimal-iso&version=4.12Do one of the following depending on your installation:
Download the ISO:
$ curl -L -s '<iso_url>' --output rhcos-live-minimal.isoReplace
<iso_url>with the URL for the ISO from the previous step.Download the iPXE boot artifacts:
$ curl -H "Authorization: Bearer ${API_TOKEN}" -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script" >> discovery.txt
-
Boot the new worker host from the downloaded
rhcos-live-minimal.isoor the iPXE boot artifacts. Get the list of hosts in the cluster that are not installed. Keep running the following command until the new host shows up:
$ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.hosts[] | select(.status != "installed").id'Example output
2294ba03-c264-4f11-ac08-2f1bb2f8c296Set the
$HOST_IDvariable for the new host, for example:$ HOST_ID=<host_id>Replace
<host_id>with the host ID from the previous step.Check that the host is ready to install by running the following command:
NoteEnsure that you copy the entire command including the complete
jqexpression.$ curl -s $API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID -H "Authorization: Bearer ${API_TOKEN}" | jq ' def host_name($host): if (.suggested_hostname // "") == "" then if (.inventory // "") == "" then "Unknown hostname, please wait" else .inventory | fromjson | .hostname end else .suggested_hostname end; def is_notable($validation): ["failure", "pending", "error"] | any(. == $validation.status); def notable_validations($validations_info): [ $validations_info // "{}" | fromjson | to_entries[].value[] | select(is_notable(.)) ]; { "Hosts validations": { "Hosts": [ .hosts[] | select(.status != "installed") | { "id": .id, "name": host_name(.), "status": .status, "notable_validations": notable_validations(.validations_info) } ] }, "Cluster validations info": { "notable_validations": notable_validations(.validations_info) } } ' -rExample output
{ "Hosts validations": { "Hosts": [ { "id": "97ec378c-3568-460c-bc22-df54534ff08f", "name": "localhost.localdomain", "status": "insufficient", "notable_validations": [ { "id": "ntp-synced", "status": "failure", "message": "Host couldn't synchronize with any NTP server" }, { "id": "api-domain-name-resolved-correctly", "status": "error", "message": "Parse error for domain name resolutions result" }, { "id": "api-int-domain-name-resolved-correctly", "status": "error", "message": "Parse error for domain name resolutions result" }, { "id": "apps-domain-name-resolved-correctly", "status": "error", "message": "Parse error for domain name resolutions result" } ] } ] }, "Cluster validations info": { "notable_validations": [] } }When the previous command shows that the host is ready, start the installation using the /v2/infra-envs/{infra_env_id}/hosts/{host_id}/actions/install API by running the following command:
$ curl -X POST -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/actions/install" -H "Authorization: Bearer ${API_TOKEN}"As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host.
ImportantYou must approve the CSRs to complete the installation.
Keep running the following API call to monitor the cluster installation:
$ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq '{ "Cluster day-2 hosts": [ .hosts[] | select(.status != "installed") | {id, requested_hostname, status, status_info, progress, status_updated_at, updated_at, infra_env_id, cluster_id, created_at} ] }'Example output
{ "Cluster day-2 hosts": [ { "id": "a1c52dde-3432-4f59-b2ae-0a530c851480", "requested_hostname": "control-plane-1", "status": "added-to-existing-cluster", "status_info": "Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs", "progress": { "current_stage": "Done", "installation_percentage": 100, "stage_started_at": "2022-07-08T10:56:20.476Z", "stage_updated_at": "2022-07-08T10:56:20.476Z" }, "status_updated_at": "2022-07-08T10:56:20.476Z", "updated_at": "2022-07-08T10:57:15.306369Z", "infra_env_id": "b74ec0c3-d5b5-4717-a866-5b6854791bd3", "cluster_id": "8f721322-419d-4eed-aa5b-61b50ea586ae", "created_at": "2022-07-06T22:54:57.161614Z" } ] }Optional: Run the following command to see all the events for the cluster:
$ curl -s "$API_URL/api/assisted-install/v2/events?cluster_id=$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -c '.[] | {severity, message, event_time, host_id}'Example output
{"severity":"info","message":"Host compute-0: updated status from insufficient to known (Host is ready to be installed)","event_time":"2022-07-08T11:21:46.346Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"} {"severity":"info","message":"Host compute-0: updated status from known to installing (Installation is in progress)","event_time":"2022-07-08T11:28:28.647Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"} {"severity":"info","message":"Host compute-0: updated status from installing to installing-in-progress (Starting installation)","event_time":"2022-07-08T11:28:52.068Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"} {"severity":"info","message":"Uploaded logs for host compute-0 cluster 8f721322-419d-4eed-aa5b-61b50ea586ae","event_time":"2022-07-08T11:29:47.802Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"} {"severity":"info","message":"Host compute-0: updated status from installing-in-progress to added-to-existing-cluster (Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs)","event_time":"2022-07-08T11:29:48.259Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"} {"severity":"info","message":"Host: compute-0, reached installation stage Rebooting","event_time":"2022-07-08T11:29:48.261Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}- Log in to the cluster and approve the pending CSRs to complete the installation.
Verification
Check that the new host was successfully added to the cluster with a status of
Ready:$ oc get nodesExample output
NAME STATUS ROLES AGE VERSION control-plane-1.example.com Ready master,worker 56m v1.25.0 compute-1.example.com Ready worker 11m v1.25.0
11.5. Replacing a control plane node in a healthy cluster Copy linkLink copied to clipboard!
You can replace a control plane (master) node in a healthy OpenShift Container Platform cluster that has three to five control plane nodes, by adding a new control plane node and removing an existing control plane node.
If the cluster is unhealthy, you must peform additional operations before you can manage the control plane nodes. See "Replacing a control plane node in an unhealthy cluster" for more information.
11.5.1. Adding a new control plane node Copy linkLink copied to clipboard!
Add the new control plane node, and verify that it is healthy. In the example below, the new node is node-5.
Prerequisites
- You are using OpenShift Container Platform 4.11 or later.
- You have installed a healthy cluster with at least three control plane nodes.
- You have created a single control plane node to be added to the cluster for Day 2.
Procedure
Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:
$ oc get csr | grep PendingExample output
csr-5sd59 8m19s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending csr-xzqts 10s kubernetes.io/kubelet-serving system:node:node-5 <none> PendingApprove all pending CSRs for the new node (
node-5in this example):$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approveImportantYou must approve the CSRs to complete the installation.
Confirm that the new control plane node is in
Readystatus:$ oc get nodesExample output
NAME STATUS ROLES AGE VERSION node-0 Ready master 4h42m v1.24.0+3882f8f node-1 Ready master 4h27m v1.24.0+3882f8f node-2 Ready master 4h43m v1.24.0+3882f8f node-3 Ready worker 4h29m v1.24.0+3882f8f node-4 Ready worker 4h30m v1.24.0+3882f8f node-5 Ready master 105s v1.24.0+3882f8fNoteThe
etcdoperator requires aMachinecustom resource (CR) that references the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three or more control plane nodes.Create the
BareMetalHostandMachineCRs and link them to the new control plane’sNodeCR.Create the
BareMetalHostCR with a unique.metadata.namevalue:apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: node-5 namespace: openshift-machine-api spec: automatedCleaningMode: metadata bootMACAddress: 00:00:00:00:00:02 bootMode: UEFI customDeploy: method: install_coreos externallyProvisioned: true online: true userData: name: master-user-data-managed namespace: openshift-machine-apiApply the
BareMetalHostCR:$ oc apply -f <filename>Replace
<filename>with the name of theBareMetalHostCR.Create the
MachineCR using the unique.metadata.namevalue:apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: annotations: machine.openshift.io/instance-state: externally provisioned metal3.io/BareMetalHost: openshift-machine-api/node-5 finalizers: - machine.machine.openshift.io labels: machine.openshift.io/cluster-api-cluster: <cluster_name> machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master name: node-5 namespace: openshift-machine-api spec: metadata: {} providerSpec: value: apiVersion: baremetal.cluster.k8s.io/v1alpha1 customDeploy: method: install_coreos hostSelector: {} image: checksum: "" url: "" kind: BareMetalMachineProviderSpec metadata: creationTimestamp: null userData: name: master-user-data-managedReplace
<cluster_name>with the name of the specific cluster, for example,test-day2-1-6qv96.To get the cluster name, run the following command:
$ oc get infrastructure cluster -o=jsonpath='{.status.infrastructureName}{"\n"}'Apply the
MachineCR:$ oc apply -f <filename>Replace
<filename>with the name of theMachineCR.Link
BareMetalHost,Machine, andNodeby running thelink-machine-and-node.shscript:Copy the
link-machine-and-node.shscript below to a local machine:#!/bin/bash # Credit goes to # https://bugzilla.redhat.com/show_bug.cgi?id=1801238. # This script will link Machine object # and Node object. This is needed # in order to have IP address of # the Node present in the status of the Machine. set -e machine="$1" node="$2" if [ -z "$machine" ] || [ -z "$node" ]; then echo "Usage: $0 MACHINE NODE" exit 1 fi node_name=$(echo "${node}" | cut -f2 -d':') oc proxy & proxy_pid=$! function kill_proxy { kill $proxy_pid } trap kill_proxy EXIT SIGINT HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts" function print_nics() { local ips local eob declare -a ips readarray -t ips < <(echo "${1}" \ | jq '.[] | select(. | .type == "InternalIP") | .address' \ | sed 's/"//g') eob=',' for (( i=0; i<${#ips[@]}; i++ )); do if [ $((i+1)) -eq ${#ips[@]} ]; then eob="" fi cat <<- EOF { "ip": "${ips[$i]}", "mac": "00:00:00:00:00:00", "model": "unknown", "speedGbps": 10, "vlanId": 0, "pxe": true, "name": "eth1" }${eob} EOF done } function wait_for_json() { local name local url local curl_opts local timeout local start_time local curr_time local time_diff name="$1" url="$2" timeout="$3" shift 3 curl_opts="$@" echo -n "Waiting for $name to respond" start_time=$(date +%s) until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do echo -n "." curr_time=$(date +%s) time_diff=$((curr_time - start_time)) if [[ $time_diff -gt $timeout ]]; then printf '\nTimed out waiting for %s' "${name}" return 1 fi sleep 5 done echo " Success!" return 0 } wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json" addresses=$(oc get node -n openshift-machine-api "${node_name}" -o json | jq -c '.status.addresses') machine_data=$(oc get machines.machine.openshift.io -n openshift-machine-api -o json "${machine}") host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g') if [ -z "$host" ]; then echo "Machine $machine is not linked to a host yet." 1>&2 exit 1 fi # The address structure on the host doesn't match the node, so extract # the values we want into separate variables so we can build the patch # we need. hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g') set +e read -r -d '' host_patch << EOF { "status": { "hardware": { "hostname": "${hostname}", "nics": [ $(print_nics "${addresses}") ], "systemVendor": { "manufacturer": "Red Hat", "productName": "product name", "serialNumber": "" }, "firmware": { "bios": { "date": "04/01/2014", "vendor": "SeaBIOS", "version": "1.11.0-2.el7" } }, "ramMebibytes": 0, "storage": [], "cpu": { "arch": "x86_64", "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz", "clockMegahertz": 2199.998, "count": 4, "flags": [] } } } } EOF set -e echo "PATCHING HOST" echo "${host_patch}" | jq . curl -s \ -X PATCH \ "${HOST_PROXY_API_PATH}/${host}/status" \ -H "Content-type: application/merge-patch+json" \ -d "${host_patch}" oc get baremetalhost -n openshift-machine-api -o yaml "${host}"Make the script executable:
$ chmod +x link-machine-and-node.shRun the script:
$ bash link-machine-and-node.sh node-5 node-5NoteThe first
node-5instance represents the machine, and the second represents the node.
Confirm members of
etcdby executing into one of the pre-existing control plane nodes:Open a remote shell session to the control plane node:
$ oc rsh -n openshift-etcd etcd-node-0List
etcdmembers:# etcdctl member list -w tableExample output
+--------+---------+--------+--------------+--------------+---------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | LEARNER | +--------+---------+--------+--------------+--------------+---------+ |76ae1d00| started |node-0 |192.168.111.24|192.168.111.24| false | |2c18942f| started |node-1 |192.168.111.26|192.168.111.26| false | |61e2a860| started |node-2 |192.168.111.25|192.168.111.25| false | |ead5f280| started |node-5 |192.168.111.28|192.168.111.28| false | +--------+---------+--------+--------------+--------------+---------+
Monitor the
etcdoperator configuration process until completion:$ oc get clusteroperator etcdExample output (upon completion)
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE etcd 4.11.5 True False False 5h54mConfirm
etcdhealth by running the following commands:Open a remote shell session to the control plane node:
$ oc rsh -n openshift-etcd etcd-node-0Check endpoint health:
# etcdctl endpoint healthExample output
192.168.111.24 is healthy: committed proposal: took = 10.383651ms 192.168.111.26 is healthy: committed proposal: took = 11.297561ms 192.168.111.25 is healthy: committed proposal: took = 13.892416ms 192.168.111.28 is healthy: committed proposal: took = 11.870755ms
Verify that all nodes are ready:
$ oc get nodesExample output
NAME STATUS ROLES AGE VERSION node-0 Ready master 6h20m v1.24.0+3882f8f node-1 Ready master 6h20m v1.24.0+3882f8f node-2 Ready master 6h4m v1.24.0+3882f8f node-3 Ready worker 6h7m v1.24.0+3882f8f node-4 Ready worker 6h7m v1.24.0+3882f8f node-5 Ready master 99m v1.24.0+3882f8fVerify that the cluster Operators are all available:
$ oc get ClusterOperatorsExample output
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MSG authentication 4.11.5 True False False 5h57m baremetal 4.11.5 True False False 6h19m cloud-controller-manager 4.11.5 True False False 6h20m cloud-credential 4.11.5 True False False 6h23m cluster-autoscaler 4.11.5 True False False 6h18m config-operator 4.11.5 True False False 6h19m console 4.11.5 True False False 6h4m csi-snapshot-controller 4.11.5 True False False 6h19m dns 4.11.5 True False False 6h18m etcd 4.11.5 True False False 6h17m image-registry 4.11.5 True False False 6h7m ingress 4.11.5 True False False 6h6m insights 4.11.5 True False False 6h12m kube-apiserver 4.11.5 True False False 6h16m kube-controller-manager 4.11.5 True False False 6h16m kube-scheduler 4.11.5 True False False 6h16m kube-storage-version-migrator 4.11.5 True False False 6h19m machine-api 4.11.5 True False False 6h15m machine-approver 4.11.5 True False False 6h19m machine-config 4.11.5 True False False 6h18m marketplace 4.11.5 True False False 6h18m monitoring 4.11.5 True False False 6h4m network 4.11.5 True False False 6h20m node-tuning 4.11.5 True False False 6h18m openshift-apiserver 4.11.5 True False False 6h8m openshift-controller-manager 4.11.5 True False False 6h7m openshift-samples 4.11.5 True False False 6h12m operator-lifecycle-manager 4.11.5 True False False 6h18m operator-lifecycle-manager-catalog 4.11.5 True False False 6h19m operator-lifecycle-manager-pkgsvr 4.11.5 True False False 6h12m service-ca 4.11.5 True False False 6h19m storage 4.11.5 True False False 6h19mVerify that the cluster version is correct:
$ oc get ClusterVersionExample output
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.5 True False 5h57m Cluster version is 4.11.5
11.5.2. Removing the existing control plane node Copy linkLink copied to clipboard!
Remove the control plane node that you are replacing. This is node-0 in the example below.
Prerequisites
- You have added a new healthy control plane node.
Procedure
Delete the
BareMetalHostCR of the pre-existing control plane node:$ oc delete bmh -n openshift-machine-api node-0Confirm that the machine is unhealthy:
$ oc get machine -AExample output
NAMESPACE NAME PHASE AGE openshift-machine-api node-0 Failed 20h openshift-machine-api node-1 Running 20h openshift-machine-api node-2 Running 20h openshift-machine-api node-3 Running 19h openshift-machine-api node-4 Running 19h openshift-machine-api node-5 Running 14hDelete the
MachineCR:$ oc delete machine -n openshift-machine-api node-0 machine.machine.openshift.io "node-0" deletedConfirm removal of the
NodeCR:$ oc get nodesExample output
NAME STATUS ROLES AGE VERSION node-1 Ready master 20h v1.24.0+3882f8f node-2 Ready master 19h v1.24.0+3882f8f node-3 Ready worker 19h v1.24.0+3882f8f node-4 Ready worker 19h v1.24.0+3882f8f node-5 Ready master 15h v1.24.0+3882f8fCheck
etcd-operatorlogs to confirm status of theetcdcluster:$ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjfExample output
E0927 07:53:10.597523 1 base_controller.go:272] ClusterMemberRemovalController reconciliation failed: cannot remove member: 192.168.111.23 because it is reported as healthy but it doesn't have a machine nor a node resourceRemove the physical machine to allow the
etcdoperator to reconcile the cluster members:Open a remote shell session to the control plane node:
$ oc rsh -n openshift-etcd etcd-node-1Monitor the progress of
etcdoperator reconciliation by checking members and endpoint health:# etcdctl member list -w table; etcdctl endpoint healthExample output
+--------+---------+--------+--------------+--------------+---------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | LEARNER | +--------+---------+--------+--------------+--------------+---------+ |2c18942f| started | node-1 |192.168.111.26|192.168.111.26| false | |61e2a860| started | node-2 |192.168.111.25|192.168.111.25| false | |ead4f280| started | node-5 |192.168.111.28|192.168.111.28| false | +--------+---------+--------+--------------+--------------+---------+ 192.168.111.26 is healthy: committed proposal: took = 10.458132ms 192.168.111.25 is healthy: committed proposal: took = 11.047349ms 192.168.111.28 is healthy: committed proposal: took = 11.414402ms
11.6. Replacing a control plane node in an unhealthy cluster Copy linkLink copied to clipboard!
You can replace an unhealthy control plane (master) node in an OpenShift Container Platform cluster that has three to five control plane nodes, by removing the unhealthy control plane node and adding a new one.
For details on replacing a control plane node in a healthy cluster, see "Replacing a control plane node in a healthy cluster".
f the cluster is unhealthy, you must peform additional operations before you can manage the control plane nodes. See "Replacing a control plane node in an unhealthy cluster" for more information.
11.6.1. Removing an unhealthy control plane node Copy linkLink copied to clipboard!
Remove the unhealthy control plane node from the cluster. This is node-0 in the example below.
Prerequisites
- You have installed a cluster with at least three control plane nodes.
- At least one of the control plane nodes is not ready.
Procedure
Check the node status to confirm that a control plane node is not ready:
$ oc get nodesExample output
NAME STATUS ROLES AGE VERSION node-0 NotReady master 20h v1.24.0+3882f8f node-1 Ready master 20h v1.24.0+3882f8f node-2 Ready master 20h v1.24.0+3882f8f node-3 Ready worker 20h v1.24.0+3882f8f node-4 Ready worker 20h v1.24.0+3882f8fConfirm in the
etcd-operatorlogs that the cluster is unhealthy:$ oc logs -n openshift-etcd-operator deployment/etcd-operatorExample output
E0927 08:24:23.983733 1 base_controller.go:272] DefragController reconciliation failed: cluster is unhealthy: 2 of 3 members are available, node-0 is unhealthyConfirm the
etcdmembers by running the following commands:Open a remote shell session to the control plane node:
$ oc rsh -n openshift-etcd node-1List the
etcdctlmembers:# etcdctl member list -w tableExample output
+--------+---------+--------+--------------+--------------+---------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | LEARNER | +--------+---------+--------+--------------+--------------+---------+ |61e2a860| started | node-0 |192.168.111.25|192.168.111.25| false | |2c18942f| started | node-1 |192.168.111.26|192.168.111.26| false | |ead4f280| started | node-2 |192.168.111.28|192.168.111.28| false | +--------+---------+--------+--------------+--------------+---------+
Confirm that
etcdctlendpoint health reports an unhealthy member of the cluster:# etcdctl endpoint healthExample output
{"level":"warn","ts":"2022-09-27T08:25:35.953Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000680380/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 12.465641ms 192.168.111.26 is healthy: committed proposal: took = 12.297059ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy clusterRemove the unhealthy control plane by deleting the
Machinecustom resource (CR):$ oc delete machine -n openshift-machine-api node-0NoteThe
MachineandNodeCRs might not be deleted because they are protected by finalizers. If this occurs, you must delete theMachineCR manually by removing all finalizers.Verify in the
etcd-operatorlogs whether the unhealthy machine has been removed:$ oc logs -n openshift-etcd-operator deployment/etcd-operatorExample output
I0927 08:58:41.249222 1 machinedeletionhooks.go:135] skip removing the deletion hook from machine node-0 since its member is still present with any of: [{InternalIP } {InternalIP 192.168.111.25}]If you see that removal has been skipped, as in the above log example, manually remove the unhealthy
etcdctlmember:Open a remote shell session to the control plane node:
$ oc rsh -n openshift-etcd node-1List the
etcdctlmembers:# etcdctl member list -w tableExample output
+--------+---------+--------+--------------+--------------+---------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | LEARNER | +--------+---------+--------+--------------+--------------+---------+ |61e2a860| started | node-0 |192.168.111.25|192.168.111.25| false | |2c18942f| started | node-1 |192.168.111.26|192.168.111.26| false | |ead4f280| started | node-2 |192.168.111.28|192.168.111.28| false | +--------+---------+--------+--------------+--------------+---------+Confirm that
etcdctlendpoint health reports an unhealthy member of the cluster:# etcdctl endpoint healthExample output
{"level":"warn","ts":"2022-09-27T10:31:07.227Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6e00/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 13.038278ms 192.168.111.26 is healthy: committed proposal: took = 12.950355ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy clusterRemove the unhealthy
etcdctlmember from the cluster:# etcdctl member remove 61e2a86084aafa62Example output
Member 61e2a86084aafa62 removed from cluster 6881c977b97990d7Verify that the unhealthy
etcdctlmember was removed by running the following command:# etcdctl member list -w tableExample output
+----------+---------+--------+--------------+--------------+-------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |LEARNER| +----------+---------+--------+--------------+--------------+-------+ | 2c18942f | started | node-1 |192.168.111.26|192.168.111.26| false | | ead4f280 | started | node-2 |192.168.111.28|192.168.111.28| false | +----------+---------+--------+--------------+--------------+-------+
11.6.2. Adding a new control plane node Copy linkLink copied to clipboard!
Add a new control plane node to replace the unhealthy node that you removed. In the example below, the new node is node-5.
Prerequisites
- You have installed a control plane node for Day 2. For more information, see Adding hosts with the web console or Adding hosts with the API.
Procedure
Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:
$ oc get csr | grep PendingExample output
csr-5sd59 8m19s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending csr-xzqts 10s kubernetes.io/kubelet-serving system:node:node-5 <none> PendingApprove all pending CSRs for the new node (
node-5in this example):$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approveNoteYou must approve the CSRs to complete the installation.
Confirm that the control plane node is in
Readystatus:$ oc get nodesExample output
NAME STATUS ROLES AGE VERSION node-1 Ready master 20h v1.24.0+3882f8f node-2 Ready master 20h v1.24.0+3882f8f node-3 Ready worker 20h v1.24.0+3882f8f node-4 Ready worker 20h v1.24.0+3882f8f node-5 Ready master 2m52s v1.24.0+3882f8fThe
etcdoperator requires aMachineCR referencing the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three control plane nodes.Create the
BareMetalHostandMachineCRs and link them to the new control plane’sNodeCR.ImportantBoot-it-yourself will not create
BareMetalHostandMachineCRs, so you must create them. Failure to create theBareMetalHostandMachineCRs will generate errors in theetcdoperator.Create the
BareMetalHostCR with a unique.metadata.namevalue:apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: node-5 namespace: openshift-machine-api spec: automatedCleaningMode: metadata bootMACAddress: 00:00:00:00:00:02 bootMode: UEFI customDeploy: method: install_coreos externallyProvisioned: true online: true userData: name: master-user-data-managed namespace: openshift-machine-apiApply the
BareMetalHostCR:$ oc apply -f <filename>Replace
<filename>with the name of theBareMetalHostCR.Create the
MachineCR using the unique.metadata.namevalue:apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: annotations: machine.openshift.io/instance-state: externally provisioned metal3.io/BareMetalHost: openshift-machine-api/node-5 finalizers: - machine.machine.openshift.io labels: machine.openshift.io/cluster-api-cluster: test-day2-1-6qv96 machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master name: node-5 namespace: openshift-machine-api spec: metadata: {} providerSpec: value: apiVersion: baremetal.cluster.k8s.io/v1alpha1 customDeploy: method: install_coreos hostSelector: {} image: checksum: "" url: "" kind: BareMetalMachineProviderSpec metadata: creationTimestamp: null userData: name: master-user-data-managedApply the
MachineCR:$ oc apply -f <filename>Replace
<filename>with the name of theMachineCR.Link
BareMetalHost,Machine, andNodeby running thelink-machine-and-node.shscript:Copy the
link-machine-and-node.shscript below to a local machine:#!/bin/bash # Credit goes to # https://bugzilla.redhat.com/show_bug.cgi?id=1801238. # This script will link Machine object # and Node object. This is needed # in order to have IP address of # the Node present in the status of the Machine. set -e machine="$1" node="$2" if [ -z "$machine" ] || [ -z "$node" ]; then echo "Usage: $0 MACHINE NODE" exit 1 fi node_name=$(echo "${node}" | cut -f2 -d':') oc proxy & proxy_pid=$! function kill_proxy { kill $proxy_pid } trap kill_proxy EXIT SIGINT HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts" function print_nics() { local ips local eob declare -a ips readarray -t ips < <(echo "${1}" \ | jq '.[] | select(. | .type == "InternalIP") | .address' \ | sed 's/"//g') eob=',' for (( i=0; i<${#ips[@]}; i++ )); do if [ $((i+1)) -eq ${#ips[@]} ]; then eob="" fi cat <<- EOF { "ip": "${ips[$i]}", "mac": "00:00:00:00:00:00", "model": "unknown", "speedGbps": 10, "vlanId": 0, "pxe": true, "name": "eth1" }${eob} EOF done } function wait_for_json() { local name local url local curl_opts local timeout local start_time local curr_time local time_diff name="$1" url="$2" timeout="$3" shift 3 curl_opts="$@" echo -n "Waiting for $name to respond" start_time=$(date +%s) until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do echo -n "." curr_time=$(date +%s) time_diff=$((curr_time - start_time)) if [[ $time_diff -gt $timeout ]]; then printf '\nTimed out waiting for %s' "${name}" return 1 fi sleep 5 done echo " Success!" return 0 } wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json" addresses=$(oc get node -n openshift-machine-api "${node_name}" -o json | jq -c '.status.addresses') machine_data=$(oc get machines.machine.openshift.io -n openshift-machine-api -o json "${machine}") host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g') if [ -z "$host" ]; then echo "Machine $machine is not linked to a host yet." 1>&2 exit 1 fi # The address structure on the host doesn't match the node, so extract # the values we want into separate variables so we can build the patch # we need. hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g') set +e read -r -d '' host_patch << EOF { "status": { "hardware": { "hostname": "${hostname}", "nics": [ $(print_nics "${addresses}") ], "systemVendor": { "manufacturer": "Red Hat", "productName": "product name", "serialNumber": "" }, "firmware": { "bios": { "date": "04/01/2014", "vendor": "SeaBIOS", "version": "1.11.0-2.el7" } }, "ramMebibytes": 0, "storage": [], "cpu": { "arch": "x86_64", "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz", "clockMegahertz": 2199.998, "count": 4, "flags": [] } } } } EOF set -e echo "PATCHING HOST" echo "${host_patch}" | jq . curl -s \ -X PATCH \ "${HOST_PROXY_API_PATH}/${host}/status" \ -H "Content-type: application/merge-patch+json" \ -d "${host_patch}" oc get baremetalhost -n openshift-machine-api -o yaml "${host}"Make the script executable:
$ chmod +x link-machine-and-node.shRun the script:
$ bash link-machine-and-node.sh node-5 node-5NoteThe first
node-5instance represents the machine, and the second represents the node.
Confirm members of
etcdby running the following commands:Open a remote shell session to the control plane node:
$ oc rsh -n openshift-etcd node-1List the
etcdctlmembers:# etcdctl member list -w tableExample output
+---------+-------+--------+--------------+--------------+-------+ | ID | STATUS| NAME | PEER ADDRS | CLIENT ADDRS |LEARNER| +---------+-------+--------+--------------+--------------+-------+ | 2c18942f|started| node-1 |192.168.111.26|192.168.111.26| false | | ead4f280|started| node-2 |192.168.111.28|192.168.111.28| false | | 79153c5a|started| node-5 |192.168.111.29|192.168.111.29| false | +---------+-------+--------+--------------+--------------+-------+
Monitor the
etcdoperator configuration process until completion:$ oc get clusteroperator etcdExample output (upon completion)
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE etcd 4.11.5 True False False 22hConfirm
etcdctlhealth by running the following commands:Open a remote shell session to the control plane node:
$ oc rsh -n openshift-etcd node-1Check endpoint health:
# etcdctl endpoint healthExample output
192.168.111.26 is healthy: committed proposal: took = 9.105375ms 192.168.111.28 is healthy: committed proposal: took = 9.15205ms 192.168.111.29 is healthy: committed proposal: took = 10.277577ms
Confirm the health of the nodes:
$ oc get NodesExample output
NAME STATUS ROLES AGE VERSION node-1 Ready master 20h v1.24.0+3882f8f node-2 Ready master 20h v1.24.0+3882f8f node-3 Ready worker 20h v1.24.0+3882f8f node-4 Ready worker 20h v1.24.0+3882f8f node-5 Ready master 40m v1.24.0+3882f8fVerify that the cluster Operators are all available:
$ oc get ClusterOperatorsExample output
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.11.5 True False False 150m baremetal 4.11.5 True False False 22h cloud-controller-manager 4.11.5 True False False 22h cloud-credential 4.11.5 True False False 22h cluster-autoscaler 4.11.5 True False False 22h config-operator 4.11.5 True False False 22h console 4.11.5 True False False 145m csi-snapshot-controller 4.11.5 True False False 22h dns 4.11.5 True False False 22h etcd 4.11.5 True False False 22h image-registry 4.11.5 True False False 22h ingress 4.11.5 True False False 22h insights 4.11.5 True False False 22h kube-apiserver 4.11.5 True False False 22h kube-controller-manager 4.11.5 True False False 22h kube-scheduler 4.11.5 True False False 22h kube-storage-version-migrator 4.11.5 True False False 148m machine-api 4.11.5 True False False 22h machine-approver 4.11.5 True False False 22h machine-config 4.11.5 True False False 110m marketplace 4.11.5 True False False 22h monitoring 4.11.5 True False False 22h network 4.11.5 True False False 22h node-tuning 4.11.5 True False False 22h openshift-apiserver 4.11.5 True False False 163m openshift-controller-manager 4.11.5 True False False 22h openshift-samples 4.11.5 True False False 22h operator-lifecycle-manager 4.11.5 True False False 22h operator-lifecycle-manager-catalog 4.11.5 True False False 22h operator-lifecycle-manager-pkgsvr 4.11.5 True False False 22h service-ca 4.11.5 True False False 22h storage 4.11.5 True False False 22hVerify that the cluster version is correct:
$ oc get ClusterVersionExample output
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.5 True False 22h Cluster version is 4.11.5
Chapter 12. Installing on Nutanix Copy linkLink copied to clipboard!
If you install OpenShift Container Platform on Nutanix, the Assisted Installer can integrate the OpenShift Container Platform cluster with the Nutanix platform, which exposes the Machine API to Nutanix and enables autoscaling and the dynamic provisioning of storage containers with the Nutanix Container Storage Interface (CSI).
To deploy an OpenShift Container Platform cluster and maintain its daily operation, you need access to a Nutanix account with the necessary environment requirements. For details, see "Environment requirements" in Additional resources.
12.1. Adding hosts on Nutanix with the UI Copy linkLink copied to clipboard!
To add hosts on Nutanix with the user interface (UI), generate the minimal discovery image ISO from the Assisted Installer. This downloads a smaller image that will fetch the data needed to boot a host with networking and is the default setting. The majority of the content downloads upon boot. The ISO image is about 100MB in size.
After this is complete, you must create an image for the Nutanix platform and create the Nutanix virtual machines.
Prerequisites
- You have created a cluster profile in the Assisted Installer UI.
- You have a Nutanix cluster environment set up, and made a note of the cluster name and subnet name.
Procedure
- In the Cluster details page, select Nutanix from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
- In the Host discovery page, click the Add hosts button.
Optional: Add an SSH public key so that you can connect to the Nutanix VMs as the
coreuser. Having a login to the cluster hosts can provide you with debugging information during the installation.- If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
-
In the SSH public key field, click Browse to upload the
id_rsa.pubfile containing the SSH public key or drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
Select the required provisioning type.
NoteMinimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.
In Networking, select Cluster-managed networking. Nutanix does not support User-managed networking.
Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.
NoteThe proxy username and password must be URL-encoded.
- Optional: Configure the discovery image if you want to boot it with an ignition file. For details, see Configuring the discovery image.
- Click Generate Discovery ISO.
- Copy the Discovery ISO URL.
- In the Nutanix Prism UI, follow the directions to upload the discovery image from the Assisted Installer.
In the Nutanix Prism UI, create the control plane (master) VMs through Prism Central.
-
Enter the Name. For example,
control-planeormaster. - Enter the Number of VMs. This should be 3, 4, or 5 for the control plane.
- Ensure the remaining settings meet the minimum requirements for control plane hosts.
-
Enter the Name. For example,
In the Nutanix Prism UI, create the worker VMs through Prism Central.
-
Enter the Name. For example,
worker. - Enter the Number of VMs. You should create at least 2 worker nodes.
- Ensure the remaining settings meet the minimum requirements for worker hosts.
-
Enter the Name. For example,
-
Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a
Readystatus. - Continue with the installation procedure.
12.2. Adding hosts on Nutanix with the API Copy linkLink copied to clipboard!
To add hosts on Nutanix with the API, generate the discovery image ISO from the Assisted Installer. Use the minimal discovery image ISO. This is the default setting. The image includes only what is required to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.
Once this is complete, you must create an image for the Nutanix platform and create the Nutanix virtual machines.
Prerequisites
- You have set up the Assisted Installer API authentication.
- You have created an Assisted Installer cluster profile.
- You have created an Assisted Installer infrastructure environment.
-
You have your infrastructure environment ID exported in your shell as
$INFRA_ENV_ID. - You have completed the Assisted Installer cluster configuration.
- You have a Nutanix cluster environment set up, and made a note of the cluster name and subnet name.
Procedure
- Configure the discovery image if you want it to boot with an ignition file.
Create a Nutanix cluster configuration file to hold the environment variables:
$ touch ~/nutanix-cluster-env.sh$ chmod +x ~/nutanix-cluster-env.shIf you have to start a new terminal session, you can reload the environment variables easily. For example:
$ source ~/nutanix-cluster-env.shAssign the Nutanix cluster’s name to the
NTX_CLUSTER_NAMEenvironment variable in the configuration file:$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_CLUSTER_NAME=<cluster_name> EOFReplace
<cluster_name>with the name of the Nutanix cluster.Assign the Nutanix cluster’s subnet name to the
NTX_SUBNET_NAMEenvironment variable in the configuration file:$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_SUBNET_NAME=<subnet_name> EOFReplace
<subnet_name>with the name of the Nutanix cluster’s subnet.Refresh the API token:
$ source refresh-tokenGet the download URL:
$ curl -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-urlCreate the Nutanix image configuration file:
$ cat << EOF > create-image.json { "spec": { "name": "ocp_ai_discovery_image.iso", "description": "ocp_ai_discovery_image.iso", "resources": { "architecture": "X86_64", "image_type": "ISO_IMAGE", "source_uri": "<image_url>", "source_options": { "allow_insecure_connection": true } } }, "metadata": { "spec_version": 3, "kind": "image" } } EOFReplace
<image_url>with the image URL downloaded from the previous step.Create the Nutanix image:
$ curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/images \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d @./create-image.json | jq '.metadata.uuid'-
Replace
<user>with the Nutanix user name. -
Replace
'<password>'with the Nutanix password. -
Replace
<domain-or-ip>with the domain name or IP address of the Nutanix plaform. -
Replace
<port>with the port for the Nutanix server. The port defaults to9440.
-
Replace
Assign the returned UUID to the
NTX_IMAGE_UUIDenvironment variable in the configuration file:$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_IMAGE_UUID=<uuid> EOFGet the Nutanix cluster UUID:
$ curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/clusters/list' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kind": "cluster" }' | jq '.entities[] | select(.spec.name=="<nutanix_cluster_name>") | .metadata.uuid'-
Replace
<user>with the Nutanix user name. -
Replace
'<password>'with the Nutanix password. -
Replace
<domain-or-ip>with the domain name or IP address of the Nutanix platform. -
Replace
<port>with the port for the Nutanix server. The port defaults to9440. -
Replace
<nutanix_cluster_name>with the name of the Nutanix cluster.
-
Replace
Assign the returned Nutanix cluster UUID to the
NTX_CLUSTER_UUIDenvironment variable in the configuration file:$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_CLUSTER_UUID=<uuid> EOFReplace
<uuid>with the returned UUID of the Nutanix cluster.Get the Nutanix cluster’s subnet UUID:
$ curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/subnets/list' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kind": "subnet", "filter": "name==<subnet_name>" }' | jq '.entities[].metadata.uuid'-
Replace
<user>with the Nutanix user name. -
Replace
'<password>'with the Nutanix password. -
Replace
<domain-or-ip>with the domain name or IP address of the Nutanix platform. -
Replace
<port>with the port for the Nutanix server. The port defaults to9440. -
Replace
<subnet_name>with the name of the cluster’s subnet.
-
Replace
Assign the returned Nutanix subnet UUID to the
NTX_CLUSTER_UUIDenvironment variable in the configuration file:$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_SUBNET_UUID=<uuid> EOFReplace
<uuid>with the returned UUID of the cluster subnet.Ensure the Nutanix environment variables are set:
$ source ~/nutanix-cluster-env.shCreate a VM configuration file for each Nutanix host. Create three to five control plane (master) VMs and at least two worker VMs. For example:
$ touch create-master-0.json$ cat << EOF > create-master-0.json { "spec": { "name": "<host_name>", "resources": { "power_state": "ON", "num_vcpus_per_socket": 1, "num_sockets": 16, "memory_size_mib": 32768, "disk_list": [ { "disk_size_mib": 122880, "device_properties": { "device_type": "DISK" } }, { "device_properties": { "device_type": "CDROM" }, "data_source_reference": { "kind": "image", "uuid": "$NTX_IMAGE_UUID" } } ], "nic_list": [ { "nic_type": "NORMAL_NIC", "is_connected": true, "ip_endpoint_list": [ { "ip_type": "DHCP" } ], "subnet_reference": { "kind": "subnet", "name": "$NTX_SUBNET_NAME", "uuid": "$NTX_SUBNET_UUID" } } ], "guest_tools": { "nutanix_guest_tools": { "state": "ENABLED", "iso_mount_state": "MOUNTED" } } }, "cluster_reference": { "kind": "cluster", "name": "$NTX_CLUSTER_NAME", "uuid": "$NTX_CLUSTER_UUID" } }, "api_version": "3.1.0", "metadata": { "kind": "vm" } } EOFReplace
<host_name>with the name of the host.Boot each Nutanix virtual machine:
$ curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/vms' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d @./<vm_config_file_name> | jq '.metadata.uuid'-
Replace
<user>with the Nutanix user name. -
Replace
'<password>'with the Nutanix password. -
Replace
<domain-or-ip>with the domain name or IP address of the Nutanix platform. -
Replace
<port>with the port for the Nutanix server. The port defaults to9440. -
Replace
<vm_config_file_name>with the name of the VM configuration file.
-
Replace
Assign the returned VM UUID to a unique environment variable in the configuration file:
$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_MASTER_0_UUID=<uuid> EOFReplace
<uuid>with the returned UUID of the VM.NoteThe environment variable must have a unique name for each VM.
Wait until the Assisted Installer has discovered each VM and they have passed validation.
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" --header "Content-Type: application/json" -H "Authorization: Bearer $API_TOKEN" | jq '.enabled_host_count'Modify the cluster definition to enable integration with Nutanix:
$ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \ -X PATCH \ -H "Authorization: Bearer ${API_TOKEN}" \ -H "Content-Type: application/json" \ -d ' { "platform_type":"nutanix" } ' | jq- Continue with the installation procedure.
12.3. Nutanix postinstallation configuration Copy linkLink copied to clipboard!
Complete and validate the OpenShift Container Platform integration with the Nutanix cloud provider.
By default, the installation program downloads and installs the Red Hat Enterprise Linux CoreOS (RHCOS) image. If Prism Central does not have internet access, you can host the RHCOS image on any HTTP server and point the installation program to the image or you can use Prism Central to upload the image manually.
12.3.1. Updating the Nutanix configuration settings Copy linkLink copied to clipboard!
After installing OpenShift Container Platform on the Nutanix platform by using the Assisted Installer, update the following Nutanix configuration settings manually.
Prerequisites
- You have your Nutanix Prism Element username.
- You have your Nutanix Prism Element password.
- You have your Nutanix Prism storage container.
- The Assisted Installer has finished installing the cluster successfully.
- You have connected the cluster to the Red Hat console.redhat.com.
- You have access to the Red Hat OpenShift Container Platform command line interface.
Procedure
In the OpenShift Container Platform command line interface, update the Nutanix cluster configuration settings:
$ oc patch infrastructure/cluster --type=merge --patch-file=/dev/stdin <<-EOF { "spec": { "platformSpec": { "nutanix": { "prismCentral": { "address": "<prismcentral_address>", "port": <prismcentral_port> }, "prismElements": [ { "endpoint": { "address": "<prismelement_address>", "port": <prismelement_port> }, "name": "<prismelement_clustername>" } ] }, "type": "Nutanix" } } } EOF-
Replace
<prismcentral_address>with the Nutanix Prism Central address. -
Replace
<prismcentral_port>with the Nutanix Prism Central port. -
Replace
<prismelement_address>with the Nutanix Prism Element address. -
Replace
<prismelement_port>with the Nutanix Prism Element port. -
Replace
<prismelement_clustername>with the Nutanix Prism Element cluster name.
Example output:
infrastructure.config.openshift.io/cluster patchedFor additional details, see Creating a compute machine set on Nutanix.
NoteOptional: You can define prism category key and value pairs. These category key-value pairs must exist in Prism Central. You can define the key-value pairs in separate categories for compute nodes, control plane nodes, or all nodes.
-
Replace
Create the Nutanix secret:
$ cat <<EOF | oc create -f - apiVersion: v1 kind: Secret metadata: name: nutanix-credentials namespace: openshift-machine-api type: Opaque stringData: credentials: | [{"type":"basic_auth","data":{"prismCentral":{"username":"${<prismcentral_username>}","password":"${<prismcentral_password>}"},"prismElements":null}}] EOF-
Replace
<prismcentral_username>with the Nutanix Prism Central username. -
Replace
<prismcentral_password>with the Nutanix Prism Central password.
Example output:
secret/nutanix-credentials created-
Replace
When installing OpenShift Container Platform version 4.13 or later, update the Nutanix cloud provider configuration:
Get the Nutanix cloud provider configuration YAML file:
$ oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config-backup.yamlCreate a backup of the configuration file:
$ cp cloud-provider-config_backup.yaml cloud-provider-config.yamlOpen the configuration YAML file:
$ vi cloud-provider-config.yamlEdit the configuration YAML file as follows:
kind: ConfigMap apiVersion: v1 metadata: name: cloud-provider-config namespace: openshift-config data: config: | { "prismCentral": { "address": "<prismcentral_address>", "port":<prismcentral_port>, "credentialRef": { "kind": "Secret", "name": "nutanix-credentials", "namespace": "openshift-cloud-controller-manager" } }, "topologyDiscovery": { "type": "Prism", "topologyCategories": null }, "enableCustomLabeling": true }Apply the configuration updates:
$ oc apply -f cloud-provider-config.yamlExample output:
Warning: resource configmaps/cloud-provider-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. configmap/cloud-provider-config configured
12.3.2. Creating the Nutanix CSI Operator group Copy linkLink copied to clipboard!
Create an Operator group for the Nutanix CSI Operator.
For a description of operator groups and related concepts, see Understanding Operators.
Prerequisites
- You have updated the Nutanix configuration settings.
Procedure
Open the Nutanix CSI Operator Group YAML file:
$ vi openshift-cluster-csi-drivers-operator-group.yamlEdit the YAML file as follows:
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: generateName: openshift-cluster-csi-drivers namespace: openshift-cluster-csi-drivers spec: targetNamespaces: - openshift-cluster-csi-drivers upgradeStrategy: DefaultCreate the Operator Group:
$ oc create -f openshift-cluster-csi-drivers-operator-group.yamlExample output:
operatorgroup.operators.coreos.com/openshift-cluster-csi-driversjw9cd created
12.3.3. Installing the Nutanix CSI Operator Copy linkLink copied to clipboard!
The Nutanix Container Storage Interface (CSI) Operator for Kubernetes deploys and manages the Nutanix CSI Driver.
For instructions on performing this step through the OpenShift Container Platform web console, see the Installing the Operator section of the Nutanix CSI Operator documentation.
Prerequisites
- You have created the Nutanix CSI Operator group.
Procedure
Get the parameter values for the Nutanix CSI Operator YAML file:
Check that the Nutanix CSI Operator exists:
$ oc get packagemanifests | grep nutanixExample output:
nutanixcsioperator Certified Operators 129mAssign the default channel for the Operator to a BASH variable:
$ DEFAULT_CHANNEL=$(oc get packagemanifests nutanixcsioperator -o jsonpath={.status.defaultChannel})Assign the starting cluster service version (CSV) for the Operator to a BASH variable:
$ STARTING_CSV=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.channels[*].currentCSV\})Assign the catalog source for the subscription to a BASH variable:
$ CATALOG_SOURCE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSource\})Assign the Nutanix CSI Operator source namespace to a BASH variable:
$ SOURCE_NAMESPACE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSourceNamespace\})
Create the Nutanix CSI Operator YAML file using the BASH variables:
$ cat << EOF > nutanixcsioperator.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: nutanixcsioperator namespace: openshift-cluster-csi-drivers spec: channel: $DEFAULT_CHANNEL installPlanApproval: Automatic name: nutanixcsioperator source: $CATALOG_SOURCE sourceNamespace: $SOURCE_NAMESPACE startingCSV: $STARTING_CSV EOFCreate the CSI Nutanix Operator:
$ oc apply -f nutanixcsioperator.yamlExample output:
subscription.operators.coreos.com/nutanixcsioperator createdRun the following command until the Operator subscription state changes to
AtLatestKnown. This indicates that the Operator subscription has been created, and might take some time.$ oc get subscription nutanixcsioperator -n openshift-cluster-csi-drivers -o 'jsonpath={..status.state}'
12.3.4. Deploying the Nutanix CSI storage driver Copy linkLink copied to clipboard!
The Nutanix Container Storage Interface (CSI) Driver for Kubernetes provides scalable and persistent storage for stateful applications.
For instructions on performing this step through the OpenShift Container Platform web console, see the Installing the CSI Driver using the Operator section of the Nutanix CSI Operator documentation.
Prerequisites
- You have installed the Nutanix CSI Operator.
Procedure
Create a
NutanixCsiStorageresource to deploy the driver:$ cat <<EOF | oc create -f - apiVersion: crd.nutanix.com/v1alpha1 kind: NutanixCsiStorage metadata: name: nutanixcsistorage namespace: openshift-cluster-csi-drivers spec: {} EOFExample output:
snutanixcsistorage.crd.nutanix.com/nutanixcsistorage createdCreate a Nutanix secret YAML file for the CSI storage driver:
$ cat <<EOF | oc create -f - apiVersion: v1 kind: Secret metadata: name: ntnx-secret namespace: openshift-cluster-csi-drivers stringData: # prism-element-ip:prism-port:admin:password key: <prismelement_address:prismelement_port:prismcentral_username:prismcentral_password> EOFReplace the
keyparameters with actual values while keeping the same format.Example output:
secret/nutanix-secret created
12.3.5. Validating the postinstallation configurations Copy linkLink copied to clipboard!
Verify that you can create a storage class and a bound persistent volume claim.
Prerequisites
- You have deployed the Nutanix CSI storage driver.
Procedure
Verify that you can create a storage class:
$ cat <<EOF | oc create -f - kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: nutanix-volume annotations: storageclass.kubernetes.io/is-default-class: 'true' provisioner: csi.nutanix.com parameters: csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/provisioner-secret-namespace: openshift-cluster-csi-drivers csi.storage.k8s.io/provisioner-secret-name: ntnx-secret storageContainer: <nutanix_storage_container> csi.storage.k8s.io/controller-expand-secret-name: ntnx-secret csi.storage.k8s.io/node-publish-secret-namespace: openshift-cluster-csi-drivers storageType: NutanixVolumes csi.storage.k8s.io/node-publish-secret-name: ntnx-secret csi.storage.k8s.io/controller-expand-secret-namespace: openshift-cluster-csi-drivers reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: Immediate EOFNoteTake
<nutanix_storage_container>from the Nutanix configuration; for example,SelfServiceContainer.Example output:
storageclass.storage.k8s.io/nutanix-volume createdVerify that you can create the Nutanix persistent volume claim (PVC):
Create the persistent volume claim (PVC):
$ cat <<EOF | oc create -f - kind: PersistentVolumeClaim apiVersion: v1 metadata: name: nutanix-volume-pvc namespace: openshift-cluster-csi-drivers annotations: volume.beta.kubernetes.io/storage-provisioner: csi.nutanix.com volume.kubernetes.io/storage-provisioner: csi.nutanix.com finalizers: - kubernetes.io/pvc-protection spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: nutanix-volume volumeMode: Filesystem EOFExample output:
persistentvolumeclaim/nutanix-volume-pvc createdValidate that the persistent volume claim (PVC) status is Bound:
$ oc get pvc -n openshift-cluster-csi-driversExample output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nutanix-volume-pvc Bound nutanix-volume 52s
Chapter 13. Installing on vSphere Copy linkLink copied to clipboard!
The Assisted Installer integrates the OpenShift Container Platform cluster with the vSphere platform, which exposes the Machine API to vSphere and enables auto-scaling.
13.1. Support ending for selected Broadcom products Copy linkLink copied to clipboard!
General support for the following Broadcom products ends on October 2nd, 2025:
- VMware vSphere 7.x
- VMware vCenter 7.x
- VMware Cloud Foundation (VCF) 4.x
This change follows Broadcom’s earlier End of General Support (EoGS) announcement, which extended support for these products up to this date.
For details, see the following references:
13.2. Adding hosts on vSphere Copy linkLink copied to clipboard!
You can add hosts to the Assisted Installer cluster using the online vSphere client or the govc vSphere CLI tool. The following procedure demonstrates adding hosts with the govc CLI tool. To use the online vSphere Client, see the documentation for vSphere.
To add hosts on vSphere with the vSphere govc CLI, generate the discovery image ISO from the Assisted Installer. The minimal discovery image ISO is the default setting. This image includes only what is required to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.
After this is complete, you must create an image for the vSphere platform and create the vSphere virtual machines.
Prerequisites
- You are using vSphere 8.0 or higher.
-
You have the vSphere
govcCLI tool installed and configured. -
You have set
clusterSet disk.EnableUUIDto TRUE in vSphere. - You have created a cluster in the Assisted Installer web console, or
- You have created an Assisted Installer cluster profile and infrastructure environment with the API.
-
You have exported your infrastructure environment ID in your shell as
$INFRA_ENV_ID.
Procedure
- Configure the discovery image if you want it to boot with an ignition file.
- In Cluster details, select vSphere from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
- In Host discovery, click the Add hosts button and select the provisioning type.
Add an SSH public key so that you can connect to the vSphere VMs as the
coreuser. Having a login to the cluster hosts can provide you with debugging information during the installation.- If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
-
In the SSH public key field, click Browse to upload the
id_rsa.pubfile containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
Select the required discovery image ISO.
NoteMinimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.
In Networking, select Cluster-managed networking or User-managed networking:
Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.
NoteThe proxy username and password must be URL-encoded.
- Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy or the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates and add the additional certificates.
- Optional: Configure the discovery image if you want to boot it with an ignition file. For details, see Additional Resources.
- Click Generate Discovery ISO.
- Copy the Discovery ISO URL.
Download the discovery ISO:
$ wget - O vsphere-discovery-image.iso <discovery_url>Replace
<discovery_url>with the Discovery ISO URL from the preceding step.On the command line, power off and delete any preexisting virtual machines:
$ for VM in $(/usr/local/bin/govc ls /<datacenter>/vm/<folder_name>) do /usr/local/bin/govc vm.power -off $VM /usr/local/bin/govc vm.destroy $VM done-
Replace
<datacenter>with the name of the data center. -
Replace
<folder_name>with the name of the VM inventory folder.
-
Replace
Remove preexisting ISO images from the data store, if there are any:
$ govc datastore.rm -ds <iso_datastore> <image>-
Replace
<iso_datastore>with the name of the data store. -
Replace
<image>with the name of the ISO image.
-
Replace
Upload the Assisted Installer discovery ISO:
$ govc datastore.upload -ds <iso_datastore> vsphere-discovery-image.isoReplace
<iso_datastore>with the name of the data store.NoteAll nodes in the cluster must boot from the discovery image.
Boot three to five control plane nodes:
$ govc vm.create -net.adapter <network_adapter_type> \ -disk.controller <disk_controller_type> \ -pool=<resource_pool> \ -c=16 \ -m=32768 \ -disk=120GB \ -disk-datastore=<datastore_file> \ -net.address="<nic_mac_address>" \ -iso-datastore=<iso_datastore> \ -iso="vsphere-discovery-image.iso" \ -folder="<inventory_folder>" \ <hostname>.<cluster_name>.example.comSee vm.create for details.
NoteThe foregoing example illustrates the minimum required resources for control plane nodes.
Boot at least two worker nodes:
$ govc vm.create -net.adapter <network_adapter_type> \ -disk.controller <disk_controller_type> \ -pool=<resource_pool> \ -c=4 \ -m=8192 \ -disk=120GB \ -disk-datastore=<datastore_file> \ -net.address="<nic_mac_address>" \ -iso-datastore=<iso_datastore> \ -iso="vsphere-discovery-image.iso" \ -folder="<inventory_folder>" \ <hostname>.<cluster_name>.example.comSee vm.create for details.
NoteThe foregoing example illustrates the minimum required resources for worker nodes.
Ensure the VMs are running:
$ govc ls /<datacenter>/vm/<folder_name>-
Replace
<datacenter>with the name of the data center. -
Replace
<folder_name>with the name of the VM inventory folder.
-
Replace
After 2 minutes, shut down the VMs:
$ for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.power -s=true $VM done-
Replace
<datacenter>with the name of the data center. -
Replace
<folder_name>with the name of the VM inventory folder.
-
Replace
Set the
disk.EnableUUIDsetting toTRUE:$ for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.change -vm $VM -e disk.EnableUUID=TRUE done-
Replace
<datacenter>with the name of the data center. Replace
<folder_name>with the name of the VM inventory folder.NoteYou must set
disk.EnableUUIDtoTRUEon all of the nodes to enable autoscaling with vSphere.
-
Replace
Restart the VMs:
$ for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.power -on=true $VM done-
Replace
<datacenter>with the name of the data center. -
Replace
<folder_name>with the name of the VM inventory folder.
-
Replace
-
Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a
Readystatus. - Select roles if needed.
- In Networking, clear the Allocate IPs via DHCP server checkbox.
- Set the API VIP address.
- Set the Ingress VIP address.
- Continue with the installation procedure.
13.3. vSphere postinstallation configuration using the CLI Copy linkLink copied to clipboard!
After installing an OpenShift Container Platform cluster by using the Assisted Installer on vSphere with the platform integration feature enabled, you must update the following vSphere configuration settings manually:
- vCenter username
- vCenter password
- vCenter address
- vCenter cluster
- Data center
- Data store
- Folder
Prerequisites
- The Assisted Installer has finished installing the cluster successfully.
- The cluster is connected to console.redhat.com.
Procedure
Generate a base64-encoded username and password for vCenter:
$ echo -n "<vcenter_username>" | base64 -w0Replace
<vcenter_username>with your vCenter username.$ echo -n "<vcenter_password>" | base64 -w0Replace
<vcenter_password>with your vCenter password.Backup the vSphere credentials:
$ oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yamlEdit the vSphere credentials:
$ cp creds_backup.yaml vsphere-creds.yaml$ vi vsphere-creds.yamlapiVersion: v1 data: <vcenter_address>.username: <vcenter_username_encoded> <vcenter_address>.password: <vcenter_password_encoded> kind: Secret metadata: annotations: cloudcredential.openshift.io/mode: passthrough creationTimestamp: "2022-01-25T17:39:50Z" name: vsphere-creds namespace: kube-system resourceVersion: "2437" uid: 06971978-e3a5-4741-87f9-2ca3602f2658 type: Opaque-
Replace
<vcenter_address>with the vCenter address. -
Replace
<vcenter_username_encoded>with the base64-encoded version of your vSphere username. -
Replace
<vcenter_password_encoded>with the base64-encoded version of your vSphere password.
-
Replace
Replace the vSphere credentials:
$ oc replace -f vsphere-creds.yamlRedeploy the kube-controller-manager pods:
$ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' \ --type=mergeBackup the vSphere cloud provider configuration:
$ oc get cm cloud-provider-config -o yaml -n openshift-config > \ cloud-provider-config_backup.yamlEdit the cloud provider configuration:
For OCP versions 4.17 and earlier, use the INI syntax for the
data.configsection of the.yamlfile.$ vi cloud-provider-config.yamlapiVersion: v1 data: config: | [Global] secret-name = "vsphere-creds" secret-namespace = "kube-system" insecure-flag = "1" [Workspace] server = "<vcenter_address>" datacenter = "<datacenter>" default-datastore = "<datastore>" folder = "/<datacenter>/vm/<folder>" [VirtualCenter "<vcenter_address>"] datacenters = "<datacenter>" kind: ConfigMap metadata: creationTimestamp: "2022-01-25T17:40:49Z" name: cloud-provider-config namespace: openshift-config resourceVersion: "2070" uid: 80bb8618-bf25-442b-b023-b31311918507-
Replace
<vcenter_address>with the vCenter address. -
Replace
<datacenter>with the name of the data center. -
Replace
<datastore>with the name of the data store. -
Replace
<folder>with the folder containing the cluster VMs.
-
Replace
For OCP versions 4.18 and later, use YAML syntax for the
data.configsection of the.yamlfile.$ cp cloud-provider-config_backup.yaml cloud-provider-config.yaml$ vi cloud-provider-config.yamlapiVersion: v1 data: config: | global: user: "" password: "" server: "" port: 0 insecureFlag: true datacenters: [] soapRoundtripCount: 0 caFile: "" thumbprint: "" secretName: vsphere-creds secretNamespace: kube-system secretsDirectory: "" apiDisable: false apiBinding: "" ipFamily: [] vcenter: <vcenter_address>: user: "" password: "" tenantref: "" server: "<vcenter_address>" port: <vcenter_port> insecureFlag: true datacenters: - <datacenter> soapRoundtripCount: 0 caFile: "" thumbprint: "" secretref: "" secretName: "" secretNamespace: "" ipFamily: [] labels: zone: "" region: "" kind: ConfigMap-
Replace
<vcenter_address>with the vCenter address. -
Replace
<vcenter_port>with the vCenter port number. Port 443 is the default. -
Replace
<datacenter>with the name of the data center.
-
Replace
Apply the cloud provider configuration:
$ oc apply -f cloud-provider-config.yamlTaint the nodes with the
uninitializedtaint:ImportantFollow steps 9 through 12 if you are installing OpenShift Container Platform 4.13 or later.
Identify the nodes to taint:
$ oc get nodesRun the following command for each node:
$ oc adm taint node <node_name> node.cloudprovider.kubernetes.io/uninitialized=true:NoScheduleReplace
<node_name>with the name of the node.
Example:
$ oc get nodes NAME STATUS ROLES AGE VERSION master-0 Ready control-plane,master 45h v1.26.3+379cd9f master-1 Ready control-plane,master 45h v1.26.3+379cd9f worker-0 Ready worker 45h v1.26.3+379cd9f worker-1 Ready worker 45h v1.26.3+379cd9f master-2 Ready control-plane,master 45h v1.26.3+379cd9f $ oc adm taint node master-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule $ oc adm taint node master-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule $ oc adm taint node master-2 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule $ oc adm taint node worker-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule $ oc adm taint node worker-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoScheduleBack up the infrastructures configuration:
$ oc get infrastructures.config.openshift.io -o yaml > infrastructures.config.openshift.io.yaml.backupEdit the infrastructures configuration:
$ cp infrastructures.config.openshift.io.yaml.backup infrastructures.config.openshift.io.yaml$ vi infrastructures.config.openshift.io.yamlapiVersion: v1 items: - apiVersion: config.openshift.io/v1 kind: Infrastructure metadata: creationTimestamp: "2022-05-07T10:19:55Z" generation: 1 name: cluster resourceVersion: "536" uid: e8a5742c-6d15-44e6-8a9e-064b26ab347d spec: cloudConfig: key: config name: cloud-provider-config platformSpec: type: VSphere vsphere: failureDomains: - name: assisted-generated-failure-domain region: assisted-generated-region server: <vcenter_address> topology: computeCluster: /<data_center>/host/<vcenter_cluster> datacenter: <data_center> datastore: /<data_center>/datastore/<datastore> folder: "/<data_center>/path/to/folder" networks: - "VM Network" resourcePool: /<data_center>/host/<vcenter_cluster>/Resources zone: assisted-generated-zone nodeNetworking: external: {} internal: {} vcenters: - datacenters: - <data_center> server: <vcenter_address> kind: List metadata: resourceVersion: ""-
Replace
<vcenter_address>with your vCenter address. -
Replace
<datacenter>with the name of your vCenter data center. -
Replace
<datastore>with the name of your vCenter data store. -
Replace
<folder>with the folder containing the cluster VMs. -
Replace
<vcenter_cluster>with the vSphere vCenter cluster where OpenShift Container Platform is installed.
-
Replace
Apply the infrastructures configuration:
$ oc apply -f infrastructures.config.openshift.io.yaml --overwrite=true
13.4. vSphere postinstallation configuration using the web console Copy linkLink copied to clipboard!
After installing an OpenShift Container Platform cluster by using the Assisted Installer on vSphere with the platform integration feature enabled, you must update the following vSphere configuration settings manually:
- vCenter address
- vCenter cluster
- vCenter username
- vCenter password
- Data center
- Default data store
- Virtual machine folder
Prerequisites
- The Assisted Installer has finished installing the cluster successfully.
- The cluster is connected to console.redhat.com.
Procedure
- In the Administrator perspective, navigate to Home → Overview.
- Under Status, click vSphere connection to open the vSphere connection configuration wizard.
-
In the vCenter field, enter the network address of the vSphere vCenter server. This can be either a domain name or an IP address. It appears in the vSphere web client URL; for example
https://[your_vCenter_address]/ui. In the vCenter cluster field, enter the name of the vSphere vCenter cluster where OpenShift Container Platform is installed.
ImportantThis step is mandatory if you installed OpenShift Container Platform 4.13 or later.
- In the Username field, enter your vSphere vCenter username.
In the Password field, enter your vSphere vCenter password.
WarningThe system stores the username and password in the
vsphere-credssecret in thekube-systemnamespace of the cluster. An incorrect vCenter username or password makes the cluster nodes unschedulable.-
In the Datacenter field, enter the name of the vSphere data center that contains the virtual machines used to host the cluster; for example,
SDDC-Datacenter. In the Default data store field, enter the vSphere data store that stores the persistent data volumes; for example,
/SDDC-Datacenter/datastore/datastorename.WarningUpdating the vSphere data center or default data store after the configuration has been saved detaches any active vSphere
PersistentVolumes.-
In the Virtual Machine Folder field, enter the data center folder that contains the virtual machine of the cluster; for example,
/SDDC-Datacenter/vm/ci-ln-hjg4vg2-c61657-t2gzr. For the OpenShift Container Platform installation to succeed, all virtual machines comprising the cluster must be located in a single data center folder. -
Click Save Configuration. This updates the
cloud-provider-configfile in theopenshift-confignamespace, and starts the configuration process. - Reopen the vSphere connection configuration wizard and expand the Monitored operators panel. Check that the status of the operators is either Progressing or Healthy.
Verification
The connection configuration process updates operator statuses and control plane nodes. It takes approximately an hour to complete. During the configuration process, the nodes will reboot. Previously bound PersistentVolumeClaims objects might become disconnected.
Follow the steps below to monitor the configuration process.
Check that the configuration process completed successfully:
- In the Administrator perspective, navigate to Home > Overview.
- Under Status click Operators. Wait for all operator statuses to change from Progressing to All succeeded. A Failed status indicates that the configuration failed.
- Under Status, click Control Plane. Wait for the response rate of all Control Pane components to return to 100%. A Failed control plane component indicates that the configuration failed.
A failure indicates that at least one of the connection settings is incorrect. Change the settings in the vSphere connection configuration wizard and save the configuration again.
Check that you are able to bind
PersistentVolumeClaimsobjects by performing the following steps:Create a
StorageClassobject using the following YAML:kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: vsphere-sc provisioner: kubernetes.io/vsphere-volume parameters: datastore: YOURVCENTERDATASTORE diskformat: thin reclaimPolicy: Delete volumeBindingMode: ImmediateCreate a
PersistentVolumeClaimsobject using the following YAML:kind: PersistentVolumeClaim apiVersion: v1 metadata: name: test-pvc namespace: openshift-config annotations: volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume finalizers: - kubernetes.io/pvc-protection spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: vsphere-sc volumeMode: Filesystem
For instructions, see Dynamic provisioning in the OpenShift Container Platform documentation. To troubleshoot a
PersistentVolumeClaimsobject, navigate to Storage → PersistentVolumeClaims in the Administrator perspective of the OpenShift Container Platform web console.
Chapter 14. Installing on Oracle Cloud Infrastructure Copy linkLink copied to clipboard!
The Oracle® Cloud Infrastructure (OCI) platform provides services that can meet your needs for regulatory compliance, performance, and cost-effectiveness. You can use Oracle Cloud Infrastructure Resource Manager to provision and configure OCI resources.
The Assisted Installer supports cluster installation on Oracle Cloud Infrastructure for these OpenShift Container Platform versions:
- From OpenShift Container Platform 4.14 and later, you can install a cluster on an Oracle Cloud Infrastructure virtual machine by using your own infrastructure.
- From OpenShift Container Platform 4.16 and later, you can also install a cluster on an Oracle Cloud Infrastructure bare-metal machine.
Bare-metal installations that use iSCSI boot drives require a secondary vNIC that is automatically created in the Terraform stack provided by Oracle.
The following sections show the OCI deployment options and their supported infrastructure in OpenShift Container Platform. Each infrastructure can have both VM and bare-metal servers.
14.1. Installing on Oracle Distributed Cloud Copy linkLink copied to clipboard!
You can use the Assisted Installer to install an OpenShift Container Platform (RHOCP) cluster on the following Oracle® Distributed Cloud infrastructure:
| Infrastructure | Description | Certified from | Support status |
|---|---|---|---|
| Commercial Public Cloud | Standard global OCI regions accessible to all customers. | RHOCP 4.14 | General Availability |
| Dedicated Region | A complete OCI region hardware and software stack located in a customer data center. | RHOCP 4.14 | General Availability |
| US Government Cloud | Physically isolated regions for US public sector, meeting FedRAMP High and DoD requirements. | RHOCP 4.20 | Technology Preview |
| UK Government Cloud | Dedicated UK regions for public sector, ensuring data residency and UK-citizen operations. | RHOCP 4.20 | Technology Preview |
| EU Sovereign Cloud | EU-based regions operated by EU residents to meet strict European data privacy standards. | RHOCP 4.20 | Technology Preview |
| Isolated Region | Air-gapped regions disconnected from the internet for top-secret/national security workloads. | RHOCP 4.20 | Technology Preview |
| Oracle Alloy | A partner-operated platform that allows third-party providers to offer their own branded OCI services. | RHOCP 4.20 | Technology Preview |
For installation instructions, see Installing a cluster on Oracle Distributed Cloud by using the Assisted Installer.
The installation of OpenShift Container Platform on US Government Cloud, UK Government Cloud, EU Sovereign Cloud, Isolated Region, and Oracle Alloy infrastructures is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
14.2. Installing on Oracle Edge Cloud Copy linkLink copied to clipboard!
You can use the Assisted Installer to install an OpenShift Container Platform (RHOCP) cluster on the following Oracle® Edge Cloud infrastructure:
| Infrastructure | Description | Certified from | Support status |
|---|---|---|---|
| Private Cloud Appliance | A customer-managed, rack-scale engineered system for on-premise private cloud modernization. | RHOCP 4.18 | General Availability |
| Oracle Compute Cloud@Customer | An Oracle-managed compute rack in the customer’s data center that consumes OCI services via subscription. | RHOCP 4.18 | General Availability |
| Roving Edge | Robust, portable tactical nodes for compute and storage in remote or mobile environments. | RHOCP 4.20 | Technology Preview |
For installation instructions, see Installing a cluster on Oracle Edge Cloud by using the Assisted Installer.
The installation of OpenShift Container Platform on a Roving Edge device is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
14.3. Installing on Oracle Database Appliance Copy linkLink copied to clipboard!
Oracle Database Appliance (ODA) is an engineered system designed to simplify the deployment, management, and support of Oracle databases. It includes built-in KVM virtualization and can host application virtual machines, such as Red Hat OpenShift control plane and worker nodes, alongside database workloads.
You can install an OpenShift Container Platform cluster on Oracle Database Appliance by using the Assisted Installer. This deployment is available for OpenShift Container Platform 4.21 and later releases.
For installation instructions, see Installing on Oracle Database Appliance.
Chapter 15. Troubleshooting Copy linkLink copied to clipboard!
There are cases where the Assisted Installer cannot begin the installation or the cluster fails to install properly. In these events, it is helpful to understand the likely failure modes and how to troubleshoot them.
15.1. Troubleshooting discovery ISO issues Copy linkLink copied to clipboard!
The Assisted Installer uses an ISO image to run an agent that registers the host to the cluster and performs hardware and network validations before attempting to install OpenShift. You can follow these procedures to troubleshoot problems related to the host discovery.
Once you start the host with the discovery ISO image, the Assisted Installer discovers the host and presents it in the Assisted Service web console. For details, see "Configuring the discovery image".
15.1.1. Verifying the discovery agent is running Copy linkLink copied to clipboard!
Check that the discovery agent is running on the host by verifying the host machine is powered on, networking is configured correctly, and the agent service is active.
Prerequisites
- You have created an infrastructure environment by using the API or have created a cluster by using the web console.
- You booted a host with the Infrastructure Environment discovery ISO and the host failed to register.
- You have SSH access to the host.
- You provided an SSH public key in the "Add hosts" dialog before generating the Discovery ISO so that you can SSH into your machine without a password.
Procedure
- Verify that your host machine is powered on.
- If you selected DHCP networking, check that the DHCP server is enabled.
- If you selected Static IP, bridges and bonds networking, check that your configurations are correct.
Verify that you can access your host machine using SSH, a console such as the BMC, or a virtual machine console:
$ ssh core@<host_ip_address>You can specify private key file by using the
-iparameter if it is not stored in the default directory.$ ssh -i <ssh_private_key_file> core@<host_ip_address>If you fail to connect over SSH to the host, the host failed during boot or it failed to configure the network.
Upon login, the following example shows the message you should see:
If you are not seeing this message it means that the host did not boot with the Assisted Installer ISO image. Make sure you configured the boot order properly (The host should boot once from the live-ISO).
Check the agent service logs:
$ sudo journalctl -u agent.serviceThe following is a screenshot of an agent service log. In the example, the errors indicate a network issue:
If there is an error pulling the agent image, check the proxy settings. Verify that the host is connected to the network. You can use
nmclito get additional information about your network configuration.
15.1.2. Verifying the agent can access the assisted-service Copy linkLink copied to clipboard!
Check the agent logs to verify that the discovery agent can access the Assisted Service.
Prerequisites
- You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
- You booted a host with the Infrastructure Environment discovery ISO and the host failed to register.
- You verified the discovery agent is running.
Procedure
Check the agent logs to verify the agent can access the Assisted Service:
$ sudo journalctl TAG=agentThe following is a screenshot of an agent log. In the example, the errors indicate that the agent failed to access the Assisted Service:
Check the proxy settings you configured for the cluster. If configured, the proxy must allow access to the Assisted Service URL.
15.2. Troubleshooting minimal discovery ISO issues Copy linkLink copied to clipboard!
Use the minimal ISO image when the virtual media connection has limited bandwidth. It includes only what the agent requires to boot a host with networking. The majority of the content is downloaded upon boot. The resulting ISO image is about 100MB in size compared to 1GB for the full ISO image.
15.2.1. Troubleshooting minimal ISO boot failure by interrupting the boot process Copy linkLink copied to clipboard!
If your environment requires static network configuration to access the Assisted Installer service, any issues with that configuration might prevent the minimal ISO from booting properly.
If the boot screen shows that the host has failed to download the root file system image, the network might not be configured correctly. You can interrupt the kernel boot early in the bootstrap process, before the root file system image is downloaded. This allows you to access the root console and review the network configurations.
The following example shows a rootfs download failure:
Procedure
Add the
.spec.kernelArgumentsstanza to theinfraEnvobject of the cluster you are deploying:NoteFor details on modifying an infrastructure environment, see Modifying an infrastructure environment.
# ... spec: clusterRef: name: sno1 namespace: sno1 cpuArchitecture: x86_64 ipxeScriptType: DiscoveryImageAlways kernelArguments: - operation: append value: rd.break=initqueue nmStateConfigLabelSelector: matchLabels: nmstate-label: sno1 pullSecretRef: name: assisted-deployment-pull-secretThe value
rd.break=initqueueinterrupts the boot at thedracutmain loop. See rd.break options for debugging kernel boot for details.-
Wait for the related nodes to reboot automatically and for the boot to stop at the
iniqueuestage, beforerootfsis downloaded. You will be redirected to the root console. Identify and change the incorrect network configurations. Here are some useful diagnostic commands:
View system logs by using
journalctl, for example:# journalctl -p err //Sorts logs by errors # journalctl -p crit //Sorts logs by critical errors # journalctl -p warning //Sorts logs by warningsView network connection information by using
nmcli, as follows:# nmcli conn showCheck the configuration files for incorrect network connections, for example:
# cat /etc/assisted/network/host0/eno3.nmconnection
-
Press
control+dto resume the bootstrap process. The server downloadsrootfsand completes the process. -
Reopen the
infraEnvobject and remove the.spec.kernelArgumentsstanza.
15.3. Correcting a host’s boot order Copy linkLink copied to clipboard!
Once the installation that runs as part of the Discovery Image completes, the Assisted Installer reboots the host. The host must boot from its installation disk to continue forming the cluster. If you have not correctly configured the host’s boot order, it will boot from another disk instead, interrupting the installation.
If the host boots the discovery image again, the Assisted Installer will immediately detect this event and set the host’s status to Installing Pending User Action. Alternatively, if the Assisted Installer does not detect that the host has booted the correct disk within the allotted time, it will also set this host status.
Procedure
- Reboot the host and set its boot order to boot from the installation disk. If you didn’t select an installation disk, the Assisted Installer selected one for you.
- To view the selected installation disk, click to expand the host’s information in the host inventory, and check which disk has the “Installation disk” role.
15.4. Troubleshooting a partially-successful installation Copy linkLink copied to clipboard!
There are cases where the Assisted Installer declares an installation to be successful even though it encountered errors:
- If you requested to install OLM operators and one or more failed to install, log in to the cluster’s console to remediate the failures.
- If you requested to install more than two worker nodes and at least one failed to install, but at least two succeeded, add the failed workers to the installed cluster. For details, see "Replacing a control plane node in an unhealthy cluster".
15.5. Resolving API connectivity failure when adding nodes to a cluster Copy linkLink copied to clipboard!
When you add a node to an existing cluster as part of Day 2 operations, the node downloads the ignition configuration file from the Day 1 cluster. If the download fails and the node is unable to connect to the cluster, the status of the host in the Host discovery step changes to Insufficient. Clicking this status displays the following error message:
The host failed to download the ignition file from <URL>. You must ensure the host can reach the URL. Check your DNS and network configuration or update the IP address or domain used to reach the cluster.
error: ignition file download failed.... no route to host
There are several possible reasons for the connectivity failure. Here are some recommended actions.
Procedure
Check the IP address and domain name of the cluster:
- Click the set the IP or domain used to reach the cluster hyperlink.
- In the Update cluster hostname window, enter the correct IP address or domain name for the cluster.
- Check your DNS settings to ensure that the DNS can resolve the domain that you provided.
-
Ensure that port
22624is open in all firewalls. Check the agent logs of the host to verify that the agent can access the Assisted Service via SSH:
$ sudo journalctl TAG=agentNoteFor more details, see the "Verifying the agent can access the Assisted Service" troubleshooting section.
15.6. Troubleshooting auto-assign validation errors Copy linkLink copied to clipboard!
When using auto-assign for host roles, you might encounter validation errors related to the internal suggested_role field.
The Assisted Installer assigns two role types to each host:
-
role: This is the primary, visible role for the host. You can explicitly set the role as either control plane, arbiter, or worker. Alternatively, you can choose auto-assign, which allows the Assisted Installer to determine the appropriate role automatically. -
suggested_role: When a host is set to auto-assign, the Assisted Installer assigns it asuggested_rolevalue at the start of the installation process. This is an internal role type and is not visible to the user. The Assisted Installer determines thesuggested_rolebased on its internal logic. For details, see Supported host roles.
The web console does not display the suggested_role field. Therefore, when using auto-assign, you might still encounter host pre-installation validation errors that suggest a host has a specific role. This happens because the Assisted Installer relies on suggested_role for certain validations.
Procedure
- If you encounter an error, either manually assign a role to each affected node instead of using auto-assign, or follow the instructions provided in the error message, where applicable.
Legal Notice
Copy linkLink copied to clipboard!
Copyright © Red Hat
OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
Modified versions must remove all Red Hat trademarks.
Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.
Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of the OpenJS Foundation.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.