Chapter 1. OpenShift Container Platform 4.16 release notes

RHCOS now uses Red Hat Enterprise Linux (RHEL) 9.4 packages in OpenShift Container Platform 4.16. These packages ensure that your OpenShift Container Platform instances receive the latest fixes, features, enhancements, hardware support, and driver updates. As an Extended Update Support (EUS) release, OpenShift Container Platform 4.14 is excluded from this change and will continue to use RHEL 9.2 EUS packages for the entirety of its lifecycle.

With this release, you can now install RHCOS to Small Computer Systems Interface (iSCSI) boot devices. Multipathing for iSCSI is also supported. For more information, see Installing RHCOS manually on an iSCSI boot device and Installing RHCOS on an iSCSI boot device using iBFT

With this release, you can now install RHCOS to Intel® VROC RAID devices. For more information about configuring RAID to an Intel® VROC device, see Configuring an Intel® Virtual RAID on CPU (VROC) data volume.

In OpenShift Container Platform 4.16, the installation program uses Cluster API instead of Terraform to provision cluster infrastructure during installations on Amazon Web Services. There are several additional required permissions as a result of this change. For more information, see Required AWS permissions for the IAM user.

Additionally, SSH access to control plane and compute machines is no longer open to the machine network, but is restricted to the security groups associated with the control plane and compute machines.

In OpenShift Container Platform 4.16, the installation program uses Cluster API instead of Terraform to provision cluster infrastructure during installations on VMware vSphere.

In OpenShift Container Platform 4.16, the installation program uses Cluster API instead of Terraform to provision cluster infrastructure during installations on Nutanix.

In OpenShift Container Platform 4.16, the installation program uses Cluster API instead of Terraform to provision cluster infrastructure during installations on GCP. This feature is available as a Technology Preview in OpenShift Container Platform 4.16. To enable Technology Preview features, set the featureSet: TechPreviewNoUpgrade parameter in the install-config.yaml file before installation. Alternatively, add the following stanza to the install-config.yaml file before installation to enable Cluster API installation without any other Technology Preview features:

featureSet: CustomNoUpgrade
featureGates:
- ClusterAPIInstall=true

For more information, see Optional configuration parameters.

With this release, the OpenShift Container Platform installation program no longer supports the installer-provisioned installation on the Alibaba Cloud platform. You can install a cluster on Alibaba Cloud by using Assisted Installer, which is currently a Technology Preview feature. For more information, see Installing on Alibaba cloud.

In OpenShift Container Platform 4.16, you can disable the cloud controller manager capability during installation. For more information, see Cloud controller manager capability.

With this update, if you install a FIPS-enabled cluster, you must run the installation program from a RHEL 9 computer that is configured to operate in FIPS mode, and you must use a FIPS-capable version of the installation program. For more information, see Support for FIPS cryptography.

In OpenShift Container Platform 4.16, you can add up to ten tags to attach to the virtual machines (VMs) provisioned by a VMware vSphere cluster. These tags are in addition to the unique cluster-specific tag that the installation program uses to identify and remove associated VMs when a cluster is decommissioned.

You can define the tags on the VMware vSphere VMs in the install-config.yaml file during cluster creation. For more information, see Sample install-config.yaml file for an installer-provisioned VMware vSphere cluster.

You can define tags for compute or control plane machines on an existing cluster by using machine sets. For more information, see "Adding tags to machines by using machine sets" for compute or control plane machine sets.

OpenShift Container Platform 4.16 uses Kubernetes 1.29, which removed several deprecated APIs.

A cluster administrator must provide manual acknowledgment before the cluster can be updated from OpenShift Container Platform 4.15 to 4.16. This is to help prevent issues after updating to OpenShift Container Platform 4.16, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All OpenShift Container Platform 4.15 clusters require this administrator acknowledgment before they can be updated to OpenShift Container Platform 4.16.

For more information, see Preparing to update to OpenShift Container Platform 4.16.

With this release, you can prevent the kubeadmin password from being displayed in the console after the installation by using the --skip-password-print flag during cluster creation. The password remains accessible in the auth directory.

With this release, the OpenShift-based Appliance Builder is available as a Technology Preview feature. The Appliance Builder enables self-contained OpenShift Container Platform cluster installations, meaning that it does not rely on internet connectivity or external registries. It is a container-based utility that builds a disk image that includes the Agent-based Installer, which can then be used to install multiple OpenShift Container Platform clusters.

For more information, see the OpenShift-based Appliance Builder User Guide.

With this release, you can enable bring your own public IPv4 addresses (BYOIP) feature when installing on Amazon Web Services (AWS) by using the publicIpv4Pool field to allocate Elastic IP addresses (EIPs). You must ensure that you have the required permissions to enable BYOIP. For more information, see Optional AWS configuration parameters.

You can deploy OpenShift Container Platform 4.16 in Google Cloud Platform (GCP) in the Dammam, Saudi Arabia (me-central2) region and in the Johannesburg, South Africa (africa-south1) region. For more information, see Supported GCP regions.

With this release, you can deploy compute nodes on GPU-enabled NVIDIA H100 machines when installing a cluster on GCP. For more information, see Tested instance types for GCP and Google’s documentation about the Accelerator-optimized machine family.

With this release, you can manage workloads on multi-architecture clusters by using the Multiarch Tuning Operator. This Operator enhances the operational experience within multi-architecture clusters, and single-architecture clusters that are migrating to a multi-architecture compute configuration. It implements the ClusterPodPlacementConfig custom resource (CR) to support architecture-aware workload scheduling.

For more information, see Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator.

This feature provides support for adding 64-bit x86 compute machines to a multi-architecture cluster with 64-bit ARM control plane machines. With this release, you can add 64-bit x86 compute machines to a cluster that uses 64-bit ARM control plane machines and already includes 64-bit ARM compute machines.

This feature provides support for installing an Agent-based Installer cluster with multi payload. After installing the Agent-based Installer cluster with multi payload, you can add compute machines with different architectures to the cluster.

With this release, French and Spanish are supported in the web console. You can update the language in the web console from the Language list on the User Preferences page.

With this release, Patternfly 4 and React Router 5 are deprecated in the web console. All plugins should migrate to Patternfly 5 and React Router 6 as soon as possible.

This release introduces the following updates to the Administrator perspective of the web console:

This release introduces the following updates to the Developer perspective of the web console:

The oc-mirror plugin v2 for OpenShift Container Platform includes new features and functionalities that improve the mirroring process for Operator images and other OpenShift Container Platform content.

The following are the key enhancements and features in oc-mirror plugin v2:

The oc-mirror OpenShift CLI (oc) plugin is used to mirror all the required OpenShift Container Platform content and other images to your mirror registry, simplifying the maintenance of disconnected clusters.

Previously, the oc adm upgrade command provided limited information about the status of a cluster update. This release adds the oc adm upgrade status command, which decouples status information from the oc adm upgrade command and provides specific information regarding a cluster update, including the status of the control plane and worker node updates.

With this release, if you query a resource by using its short name, the OpenShift CLI (oc) returns a warning if more than one custom resource definition (CRD) with the same short name exists in the cluster.

Warning: short name "ex" could also match lower priority resource examples.test.com

This release introduces a new --interactive flag for the oc delete command. When the --interactive flag is set to true, the resource is deleted only if the user confirms the deletion. This flag is available as a Technology Preview feature.

The IBM Z® and IBM® LinuxONE release on OpenShift Container Platform 4.16 adds improvements and new capabilities to OpenShift Container Platform components and concepts.

This release introduces support for the following features on IBM Z® and IBM® LinuxONE:

The IBM Power® release on OpenShift Container Platform 4.16 adds improvements and new capabilities to OpenShift Container Platform components.

This release introduces support for the following features on IBM Power®:

Starting in OpenShift Container Platform 4.14, Extended Update Support (EUS) is extended to the IBM Power® and the IBM Z® platform. For more information, see the OpenShift EUS Overview.

In this release, you can enable Microsoft Entra Workload ID to use short-term credentials on an existing Microsoft Azure OpenShift Container Platform cluster. This functionality is now also supported in versions 4.14 and 4.15 of OpenShift Container Platform. For more information, see Enabling token-based authentication.

As part of the OpenShift Container Platform move to OVN-Kubernetes as the only supported network plugin, starting with OpenShift Container Platform 4.16, if your cluster uses the OpenShift SDN network plugin, you cannot upgrade to future major versions of OpenShift Container Platform without migrating to OVN-Kubernetes. For more information about migrating to OVN-Kubernetes, see Migrating from the OpenShift SDN network plugin.

If you try an upgrade, the Cluster Network Operator reports the following status:

- lastTransitionTime: "2024-04-11T05:54:37Z"
  message: Cluster is configured with OpenShiftSDN, which is not supported in the
    next version. Please follow the documented steps to migrate from OpenShiftSDN
    to OVN-Kubernetes in order to be able to upgrade. https://docs.openshift.com/container-platform/4.16/networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.html
  reason: OpenShiftSDNConfigured
  status: "False"
  type: Upgradeable

Configuring linuxptp services as grandmaster clock (T-GM) for dual Intel E810 Westport Channel network interface controllers (NICs) is now a generally available feature in OpenShift Container Platform. The host system clock is synchronized from the NIC that is connected to the Global Navigation Satellite Systems (GNSS) time source. The second NIC is synced to the 1PPS timing output provided by the NIC that is connected to GNSS. For more information see Configuring linuxptp services as a grandmaster clock for dual E810 Westport Channel NICs.

You can configure the linuxptp services ptp4l and phc2sys as a highly available (HA) system clock for dual PTP boundary clocks (T-BC).

For more information, see Configuring linuxptp as a highly available system clock for dual-NIC Intel E810 PTP boundary clocks.

To periodically test network connectivity among cluster components, the Cluster Network Operator (CNO) creates the network-check-source deployment and the network-check-target daemon set. In OpenShift Container Platform 4.16, you can configure the nodes by setting node selectors and run the source and target pods to check the network connectivity. For more information, see Verifying connectivity to an endpoint.

With this release, IP addresses and ranges are handled more efficiently in NSGs for OpenShift Container Platform clusters hosted on Microsoft Azure. As a result, the maximum limit of Classless Inter-Domain Routings (CIDRs) for all Ingress Controllers in Microsoft Azure clusters, using the allowedSourceRanges field, increases from approximately 1000 to 4000 CIDRs.

With this release, migration from the OpenShift SDN network plugin to OVN-Kubernetes is now supported on Nutanix platforms. For more information, see Migration to the OVN-Kubernetes network plugin.

With this release, OVN-Kubernetes uses a new DNSNameResolver custom resource to keep track of DNS records in your egress firewall rules, and is available as a Technology Preview. This custom resource supports the use of both wildcard DNS names and regular DNS names and allows access to DNS names regardless of the IP addresses associated with its change.

For more information, see Improved DNS resolution and resolving wildcard domain names.

With this release, you can configure the SR-IOV Network Operator to drain nodes in parallel during network policy updates. The option to drain nodes in parallel enables faster rollouts of SR-IOV network configurations. You can use the SriovNetworkPoolConfig custom resource to configure parallel node draining and define the maximum number of nodes in the pool that the Operator can drain in parallel.

For further information, see Configuring parallel node draining during SR-IOV network policy updates.

As of OpenShift Container Platform 4.16, the SR-IOV Network Operator no longer automatically creates a SriovOperatorConfig custom resource (CR). Create the SriovOperatorConfig CR by using the procedure described in Configuring the SR-IOV Network Operator

This release introduces 802.1Q-in-802.1Q also known as QinQ support. QinQ introduces a second VLAN tag, where the service provider designates the outer tag for their use, offering them flexibility, while the inner tag remains dedicated to the customer’s VLAN. When two VLAN tags are present in a packet, the outer VLAN tag can be either 802.1Q or 802.1ad. The inner VLAN tag must always be 802.1Q.

For more information, see Configuring QinQ support for SR-IOV enabled workloads.

With this release, you can configure an OpenShift Container Platform cluster on any on-premise infrastructure, such as bare metal, VMware vSphere, Red Hat OpenStack Platform (RHOSP), or Nutanix, to use a user-managed load balancer in place of the default load balancer. For this configuration, you must specify loadBalancer.type: UserManaged in your cluster’s install-config.yaml file.

For more information about this feature on bare-metal infrastructure, see Services for a user-managed load balancer in Setting up the environment for an OpenShift installation.

With this release, if you have pods in your cluster using iptables rules the following event message is given to warn against future deprecation:

This pod appears to have created one or more iptables rules. IPTables is deprecated and will no longer be available in RHEL 10 and later. You should consider migrating to another API such as nftables or eBPF.

For more information, see Getting started with nftables. If you are running third-party software, check with your vendor to ensure they will have an nftables based version available soon.

With this release, you can view the ingress network flows for OpenShift Container Platform services. You can use this information to manage ingress traffic for your network and improve network security.

For more information, see OpenShift Container Platform network flow matrix.

With this release, you can add IPv6 virtual IPs (VIPs) for API and Ingress services to an existing dual-stack-configured cluster by patching the cluster infrastructure.

If you have already upgraded your cluster to OpenShift Container Platform 4.16 and you need to convert the single-stack cluster network to a dual-stack cluster network, you must specify the following for your cluster in the YAML configuration patch file:

For more information, see Converting to a dual-stack cluster network in Converting to IPv4/IPv6 dual-stack networking.

This release introduces FRR-K8s, a Kubernetes based DaemonSet that exposes a subset of the FRR API in a Kubernetes-compliant manner. As a cluster administrator, you can use the FRRConfiguration custom resource (CR) to configure the MetalLB Operator to use the FRR-K8s daemon set as the backend. You can use this to operate FRR services, such as receiving routes.

For more information, see Configuring the integration of MetalLB and FRR-K8s.

With this release, OpenShift Container Platform routes can be configured with third-party certificate management solutions, utilising the .spec.tls.externalCertificate field in the route API. This allows you to reference externally managed TLS certificates through secrets, streamlining the process by eliminating manual certificate management. By using externally managed certificates, you reduce errors, ensure a smoother certificate update process, and enable the OpenShift router to promptly serve renewed certificates. For more information, see Creating a route with externally managed certificate.

This feature provides two new APIs, AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP). Before namespaces are created, cluster Administrators can use ANP and BANP to apply cluster-scoped network policies and safeguards for an entire cluster. Because it is cluster scoped, ANP provides Administrators a solution to manage the security of their network at scale without having to duplicate their network policies on each namespace.

For more information, see AdminNetworkPolicy in Network security.

Previously, when migrating from OpenShift SDN to OVN-Kubernetes, the only available option was an offline migration method. This process included some downtime, during which clusters were unreachable.

This release introduces a limited live migration method. The limited live migration method is the process in which the OpenShift SDN network plugin and its network configurations, connections, and associated resources are migrated to the OVN-Kubernetes network plugin without service interruption. It is available for OpenShift Container Platform. It is not available for hosted control plane deployment types. This migration method is valuable for deployment types that require constant service availability and offers the following benefits:

Migration to OVN-Kubernetes is intended to be a one-way process.

For more information, see Limited live migration to the OVN-Kubernetes network plugin overview.

Previously, it was not possible to configure the same CIDR range twice and to have the Whereabouts CNI plugin assign IP addresses from these ranges independently. This limitation caused issues in multi-tenant environments where different groups might need to select overlapping CIDR ranges.

With this release, the Whereabouts CNI plugin supports overlapping IP address ranges through the inclusion of a network_name parameter. Administrators can use the network_name parameter to configure the same CIDR range multiple times within separate NetworkAttachmentDefinitions, which enables independent IP address assignments for each range.

This feature also includes enhanced namespace handling, stores IPPool custom resources (CRs) in the appropriate namespaces, and supports cross-namespaces when permitted by Multus. These improvements provide greater flexibility and management capabilities in multi-tenant environments.

For more information about this feature, see Dynamic IP address assignment configuration with Whereabouts.

If you use the OVN-Kubernetes network plugin, you can configure the transit, join, and masquerade subnets. The transit, join and masquerade subnets can be configured either during cluster installation or after. The subnet defaults are:

For more information about these configuration fields, see Cluster Network Operator configuration object. For more information about configuring the transit and join subnets on an existing cluster, see Configure OVN-Kubernetes internal IP address subnets.

The Telemetry and the Insights Operator collects telemetry on IPsec connections. For more information, see Showing data collected by Telemetry.

You can now use the Secrets Store CSI Driver Operator to mount secrets from HashiCorp Vault to a Container Storage Interface (CSI) volume in OpenShift Container Platform. The Secrets Store CSI Driver Operator is available as a Technology Preview feature.

For the full list of available secrets store providers, see Secrets store providers.

For information about using the Secrets Store CSI Driver Operator to mount secrets from HashiCorp Vault, see Mounting secrets from HashiCorp Vault.

OpenShift Container Platform 4.16 introduces volume cloning for the Microsoft Azure File Container Storage Interface (CSI) Driver Operator as a Technology Preview feature. Volume cloning duplicates an existing persistent volume (PV) to help protect against data loss in OpenShift Container Platform. You can also use a volume clone just as you would use any standard volume.

For more information, see Azure File CSI Driver Operator and CSI volume cloning.

The Node Expansion Secret feature allows your cluster to expand storage of mounted volumes, even when access to those volumes requires a secret (for example, a credential for accessing a Storage Area Network (SAN) fabric) to perform the node expand operation. OpenShift Container Platform 4.16 supports this feature as generally available.

The default maximum number of snapshots in VMware vSphere Container Storage Interface (CSI) is 3 per volume. In OpenShift Container Platform 4.16, you can now change this maximum number of snapshots to a maximum of 32 per volume. You also have granular control of the maximum number of snapshots for vSAN and Virtual Volume datastores. OpenShift Container Platform 4.16 supports this feature as generally available.

For more information, see Changing the maximum number of snapshots for vSphere.

In OpenShift Container Platform 4.16 introduces a new parameter, LastPhaseTransitionTime, which has a timestamp that is updated every time a persistent volume (PV) transitions to a different phase (pv.Status.Phase). This feature is being released with Technology Preview status.

OpenShift Container Platform is capable of provisioning persistent volumes (PVs) with a Container Storage Interface (CSI) driver for the Common Internet File System (CIFS) dialect/Server Message Block (SMB) protocol. The CIFS/SMB CSI Driver Operator that manages this driver is in Technology Preview status.

For more information, see CIFS/SMB CSI Driver Operator.

OpenShift Container Platform 4.14 introduced a new access mode with Technical Preview status for persistent volumes (PVs) and persistent volume claims (PVCs) called ReadWriteOncePod (RWOP). RWOP can be used only in a single pod on a single node compared to the existing ReadWriteOnce access mode where a PV or PVC can be used on a single node by many pods. If the driver enables it, RWOP uses the SELinux context mount set in the PodSpec or container, which allows the driver to mount the volume directly with the correct SELinux labels. This eliminates the need to recursively relabel the volume, and pod startup can be significantly faster.

In OpenShift Container Platform 4.16, this feature is generally available.

For more information, see Access modes.

To support VMware vSphere Container Storage Interface (CSI) volume provisioning and usage in multi-zonal clusters, the deployment should match certain requirements imposed by CSI driver. These requirements have changed starting with 3.1.0, and although OpenShift Container Platform 4.16 accepts both the old and new tagging methods, you should use the new tagging method since VMware vSphere considers the old way an invalid configuration. To prevent problems, you should not use the old tagging method.

For more information, see vSphere CSI topology requirements.

This feature provides support for configuring thick-provisioned storage. If you exclude the deviceClasses.thinPoolConfig field in the LVMCluster custom resource (CR), logical volumes are thick provisioned. Using thick-provisioned storage includes the following limitations:

For information about configuring the LVMCluster CR, see About the LVMCluster custom resource.

This update provides a new warning message when you do not configure the deviceSelector field in the LVMCluster custom resource (CR).

The LVMCluster CR supports a new field, deviceDiscoveryPolicy, which indicates whether the deviceSelector field is configured. If you do not configure the deviceSelector field, LVM Storage automatically sets the deviceDiscoveryPolicy field to RuntimeDynamic. Otherwise, the deviceDiscoveryPolicy field is set to Preconfigured.

It is not recommended to exclude the deviceSelector field from the LMVCluster CR. For more information about the limitations of not configuring the deviceSelector field, see About adding devices to a volume group.

This feature provides support for adding encrypted devices to a volume group. You can enable disk encryption on the cluster nodes during an OpenShift Container Platform installation. After encrypting a device, you can specify the path to the LUKS encrypted device in the deviceSelector field in the LVMCluster custom resource. For information about disk encryption, About disk encryption and Configuring disk encryption and mirroring.

For more information about adding devices to a volume group, see About adding devices to a volume group.

Earlier Technology Preview phases of Operator Lifecycle Manager (OLM) 1.0 introduced a new Operator API, provided as operator.operators.operatorframework.io by the Operator Controller component. In OpenShift Container Platform 4.16, this API is renamed ClusterExtension, provided as clusterextension.olm.operatorframework.io, for this Technology Preview phase of OLM 1.0.

This API still streamlines management of installed extensions, which includes Operators via the registry+v1 bundle format, by consolidating user-facing APIs into a single object. The rename to ClusterExtension addresses the following:

For more information, see Operator Controller.

With this release, OLM 1.0 displays the following status condition messages for installed cluster extensions:

When determining upgrade edges for an installed cluster extension, Operator Lifecycle Manager (OLM) 1.0 supports legacy OLM semantics starting in OpenShift Container Platform 4.16. This support follows the behavior from legacy OLM, including replaces, skips, and skipRange directives, with a few noted differences.

By supporting legacy OLM semantics, OLM 1.0 now honors the upgrade graph from catalogs accurately.

For more information, see Upgrade constraint semantics.

With this release, unauthenticated users no longer have access to the system:webhook role binding. Before OpenShift Container Platform 4.16, unauthenticated users could access the system:webhook role binding. Changing this access for unauthenticated users adds an additional layer of security and should only be enabled by users when necessary. This change is for new clusters; previous clusters are not affected.

There are use cases where you might want to allow unauthenticated users the system:webhook role binding for specific namespaces. The system:webhook cluster role allows users to trigger builds from external systems that do not use OpenShift Container Platform authentication mechanisms, such as GitHub, GitLab, and Bitbucket. Cluster admins can grant unauthenticated users access to the system:webhook role binding to facilitate this use case.

To grant unauthenticated users access to the system:webhook role binding in specific namespaces, see Adding unauthenticated users to the system:webhook role binding.

With this release, you can now garbage collect unused rendered machine configs. By using the oc adm prune renderedmachineconfigs command, you can view the unused rendered machine configs, determine which to remove, then batch delete the rendered machine configs that you no longer need. Having too many machine configs can make working with the machine configs confusing and can also contribute to disk space and performance issues. For more information, see Managing unused rendered machine configs.

By default, when you make certain changes to the parameters in a MachineConfig object, the Machine Config Operator (MCO) drains and reboots the nodes associated with that machine config. However, you can create a node disruption policy in the MCO namespace that defines a set of Ignition config objects changes that would require little or no disruption to your workloads. For more information, see Using node disruption policies to minimize disruption from machine config changes.

With Red Hat Enterprise Linux CoreOS (RHCOS) image layering, you can now automatically build the custom layered image directly in your cluster, as a Technology Preview feature. Previously, you needed to build the custom layered image outside of the cluster, then pull the image into the cluster. You can use the image layering feature to extend the functionality of your base RHCOS image by layering additional images onto the base image. For more information, see RHCOS image layering.

By default, the MCO does not delete the boot image it uses to bring up a Red Hat Enterprise Linux CoreOS (RHCOS) node. Consequently, the boot image in your cluster is not updated along with your cluster. You can now configure your cluster to update the boot image whenever you update your cluster. For more information, see Updating boot images.

With this release, the cluster autoscaler can use the LeastWaste, Priority, and Random expanders. You can configure these expanders to influence the selection of machine sets when scaling the cluster. For more information, see Configuring the cluster autoscaler.

This release introduces the ability to manage machines by using the upstream Cluster API, integrated into OpenShift Container Platform, as a Technology Preview for VMware vSphere clusters. This capability is in addition or an alternative to managing machines with the Machine API. For more information, see About the Cluster API.

With this release, the previously Technology Preview feature of defining a vSphere failure domain for a control plane machine set is Generally Available. For more information, see Control plane configuration options for VMware vSphere.

The Vertical Pod Autoscaler Operator (VPA) consists of three components: the recommender, updater, and admission controller. The Operator and each component has its own pod in the VPA namespace on the control plane nodes. You can move the VPA Operator and component pods to infrastructure or worker nodes. For more information, see Moving the Vertical Pod Autoscaler Operator components.

With this release, the oc adm must-gather command collects the following additional information:

These additions help identify issues that might stem from using a specific version of oc. The oc adm must-gather command also lists what image was used and if any data could not be gathered in the must-gather logs.

For more information, see About the must-gather tool.

In OpenShift Container Platform 4.16 and later, you can edit the baseboard management controller (BMC) address in the BareMetalHost resource of a bare-metal node. The node must be in the Provisioned, ExternallyProvisioned, Registering, or Available state. Editing the BMC address in the BareMetalHost resource will not result in deprovisioning the node. See Editing a BareMetalHost resource for additional details.

In OpenShift Container Platform 4.16 and later, you can attach a generic, non-bootable ISO virtual media image to a provisioned node by using the DataImage resource. After you apply the resource, the ISO image becomes accessible to the operating system on the next reboot. The node must use Redfish or drivers derived from it to support this feature. The node must be in the Provisioned or ExternallyProvisioned state. See Attaching a non-bootable ISO to a bare-metal node for additional details.

This release includes the following version updates for in-cluster monitoring stack components and dependencies:

The Metrics Server component is now generally available and automatically installed instead of the deprecated Prometheus Adapter. Metrics Server collects resource metrics and exposes them in the metrics.k8s.io Metrics API service for use by other tools and APIs, which frees the core platform Prometheus stack from handling this functionality. For more information, see MetricsServerConfig in the config map API reference for the Cluster Monitoring Operator.

This release introduces a new monitoring-alertmanager-view role to allow read-only access to the Alertmanager API in the openshift-monitoring project.

Vertical Pod Autoscaler (VPA) metrics are now available through the kube-state-metrics agent. VPA metrics follow a similar exposition format just as they did before being deprecated and removed from native support upstream.

With this release, the proxy service in front of Prometheus, Alertmanager, and Thanos Ruler has been updated from OAuth to kube-rbac-proxy. This change might affect service accounts and users accessing these API endpoints without the appropriate roles and cluster roles.

With this release, when Prometheus scrapes a target, duplicate samples are no longer silently ignored, even if they have the same value. The first sample is accepted and the prometheus_target_scrapes_sample_duplicate_timestamp_total counter is incremented, which might trigger the PrometheusDuplicateTimestamps alert.

With this release, platform pods deployed with a workload annotation that includes both CPU limits and CPU requests will have the CPU limits accurately calculated and applied as a CPU quota for the specific pod. In prior releases, if a workload partitioned pod had both CPU limits and requests set, they were ignored by the webhook. The pod did not benefit from workload partitioning and was not locked down to specific cores. This update ensures the requests and limits are now interpreted correctly by the webhook.

For more information, see Workload partitioning.

Beginning with OpenShift Container Platform 4.16, Control Groups version 2 (cgroup v2), also known as cgroup2 or cgroupsv2, is enabled by default for all new deployments, even when performance profiles are present.

Since OpenShift Container Platform 4.14, cgroups v2 has been the default, but the performance profile feature required the use of cgroups v1. This issue has been resolved.

cgroup v1 is still used in upgraded clusters with performance profiles that have initial installation dates before OpenShift Container Platform 4.16. cgroup v1 can still be used in the current version by changing the cgroupMode field in the node.config object to v1.

For more information, see Configuring the Linux cgroup version on your nodes.

With this release, you can increase the disk quota in etcd. This is a Technology Preview feature. For more information, see Increasing the database size for etcd.

With this release, the Node Tuning Operator supports setting CPU frequencies in the PerformanceProfile for reserved and isolated core CPUs. This is an optional feature that you can use to define specific frequencies. The Node Tuning Operator then sets those frequencies by enabling the intel_pstate CPUFreq driver in the Intel hardware. You must follow Intel’s recommendations on frequencies for FlexRAN-like applications, which require the default CPU frequency to be set to a lower value than the default running frequency.

Previously, for the RAN DU-profile, setting the realTime workload hint to true in the PerformanceProfile always disabled the intel_pstate. With this release, the Node Tuning Operator detects the underlying Intel hardware using TuneD and appropriately sets the intel_pstate kernel parameter based on the processor’s generation. This decouples the intel_pstate from the realTime and highPowerConsumption workload hints. The intel_pstate now depends only on the underlying processor generation.

For pre-IceLake processors, the intel_pstate is deactivated by default, whereas for IceLake and later generation processors, the intel_pstate is set to active.

You can now use PolicyGenerator resources and Red Hat Advanced Cluster Management (RHACM) to deploy polices for managed clusters with GitOps ZTP. The PolicyGenerator API is part of the Open Cluster Management standard and provides a generic way of patching resources which is not possible with the PolicyGenTemplate API. Using PolicyGenTemplate resources to manage and deploy polices will be deprecated in an upcoming OpenShift Container Platform release.

For more information, see Configuring managed cluster policies by using PolicyGenerator resources.

With this release, Topology Aware Lifecycle Manager (TALM) uses a Red Hat Advanced Cluster Management (RHACM) feature to remediate inform policies on managed clusters. This enhancement removes the need for the Operator to create enforce copies of inform policies during policy remediation. This enhancement also reduces the workload on the hub cluster due to copied policies, and can reduce the overall time required to remediate policies on managed clusters.

For more information, see Update policies on managed clusters.

With this release, you can reduce the time taken for cluster installation by using accelerated provisioning of GitOps ZTP for single-node OpenShift. Accelerated ZTP speeds up installation by applying Day 2 manifests derived from policies at an earlier stage.

The benefits of accelerated provisioning of GitOps ZTP increase with the scale of your deployment. Full acceleration gives more benefit on a larger number of clusters. With a smaller number of clusters, the reduction in installation time is less significant.

For more information, see Accelerated provisioning of GitOps ZTP.

With this release, you can use the Lifecycle Agent to orchestrate an image-based upgrade for single-node OpenShift clusters from OpenShift Container Platform <4.y> to <4.y+2>, and <4.y.z> to <4.y.z+n>. The Lifecycle Agent generates an Open Container Initiative (OCI) image that matches the configuration of participating clusters. In addition to the OCI image, the image-based upgrade uses the ostree library and the OADP Operator to reduce upgrade and service outage duration when transitioning between the original and target platform versions.

For more information, see Understanding the image-based upgrade for single-node OpenShift clusters.

You can now enable IPsec encryption in managed single-node OpenShift clusters that you deploy with GitOps ZTP and Red Hat Advanced Cluster Management (RHACM). You can encrypt external traffic between pods and IPsec endpoints external to the managed cluster. All pod-to-pod network traffic between nodes on the OVN-Kubernetes cluster network is encrypted with IPsec in Transport mode.

For more information, see Configuring IPsec encryption for single-node OpenShift clusters using GitOps ZTP and SiteConfig resources.

Hosted control planes for OpenShift Container Platform 4.16 is now Generally Available on the AWS platform.

In Red Hat Enterprise Linux (RHEL) 9, the default mode is cgroup v2. When Red Hat Enterprise Linux (RHEL) 10 is released, systemd will not support booting in the cgroup v1 mode and only cgroup v2 mode will be available. As such, cgroup v1 is deprecated in OpenShift Container Platform 4.16 and later. cgroup v1 will be removed in a future OpenShift Container Platform release.

The Cluster Samples Operator is deprecated with the OpenShift Container Platform 4.16 release. The Cluster Samples Operator will stop managing and providing support to the non-S2I samples (image streams and templates). No new templates, samples or non-Source-to-Image (Non-S2I) image streams will be added to the Cluster Samples Operator. However, the existing S2I builder image streams and templates will continue to receive updates until the Cluster Samples Operator is removed in a future release.

With this release, installation of package-based RHEL worker nodes is deprecated. In a subsequent future release, RHEL worker nodes will be removed and no longer supported.

RHCOS image layering will replace this feature and supports installing additional packages on the base operating system of your worker nodes.

For more information about image layering, see RHCOS image layering.

The Red Hat-supported version of the Operator SDK CLI tool, including the related scaffolding and testing tools for Operator projects, is deprecated and is planned to be removed in a future release of OpenShift Container Platform. Red Hat will provide bug fixes and support for this feature during the current release lifecycle, but this feature will no longer receive enhancements and will be removed from future OpenShift Container Platform releases.

The Red Hat-supported version of the Operator SDK is not recommended for creating new Operator projects. Operator authors with existing Operator projects can use the version of the Operator SDK CLI tool released with OpenShift Container Platform 4.16 to maintain their projects and create Operator releases targeting newer versions of OpenShift Container Platform.

The following related base images for Operator projects are not deprecated. The runtime functionality and configuration APIs for these base images are still supported for bug fixes and for addressing CVEs.

For information about the unsupported, community-maintained, version of the Operator SDK, see Operator SDK (Operator Framework).

The preserveBootstrapIgnition parameter for Amazon Web Services in the install-config.yaml file has been deprecated. You can use the bestEffortDeleteIgnition parameter instead.

The nodes.diskPartition section in the SiteConfig custom resource (CR) is deprecated with the OpenShift Container Platform 4.16 release. This configuration has been replaced with the ignitionConfigOverride method, which provides a more flexible way of creating a disk partition for any use case.

For more information, see Configuring disk partitioning with SiteConfig.

OpenShift Container Platform 4.16 removes platform Operators (Technology Preview) and plain bundles (Technology Preview), which were prototypes for Operator Lifecycle Manager (OLM) 1.0 (Technology Preview).

OpenShift Container Platform 4.16 supports baseboard management controller (BMC) addressing with Dell servers as documented in BMC addressing for Dell iDRAC. Specifically, it supports idrac-virtualmedia, redfish, and ipmi. In previous versions, idrac was included, but not documented or supported. In OpenShift Container Platform 4.16, idrac has been removed.

With this release, the dedicated service monitors feature for core platform monitoring has been removed. You can no longer enable this feature in the cluster-monitoring-config config map object in the openshift-monitoring namespace. To replace this feature, Prometheus functionality has been improved to ensure that alerts and time aggregations are accurate. This improved functionality is active by default and makes the dedicated service monitors feature obsolete.

With this release, the Prometheus Adapter component for core platform monitoring has been removed. It has been replaced by the new Metrics Server component.

The MetalLB AddressPool custom resource definition (CRD) has been deprecated for several versions. However, in this release, the CRD is completely removed. The sole supported method of configuring MetalLB address pools is by using the IPAddressPools CRD.

With this release, the documentation for the Service Binding Operator (SBO) has been removed because this Operator is no longer supported.

OpenShift Container Platform 4.16 no longer supports AliCloud Container Storage Interface (CSI) Driver Operator.

Kubernetes 1.29 removed the following deprecated APIs, so you must migrate manifests and API clients to use the appropriate API version. For more information about migrating removed APIs, see the Kubernetes documentation.

OpenShift Container Platform 4.16 removes support for managing machines with Machine API for Alibaba Cloud clusters. This change includes removing support for the cloud controller manager for Alibaba Cloud, which was previously a Technology Preview feature.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster by using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

The following enhancement is included in this z-stream release:

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

The following enhancements are included in this z-stream release:

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

The following enhancements are included in this z-stream release:

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

The following enhancements are included in this z-stream release:

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.

To update an existing OpenShift Container Platform 4.16 cluster to this latest release, see Updating a cluster using the CLI.