Reference Architecture for deploying Red Hat OpenShift Container Platform on Red Hat OpenStack Platform

With installer-provisioned infrastructure (IPI), the installer manages all aspects of the installation, including infrastructure provisioning with an opinionated best practices deployment of OpenShift. The installer is able to request, deploy, and manage the underlying infrastructure directly via an infrastructure-aware provider that has knowledge of that infrastructure. IPI results in production-ready installations from minimal configuration, which provides a managed service experience to deploying self-managed clusters.

IPI is also capable of using pre-existing networking infrastructure.

Figure 3.1. The OpenShift IPI workflow

With the user-provisioned infrastructure (UPI) installation method, administrators are responsible for creating and managing their own underlying infrastructure before installation. The installer uses an infrastructure-aware provider, but the administrator must prepare the components in advance. The OpenShift installer can then use the infrastructure provider to interact with the prepared infrastructure. For OpenShift on OpenStack, Red Hat provides sample Ansible scripts for this purpose.

A platform agnostic installation can be used on any underlying infrastructure because it does not use any infrastructure provider. The installer is not able to control any of the underlying infrastructure operations, which means that an administrator must both prepare the infrastructure and orchestrate the installation via their own custom methods. This type of installation is the most flexible, but requires additional custom automation as it does not come with any cloud integration.

Installer-provisioned infrastructure, user-provisioned infrastructure, and platform agnostic installation methods are fully supported by Red Hat.

For a comprehensive overview of installation patterns and platforms, see Selecting a cluster installation method and preparing it for users in the official documentation.

When you deploy OpenShift on OpenStack, you must consider the placement of OpenShift nodes carefully:

Red Hat offers a few supported ways to manage high availability for the OpenShift on OpenStack control plane VMs:

Using OpenStack availability zones is the preferred way to ensure that control plane nodes are properly managed for HA.

Figure 3.2. Placing OpenShift control plane nodes across compute hosts using nova availability zones.

In OpenStack, the concept of availability zones allow administrators to logically partition their cloud and provide a simple abstraction of the physical infrastructure for cloud tenants to easily use. Availability zones are very granular in OpenStack. Some OpenStack services, such as nova, cinder, and neutron, have their own availability zone implementations.

For an HA OpenShift control plane, an openstack cloud administrator must provide a minimum of three availability zones and ensure that compute resources offered are properly configured.

OpenShift 4.12 requires that the control plane nodes belong to the same L2 network. When distributing the control plane nodes across three availability zones, we typically connect them to a stretched L2 provider network. It usually involves configuring the infrastructure with a small and dedicated L2 network segmented with VLAN or VXLAN or Geneve protocols with failure domain fabrics able to interconnect this network. The latency between the zones has to be less than 100ms.

When using compute availability zones, you should have corresponding volume availability zones with the same name. This is a requirement when using CSI topology awareness, which is enabled in OpenShift 4.12.

For an IPI install, you must add control plane availability zones to the install-config.yaml file manually before installation as the guided install does not prompt for them. Zones are added to the configuration file as a YAML array.

controlPlane:
  name: master
  platform:
    openstack:
      zones: ['AZ0', 'AZ1', 'AZ2']
  replicas: 3

At install time, the OpenShift installer assigns each control plane node to a different AZ upon VM creation.

OpenStack supports the ability to deterministically place instances through affinity policies. Affinity rules are grouped together into detailed policies allowing an administrator to control placement. Through the use of these policies and OpenStack server groups, deployments without nova availability zones still remain HA.

The OpenShift on OpenStack installer creates server groups for both control-plane and compute nodes. These server groups are then used with an soft-anti-affinity policy to request the instances to land on different OpenStack compute hosts.

There are two ways to use availability zones for worker nodes:

For worker nodes, the installer has the following placement rules:

The following diagram shows one way that you can configure your nodes into availability zones:

Figure 3.3. An example of OpenShift control plane and worker nodes deployed across three nova availability zones.

The worker nodes run different types of workload than the control-plane nodes which allows them to take advantage of root-volumes backed by Ceph.

To do this, explicitly set the rootVolume for the workers in the compute section of the OpenShift install-config.yaml installation file. This will instruct the installer to use Ceph and not ephemeral disks like the control plane nodes.

Additionally, volume zones should match the compute zones names:

compute:
- name: worker
  platform:
    openstack:
      zones: ['AZ0', 'AZ1', 'AZ2']
      rootVolume:
        size: 100
        zones: ['AZ0', 'AZ1', 'AZ2']
  replicas: 3

By default, a Red Hat Ceph Storage (RHCS) deployment is highly available. For this solution, we deploy RHCS with director. Deploying RHCS this way can automatically integrate and provide HA storage for the Compute, Image, Block Storage and Object Storage services.

Implementing high availability for our solution for OpenStack is done through multiple components:

To implement external load balancing for your OpenShift applications, OpenStack provides the Octavia load balancing solution. Octavia is the fully supported and tested load balancing-as-a-service component for Red Hat OpenStack Platform 16.2.

We use nova availability zones and anti-affinity policies to implement high availability for our OpenShift solution. We place each controller node in a separate availability zone to ensure that they are never on the same physical servers.

Worker nodes are distributed as evenly as possible across the same availability zones, which ensures an even distribution across physical infrastructure. We use anti-affinity policies to request them to land on different physical servers. The worker nodes use root volumes with different types per failure domain. The cinder availability zone names match the nova availability zone names.

Red Hat OpenStack Platform 16.2 uses Red Hat Ceph Storage (RHCS) 4 as part of a director driven storage solution. The BlueStore back end is used by default. Installations of RHCS 4 and later only support the use of the more performant BlueStore back end.

For more information about the storage back ends supported by RHCS, see Ceph ObjectStore in the Architecture Guide.

For more information about configuring RHCS, see Supported Configurations.

The OpenShift installer follows the recommended practice for a scaled registry and attempts to use OpenStack’s object storage service as the preferred location for the internal image registry back end. To do this, the installer checks for an accessible object storage location and uses that if found. If it cannot find one, it uses the recommended best practice for a non-scaled registry and creates a ReadWriteOnce (RWO) cinder volume.

OpenShift supports the OpenStack Manila CSI Driver Operator, so that OpenShift can interact with OpenStack’s Shared File Systems service (manila) to provision PVs from remote, shareable file systems.

Accessing remote shared file systems by using the Shared File Systems service means that container workloads can use a storage back end that allows simultaneous access for compute instances, bare metal nodes, and containers. This kind of access, known to an OpenShift PV as its access mode, is called ReadWriteMany (RWX), and is required for complex container use cases.

With Manila-CSI, an OpenShift user can consume a remote shared file system easily via the OpenStack API and the Shared File Systems service.

Figure 3.4. Using Manila for Container Workloads

The OpenShift installer creates a storage class automatically if OpenShift detects Manila in the underlying OpenStack cloud. The installer does not set that storage class as the default.

$ oc get sc
NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-default     manila.csi.openstack.org   Delete          Immediate              false                  37d
standard-csi (default) cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   37d

For more information on setting a default StorageClass, see Changing the default storage class.

Openshift supports the Cinder CSI driver to provision volumes. PVs provisioned with this driver are dynamically allocated. These volumes are typically used with ReadWriteOnce (RWO) Access Mode, although it is possible to attach the volumes to multiple instances at the same time provided they use a multiattach volume type. It is generally better for applications requiring block storage and/or lower latency.

This document takes this into account and uses the best options where available for their purpose.

The control plane VMs must meet the known resource requirements to run the etcd key-value store. Low write latency is a requirement for etcd stability and performance. Red Hat requires that all controllers have access to fast disk (SSD or better) to ensure stability and guarantee supportability. Note that regardless of disk technology used by OSDs, using Ceph requires additional consideration when used with OpenShift control planes. If you require Ceph for your control plane instances please contact Red Hat support for guidance and supportability details.

To learn more about the etcd disk requirements, see the etcd documentation.

You have several options for providing storage to your cluster. To help you make a decision, we highlight the following three options:

# cat /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml
...
CinderEnableIscsiBackend: false
CinderEnableRbdBackend: true
CinderBackupBackend: ceph
NovaEnableRbdBackend: true
GlanceBackend: rbd
...

Like many other OpenStack components, the networking subsystem (neutron) is pluggable so that customers can choose the solution that best fits their business and technological requirements. For Red Hat OpenStack Platform, customers can choose from supported options.

The default neutron back end implemented by Red Hat OpenStack Platform 16.2 is OVN.

You should choose a neutron back end that meets your unique circumstances.

When planning for OpenShift 4.12 on OpenStack 16.2, we strongly recommend the use of the default back end of OVN as it’s the most tested configuration and allows for using the Octavia OVN provider for LoadBalancer Services if desired.

To assist with choosing a neutron back end when deploying OpenShift on OpenStack, consider the following:

Red Hat OpenStack Platform uses the Octavia project for its Load Balancing as a Service (LBaaS) implementation. In OSP 16.2 deployed with OVN there are two Octavia backends available, the default Amphora and OVN Provider. The main difference is that Amphora deploys each load balancer by creating a VM and configuring a HAProxy instance on it, while OVN Provider implements them using OpenFlow rules.

By default OpenShift will use Amphora as the Octavia backend.

For more details on the OVN Octavia Provider, including its limitations, review the upstream documentation and consult a Red Hat Solution Architect or Support Associate.

In our solution, we use the default neutron networking back end of Red Hat OpenStack Platform 16.2: OVN.

LoadBalancer Services in OpenShift are handled by cloud-provider-openstack and OpenStack Octavia. For each service of type LoadBalancer, a load balancer will be created in Octavia. By default OpenShift uses Amphora backend as a more feature-rich solution. The downside of this is a higher resource consumption as Amphora creates a VM for each of the load balancers.

As load balancer services tend to be fairly static, we have found that Amphora is a good choice for most use cases.

Please also note that Amphora always sets the source IP for the traffic incoming to pods to the IP of the load balancer. This means that even with .spec.externalTrafficPolicy set to Local you will not be able to get the original source IP without using additional features such as PROXY protocol.

If resource consumption is a concern, you can configure to your Openshift cluster to use the OVN Octavia Provider. Before changing the Octavia Provider, you need to consider these limitations:

With OSP 16.2 and OpenShift 4.12 we recommend that you use the default of Amphora as the Octavia backend used by cloud-provider-openstack.

For our solution, we use OVN-Kubernetes for OpenShift networking and Amphora as Octavia backend.

A reachable IP address is required to be in DNS before you install the cluster. The following address spaces must resolve:

api.<cluster name>.<base domain>

The installer can automatically associate an existing floating IP to the API port via the apiFloatingIP value of the OpenStack platform section of install-config.yaml:

platform:
  openstack:
...
    apiFloatingIP:     "10.46.43.176"

This domain is used for access to the applications running in OpenShift. In your DNS, you create a wildcard entry to resolve the following naming structure:

*.apps.<cluster name>.<base domain>.

You can automatically associate this OpenShift apps IP to your cluster by using the ingressFloatingIP value in your install-config.yaml. Do not confuse ingressVIP with ingressFloatingIP: ingressVIP is for the Machine network and ingressFloatingIP is from the external network.

The bootstrap node must be able to resolve registry domain names directly because it retrieves the basic resources to stand up the bootstrap and production clusters.

When installing OpenShift on OpenStack, the installer creates the necessary OpenStack security groups for a functional installation. The installer creates security groups for the control plane nodes and the worker nodes.

You should review the security groups for a clear understanding of how they might affect your own internal security requirements.

For details of the rules created, you can review the upstream code for both control plane nodes and worker nodes.

For details on how these groups are used, see Security Group Rules in the upstream Cluster API documentation for OpenStack.

To learn how to add additional security groups directly at install time review the Optional RHOSP configuration parameters section of the installation guide.