Chapter 4. Installing a cluster on OpenStack with Kuryr

Procedure

1. If you are using the local registry, create a template to upload the images to the registry. For example:

   (undercloud) $ openstack overcloud container image prepare \
   -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
   --namespace=registry.access.redhat.com/rhosp13 \
   --push-destination=<local-ip-from-undercloud.conf>:8787 \
   --prefix=openstack- \
   --tag-from-label {version}-{product-version} \
   --output-env-file=/home/stack/templates/overcloud_images.yaml \
   --output-images-file /home/stack/local_registry_images.yaml

2. Verify that the local_registry_images.yaml file contains the Octavia images. For example:

   ...
   - imagename: registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-43
     push_destination: <local-ip-from-undercloud.conf>:8787
   - imagename: registry.access.redhat.com/rhosp13/openstack-octavia-health-manager:13.0-45
     push_destination: <local-ip-from-undercloud.conf>:8787
   - imagename: registry.access.redhat.com/rhosp13/openstack-octavia-housekeeping:13.0-45
     push_destination: <local-ip-from-undercloud.conf>:8787
   - imagename: registry.access.redhat.com/rhosp13/openstack-octavia-worker:13.0-44
     push_destination: <local-ip-from-undercloud.conf>:8787

   Note

   The Octavia container versions vary depending upon the specific RHOSP release installed.

3. Pull the container images from registry.redhat.io to the Undercloud node:

   (undercloud) $ sudo openstack overcloud container image upload \
     --config-file  /home/stack/local_registry_images.yaml \
     --verbose

   This may take some time depending on the speed of your network and Undercloud disk.

4. Install or update your Overcloud environment with Octavia:

   $ openstack overcloud deploy --templates \
     -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
     -e octavia_timeouts.yaml

   Note

   This command only includes the files associated with Octavia; it varies based on your specific installation of RHOSP. See the RHOSP documentation for further information. For more information on customizing your Octavia installation, see installation of Octavia using Director.

   Note

   When leveraging Kuryr SDN, the Overcloud installation requires the Neutron trunk extension. This is available by default on director deployments. Use the openvswitch firewall instead of the default ovs-hybrid when the Neutron backend is ML2/OVS. There is no need for modifications if the backend is ML2/OVN.

1. API load balancer: Provides a common endpoint for users, both human and machine, to interact with and configure the platform. Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
   • A stateless load balancing algorithm. The options vary based on the load balancer implementation.
   Important

   Do not configure session persistence for an API load balancer. Configuring session persistence for a Kubernetes API server might cause performance issues from excess application traffic for your OpenShift Container Platform cluster and the Kubernetes API that runs inside the cluster.

   Configure the following ports on both the front and back of the load balancers:

   Table 4.2. API load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   6443	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane. You must configure the /readyz endpoint for the API server health check probe.	X	X	Kubernetes API server
   22623	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	X		Machine config server

   Note

   The load balancer must be configured to take a maximum of 30 seconds from the time the API server turns off the /readyz endpoint to the removal of the API server instance from the pool. Within the time frame after /readyz returns an error or becomes healthy, the endpoint must have been removed or added. Probing every 5 or 10 seconds, with two successful requests to become healthy and three to become unhealthy, are well-tested values.

2. Application Ingress load balancer: Provides an ingress point for application traffic flowing in from outside the cluster. A working configuration for the Ingress router is required for an OpenShift Container Platform cluster.

   Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
   • A connection-based or session-based persistence is recommended, based on the options available and types of applications that will be hosted on the platform.
   Tip

   If the true IP address of the client can be seen by the application Ingress load balancer, enabling source IP-based session persistence can improve performance for applications that use end-to-end TLS encryption.

   Configure the following ports on both the front and back of the load balancers:

   Table 4.3. Application Ingress load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   443	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTPS traffic
   80	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTP traffic

   Note

   If you are deploying a three-node cluster with zero compute nodes, the Ingress Controller pods run on the control plane nodes. In three-node cluster deployments, you must configure your application Ingress load balancer to route HTTP and HTTPS traffic to the control plane nodes.

Procedure

1. Edit your install-config.yaml file and add the proxy settings. For example:

   apiVersion: v1
   baseDomain: my.domain.com
   proxy:
     httpProxy: http://<username>:<pswd>@<ip>:<port> 1
     httpsProxy: https://<username>:<pswd>@<ip>:<port> 2
     noProxy: example.com 3
   additionalTrustBundle: | 4
       -----BEGIN CERTIFICATE-----
       <MY_TRUSTED_CA_CERT>
       -----END CERTIFICATE-----
   additionalTrustBundlePolicy: <policy_to_add_additionalTrustBundle> 5

   1

   A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be http.

   2

   A proxy URL to use for creating HTTPS connections outside the cluster.

   3

   A comma-separated list of destination domain names, IP addresses, or other network CIDRs to exclude from proxying. Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass the proxy for all destinations.

   4

   If provided, the installation program generates a config map that is named user-ca-bundle in the openshift-config namespace that contains one or more additional CA certificates that are required for proxying HTTPS connections. The Cluster Network Operator then creates a trusted-ca-bundle config map that merges these contents with the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle, and this config map is referenced in the trustedCA field of the Proxy object. The additionalTrustBundle field is required unless the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.

   5

   Optional: The policy to determine the configuration of the Proxy object to reference the user-ca-bundle config map in the trustedCA field. The allowed values are Proxyonly and Always. Use Proxyonly to reference the user-ca-bundle config map only when http/https proxy is configured. Use Always to always reference the user-ca-bundle config map. The default value is Proxyonly.

   Note

   The installation program does not support the proxy readinessEndpoints field.

   Note

   If the installer times out, restart and then complete the deployment by using the wait-for command of the installer. For example:

   $ ./openshift-install wait-for install-complete --log-level debug

2. Save the file and reference it when installing OpenShift Container Platform.

Procedure

1. From a command line, create the manifest files:

   $ ./openshift-install create manifests --dir <installation_directory> 1

   1

   For <installation_directory>, specify the name of the directory that contains the install-config.yaml file for your cluster.

2. Create a file that is named cluster-network-03-config.yml in the <installation_directory>/manifests/ directory:

   $ touch <installation_directory>/manifests/cluster-network-03-config.yml 1

   1

   For <installation_directory>, specify the directory name that contains the manifests/ directory for your cluster.

   After creating the file, several network configuration files are in the manifests/ directory, as shown:

   $ ls <installation_directory>/manifests/cluster-network-*

   Example output

   cluster-network-01-crd.yml
   cluster-network-02-config.yml
   cluster-network-03-config.yml

3. Open the cluster-network-03-config.yml file in an editor, and enter a custom resource (CR) that describes the Cluster Network Operator configuration that you want:

   $ oc edit networks.operator.openshift.io cluster

4. Edit the settings to meet your requirements. The following file is provided as an example:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     serviceNetwork:
     - 172.30.0.0/16
     defaultNetwork:
       type: Kuryr
       kuryrConfig:
         enablePortPoolsPrepopulation: false 1
         poolMinPorts: 1 2
         poolBatchPorts: 3 3
         poolMaxPorts: 5 4
         openstackServiceNetwork: 172.30.0.0/15 5

   1

   Set enablePortPoolsPrepopulation to true to make Kuryr create new Neutron ports when the first pod on the network for pods is created in a namespace. This setting raises the Neutron ports quota but can reduce the time that is required to spawn pods. The default value is false.

   2

   Kuryr creates new ports for a pool if the number of free ports in that pool is lower than the value of poolMinPorts. The default value is 1.

   3

   poolBatchPorts controls the number of new ports that are created if the number of free ports is lower than the value of poolMinPorts. The default value is 3.

   4

   If the number of free ports in a pool is higher than the value of poolMaxPorts, Kuryr deletes them until the number matches that value. Setting this value to 0 disables this upper bound, preventing pools from shrinking. The default value is 0.

   5

   The openStackServiceNetwork parameter defines the CIDR range of the network from which IP addresses are allocated to RHOSP Octavia’s LoadBalancers.

   If this parameter is used with the Amphora driver, Octavia takes two IP addresses from this network for each load balancer: one for OpenShift and the other for VRRP connections. Because these IP addresses are managed by OpenShift Container Platform and Neutron respectively, they must come from different pools. Therefore, the value of openStackServiceNetwork must be at least twice the size of the value of serviceNetwork, and the value of serviceNetwork must overlap entirely with the range that is defined by openStackServiceNetwork.

   The CNO verifies that VRRP IP addresses that are taken from the range that is defined by this parameter do not overlap with the range that is defined by the serviceNetwork parameter.

   If this parameter is not set, the CNO uses an expanded value of serviceNetwork that is determined by decrementing the prefix size by 1.

5. Save the cluster-network-03-config.yml file, and exit the text editor.
6. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program deletes the manifests/ directory while creating the cluster.

Procedure

1. Using the Red Hat OpenStack Platform (RHOSP) CLI, create the API FIP:

   $ openstack floating ip create --description "API <cluster_name>.<base_domain>" <external_network>

2. Using the Red Hat OpenStack Platform (RHOSP) CLI, create the apps, or Ingress, FIP:

   $ openstack floating ip create --description "Ingress <cluster_name>.<base_domain>" <external_network>

3. Add records that follow these patterns to your DNS server for the API and Ingress FIPs:

   api.<cluster_name>.<base_domain>.  IN  A  <API_FIP>
   *.apps.<cluster_name>.<base_domain>. IN  A <apps_FIP>

   Note

   If you do not control the DNS server, you can access the cluster by adding the cluster domain names such as the following to your /etc/hosts file:

   • <api_floating_ip> api.<cluster_name>.<base_domain>
   • <application_floating_ip> grafana-openshift-monitoring.apps.<cluster_name>.<base_domain>
   • <application_floating_ip> prometheus-k8s-openshift-monitoring.apps.<cluster_name>.<base_domain>
   • <application_floating_ip> oauth-openshift.apps.<cluster_name>.<base_domain>
   • <application_floating_ip> console-openshift-console.apps.<cluster_name>.<base_domain>
   • application_floating_ip integrated-oauth-server-openshift-authentication.apps.<cluster_name>.<base_domain>

   The cluster domain names in the /etc/hosts file grant access to the web console and the monitoring interface of your cluster locally. You can also use the kubectl or oc. You can access the user applications by using the additional entries pointing to the <application_floating_ip>. This action makes the API and applications accessible to only you, which is not suitable for production deployment, but does allow installation for development and testing.

4. Add the FIPs to the install-config.yaml file as the values of the following parameters:

   • platform.openstack.ingressFloatingIP
   • platform.openstack.apiFloatingIP