Red Hat SummitChapter 22. Configuring for OpenStack
22.1. Overview
When deployed on OpenStack, OpenShift Container Platform can be configured to access the OpenStack infrastructure, including using OpenStack Cinder volumes as persistent storage for application data.
OpenShift Container Platform 3.11 is supported for use with Red Hat OpenStack Platform 13.
The latest OpenShift Container Platform release supports both the latest Red Hat OpenStack Platform long life release and intermediate release. The release cycles of OpenShift Container Platform and Red Hat OpenStack Platform are different and versions tested may vary in the future depending on the release dates of both products.
22.2. Before you Begin
22.2.1. OpenShift Container Platform SDN
The default OpenShift Container Platform SDN is OpenShiftSDN. There is another option: use Kuryr SDN.
22.2.2. Kuryr SDN
Kuryr is a CNI plug-in that uses Neutron and Octavia to provide networking for pods and services. It is primarily designed for OpenShift Container Platform clusters that run on OpenStack virtual machines. Kuryr improves the network performance by plugging OpenShift Container Platform pods into OpenStack SDN. In addition it provides interconnectivity between OpenShift Container Platform pods and OpenStack virtual instances.
Kuryr is recommended for OpenShift Container Platform deployments on encapsulated OpenStack tenant networks in order to avoid double encapsulation, such as running an encapsulated OpenShift SDN over an OpenStack network. Kuryr is recommended whenever VXLAN, GRE, or GENEVE are required.
Conversely, implementing Kuryr does not make sense in the following cases:
- You use provider networks, tenant VLANs, or a third party commercial SDN such as Cisco ACI or Juniper Contrail.
- The deployment will use many services on a few hypervisors, or OpenShift Container Platform virtual machine nodes. Each OpenShift Container Platform service creates an Octavia Amphora virtual machine in OpenStack that hosts a required load balancer.
To enable Kuryr SDN, your environment must meet the following requirements:
- Running OpenStack 13 or later
- Overcloud with Octavia
- Neutron Trunk ports extension enabled
- If ML2/OVS Neutron driver is used the OpenvSwitch firewall driver must be used, instead of the ovs-hybrid one.
To use Kuryr with OpenStack 13.0.13, the Kuryr container images must be version 3.11.306 or higher.
22.2.3. OpenShift Container Platform Prerequisites
A successful deployment of OpenShift Container Platform requires many prerequisites. This consists of a set of infrastructure and host configuration steps prior to the actual installation of OpenShift Container Platform using Ansible. In the following subsequent sections, details regarding the prerequisites and configuration changes required for an OpenShift Container Platform on a OpenStack environment are discussed in detail.
All of the OpenStack CLI commands in this reference environment are executed using the CLI openstack commands within a different node from the director node. The commands are executed in the other node to avoid package conflicts with Ansible version 2.6 and above. Be sure to install the following packages in the specified repositories.
Example:
Enable the rhel-7-server-openstack-13-tools-rpms and the required OpenShift Container Platform repositories from Set Up Repositories.
sudo subscription-manager repos \
--enable rhel-7-server-openstack-{rhosp_version}-tools-rpms \
--enable rhel-7-server-openstack-14-tools-rpms
sudo subscription-manager repo-override --repo=rhel-7-server-openstack-14-tools-rpms --add=includepkgs:"python2-openstacksdk.* python2-keystoneauth1.* python2-os-service-types.*"
sudo yum install -y python2-openstackclient python2-heatclient python2-octaviaclient ansible
$ sudo subscription-manager repos \
--enable rhel-7-server-openstack-{rhosp_version}-tools-rpms \
--enable rhel-7-server-openstack-14-tools-rpms
$ sudo subscription-manager repo-override --repo=rhel-7-server-openstack-14-tools-rpms --add=includepkgs:"python2-openstacksdk.* python2-keystoneauth1.* python2-os-service-types.*"
$ sudo yum install -y python2-openstackclient python2-heatclient python2-octaviaclient ansible
Verify the packages are of at least the following versions (use rpm -q <package_name>):
-
python2-openstackclient-3.14.1.-1 -
python2-heatclient1.14.0-1 -
python2-octaviaclient1.4.0-1 -
python2-openstacksdk0.17.2
22.2.3.1. Enabling Octavia: OpenStack Load Balancing as a Service (LBaaS)
Octavia is a supported load balancer solution that is recommended to be used in conjunction with OpenShift Container Platform in order to load balance the external incoming traffic and provide a single view of the OpenShift Container Platform master services for the applications.
In order to enable Octavia, the Octavia service must be included during the installation of the OpenStack overcloud or upgraded if the overcloud already exists. The following steps provide basic non-custom steps in enabling Octavia and apply to both either a clean install of the overcloud or an overcloud update.
The following steps only capture the key pieces required during the deployment of OpenStack when dealing with Octavia. For more information visit the documentation of Installation of OpenStack. It is also important to note that registry methods vary. For more information visit the documentation on Registry Methods. This example used the local registry method.
If using the local registry, create a template to upload the images to the registry. Example shown below.
Verify that the created local_registry_images.yaml contains the Octavia images.
Octavia images in local registry file
The versions of the Octavia containers will vary depending upon the specific Red Hat OpenStack Platform release installed.
The following step pulls the container images from registry.redhat.io to the undercloud node. This process might take some time depending on the speed of the network and undercloud disk.
(undercloud) $ sudo openstack overcloud container image upload \ --config-file /home/stack/local_registry_images.yaml \ --verbose
(undercloud) $ sudo openstack overcloud container image upload \
--config-file /home/stack/local_registry_images.yaml \
--verboseAs an Octavia Load Balancer is used to access the OpenShift API, there is a need to increase their listeners default timeouts for the connections. The default timeout is 50 seconds. Increase the timeout to 20 minutes by passying the following file to the overcloud deploy command:
(undercloud) $ cat octavia_timeouts.yaml parameter_defaults: OctaviaTimeoutClientData: 1200000 OctaviaTimeoutMemberData: 1200000
(undercloud) $ cat octavia_timeouts.yaml
parameter_defaults:
OctaviaTimeoutClientData: 1200000
OctaviaTimeoutMemberData: 1200000This is not needed from Red Hat OpenStack Platform 14 and onwards.
Install or update your overcloud environment with Octavia:
The command above only includes the files associated with Octavia. This command will vary based upon your specifc installation of OpenStack. See the official OpenStack documentation for further information. For more information on customizing your Octavia installation, see installation of Octavia using Director.
If Kuryr SDN is used, the overcloud installation requires the "trunk" extension to be enabled at Neutron. This is enabled by default on Director deployments. Use the openvswitch firewall instead of the default ovs-hybrid when the Neutron backend is ML2/OVS. There is no need for modifications if the backend is ML2/OVN.
22.2.3.2. Creating OpenStack User Accounts, Projects, and Roles
Before installing OpenShift Container Platform, the Red Hat OpenStack Platform (RHOSP) environment requires a project, often referred to as a tenant, that stores the OpenStack instances that are to install the OpenShift Container Platform. This project requires ownership by a user and the role of that user to be set to _member_.
The following steps show how to accomplish the above.
As the OpenStack overcloud administrator,
Create a project (tenant) that is to store the RHOSP instances
openstack project create <project>
$ openstack project create <project>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a RHOSP user that has ownership of the previously created project:
openstack user create --password <password> <username>
$ openstack user create --password <password> <username>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the role of the user:
openstack role add --user <username> --project <project> _member_
$ openstack role add --user <username> --project <project> _member_Copy to Clipboard Copied! Toggle word wrap Toggle overflow
The default quotas assigned to new RH OSP projects are not high enough for OpenShift Container Platform installations. Increase the quotas to at least 30 security groups, 200 security group rules, and 200 ports.
openstack quota set --secgroups 30 --secgroup-rules 200 --ports 200 <project>
$ openstack quota set --secgroups 30 --secgroup-rules 200 --ports 200 <project>
- 1
- For
<project>, specify the name of the project to modify
22.2.3.3. Extra steps for Kuryr SDN
If Kuryr SDN is enabled, especially if you are using namespace isolation, increase your project’s quotas to meet these minimum requirements:
- 300 security groups - one for each namespace plus one for each load balancer
- 150 networks - one for each namespace
- 150 subnets - one for each namespace
- 500 security group rules
- 500 ports - one port per Pod and additional ports for pools to speed up Pod creation
This is not a global recommendation. Adjust your quotas to meet your requirements.
If you are using namespace isolation, each namespace is given a new network and subnet. Additionally, a security group is created to enable traffic between Pods in the namespace.
openstack quota set --networks 150 --subnets 150 --secgroups 300 --secgroup-rules 500 --ports 500 <project>
$ openstack quota set --networks 150 --subnets 150 --secgroups 300 --secgroup-rules 500 --ports 500 <project>
- 1
- For
<project>, specify the name of the project to modify
If you enabled namespace isolation, you must add the project ID to the octavia.conf configuration file after you create the project. This step ensures that required LoadBalancer security groups belong to that project and that they can be updated to enforce services isolation across namespaces.
Get the project ID
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add the project ID to [filename]octavia.conf on the controllers and restart octavia worker.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
22.2.3.4. Configuring the RC file
After you configure the project, an OpenStack administrator can create an RC file with all the required information to the user(s) implementing the OpenShift Container Platform environment.
An example RC file:
Changing _OS_PROJECT_DOMAIN_NAME and _OS_USER_DOMAIN_NAME from the Default value is supported as long as both reference the same domain.
As the user(s) implementing the OpenShift Container Platform environment, within the OpenStack director node or workstation, ensure to source the credentials as follows:
source path/to/examplerc
$ source path/to/examplerc22.2.3.5. Create an OpenStack Flavor
Within OpenStack, flavors define the size of a virtual server by defining the compute, memory, and storage capacity of nova computing instances. Since the base image within this reference architecture is Red Hat Enterprise Linux 7.5, a m1.node and m1.master sized flavor is created with the following specifications as shown in Table 22.1, “Minimum System Requirements for OpenShift”.
Although the minimum system requirements are sufficient to run a cluster, to improve performance, it is recommended to increase vCPU on master nodes. Additionally, more memory is recommended if etcd is co-located on the master nodes.
| Node Type | CPU | RAM | Root Disk | Flavor |
|---|---|---|---|---|
| Masters | 4 | 16 GB | 45 GB |
|
| Nodes | 1 | 8 GB | 20 GB |
|
As an OpenStack administrator,
openstack flavor create <flavor_name> \
--id auto \
--ram <ram_in_MB> \
--disk <disk_in_GB> \
--vcpus <num_vcpus>
$ openstack flavor create <flavor_name> \
--id auto \
--ram <ram_in_MB> \
--disk <disk_in_GB> \
--vcpus <num_vcpus>An example below showing the creation of flavors within this reference environment.
If access to OpenStack administrator privileges to create new flavors is unavailable, use existing flavors within the OpenStack environment that meet the requirements in Table 22.1, “Minimum System Requirements for OpenShift”.
Verification of the OpenStack flavors via:
openstack flavor list
$ openstack flavor list22.2.3.6. Creating an OpenStack Keypair
Red Hat OpenStack Platform uses cloud-init to place an ssh public key on each instance as it is created to allow ssh access to the instance. Red Hat OpenStack Platform expects the user to hold the private key.
Losing the private key will cause the inability to access the instances.
To generate a keypair, use the following command:
openstack keypair create <keypair-name> > /path/to/<keypair-name>.pem
$ openstack keypair create <keypair-name> > /path/to/<keypair-name>.pemVerification of the keypair creation can be done via:
openstack keypair list
$ openstack keypair list
Once the keypair is created, set the permissions to 600 thus only allowing the owner of the file to read and write to that file.
chmod 600 /path/to/<keypair-name>.pem
$ chmod 600 /path/to/<keypair-name>.pem22.2.3.7. Setting up DNS for OpenShift Container Platform
DNS service is an important component in the OpenShift Container Platform environment. Regardless of the provider of DNS, an organization is required to have certain records in place to serve the various OpenShift Container Platform components.
Using /etc/hosts is not valid, a proper DNS service must exist.
Using the key secret of the DNS, you can provide the information to the OpenShift Ansible Installer and it will automatically add A records for the target instances and the various OpenShift Container Platform components. This process setup is described later when configuring the OpenShift Ansible Installer.
Access to a DNS server is expected. You can use Red Hat Labs DNS Helper for assistance with access.
Application DNS
Applications served by OpenShift are accessible by the router on ports 80/TCP and 443/TCP. The router uses a wildcard record to map all host names under a specific sub domain to the same IP address without requiring a separate record for each name.
This allows OpenShift Container Platform to add applications with arbitrary names as long as they are under that sub domain.
For example, a wildcard record for *.apps.example.com causes DNS name lookups for tax.apps.example.com and home-goods.apps.example.com to both return the same IP address: 10.19.x.y. All traffic is forwarded to the OpenShift Routers. The Routers examine the HTTP headers of the queries and forward them to the correct destination.
With a load-balancer such as Octavia, host address of 10.19.x.y, the wildcard DNS record can be added as follows:
| IP Address | Hostname | Purpose |
|---|---|---|
| 10.19.x.y |
| User access to application web services |
22.2.3.8. Creation of OpenShift Container Platform Networks via OpenStack
When deploying OpenShift Container Platform on Red Hat OpenStack Platform as described in this segment, the requirements are two networks — public and internal network.
Public Network
The public network is a network that contains external access and can be reached by the outside world. The public network creation can be only done by an OpenStack administrator.
The following commands provide an example of creating an OpenStack provider network for public network access.
As an OpenStack administrator (overcloudrc access),
Once the network and subnet have been created verify via:
openstack network list openstack subnet list
$ openstack network list
$ openstack subnet list
<float_start_ip> and <float_end_ip> are the associated floating IP pool provided to the network labeled public network. The Classless Inter-Domain Routing (CIDR) uses the format <ip>/<routing_prefix>, i.e. 10.0.0.1/24.
Internal Network
The internal network is connected to the public network via a router during the network setup. This allows each Red Hat OpenStack Platform instance attached to the internal network the ability to request a floating IP from the public network for public access. The internal network is created automically by the OpenShift Ansible installer via setting the openshift_openstack_private_network_name. More information regarding changes required for the OpenShift Ansible installer are described later.
22.2.3.9. Creating OpenStack Deployment Host Security Group
OpenStack networking allows the user to define inbound and outbound traffic filters that can be applied to each instance on a network. This allows the user to limit network traffic to each instance based on the function of the instance services and not depend on host based filtering. The OpenShift Ansible installer handles the proper creation of all the ports and services required for each type of host that is part of the OpenShift Container Platform cluster except for the deployment host.
The following command creates an empty security group with no rules set for the deployment host.
source path/to/examplerc openstack security group create <deployment-sg-name>
$ source path/to/examplerc
$ openstack security group create <deployment-sg-name>Verify the creation of the security group:
openstack security group list
$ openstack security group listDeployment Host Security Group
The deployment instance only needs to allow inbound ssh. This instance exists to give operators a stable base to deploy, monitor and manage the OpenShift Container Platform environment.
| Port/Protocol | Service | Remote source | Purpose |
|---|---|---|---|
| ICMP | ICMP | Any | Allow ping, traceroute, etc. |
| 22/TCP | SSH | Any | Secure shell login |
Creation of the above security group rules is as follows:
Verification of the security group rules is as follows:
22.2.3.10. OpenStack Cinder Volumes
OpenStack Block Storage provides persistent block storage management via the cinder service. Block storage enables the OpenStack user to create a volume that may be attached to different OpenStack instances.
22.2.3.10.1. Docker Volume
The master and node instances contain a volume to store docker images. The purpose of the volume is to ensure that a large image or container does not compromise node performance or abilities of the existing node.
A docker volume of a minimum of 15GB is required for running containers. This may need adjustment depending on the size and number of containers each node will run.
The docker volume is created by the OpenShift Ansible installer via the variable openshift_openstack_docker_volume_size. More information regarding changes required for the OpenShift Ansible installer are described later.
22.2.3.10.2. Registry volume
The OpenShift image registry requires a cinder volume to ensure that images are saved in the event that the registry needs to migrate to another node. The following steps show how to create the image registry via OpenStack. Once the volume is created, the volume ID will be included in the OpenShift Ansible Installer OSEv3.yml file via the parameter openshift_hosted_registry_storage_openstack_volumeID as described later.
source /path/to/examplerc openstack volume create --size <volume-size-in-GB> <registry-name>
$ source /path/to/examplerc
$ openstack volume create --size <volume-size-in-GB> <registry-name>The registry volume size should be at least 30GB.
Verify the creation of the volume.
22.2.3.11. Creating and Configuring the Deployment Instance
The role of the deployment instance is to serve as a utility host for the deployment and management of OpenShift Container Platform.
Creating the Deployment Host Network and Router
Prior to instance creation, an internal network and router must be created for communication with the deployment host. The following commands create that network and router.
Deploying the Deployment Instance
With the network and security group created, deploy the instance.
If the m1.small flavor does not exist by default then use an existing flavor that meets the requirements of 1 vCPU and 2GB of RAM.
Creating and Adding Floating IP to the Deployment Instance
Once the deployment instance is created, a floating IP must be created and then allocated to the instance. The following shows an example.
Within the above output, the floating_ip_address field shows that the floating IP 10.20.120.150 is created. In order to assign this IP to the deployment instance, run the following command:
source /path/to/examplerc openstack server add floating ip <deployment-instance-name> <ip>
$ source /path/to/examplerc
$ openstack server add floating ip <deployment-instance-name> <ip>
For example, if instance deployment.example.com is to be assigned IP 10.20.120.150 the command would be:
source /path/to/examplerc openstack server add floating ip deployment.example.com 10.20.120.150
$ source /path/to/examplerc
$ openstack server add floating ip deployment.example.com 10.20.120.150Adding the RC File to the Deployment Host
Once the deployment host exists, copy the RC file created earlier to the deployment host via scp as follows
scp <rc-file-deployment-host> cloud-user@<ip>:/home/cloud-user/
scp <rc-file-deployment-host> cloud-user@<ip>:/home/cloud-user/22.2.3.12. Deployment Host Configuration for OpenShift Container Platform
The following subsections describe all the steps needed to properly configure the deployment instance.
Configure ~/.ssh/config to use Deployment Host as a Jumphost
To easily connect to the OpenShift Container Platform environment, follow the steps below.
On the OpenStack director node or local workstation with the private key, <keypair-name>.pem:
exec ssh-agent bash ssh-add /path/to/<keypair-name>.pem
$ exec ssh-agent bash
$ ssh-add /path/to/<keypair-name>.pem
Identity added: /path/to/<keypair-name>.pem (/path/to/<keypair-name>.pem)
Add to the ~/.ssh/config file:
Host deployment
HostName <deployment_fqdn_hostname OR IP address>
User cloud-user
IdentityFile /path/to/<keypair-name>.pem
ForwardAgent yes
Host deployment
HostName <deployment_fqdn_hostname OR IP address>
User cloud-user
IdentityFile /path/to/<keypair-name>.pem
ForwardAgent yes
ssh into the deployment host with the -A option that enables forwarding of the authentication agent connection.
Ensure the permissions are read write only for the owner of the ~/.ssh/config file:
chmod 600 ~/.ssh/config
$ chmod 600 ~/.ssh/configssh -A cloud-user@deployment
$ ssh -A cloud-user@deployment
Once logged into the deployment host, verify the ssh agent forwarding is working via checking for the SSH_AUTH_SOCK
echo "$SSH_AUTH_SOCK"
$ echo "$SSH_AUTH_SOCK"
/tmp/ssh-NDFDQD02qB/agent.1387Subscription Manager and Enabling OpenShift Container Platform Repositories
Within the deployment instance, register it with the Red Hat Subscription Manager. This can be accomplished by using credentials:
sudo subscription-manager register --username <user> --password '<password>'
$ sudo subscription-manager register --username <user> --password '<password>'Alternatively, you can use an activation key:
sudo subscription-manager register --org="<org_id>" --activationkey=<keyname>
$ sudo subscription-manager register --org="<org_id>" --activationkey=<keyname>Once registered, enable the following repositories as follows.
Refer to the Set Up Repositories to confirm the proper OpenShift Container Platform repositories and Ansible versions to enable. The above file is just a sample.
Required Packages on the Deployment Host
The following packages are required to be installed on the deployment host.
Install the following packages:
-
openshift-ansible -
python-openstackclient -
python2-heatclient -
python2-octaviaclient -
python2-shade -
python-dns -
git -
ansible
sudo yum -y install openshift-ansible python-openstackclient python2-heatclient python2-octaviaclient python2-shade python-dns git ansible
$ sudo yum -y install openshift-ansible python-openstackclient python2-heatclient python2-octaviaclient python2-shade python-dns git ansibleConfigure Ansible
ansible is installed on the deployment instance to perform the registration, installation of packages, and the deployment of the OpenShift Container Platform environment on the master and node instances.
Before running playbooks, it is important to create an ansible.cfg file to reflect the environment you wish to deploy:
The following parameters values are important to the ansible.cfg file.
-
The
remote_usermust remain as the user openshift. - The inventory parameter ensure that there is no space between the two inventories.
Example: inventory = path/to/inventory1,path/to/inventory2
The code block above can overwrite the default values in the file. Ensure to populate <keypair-name> with the keypair that was copied to the deployment instance.
The inventory folder is created in Section 22.3.1, “Preparing the Inventory for Provisioning”.
OpenShift Authentication
OpenShift Container Platform provides the ability to use many different authentication platforms. A listing of authentication options are available at Configuring Authentication and User Agent.
Configuring the default identity provider is important as the default configuration is to Deny All.
22.3. Provisioning OpenShift Container Platform Instances using the OpenShift Ansible Playbooks
Once the creation and configuration of the deployment host is complete, we turn to preparing the environment for the deployment of OpenShift Container Platform using Ansible. In the following subsections, Ansible is configured and certain YAML files are modified to achieve a successful OpenShift Container Platform on OpenStack deployment.
22.3.1. Preparing the Inventory for Provisioning
With the installation of the openshift-ansible package complete via our previous steps, there resides a sample-inventory directory that we will copy to our cloud-user home directory of the deployment host.
On the deployment host,
cp -r /usr/share/ansible/openshift-ansible/playbooks/openstack/sample-inventory/ ~/inventory
$ cp -r /usr/share/ansible/openshift-ansible/playbooks/openstack/sample-inventory/ ~/inventoryWithin this inventory directory, the all.yml file contains all the different parameters that must be set in to order to achieve successful provisioning of the RHOCP instances. The OSEv3.yml file contains some references required by the all.yml file and all the available OpenShift Container Platform cluster parameters that you can customize.
22.3.1.1. OpenShiftSDN All YAML file
The all.yml file has many options that can be modified to meet your specific needs. The information gathered in this file is for the provisioning portion of the instances required for a successful deployment of OpenShift Container Platform. It is important to review these carefully. This document will provide a condensed version of the All YAML file and focus on the most critical parameters that need to be set for a successful deployment.
Due to using an external DNS server, the private and public sections use the public IP address of the DNS server as the DNS server does not reside in the OpenStack environment.
The values above that are enclosed by asterisks (*) require modification based upon your OpenStack environment and DNS server.
In order to properly modify the DNS portion of the All YAML file, login to the DNS server and perform the following commands to capture the key name, key algorithm and key secret:
The key name may vary and the above is only an example.
22.3.1.2. KuryrSDN All YAML file
The following all.yml file enables Kuryr SDN instead of the default OpenShiftSDN. Note that the example below is a condensed version and it is important to review the default template carefully.
If you are using namespace isolation, the Kuryr-controller creates a new Neutron network and subnet for each namespace.
Network policies and nodeport services are not supported when Kuryr SDN is enabled.
If Kuryr is enabled, OpenShift Container Platform services are implemented through OpenStack Octavia Amphora VMs.
Octavia does not support UDP load balancing. Services that expose UDP ports are not supported.
22.3.1.2.1. Configuring global namespace access
The kuryr_openstack_global_namespace parameter contains a list that defines global namespaces. By default, only the default and openshift-monitoring namespaces are included in this list.
If you are upgrading from a previous z-release of OpenShift Container Platform 3.11, note that access to other namespaces from global namespaces is controlled by the security group *-allow_from_default.
Although the remote_group_id rule can control access to other namespaces from global namespaces, using it can cause scaling and connectivity problems. To avoid these problems, switch from using remote_group_id at *_allow_from_default to remote_ip_prefix:
From a command line, retrieve your networks'
subnetCIDRvalue:oc get kuryrnets ns-default -o yaml | grep subnetCIDR
$ oc get kuryrnets ns-default -o yaml | grep subnetCIDR subnetCIDR: 10.11.13.0/24Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create TCP and UDP rules for this range:
openstack security group rule create --remote-ip 10.11.13.0/24 --protocol tcp openshift-ansible-openshift.example.com-allow_from_default openstack security group rule create --remote-ip 10.11.13.0/24 --protocol udp openshift-ansible-openshift.example.com-allow_from_default
$ openstack security group rule create --remote-ip 10.11.13.0/24 --protocol tcp openshift-ansible-openshift.example.com-allow_from_default $ openstack security group rule create --remote-ip 10.11.13.0/24 --protocol udp openshift-ansible-openshift.example.com-allow_from_defaultCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the security group rule that uses
remote_group_id:openstack security group show *-allow_from_default | grep remote_group_id openstack security group rule delete REMOTE_GROUP_ID
$ openstack security group show *-allow_from_default | grep remote_group_id $ openstack security group rule delete REMOTE_GROUP_IDCopy to Clipboard Copied! Toggle word wrap Toggle overflow
| Variable | Description |
|---|---|
| openshift_openstack_clusterid | Cluster identification name |
| openshift_openstack_public_dns_domain | Public DNS domain name |
| openshift_openstack_dns_nameservers | IP of DNS nameservers |
| openshift_openstack_public_hostname_suffix | Adds a suffix to the node hostname in the DNS record for both public and private |
| openshift_openstack_nsupdate_zone | Zone to be updated with OCP instance IPs |
| openshift_openstack_keypair_name | Keypair name used to log in to OCP instances |
| openshift_openstack_external_network_name | OpenStack public network name |
| openshift_openstack_default_image_name | OpenStack image used for OCP instances |
| openshift_openstack_num_masters | Number of master nodes to deploy |
| openshift_openstack_num_infra | Number of infrastructure nodes to deploy |
| openshift_openstack_num_cns | Number of container native storage nodes to deploy |
| openshift_openstack_num_nodes | Number of application nodes to deploy |
| openshift_openstack_master_flavor | Name of the OpenStack flavor used for master instances |
| openshift_openstack_default_flavor | Name of the Openstack flavor used for all instances, if specific flavor not specified. |
| openshift_openstack_use_lbaas_load_balancer | Boolean value enabling Octavia load balancer (Octavia must be installed) |
| openshift_openstack_docker_volume_size | Minimum size of the Docker volume (required variable) |
| openshift_openstack_external_nsupdate_keys | Updating the DNS with the instance IP addresses |
| ansible_user | Ansible user used to deploy OpenShift Container Platform. "openshift" is the required name and must not be changed. |
| openshift_openstack_disable_root | Boolean value that disables root access |
| openshift_openstack_user | OCP instances created with this user |
| openshift_openstack_node_subnet_name | Name of existing OpenShift subnet to use for deployment. This should be the same subnet name used for your deployment host. |
| openshift_openstack_router_name | Name of existing OpenShift router to use for deployment. This should be the same router name used for your deployment host. |
| openshift_openstack_master_floating_ip |
Default is |
| openshift_openstack_infra_floating_ip |
Default is |
| openshift_openstack_compute_floating_ip |
Default is |
| openshift_use_openshift_sdn |
Must set to |
| openshift_use_kuryr |
Must set to |
| use_trunk_ports |
Must be set to |
| os_sdn_network_plugin_name |
selection of the SDN behavior. Must set to |
| openshift_node_proxy_mode |
Must set to |
| openshift_master_open_ports | Ports to be opened on the VMs when using Kuryr |
| kuryr_openstack_public_net_id | Need by Kuryr. ID of the public OpenStack network from where FIPs are obtained |
| openshift_kuryr_subnet_driver |
Kuryr Subnet driver. Must be |
| openshift_kuryr_sg_driver |
Kuryr Security Group driver. Must be |
| kuryr_openstack_global_namespaces |
Global namespaces to use for namespace isolation. The default values are |
| kuryr_openstack_ca | Path to the CA certificate of the cloud. Required if OpenStack cloud endpoints are accessible over HTTPS. |
22.3.1.3. OSEv3 YAML file
The OSEv3 YAML file specifies all the different parameters and customizations relating the installation of OpenShift.
Below is a condensed version of the file with all required variables for a successful deployment. Additional variables may be required depending on what customization is required for your specific OpenShift Container Platform deployment.
For further details on any of the variables listed, see an example OpenShift-Ansible host inventory.
22.3.2. OpenStack Prerequisites Playbook
The OpenShift Container Platform Ansible Installer provides a playbook to ensure all the provisioning steps of the OpenStack instances have been met.
Prior to running the playbook, ensure to source the RC file
source path/to/examplerc
$ source path/to/examplerc
Via the ansible-playbook command on the deployment host, ensure all the prerequisites are met using prerequisites.yml playbook:
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml
$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.ymlOnce the prerequisite playbook completes successfully, run the provision playbook as follows:
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml
$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.ymlIf provision.yml prematurely errors, check if the status of the OpenStack stack and wait for it finish
If the stack shows a CREATE_IN_PROGRESS, wait for the stack to complete with a final result such as CREATE_COMPLETE. If the stack does complete successfully, re-run the provision.yml playbook for it to finish all the additional required steps.
If the stack shows a CREATE_FAILED, make sure to run the following command to see what caused the errors:
openstack stack failures list openshift-cluster
$ openstack stack failures list openshift-cluster22.3.3. Stack Name Configuration
By default, the Heat stack that is created by OpenStack for the OpenShift Container Platform cluster is named openshift-cluster. If you want to use a different name then you must set the OPENSHIFT_CLUSTER environment variable before running the playbooks:
export OPENSHIFT_CLUSTER=openshift.example.com
$ export OPENSHIFT_CLUSTER=openshift.example.com
If you use a non-default stack name and run the openshift-ansible playbooks to update your deployment, you must set OPENSHIFT_CLUSTER to your stack name to avoid errors.
22.4. Registering with Subscription Manager the OpenShift Container Platform Instances
With the nodes successfully provisioned, the next step is to ensure all the nodes are successfully registered via subscription-manager to install all the required packages for a successful OpenShift Container Platform installation. For simplicity, a repos.yml file has been created and provided.
Refer to the Set Up Repositories to confirm the proper repositories and versions to enable. The above file is just a sample.
With the repos.yml, run the ansible-playbook command:
ansible-playbook repos.yml
$ ansible-playbook repos.yml
The above example uses Ansible’s redhat_subscription and rhsm_repository modules for all registration, disabling and enabling of repositories. This specific example takes advantage of using a Red Hat activation key. If you don’t have an activation key, ensure to visit the Ansible redhat_subscription module to modify using a username and password instead as shown in the examples: https://docs.ansible.com/ansible/2.6/modules/redhat_subscription_module.html
At times, the redhat_subscription module may fail on certain nodes. If this issue occurs, please manually register that OpenShift Container Platform instance using subscription-manager.
22.5. Installing OpenShift Container Platform by Using an Ansible Playbook
With the OpenStack instances provisioned, the focus shifts to the installation OpenShift Container Platform. The installation and configuration is done via a series of Ansible playbooks and roles provided by the OpenShift RPM packages. Review the OSEv3.yml file that was previous configured to ensure all the options have been properly set.
Prior to running the installer playbook, ensure all the {rhocp} prerequisites are met via:
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml
$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.ymlRun the installer playbook to install Red Hat OpenShift Container Platform:
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml
$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.ymlOpenShift Container Platform version 3.11 is supported on RH OSP 14 and RH OSP 13. OpenShift Container Platform version 3.10 is supported on RH OSP 13.
22.6. Applying Configuration Changes to Existing OpenShift Container Platform Environment
Start or restart OpenShift Container Platform services on all master and node hosts to apply your configuration changes, see Restarting OpenShift Container Platform services:
master-restart api master-restart controllers systemctl restart atomic-openshift-node
# master-restart api
# master-restart controllers
# systemctl restart atomic-openshift-nodeKubernetes architecture expects reliable endpoints from cloud providers. When a cloud provider is down, the kubelet prevents OpenShift Container Platform from restarting. If the underlying cloud provider endpoints are not reliable, do not install a cluster that uses the cloud provider integration. Install the cluster as if it is a bare metal environment. It is not recommended to toggle cloud provider integration on or off in an installed cluster. However, if that scenario is unavoidable, then complete the following process.
Switching from not using a cloud provider to using a cloud provider produces an error message. Adding the cloud provider tries to delete the node because the node switches from using the hostname as the externalID (which would have been the case when no cloud provider was being used) to using the cloud provider’s instance-id (which is what the cloud provider specifies). To resolve this issue:
- Log in to the CLI as a cluster administrator.
Check and back up existing node labels:
oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'
$ oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the nodes:
oc delete node <node_name>
$ oc delete node <node_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow On each node host, restart the OpenShift Container Platform service.
systemctl restart atomic-openshift-node
# systemctl restart atomic-openshift-nodeCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Add back any labels on each node that you previously had.
22.6.1. Configuring OpenStack Variables on an existing OpenShift Environment
To set the required OpenStack variables, modify the /etc/origin/cloudprovider/openstack.conf file with the following contents on all of your OpenShift Container Platform hosts, both masters and nodes:
Consult your OpenStack administrators for values of the OS_ variables, which are commonly used in OpenStack configuration.
22.6.2. Configuring Zone Labels for Dynamically Created OpenStack PVs
Administrators can configure zone labels for dynamically created OpenStack PVs. This option is useful if the OpenStack Cinder zone name does not match the compute zone names, for example, if there is only one Cinder zone and many compute zones. Administrators can create Cinder volumes dynamically and then check the labels.
To view the zone labels for the PVs:
oc get pv --show-labels
# oc get pv --show-labels
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE LABELS
pvc-1faa6f93-64ac-11e8-930c-fa163e3c373c 1Gi RWO Delete Bound openshift-node/pvc1 standard 12s failure-domain.beta.kubernetes.io/zone=nova
The default setting is enabled. Using the oc get pv --show-labels command returns the failure-domain.beta.kubernetes.io/zone=nova label.
To disable the zone label, update the openstack.conf file by adding:
[BlockStorage] ignore-volume-az = yes
[BlockStorage]
ignore-volume-az = yesThe PVs created after restarting the master services will not have the zone label.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}