Deploying multiple RHOSO environments on a single RHOCP cluster

1. Review the minimum network requirements for a RHOSO environment. For more information about the RHOSO network requirements, see RHOCP network requirements and Planning your networks in Planning your deployment.
2. Plan the network topology to support multiple RHOSO environments on the same RHOCP cluster, and plan the RHOSO networks for each namespace.
3. Ensure that RHOSO-dedicated NICs are present on all RHOCP worker nodes, one for each namespace, to reinforce segregation between the RHOSO environments. These interfaces use VLANs for further network isolation.
4. Ensure that there is connectivity between the namespace interfaces and the Compute nodes for that namespace.
5. Ensure that VLAN IDs and subnets for each namespace do not overlap.
6. Plan the networks for each RHOSO environment for each namespace. Each namespace must have unique network IP addresses for all networks. Do not overlap IP addresses and ranges between the namespaces. For more information about the required network values, see the "Default RHOSO networks" table in Default Red Hat OpenStack Services on OpenShift networks in Deploying Red Hat OpenStack Services on OpenShift.

Procedure

1. Create the openstack project for the deployed RHOSO environment:

   $ oc new-project openstack

2. Ensure the openstack namespace is labeled to enable privileged pod creation by the OpenStack Operators:

   $ oc get namespace openstack -ojsonpath='{.metadata.labels}' | jq
   {
     "kubernetes.io/metadata.name": "openstack",
     "pod-security.kubernetes.io/enforce": "privileged",
     "security.openshift.io/scc.podSecurityLabelSync": "false"
   }

   If the security context constraint (SCC) is not "privileged", use the following commands to change it:

   $ oc label ns openstack security.openshift.io/scc.podSecurityLabelSync=false --overwrite
   $ oc label ns openstack pod-security.kubernetes.io/enforce=privileged --overwrite

3. Optional: To remove the need to specify the namespace when executing commands on the openstack namespace, set the default namespace to openstack:

   $ oc project openstack

Procedure

1. Create a Secret CR on your workstation, for example, openstack_service_secret.yaml.

2. Add the following initial configuration to openstack_service_secret.yaml:

   apiVersion: v1
   data:
     AdminPassword: <base64_password>
     AodhPassword: <base64_password>
     BarbicanPassword: <base64_password>
     BarbicanSimpleCryptoKEK: <base64_fernet_key>
     CeilometerPassword: <base64_password>
     CinderPassword: <base64_password>
     DbRootPassword: <base64_password>
     DesignatePassword: <base64_password>
     GlancePassword: <base64_password>
     HeatAuthEncryptionKey: <base64_password>
     HeatPassword: <base64_password>
     IronicInspectorPassword: <base64_password>
     IronicPassword: <base64_password>
     ManilaPassword: <base64_password>
     MetadataSecret: <base64_password>
     NeutronPassword: <base64_password>
     NovaPassword: <base64_password>
     OctaviaPassword: <base64_password>
     PlacementPassword: <base64_password>
     SwiftPassword: <base64_password>
   kind: Secret
   metadata:
     name: osp-secret
     namespace: openstack
   type: Opaque

   • Replace <base64_password> with a 32-character key that is base64 encoded.

     Note

     The HeatAuthEncryptionKey password must be a 32-character key for Orchestration service (heat) encryption. If you increase the length of the passwords for all other services, ensure that the HeatAuthEncryptionKey password remains at length 32.

     You can use the following command to manually generate a base64 encoded password:

     $ echo -n <password> | base64

     Alternatively, if you are using a Linux workstation and you are generating the Secret CR by using a Bash command such as cat, you can replace <base64_password> with the following command to auto-generate random passwords for each service:

     $(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32 | base64)

   • Replace the <base64_fernet_key> with a base64 encoded fernet key. You can use the following command to manually generate it:

     $(python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode('UTF-8'))" | base64)

3. Create the Secret CR in the cluster:

   $ oc create -f openstack_service_secret.yaml -n openstack

4. Verify that the Secret CR is created:

   $ oc describe secret osp-secret -n openstack

Procedure

1. Create a file named netconfig_osp_ns_1.yaml on your workstation.

2. Add the following configuration to netconfig_osp_ns_1.yaml to create the NetConfig CR:

   apiVersion: network.openstack.org/v1beta1
   kind: NetConfig
   metadata:
     name: openstacknetconfig
     namespace: osp_ns_1

3. In the netconfig_osp_ns_1.yaml file, define the topology for each data plane network. The following example creates isolated networks for the data plane:

   spec:
     networks:
     - name: ctlplane
       dnsDomain: ctlplane_1.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 192.168.122.120
           start: 192.168.122.100
         - end: 192.168.122.200
           start: 192.168.122.150
         cidr: 192.168.122.0/24
         gateway: 192.168.122.1
     - name: internalapi
       dnsDomain: internalapi_1.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 172.17.0.250
           start: 172.17.0.100
         excludeAddresses:
         - 172.17.0.10
         - 172.17.0.12
         cidr: 172.17.0.0/24
         vlan: 20
     - name: external
       dnsDomain: external_1.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 10.0.0.250
           start: 10.0.0.100
         cidr: 10.0.0.0/24
         gateway: 10.0.0.1
     - name: storage
       dnsDomain: storage_1.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 172.18.0.250
           start: 172.18.0.100
         cidr: 172.18.0.0/24
         vlan: 21
     - name: tenant
       dnsDomain: tenant_1.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 172.19.0.250
           start: 172.19.0.100
         cidr: 172.19.0.0/24
         vlan: 22

   • spec.networks.name: The name of the network, for example, ctlplane.
   • spec.networks.subnets: The IPv4 subnet specification.
   • spec.networks.subnets.name: The name of the subnet, for example, subnet1.
   • spec.networks.subnets.allocationRanges: The NetConfig allocationRange. The allocationRange must not overlap with the MetalLB IPAddressPool range and the IP address pool range.
   • spec.networks.subnets.excludeAddresses: An optional list of IP addresses from the allocation range that must not be used by data plane nodes.
   • spec.networks.subnets.vlan: The network VLAN. For information about the default RHOSO networks, see Default Red Hat OpenStack Services on OpenShift networks.
4. Save the netconfig_osp_ns_1.yaml definition file.

5. Create the data plane network:

   $ oc create -f netconfig_osp_ns_1.yaml -n openstack

6. To verify that the data plane network is created, view the openstacknetconfig resource:

   $ oc get netconfig/openstacknetconfig -n openstack

   If you see errors, check the underlying network-attach-definition and node network configuration policies:

   $ oc get network-attachment-definitions -n openstack
   $ oc get nncp

Procedure

1. Create an OpenStackControlPlane CR for the osp_ns_1 namespace by following the Creating the control plane procedure in Deploying Red Hat OpenStack Services on OpenShift.

2. Locate the load balancer configuration for each service and update it with the networks for that namespace:

     <service>:
       ...
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi_1
                     metallb.universe.tf/allow-shared-ip: internalapi_1
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer

   • metallb.universe.tf/address-pool: The IpAddressPool to assign the IP address from to load balance the service.

   • metallb.universe.tf/allow-shared-ip: The IpAddressPool to use a shared IP address from for load balancing.

     Note

     You cannot share a VIP with services that use the same port, such as the multiple instances of rabbitmq.

   • metallb.universe.tf/loadBalancerIPs: Assigns a specific VIP from the IpAddressPool range for use when load balancing the service.

3. Update the NIC mappings for the ovn service to ensure that the OVNController resources for the namespace are segregated from the other namespaces, by configuring the nicMappings for the datacentre network to correspond to the dedicated interface for the namespace:

   ...
       ovn:
         enabled: true
         template:
           ovnController:
             nicMappings:
               datacentre: ospbr_1
           ovnDBCluster:
             ovndbcluster-nb:

4. Update the control plane:

   $ oc apply -f osp_ns_1_control_plane.yaml -n openstack

5. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n osp_ns_1
   NAME 						    STATUS 	  MESSAGE
   control-plane-ns-1 	Unknown 	Setup started

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

6. Verify that the service pods are running on the correct worker nodes.

Procedure

1. Create an OpenStackDataPlaneNodeSet CR for the osp_ns_1 namespace by following the Creating the data plane procedures in Deploying Red Hat OpenStack Services on OpenShift.

2. Locate the nodes section and update the hostName and the ansibleVars.fqdn_internal_api fields with unique names for each node that indicate the namespace the node belongs to:

     nodes:
       edpm-compute-0:
         hostName: edpm_1-compute-0
         networks:
         - name: ctlplane
           subnetName: subnet1
           defaultRoute: true
           fixedIP: 192.168.122.100
         - name: internalapi
           subnetName: subnet1
           fixedIP: 172.17.0.100
         - name: storage
           subnetName: subnet1
           fixedIP: 172.18.0.100
         - name: tenant
           subnetName: subnet1
           fixedIP: 172.19.0.100
         ansible:
           ansibleHost: 192.168.122.100
           ansibleUser: cloud-admin
           ansibleVars:
             fqdn_internal_api: edpm_1-compute-0.example.com

3. Update the IP addresses of the networks defined for each node defined for the namespace:

     nodes:
       edpm-compute-0:
         ...
         networks:
         - name: ctlplane
           subnetName: subnet1
           defaultRoute: true
           fixedIP: 192.168.122.100
         - name: internalapi
           subnetName: subnet1
           fixedIP: 172.17.0.100
         - name: storage
           subnetName: subnet1
           fixedIP: 172.18.0.100
         - name: tenant
           subnetName: subnet1
           fixedIP: 172.19.0.100
         ansible:
           ansibleHost: 192.168.122.100
           ...

   • fixedIP: Set to the IP address for the network for the osp_ns_1 namespace.
4. Save the updated OpenStackDataPlaneNodeSet CR definition file.

5. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

   $ oc apply -f osp_ns_1_data_plane.yaml

6. Verify that the data plane resource has been updated by confirming that the status is SetupReady:

   $ oc wait openstackdataplanenodeset data-plane-ns-1 --for condition=SetupReady --timeout=10m

   When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

7. Create a file on your workstation to define the new OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

8. Add the OpenStackDataPlaneNodeSet CR that you modified:

   spec:
     nodeSets:
       - <nodeSet_name>

9. Save the OpenStackDataPlaneDeployment CR deployment file.

10. Deploy the modified OpenStackDataPlaneNodeSet CR:

    $ oc create -f osp_ns_1_data_plane_deploy.yaml -n openstack

    You can view the Ansible logs while the deployment executes:

    $ oc get pod -l app=openstackansibleee -w
    $ oc logs -l app=openstackansibleee -f --max-log-requests 10

    If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

    error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

11. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME             	STATUS   MESSAGE
    data-plane-ns-1     True     Setup Complete
    
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME             	STATUS   MESSAGE
    data-plane-ns-1     True     NodeSet Ready

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in the Deploying Red Hat OpenStack Services on OpenShift guide.