Chapter 9. Creating the control plane for NFV environments

Procedure

1. Create the openstack project for the deployed RHOSO environment:

   $ oc new-project openstack

2. Ensure the openstack namespace is labeled to enable privileged pod creation by the OpenStack Operators:

   $ oc get namespace openstack -ojsonpath='{.metadata.labels}' | jq
   {
     "kubernetes.io/metadata.name": "openstack",
     "pod-security.kubernetes.io/enforce": "privileged",
     "security.openshift.io/scc.podSecurityLabelSync": "false"
   }

   If the security context constraint (SCC) is not "privileged", use the following commands to change it:

   $ oc label ns openstack security.openshift.io/scc.podSecurityLabelSync=false --overwrite
   $ oc label ns openstack pod-security.kubernetes.io/enforce=privileged --overwrite

3. Create a file on your workstation named openstack_control_plane.yaml to define the OpenStackControlPlane CR:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack

4. Specify the Secret CR you created to provide secure access to the RHOSO service pods in Providing secure access to the Red Hat OpenStack Services on OpenShift services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
   spec:
     secret: osp-secret

5. Specify the storageClass you created for your Red Hat OpenShift Container Platform (RHOCP) cluster storage back end:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
   spec:
     secret: osp-secret
     storageClass: your-RHOCP-storage-class

   Note

   For information about storage classes, see Creating a storage class.

6. Add the following service configurations:

   Block Storage service (cinder)

     cinder:
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         secret: osp-secret
         cinderAPI:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
         cinderScheduler:
           replicas: 1
         cinderBackup:
           networkAttachments:
           - storage
           replicas: 0 # backend needs to be configured to activate the service
         cinderVolumes:
           volume1:
             networkAttachments:
             - storage
             replicas: 0 # backend needs to be configured to activate the service

   Important

   This definition for the Block Storage service is only a sample. You might need to modify it for your NFV environment. For more information, see Planning storage and shared file systems in Planning your deployment.

   Note

   For the initial control plane deployment, the cinderBackup and cinderVolumes services are deployed but not activated (replicas: 0). You can configure your control plane post-deployment with a back end for the Block Storage service and the backup service.

   Compute service (nova)

     nova:
       apiOverride:
         route: {}
       template:
         apiServiceTemplate:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
         schedulerServiceTemplate:
           customServiceConfig: |
             [filter_scheduler]
             enabled_filters = AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter, AggregateInstanceExtraSpecsFilter
             available_filters = nova.scheduler.filters.all_filters
         metadataServiceTemplate:
           replicas: 3
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.80
               spec:
                 type: LoadBalancer
         schedulerServiceTemplate:
           replicas: 3
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.80
               spec:
                 type: LoadBalancer
         cellTemplates:
           cell1:
             noVNCProxyServiceTemplate:
               enabled: true
               networkAttachments:
               - ctlplane
         secret: osp-secret

   Note

   A full set of Compute services (nova) are deployed by default for each of the default cells, cell0 and cell1: nova-api, nova-metadata, nova-scheduler, and nova-conductor. The novncproxy service is also enabled for cell1 by default.

   DNS service for the data plane

     dns:
       template:
         options:
         - key: server
           values:
           - <IP address for DNS server reachable from dnsmasq pod>
         override:
           service:
             metadata:
               annotations:
                 metallb.universe.tf/address-pool: ctlplane
                 metallb.universe.tf/allow-shared-ip: ctlplane
                 metallb.universe.tf/loadBalancerIPs: 192.168.122.80
             spec:
               type: LoadBalancer
         replicas: 2

   • options: Defines the dnsmasq instances required for each DNS server by using key-value pairs. In this example, there is one key-value pair defined because there is only one DNS server configured to forward requests to.

   • key: Specifies the dnsmasq parameter to customize for the deployed dnsmasq instance. Set to one of the following valid values:

     • server
     • rev-server
     • srv-host
     • txt-record
     • ptr-record
     • rebind-domain-ok
     • naptr-record
     • cname
     • host-record
     • caa-record
     • dns-rr
     • auth-zone
     • synth-domain
     • no-negcache
     • local

   • values: Specifies the value for the DNS server reachable from the dnsmasq pod on the RHOCP cluster network. You can specify a generic DNS server as the value, for example, 1.1.1.1, or a DNS server for a specific domain, for example, /google.com/8.8.8.8.

     Note

     This DNS service, dnsmasq, provides DNS services for nodes on the RHOSO data plane. dnsmasq is different from the RHOSO DNS service (designate) that provides DNS as a service for cloud tenants.

   Galera cluster

   A Galera cluster for use by all RHOSO services (openstack), and a Galera cluster for use by the Compute service for cell1 (openstack-cell1):

     galera:
       templates:
         openstack:
           storageRequest: 5000M
           secret: osp-secret
           replicas: 3
         openstack-cell1:
           storageRequest: 5000M
           secret: osp-secret
           replicas: 3

   Identity service (keystone)

     keystone:
       apiOverride:
         route: {}
       template:
         override:
           service:
             internal:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.80
               spec:
                 type: LoadBalancer
         databaseInstance: openstack
         secret: osp-secret
         replicas: 3

   Image service (glance)

     glance:
       apiOverrides:
         default:
           route: {}
       template:
         databaseInstance: openstack
         storage:
           storageRequest: 10G
         secret: osp-secret
         keystoneEndpoint: default
         glanceAPIs:
           default:
             replicas: 0 # backend needs to be configured to activate the service
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
             networkAttachments:
             - storage

   Note

   For the initial control plane deployment, the Image service is deployed but not activated (replicas: 0). You can configure your control plane post-deployment with a back end for the Image service.

   Key Management service (barbican)

     barbican:
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         secret: osp-secret
         barbicanAPI:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
         barbicanWorker:
           replicas: 3
         barbicanKeystoneListener:
           replicas: 1

   Memcached

     memcached:
       templates:
         memcached:
            replicas: 3

   Networking service (neutron)

     neutron:
       apiOverride:
         route: {}
       template:
         replicas: 3
         override:
           service:
             internal:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.80
               spec:
                 type: LoadBalancer
         databaseInstance: openstack
         secret: osp-secret
         networkAttachments:
         - internalapi
         customServiceConfig: |
           [DEFAULT]
           global_physnet_mtu = 9000
           [ml2]
           mechanism_drivers = ovn
           [ovn]
           vhost_sock_dir = <path>
           [ml2_type_vlan]
           network_vlan_ranges = <network_name1>:<VLAN-ID1>:<VLAN-ID2>,<network_name2>:<VLAN-ID1>:<VLAN-ID2>

   • mechanism_drivers - If you are using SR-IOV, you must also add the sriovnicswitch mechanism driver, for example, mechanism_drivers = ovn,sriovnicswitch.
   • vhost_sock_dir - Replace <path> with the absolute path to the vhost sockets, for example, /var/lib/vhost_sockets.
   • network_vlan_ranges - Replace <network_name1> and <network_name2> with the names of the physical networks that your gateways are on. (This network is set in the neutron network provider:*name field.)
   • <VLAN-ID1> - Replace <VLAN-ID1> and`<VLAN-ID2>` with the VLAN IDs you are using.

   Object Storage service (swift)

     swift:
       enabled: true
       proxyOverride:
         route: {}
       template:
         swiftProxy:
           networkAttachments:
           - storage
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           replicas: 2
         swiftRing:
           ringReplicas: 3
         swiftStorage:
           networkAttachments:
           - storage
           replicas: 3
           storageClass: local-storage
           storageRequest: 100Gi

   OVN

     ovn:
       template:
         ovnDBCluster:
           ovndbcluster-nb:
             replicas: 3
             dbType: NB
             storageRequest: 10G
             networkAttachment: internalapi
           ovndbcluster-sb:
             replicas: 3
             dbType: SB
             storageRequest: 10G
             networkAttachment: internalapi
         ovnNorthd: {}
         ovnController:
           networkAttachment: tenant
           nicMappings:
             <network_name>: <NIC_name>

   • nicMappings - Replace <network_name> with the name of the physical network your gateway is on. (This network is set in the neutron network provider:*name field.)
   • <NIC_name> - Replace <NIC_name> with the name of the NIC connecting to the gateway network.
   • <network_name>: <NIC_name> - Optional: add additional <network_name>:<NIC_name> pairs under nicMappings as required.

   Placement service (placement)

     placement:
       apiOverride:
         route: {}
       template:
         override:
           service:
             internal:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.80
               spec:
                 type: LoadBalancer
         databaseInstance: openstack
         replicas: 3
         secret: osp-secret

   RabbitMQ

     rabbitmq:
       templates:
         rabbitmq:
           replicas: 3
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.85
               spec:
                 type: LoadBalancer
         rabbitmq-cell1:
           replicas: 3
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.86
               spec:
                 type: LoadBalancer

   Telemetry service (ceilometer, prometheus)

     telemetry:
       enabled: true
       template:
         metricStorage:
           enabled: true
           monitoringStack:
             alertingEnabled: true
             scrapeInterval: 30s
             storage:
               strategy: persistent
               retention: 24h
               persistent:
                 pvcStorageRequest: 20G
         autoscaling:
           enabled: false
           aodh:
             passwordSelectors:
             databaseAccount: aodh
             databaseInstance: openstack
             memcachedInstance: memcached
             secret: osp-secret
           heatInstance: heat
         ceilometer:
           enabled: true
           secret: osp-secret
         logging:
           enabled: false

   • autoscaling - You must have the autoscaling field present, even if autoscaling is disabled.

7. Create the control plane:

   $ oc create -f openstack_control_plane.yaml -n openstack

   Note

   Creating the control plane also creates an OpenStackClient pod that you can access through a remote shell (rsh) to run RHOSO CLI commands.

   $ oc rsh -n openstack openstackclient

8. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Check the status of the control plane deployment:

   $ oc get openstackcontrolplane -n openstack

   Sample output

   NAME                      STATUS    MESSAGE
   openstack-control-plane   Unknown   Setup started

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

   Note

   Creating the control plane also creates an OpenStackClient pod that you can access through a remote shell (rsh) to run RHOSO CLI commands.

   $ oc rsh -n openstack openstackclient

9. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

   The control plane is deployed when all the pods are either completed or running.

Verification

1. Open a remote shell connection to the OpenStackClient pod:

   $ oc rsh -n openstack openstackclient

2. Confirm that the internal service endpoints are registered with each service:

   $ openstack endpoint list -c 'Service Name' -c Interface -c URL --service glance

   Sample output

   +--------------+-----------+---------------------------------------------------------------+
   | Service Name | Interface | URL                                                           |
   +--------------+-----------+---------------------------------------------------------------+
   | glance       | internal  | http://glance-internal.openstack.svc:9292                     |
   | glance       | public    | http://glance-public-openstack.apps.ostest.test.metalkube.org |
   +--------------+-----------+---------------------------------------------------------------+

3. Exit the OpenStackClient pod:

   $ exit

Procedure

1. Open the OpenStackControlPlane CR file on your workstation.

2. Locate the service you want to remove from the control plane and disable it:

     cinder:
       enabled: false
       apiOverride:
         route: {}
         ...

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP removes the resource related to the disabled service. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 							STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resource is updated with the disabled service when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

5. Optional: Confirm that the pods from the disabled service are no longer listed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

6. Check that the service is removed:

   $ oc get cinder -n openstack

   This command returns the following message when the service is successfully removed:

   No resources found in openstack namespace.

7. Check that the API endpoints for the service are removed from the Identity service (keystone):

   $ oc rsh -n openstack openstackclient
   $ openstack endpoint list --service volumev3

   This command returns the following message when the API endpoints for the service are successfully removed:

   No service with a type, name or ID of 'volumev3' exists.