Chapter 1. Planning the new deployment

There is a fundamental difference between the director and operator deployments regarding the configuration of the services.

In director deployments many of the service configurations are abstracted by director-specific configuration options. A single director option may trigger changes for multiple services and support for drivers, for example, the Block Storage service (cinder), that require patches to the director code base.

In operator deployments this approach has changed: reduce the installer specific knowledge and leverage Red Hat OpenShift Container Platform (RHOCP) and Red Hat OpenStack Platform (RHOSP) service specific knowledge whenever possible.

To this effect RHOSP services will have sensible defaults for RHOCP deployments and human operators will provide configuration snippets to provide necessary configuration, such as the Block Storage service backend configuration, or to override the defaults.

This shortens the distance between a service specific configuration file (such as cinder.conf) and what the human operator provides in the manifests.

These configuration snippets are passed to the operators in the different customServiceConfig sections available in the manifests, and then they are layered in the services available in the following levels. To illustrate this, if you were to set a configuration at the top Block Storage service level (spec: cinder: template:) then it would be applied to all the Block Storage services; for example to enable debug in all the Block Storage services you would do:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: openstack
spec:
  cinder:
    template:
      customServiceConfig: |
        [DEFAULT]
        debug = True
< . . . >

If you only want to set it for one of the Block Storage services, for example the scheduler, then you use the cinderScheduler section instead:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: openstack
spec:
  cinder:
    template:
      cinderScheduler:
        customServiceConfig: |
          [DEFAULT]
          debug = True
< . . . >

In Red Hat OpenShift Container Platform it is not recommended to store sensitive information like the credentials to the Block Storage service storage array in the CRs, so most RHOSP operators have a mechanism to use the Red Hat OpenShift Container Platform Secrets for sensitive configuration parameters of the services and then use them by reference in the customServiceConfigSecrets section which is analogous to the customServiceConfig.

The contents of the Secret references passed in the customServiceConfigSecrets will have the same format as customServiceConfig: a snippet with the section/s and configuration options.

When there are sensitive information in the service configuration then it becomes a matter of personal preference whether to store all the configuration in the Secret or only the sensitive parts. However, if you split the configuration between Secret and customServiceConfig you still need the section header (eg: [DEFAULT]) to be present in both places.

Attention should be paid to each service’s adoption process as they may have some particularities regarding their configuration.

In director deployments you had 4 different standard roles for the nodes: Controller, Compute, Ceph Storage, Swift Storage, but in the control plane you make a distinction based on where things are running, in Red Hat OpenShift Container Platform (RHOCP) or external to it.

When adopting a director Red Hat OpenStack Platform (RHOSP) your Compute nodes will directly become external nodes, so there should not be much additional planning needed there.

In many deployments being adopted the Controller nodes will require some thought because you have many RHOCP nodes where the Controller services could run, and you have to decide which ones you want to use, how you are going to use them, and make sure those nodes are ready to run the services.

In most deployments running RHOSP services on master nodes can have a seriously adverse impact on the RHOCP cluster, so it is recommended that you place RHOSP services on non master nodes.

By default RHOSP Operators deploy RHOSP services on any worker node, but that is not necessarily what’s best for all deployments, and there may be even services that won’t even work deployed like that.

When planing a deployment it’s good to remember that not all the services on an RHOSP deployments are the same as they have very different requirements.

Looking at the Block Storage service (cinder) component you can clearly see different requirements for its services: the cinder-scheduler is a very light service with low memory, disk, network, and CPU usage; cinder-api service has a higher network usage due to resource listing requests; the cinder-volume service will have a high disk and network usage since many of its operations are in the data path (offline volume migration, create volume from image, etc.), and then you have the cinder-backup service which has high memory, network, and CPU (to compress data) requirements.

The Image Service (glance) and Object Storage service (swift) components are in the data path, as well as RabbitMQ and Galera services.

Given these requirements it may be preferable not to let these services wander all over your RHOCP worker nodes with the possibility of impacting other workloads, or maybe you don’t mind the light services wandering around but you want to pin down the heavy ones to a set of infrastructure nodes.

There are also hardware restrictions to take into consideration, because if you are using a Fibre Channel (FC) Block Storage service backend you need the cinder-volume, cinder-backup, and maybe even the Image Service (glance) (if it’s using the Block Storage service as a backend) services to run on a RHOCP host that has an HBA.

The RHOSP Operators allow a great deal of flexibility on where to run the RHOSP services, as you can use node labels to define which RHOCP nodes are eligible to run the different RHOSP services. Refer to the About node selector to learn more about using labels to define placement of the RHOSP services.

There are a variety of reasons why you might want to restrict the nodes where Red Hat OpenStack Platform (RHOSP) services can be placed:

The mechanism provided by the RHOSP operators to achieve this is through the use of labels.

You either label the RHOCP nodes or use existing labels, and then use those labels in the RHOSP manifests in the nodeSelector field.

The nodeSelector field in the RHOSP manifests follows the standard RHOCP nodeSelector field. For more information, see About node selectors in OpenShift Container Platform 4.15 Documentation.

This field is present at all the different levels of the RHOSP manifests:

This allows a fine grained control of the placement of the RHOSP services with minimal repetition.

Values of the nodeSelector are propagated to the next levels unless they are overwritten. This means that a nodeSelector value at the deployment level will affect all the RHOSP services.

For example, you can add label type: openstack to any 3 RHOCP nodes:

$ oc label nodes worker0 type=openstack
$ oc label nodes worker1 type=openstack
$ oc label nodes worker2 type=openstack

And then in our OpenStackControlPlane you can use the label to place all the services in those 3 nodes:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: openstack
spec:
  secret: osp-secret
  storageClass: local-storage
  nodeSelector:
    type: openstack
< . . . >

You can use the selector for specific services. For example, you might want to place your the Block Storage service (cinder) volume and backup services on certain nodes if you are using FC and only have HBAs on a subset of nodes. The following example assumes that you have the label fc_card: true:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: openstack
spec:
  secret: osp-secret
  storageClass: local-storage
  cinder:
    template:
      cinderVolumes:
          pure_fc:
            nodeSelector:
              fc_card: true
< . . . >
          lvm-iscsi:
            nodeSelector:
              fc_card: true
< . . . >
      cinderBackup:
          nodeSelector:
            fc_card: true
< . . . >

The Block Storage service operator does not currently have the possibility of defining the nodeSelector in cinderVolumes, so you need to specify it on each of the backends.

It is possible to leverage labels added by the Node Feature Discovery (NFD) Operator to place RHOSP services. For more information, see Node Feature Discovery Operator in OpenShift Container Platform 4.15 Documentation.

Some services require you to have services or kernel modules running on the hosts where they run, for example iscsid or multipathd daemons, or the nvme-fabrics kernel module.

For those cases you use MachineConfig manifests, and if you are restricting the nodes that you are placing the Red Hat OpenStack Platform services on using the nodeSelector then you also want to limit where the MachineConfig is applied.

To define where the MachineConfig can be applied, you need to use a MachineConfigPool that links the MachineConfig to the nodes.

For example to be able to limit MachineConfig to the 3 Red Hat OpenShift Container Platform (RHOCP) nodes that you marked with the type: openstack label, you create the MachineConfigPool like this:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: openstack
spec:
  machineConfigSelector:
    matchLabels:
      machineconfiguration.openshift.io/role: openstack
  nodeSelector:
    matchLabels:
      type: openstack

And then you use it in the MachineConfig:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: openstack
< . . . >

Refer to the Postinstallation machine configuration tasks in OpenShift Container Platform 4.15 Documentation.

The Key Manager service (barbican) does not yet support all of the crypto plug-ins available in director.

With Red Hat OpenShift Container Platform (RHOCP), the network is a very important aspect of the deployment, and it is important to plan it carefully. The general network requirements for the Red Hat OpenStack Platform (RHOSP) services are not much different from the ones in a director deployment, but the way you handle them is.

When adopting a new RHOSP deployment, it is important to align the network configuration with the adopted cluster to maintain connectivity for existing workloads.

The following logical configuration steps will incorporate the existing network configuration:

Specifically,

- name: InternalApi
  mtu: 1500
  vip: true
  vlan: 20
  name_lower: internal_api
  dns_domain: internal.mydomain.tld.
  service_net_map_replace: internal
  subnets:
    internal_api_subnet:
      ip_subnet: '172.17.0.0/24'
      allocation_pools: [{'start': '172.17.0.4', 'end': '172.17.0.250'}]

Standalone:
  hosts:
    standalone:
      ...
      internal_api_ip: 172.17.0.100
    ...
  ...
standalone:
  children:
    Standalone: {}
  vars:
    ...
    internal_api_vip: 172.17.0.2
    ...

network_config:
  - type: ovs_bridge
    name: br-ctlplane
    routes:
    - ip_netmask: 0.0.0.0/0
      next_hop: 192.168.1.1
    - ip_netmask: 172.31.0.0/24  # <- new ctlplane subnet
      next_hop: 192.168.1.100    # <- adopted node ctlplane IP address

# ip route | grep 172
172.31.0.0/24 via 192.168.122.100 dev br-ctlplane

      - destination: 192.168.122.0/24
        next-hop-interface: ospbr

# ip route | grep 192
192.168.122.0/24 dev ospbr proto static scope link

  nodeTemplate:
    ansible:
      ansibleUser: root
      ansibleVars:
        additional_ctlplane_host_routes:
        - ip_netmask: 172.31.0.0/24
          next_hop: '{{ ctlplane_ip }}'
        edpm_network_config_template: |
          network_config:
          - type: ovs_bridge
            routes: {{ ctlplane_host_routes + additional_ctlplane_host_routes }}
            ...

  nodes:
    standalone:
      ansible:
        ansibleHost: 192.168.122.100
        ansibleUser: ""
      hostName: standalone
      networks:
      - defaultRoute: true
        fixedIP: 192.168.122.100
        name: ctlplane
        subnetName: subnet1

        edpm_sshd_allowed_ranges:
        - 192.168.122.0/24
        - 172.31.0.0/24

apiVersion: v1
items:
- apiVersion: nmstate.io/v1
  kind: NodeNetworkConfigurationPolicy
  spec:
      interfaces:
      - description: internalapi vlan interface
        ipv4:
          address:
          - ip: 172.17.0.10
            prefix-length: 24
          dhcp: false
          enabled: true
        ipv6:
          enabled: false
        name: enp6s0.20
        state: up
        type: vlan
        vlan:
          base-iface: enp6s0
          id: 20
          reorder-headers: true
      - description: storage vlan interface
        ipv4:
          address:
          - ip: 172.18.0.10
            prefix-length: 24
          dhcp: false
          enabled: true
        ipv6:
          enabled: false
        name: enp6s0.21
        state: up
        type: vlan
        vlan:
          base-iface: enp6s0
          id: 21
          reorder-headers: true
      - description: tenant vlan interface
        ipv4:
          address:
          - ip: 172.19.0.10
            prefix-length: 24
          dhcp: false
          enabled: true
        ipv6:
          enabled: false
        name: enp6s0.22
        state: up
        type: vlan
        vlan:
          base-iface: enp6s0
          id: 22
          reorder-headers: true
    nodeSelector:
      kubernetes.io/hostname: ocp-worker-0
      node-role.kubernetes.io/worker: ""

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "internalapi",
      "type": "macvlan",
      "master": "enp6s0.20",
      "ipam": {
        "type": "whereabouts",
        "range": "172.17.0.0/24",
        "range_start": "172.17.0.20",
        "range_end": "172.17.0.50"
      }
    }

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "internalapi",
      "type": "macvlan",
      "master": "enp6s0.20",
      "ipam": {
        "type": "whereabouts",
        "range": "172.17.0.0/24",
        "range_start": "172.17.0.20",
        "range_end": "172.17.0.50",
        "exclude": [
          "172.17.0.24/32",
          "172.17.0.44/31"
        ]
      }
    }

- apiVersion: metallb.io/v1beta1
  kind: IPAddressPool
  spec:
    addresses:
    - 172.17.0.60-172.17.0.70

- apiVersion: metallb.io/v1beta1
  kind: IPAddressPool
  spec:
    addresses:
    - 172.17.0.60-172.17.0.64
    - 172.17.0.66-172.17.0.70

apiVersion: network.openstack.org/v1beta1
kind: NetConfig
metadata:
  name: netconfig
spec:
  networks:
  - name: internalapi
    dnsDomain: internalapi.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.17.0.250
        start: 172.17.0.100
      cidr: 172.17.0.0/24
      vlan: 20
  - name: storage
    dnsDomain: storage.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.18.0.250
        start: 172.18.0.100
      cidr: 172.18.0.0/24
      vlan: 21
  - name: tenant
    dnsDomain: tenant.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.19.0.250
        start: 172.19.0.100
      cidr: 172.19.0.0/24
      vlan: 22

apiVersion: network.openstack.org/v1beta1
kind: NetConfig
metadata:
  name: netconfig
spec:
  networks:
  - name: internalapi
    dnsDomain: internalapi.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.17.0.199
        start: 172.17.0.100
      - end: 172.17.0.250
        start: 172.17.0.201
      cidr: 172.17.0.0/24
      vlan: 20

When looking into the storage in an Red Hat OpenStack Platform (RHOSP) deployment you can differentiate two kinds, the storage requirements of the services themselves and the storage used for the RHOSP users that the services will manage.

These requirements may drive your Red Hat OpenShift Container Platform (RHOCP) node selection, as mentioned above, and may require you to do some preparations on the RHOCP nodes before you can deploy the services.

In order to help users to handle the configuration for the director and Red Hat OpenStack Platform services the tool: https://github.com/openstack-k8s-operators/os-diff has been develop to compare the configuration files between the director deployment and the Red Hat OpenStack Services on OpenShift (RHOSO) cloud. Make sure Golang is installed and configured on your environment:

dnf install -y golang-github-openstack-k8s-operators-os-diff

Then configure the /etc/os-diff/os-diff.cfg file and the /etc/os-diff/ssh.config file according to your environment. To allow os-diff to connect to your clouds and pull files from the services that you describe in the config.yaml file you need to properly set the option in the os-diff.cfg file:

[Default]

local_config_dir=/tmp/
service_config_file=config.yaml

[Tripleo]

ssh_cmd=ssh -F ssh.config
director_host=standalone
container_engine=podman
connection=ssh
remote_config_path=/tmp/tripleo
local_config_path=/tmp/

[Openshift]

ocp_local_config_path=/tmp/ocp
connection=local
ssh_cmd=""

Os-diff uses ssh_cmd to access your director host via SSH, or the host where your cloud is accessible and the podman/docker binary is installed and allowed to interact with the running containers. This option could have a different form:

ssh_cmd=ssh -F ssh.config standalone
director_host=

ssh_cmd=ssh -F ssh.config
director_host=standalone

or without an SSH config file:

ssh_cmd=ssh -i /home/user/.ssh/id_rsa stack@my.undercloud.local
director_host=

or

ssh_cmd=ssh -i /home/user/.ssh/id_rsa stack@
director_host=my.undercloud.local

Note that the result of using ssh_cmd and director_host should be a "successful ssh access".

Configure or generate the ssh.config file from inventory or hosts file, for example:

Host *
    IdentitiesOnly yes

Host virthost
    Hostname virthost
    IdentityFile ~/.ssh/id_rsa
    User root
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null


Host standalone
    Hostname standalone
    IdentityFile <path to SSH key>
    User root
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null

Host crc
    Hostname crc
    IdentityFile ~/.ssh/id_rsa
    User stack
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null

Os-diff can use an ssh.config file for getting access to your Red Hat OpenStack Platform environment. The following command can help you generate this SSH config file from your Ansible inventory, for example, tripleo-ansible-inventory.yaml file:

os-diff configure -i tripleo-ansible-inventory.yaml -o ssh.config --yaml

Host standalone
  HostName standalone
  User root
  IdentityFile ~/.ssh/id_rsa
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

Host undercloud
  HostName undercloud
  User root
  IdentityFile ~/.ssh/id_rsa
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

And test your connection:

ssh -F ssh.config standalone