Deploying a Distributed Compute Node (DCN) architecture

Procedure

1. Log in to the undercloud as the stack user.

2. If you do not already have an undercloud.conf file, copy the sample template file:

   [stack@director ~]$ cp /usr/share/python-tripleoclient/undercloud.conf.sample ~/undercloud.conf

3. Edit the undercloud.conf file.

4. Set the following values in the [DEFAULT] section:

   1. Set local_ip to the undercloud IP on leaf0:

      local_ip = 192.168.10.1/24

   2. Set undercloud_public_host to the externally facing IP address of the undercloud:

      undercloud_public_host = 10.1.1.1

   3. Set undercloud_admin_host to the administration IP address of the undercloud. This IP address is usually on leaf0:

      undercloud_admin_host = 192.168.10.2

   4. Set local_interface to the interface to bridge for the local network:

      local_interface = eth1

   5. Set enable_routed_networks to true:

      enable_routed_networks = true

   6. Define your list of subnets using the subnets parameter. Define one subnet for each L2 segment in the routed spine and leaf:

      subnets = leaf0,leaf1,leaf2

   7. Specify the subnet associated with the physical L2 segment local to the undercloud using the local_subnet parameter:

      local_subnet = leaf0

   8. Set the value of undercloud_nameservers.

      undercloud_nameservers = 10.11.5.19,10.11.5.20

      Tip

      You can find the current IP addresses of the DNS servers that are used for the undercloud nameserver by looking in /etc/resolv.conf.

5. Create a new section for each subnet that you define in the subnets parameter:

   [leaf0]
   cidr = 192.168.10.0/24
   dhcp_start = 192.168.10.10
   dhcp_end = 192.168.10.90
   inspection_iprange = 192.168.10.100,192.168.10.190
   gateway = 192.168.10.1
   masquerade = False
   
   [leaf1]
   cidr = 192.168.11.0/24
   dhcp_start = 192.168.11.10
   dhcp_end = 192.168.11.90
   inspection_iprange = 192.168.11.100,192.168.11.190
   gateway = 192.168.11.1
   masquerade = False
   
   [leaf2]
   cidr = 192.168.12.0/24
   dhcp_start = 192.168.12.10
   dhcp_end = 192.168.12.90
   inspection_iprange = 192.168.12.100,192.168.12.190
   gateway = 192.168.12.1
   masquerade = False

6. Save the undercloud.conf file.

7. Run the undercloud installation command:

   [stack@director ~]$ openstack undercloud install

Procedure

1. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

2. Retrieve a list of your nodes to identify their UUIDs:

   (undercloud)$ openstack baremetal node list

3. Assign each bare metal node that you want to designate for a role with a custom resource class that identifies its leaf network and role.

   openstack baremetal node set \
    --resource-class baremetal.<ROLE> <node>

   • Replace <ROLE> with a name that identifies the role.

   • Replace <node> with the ID of the bare metal node.

     For example, enter the following command to tag a node with UUID 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13 to the Compute role on Leaf2:

     (undercloud)$ openstack baremetal node set \
      --resource-class baremetal.COMPUTE-LEAF2 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13

4. Add each role to your overcloud-baremetal-deploy.yaml if it is not already defined.

5. Define the resource class that you want to assign to the nodes for the role:

   - name: <role>
     count: 1
     defaults:
       resource_class: baremetal.<ROLE>

   • Replace <role> with the name of the role.
   • Replace <ROLE> with a name that identifies the role.

6. In a baremetal-deploy.yaml file, define the resource class that you want to assign to the nodes for the role. Specify the role, profile, quantity, and associated networks that you are deploying:

   - name: <role>
     count: 1
     hostname_format: <role>-%index%
     ansible_playbooks:
       - playbook: bm-deploy-playbook.yaml
     defaults:
       resource_class: baremetal.<ROLE>
       profile: control
       networks:
         - network: external
           subnet: external_subnet
         - network: internal_api
           subnet: internal_api_subnet01
         - network: storage
           subnet: storage_subnet01
         - network: storage_mgmt
           subnet: storage_mgmt_subnet01
         - network: tenant
           subnet: tenant_subnet01
       network_config:
         template: templates/multiple_nics/multiple_nics_dvr.j2
         default_route_network:
           - external

   • Replace <role> with the name of the role.

   • Replace <ROLE> with a name that identifies the role.

     Note

     You must create a baremetal-deploy.yaml environment file for every stack you are deploying, in /home/stack/<stack>.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Check the bare metal nodes:

   $ openstack baremetal node list

3. Ensure that the bare metal nodes are either in enroll or manageable state. If the bare metal node is not in one of these states, the command that sets the physical_network property on the baremetal port fails. To set all nodes to manageable state, run the following command:

   $ for node in $(openstack baremetal node list -f value -c Name); do openstack baremetal node manage $node --wait; done

4. Check which baremetal ports are associated with which baremetal node:

   $ openstack baremetal port list --node <node-uuid>

5. Set the physical-network parameter for the ports. In the example below, three subnets are defined in the configuration: leaf0, leaf1, and leaf2. The local_subnet is leaf0. Because the physical network for the local_subnet is always ctlplane, the baremetal port connected to leaf0 uses ctlplane. The remaining ports use the other leaf names:

   $ openstack baremetal port set --physical-network ctlplane <port-uuid>
   $ openstack baremetal port set --physical-network leaf1 <port-uuid>
   $ openstack baremetal port set --physical-network leaf2 <port-uuid>

6. Introspect the nodes before you deploy the overcloud. Include the --all-manageable and --provide options to set the nodes as available for deployment:

   $ openstack overcloud node introspect --all-manageable --provide

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In the /home/stack/undercloud.conf file, do the following:

   1. Locate the subnets parameter, and add a new subnet for the leaf that you are adding.

      A subnet represents an L2 segment in the routed spine and leaf:

      Example

      In this example, a new subnet (leaf3) is added for the new leaf (leaf3):

      subnets = leaf0,leaf1,leaf2,leaf3

   2. Create a section for the subnet that you added.

      Example

      In this example, the section [leaf3] is added for the new subnet (leaf3):

      [leaf0]
      cidr = 192.168.10.0/24
      dhcp_start = 192.168.10.10
      dhcp_end = 192.168.10.90
      inspection_iprange = 192.168.10.100,192.168.10.190
      gateway = 192.168.10.1
      masquerade = False
      
      [leaf1]
      cidr = 192.168.11.0/24
      dhcp_start = 192.168.11.10
      dhcp_end = 192.168.11.90
      inspection_iprange = 192.168.11.100,192.168.11.190
      gateway = 192.168.11.1
      masquerade = False
      
      [leaf2]
      cidr = 192.168.12.0/24
      dhcp_start = 192.168.12.10
      dhcp_end = 192.168.12.90
      inspection_iprange = 192.168.12.100,192.168.12.190
      gateway = 192.168.12.1
      masquerade = False
      
      [leaf3]
      cidr = 192.168.13.0/24
      dhcp_start = 192.168.13.10
      dhcp_end = 192.168.13.90
      inspection_iprange = 192.168.13.100,192.168.13.190
      gateway = 192.168.13.1
      masquerade = False

4. Save the undercloud.conf file.

5. Reinstall your undercloud:

   $ openstack undercloud install

Procedure

1. Define the central heat stack:

   $ mkdir central
   $ touch central/overrides.yaml

2. Extract data from the central heat stack into a common directory for all DCN sites:

   $ mkdir dcn-common
   $ touch dcn-common/overrides.yaml

3. Define the dcn0 site.

   $ mkdir dcn0
   $ touch dcn0/overrides.yaml

   Note

   The touch is used to provide an example of file organization. Each file must contain the appropriate content for successful deployments.

Procedure

1. Add your Registry Service Account credentials to containers.yaml.

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
         ceph_namespace: registry.redhat.io/rhceph
         ceph_image: rhceph-6-rhel9
         ceph_tag: latest
         name_prefix: openstack-
         namespace: registry.redhat.io/rhosp17-rhel9
         tag: latest
     ContainerImageRegistryCredentials:
       # https://access.redhat.com/RegistryAuthentication
       registry.redhat.io:
         registry-service-account-username: registry-service-account-password

2. Generate the environment file as images-env.yaml:

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file images-env.yaml

   The resulting images-env.yaml file is included as part of the overcloud deployment procedure for the stack for which it is generated.

Procedure

1. Log in to the undercloud host as the stack user.

2. Copy the default roles directory:

   cp -r /usr/share/openstack-tripleo-heat-templates/roles ~/.

3. Create a new file named DistributedComputeDpdk.yaml from the DistributedCompute.yaml file:

   cp roles/DistributedCompute.yaml roles/DistributedComputeDpdk.yaml

4. Add DPDK services to the new DistributedComputeDpdk.yaml file. You can identify the parameters that you need to add by identifying the parameters in the ComputeOvsDpdk.yaml file that are not present in the DistributedComputeDpdk.yaml file.

   diff -u roles/DistributedComputeDpdk.yaml roles/ComputeOvsDpdk.yaml

   In the output, the parameters that are preceded by + are present in the ComputeOvsDpdk.yaml file but are not present in the DistributedComputeDpdk.yaml file. Include these parameters in the new DistributedComputeDpdk.yaml file.

5. Use the DistributedComputeDpdk.yaml to create a DistributedComputeDpdk roles file :

   openstack overcloud roles generate --roles-path ~/roles/ -o ~/roles/roles-custom.yaml DistributedComputeDpdk

Procedure

1. Log in to the undercloud node as the stack user.
2. Locate the network definition file.

3. Modify the network definition file to extend the template to include the StorageMgmtIpSubnet and StorageMgmtNetworkVlanID attributes of the Storage Management network. Set the mtu attribute of the interfaces to 9000.

   The following is an example of implementing these interface settings:

   -
       type: interface
       name: em2
       use_dhcp: false
       mtu: 9000
   -
       type: vlan
       device: em2
       mtu: 9000
       use_dhcp: false
       vlan_id: {get_param: StorageMgmtNetworkVlanID}
       addresses:
       -
           ip_netmask: {get_param: StorageMgmtIpSubnet}
   -
       type: vlan
       device: em2
       mtu: 9000
       use_dhcp: false
       vlan_id: {get_param: StorageNetworkVlanID}
       addresses:
       -
           ip_netmask: {get_param: StorageIpSubnet}

   Note

   For a complete example of a network definition file, see Example network definition file.

4. Save the changes to the network definition file.

   Note

   All network switch ports between servers using the interface with the new MTU setting must be updated to support jumbo frames. If these switch changes are not made, problems will develop at the application layer that can cause the Red Hat Ceph Storage cluster to not reach quorum. If these settings are made and these problems are still observed, verify all hosts using the network configured for jumbo frames can communicate at the configured MTU setting. Use a command like the following example to perform this task:

   ping -M do -s 8972 172.16.1.11

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source /home/stack/stackrc

3. Generate an environment file:

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file /home/stack/central/central-images-env.yaml

4. In the home directory, create directories for each stack that you plan to deploy. Move the network_data.yaml, vip_data.yaml, and overcloud-baremetal-deploy.yaml templates for the central location to /home/stack/central/.

   mkdir /home/stack/central
   mkdir /home/stack/dcn0
   mkdir /home/stack/dcn1
   
   mv network_data.yaml /home/stack/central
   mv vip_data.yaml /home/stack/central
   mv overcloud-baremetal-deploy.yaml /home/stack/central

5. Provision networks for the overcloud. This command takes a definition file for overcloud networks as input. You must use the output file in your command to deploy the overcloud:

   (undercloud)$ openstack overcloud network provision \
   --output /home/stack/central/overcloud-networks-deployed.yaml \
   /home/stack/central/network_data.yaml

6. Provision virtual IPs for the overcloud. This command takes a definition file for virtual IPs as input. You must use the output file in your command to deploy the overcloud:

   (undercloud)$ openstack overcloud network vip provision \
   --stack central \
   --output /home/stack/central/overcloud-vip-deployed.yaml \
   /home/stack/central/vip_data.yaml

7. Provision bare metal instances. This command takes a definition file for bare metal nodes as input. You must use the output file in your command to deploy the overcloud:

   (undercloud)$ openstack overcloud node provision \
   --stack central \
   --network-config \
   -o /home/stack/central/deployed_metal.yaml \
   /home/stack/central/overcloud-baremetal-deploy.yaml

8. Create a file called central/overrides.yaml with settings similar to the following:

   parameter_defaults:
     NtpServer:
       - 0.pool.ntp.org
       - 1.pool.ntp.org
     GlanceBackend: swift

   • ControllerCount: 3 specifies that three nodes will be deployed. These will use swift for glance, lvm for cinder, and host the control-plane services for edge compute nodes.
   • ComputeCount: 0 is an optional parameter to prevent Compute nodes from being deployed with the central Controller nodes.

   • GlanceBackend: swift uses Object Storage (swift) as the Image Service (glance) back end.

     The resulting configuration interacts with the distributed compute nodes (DCNs) in the following ways:

   • The Image service on the DCN creates a cached copy of the image it receives from the central Object Storage back end. The Image service uses HTTP to copy the image from Object Storage to the local disk cache.

     Note

     The central Controller node must be able to connect to the distributed compute node (DCN) site. The central Controller node can use a routed layer 3 connection.

9. Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone, Cinder storage availability zone must match:

   cat > /home/stack/central/site-name.yaml << EOF
   parameter_defaults:
       NovaComputeAvailabilityZone: central
       ControllerExtraConfig:
           nova::availability_zone::default_schedule_zone: central
       NovaCrossAZAttach: false
   EOF

10. Deploy the central Controller node. For example, you can use a deploy.sh file with the following contents:

    openstack overcloud deploy \
    --stack central \
    --templates /usr/share/openstack-tripleo-heat-templates/ \
    -n /home/stack/central/network_data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
    -e /home/stack/central/overcloud-networks-deployed.yaml \
    -e /home/stack/central/overcloud-vip-deployed.yaml \
    -e /home/stack/central/deployed_metal.yaml

    Note

    You must include heat templates for the configuration of networking in your openstack overcloud deploy command. Designing for edge architecture requires spine and leaf networking. See Configure spine-leaf networking for more details.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source /home/stack/stackrc

3. Generate an environment file /home/stack/central/central-images-env.yaml

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file /home/stack/central/central-images-env.yaml

4. Generate roles for the central location using roles appropriate for your environment:

   openstack overcloud roles generate Compute Controller CephStorage \
   -o /home/stack/central/central_roles.yaml

5. In the home directory, create directories for each stack that you plan to deploy. Move the network_data.yaml, vip_data.yaml, and overcloud-baremetal-deploy.yaml templates for the central location to /home/stack/central/.

   mkdir /home/stack/central
   mkdir /home/stack/dcn0
   mkdir /home/stack/dcn1
   
   mv network_data.yaml /home/stack/central
   mv vip_data.yaml /home/stack/central
   mv overcloud-baremetal-deploy.yaml /home/stack/central

6. Provision networks for the overcloud. This command takes a definition file for overcloud networks as input. You must use the output file in your command to deploy the overcloud:

   openstack overcloud network provision \
   --output /home/stack/central/overcloud-networks-deployed.yaml \
   /home/stack/central/network_data.yaml

7. Provision virtual IPs for the overcloud. This command takes a definition file for virtual IPs as input. You must use the output file in your command to deploy the overcloud:

   openstack overcloud network vip provision \
   --stack central \
   --output /home/stack/central/overcloud-vip-deployed.yaml \
   /home/stack/central/vip_data.yaml

8. Provision bare metal instances. This command takes a definition file for bare metal nodes as input. You must use the output file in your command to deploy the overcloud:

   openstack overcloud node provision \
   --stack central \
   --network-config \
   -o /home/stack/central/deployed_metal.yaml \
   /home/stack/central/overcloud-baremetal-deploy.yaml

9. If you are deploying the central location with hyperconverged storage, you must create an initial-ceph.conf configuration file using the following parameters. For more information see Configuring the Red Hat Ceph Storage cluster for HCI:

   [osd]
   osd_memory_target_autotune = true
   osd_numa_auto_affinity = true
   [mgr]
   mgr/cephadm/autotune_memory_target_ratio = 0.2

10. Use the deployed_metal.yaml file as input to the openstack overcloud ceph deploy command. The openstack overcloud ceph deploy command outputs a yaml file that describes the deployed Ceph cluster:

    openstack overcloud ceph deploy \
    --stack central \
    /home/stack/central/deployed_metal.yaml \
    --config /home/stack/central/initial-ceph.conf \
    --output /home/stack/central/deployed_ceph.yaml \
    --container-image-prepare /home/stack/containers.yaml \
    --network-data /home/stack/network-data.yaml \
    --cluster central \
    --roles-data /home/stack/central/central_roles.yaml

    • Include initial-ceph.com only when deploying hyperconverged infrastructure.

11. Verify a functional Ceph deployment before continuing. Use ssh to connect to a server running the ceph-mon service. In an HCI deployment, this is a controller node. Run the following command:

    cephadm shell --config /etc/ceph/central.conf \
    --keyring /etc/ceph/central.client.admin.keyring

    Note

    You must use the --config and --keyring parameters.

12. Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone and the Cinder storage availability zone must match:

    parameter_defaults:
        NovaComputeAvailabilityZone: central
        ControllerExtraConfig:
            nova::availability_zone::default_schedule_zone: central
        NovaCrossAZAttach: false
        CinderStorageAvailabilityZone: central
        GlanceBackendID: central

13. Configure a glance.yaml template with contents similar to the following:

    parameter_defaults:
        GlanceEnabledImportMethods: web-download,copy-image
        GlanceBackend: rbd
        GlanceStoreDescription: 'central rbd glance store'
        GlanceBackendID: central
        CephClusterName: central

14. Deploy the stack for the central location:

    openstack overcloud deploy \
    --stack central \
    --templates /usr/share/openstack-tripleo-heat-templates/ \
    -r /home/stack/central/central_roles.yaml \
    -n ~/network-data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
    -e /home/stack/central/overcloud-networks-deployed.yaml \
    -e /home/stack/central/overcloud-vip-deployed.yaml \
    -e /home/stack/central/deployed_metal.yaml \
    -e /home/stack/central/deployed_ceph.yaml \
    -e ~/central/glance.yaml

15. After you have deployed the overcloud for the central location, data that is needed as input for additional stack deployments for edge sites is exported and placed in the /home/stack/overcloud-deploy directory. Ensure that the central-export.yaml file is present:

    stat /home/stack/overcloud-deploy/central/central-export.yaml

16. Export Ceph specific data:

    openstack overcloud export ceph \
    --stack central \
    --output-file /home/stack/dcn-common/central_ceph_external.yaml

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate an environment file ~/central/central-images-env.yaml

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file ~/central/central-images-env.yaml

4. In the home directory, create directories for each stack that you plan to deploy. Use this to separate templates designed for their respective sites. Move the network_data.yaml, vip_data.yaml, and overcloud-baremetal-deploy.yaml templates for the central location to /home/stack/central/.

   mkdir /home/stack/central
   mkdir /home/stack/dcn0
   mkdir /home/stack/dcn1
   
   mv network_data.yaml /home/stack/central
   mv vip_data.yaml /home/stack/central
   mv overcloud-baremetal-deploy.yaml /home/stack/central

5. Provision networks for the overcloud. This command takes a definition file for overcloud networks as input. You must use the output file in your command to deploy the overcloud:

   openstack overcloud network provision \
   --output /home/stack/central/overcloud-networks-deployed.yaml \
   /home/stack/central/network_data.yaml

6. Provision virtual IPs for the overcloud. This command takes a definition file for virtual IPs as input. You must use the output file in your command to deploy the overcloud:

   openstack overcloud network vip provision \
   --stack central \
   --output /home/stack/central/overcloud-vip-deployed.yaml \
   /home/stack/central/vip_data.yaml

7. Provision bare metal instances. This command takes a definition file for bare metal nodes as input. You must use the output file in your command to deploy the overcloud:

   openstack overcloud node provision \
   --stack central \
   --network-config \
   -o /home/stack/central/deployed_metal.yaml \
   /home/stack/central/overcloud-baremetal-deploy.yaml

8. Configure the naming conventions for your site in the site-name.yaml environment file. The Compute (nova) availability zone and the Block Storage (cinder) availability zone must match:

   cat > /home/stack/central/site-name.yaml << EOF
   parameter_defaults:
       NovaComputeAvailabilityZone: central
       ControllerExtraConfig:
           nova::availability_zone::default_schedule_zone: central
       NovaCrossAZAttach: false
       CinderStorageAvailabilityZone: central
       GlanceBackendID: central
   EOF

9. Configure an external-ceph.yaml template with contents similar to the following:

   parameter_defaults:
       CinderEnableIscsiBackend: false
       CinderEnableRbdBackend: true
       CinderEnableNfsBackend: false
       NovaEnableRbdBackend: true
       GlanceBackend: rbd
       GlanceBackendID: central
       GlanceEnabledImportMethods: web-download,copy-image
       GlanceStoreDescription: 'central rbd glance store'
       CinderRbdPoolName: "openstack-cinder"
       NovaRbdPoolName: "openstack-nova"
       GlanceRbdPoolName: "openstack-images"
       CinderBackupRbdPoolName: "automation-backups"
       GnocchiRbdPoolName: "automation-metrics"
       CephClusterFSID: 38dd387e-837a-437c-891c-7fc69e17a3c
       CephClusterName: central
       CephExternalMonHost: 10.9.0.1,10.9.0.2,10.9.0.3
       CephClientKey: "AQAKtECeLemfiBBdQp7cjNYQRGW9y8GnhhFZg=="
       CephClientUserName: "openstack

10. Deploy the central location:

    openstack overcloud deploy \
    --stack central \
    --templates /usr/share/openstack-tripleo-heat-templates/ \
    -n /home/stack/central/network-data.yaml \
    ...
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
    -e /home/stack/central/overcloud-networks-deployed.yaml \
    -e /home/stack/central/overcloud-vip-deployed.yaml \
    -e /home/stack/central/deployed_metal.yaml \
    -e /home/stack/central/external-ceph.yaml \
    -e /home/stack/central/overcloud-networks-deployed.yaml \
    -e /home/stack/central/central_roles.yaml

11. After you have deployed the overcloud for the central location, data that is needed as input for additional stack deployments for edge sites is exported and placed in the /home/stack/overcloud-deploy directory. Ensure that this control-plane-export.yaml file is present:

    stat ~/overcloud-deploy/control-plane/control-plane-export.yaml

12. Create an environment file called central_ceph_external.yaml with details about the Red Hat Ceph Storage deployment. This file can be passed to additional stack deployments for edge sites.

    parameter_defaults:
      CephExternalMultiConfig:
        - cluster: "central"
          fsid: "3161a3b4-e5ff-42a0-9f53-860403b29a33"
          external_cluster_mon_ips: "172.16.11.84, 172.16.11.87, 172.16.11.92"
          keys:
            - name: "client.openstack"
              caps:
                mgr: "allow *"
                mon: "profile rbd"
                osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
              key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q=="
              mode: "0600"
          dashboard_enabled: false
          ceph_conf_overrides:
            client:
              keyring: /etc/ceph/central.client.openstack.keyring

    • The fsid parameter is the file system ID of your Ceph Storage cluster: This value is specified in the cluster configuration file in the [global] section:

      [global]
      fsid = 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
      ...

    • The key parameter is the ceph client key for the openstack account:

      [root@ceph ~]# ceph auth list
      ...
      [client.openstack]
          key = AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw==
          caps mgr = "allow *"
          caps mon = "profile rbd"
          caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups, profile rbd pool=metrics"
      ...

      For more information about the parameters shown in the sample central_ceph_external.yaml file, see Creating a custom environment file.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate an environment file ~/dcn0/dcn0-images-env.yaml[d]:

   sudo[e] openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file ~/dcn0/dcn0-images-env.yaml

4. Generate a roles file for the edge location. Generate roles for the edge location using roles appropriate for your environment:

   (undercloud)$ openstack overcloud roles \
    generate Compute \
    -o /home/stack/dcn0/dcn0_roles.yaml

5. If you are using ML2/OVS for networking overlay, you must edit the Compute role include the NeutronDhcpAgent and NeutronMetadataAgent services:

   1. Create a role file for the Compute role:

      openstack overcloud roles \
      generate Compute \
      -o /home/stack/dcn0/dcn0_roles.yaml

   2. Edit the /home/stack/dcn0/dcn0_roles.yaml file to include the NeutronDhcpAgent and NeutronMetadataAgent services:

      ...
          - OS::TripleO::Services::MySQLClient
          - OS::TripleO::Services::NeutronBgpVpnBagpipe
      +   - OS::TripleO::Services::NeutronDhcpAgent
      +   - OS::TripleO::Services::NeutronMetadataAgent
          - OS::TripleO::Services::NeutronLinuxbridgeAgent
          - OS::TripleO::Services::NeutronVppAgent
          - OS::TripleO::Services::NovaAZConfig
          - OS::TripleO::Services::NovaCompute
      ...

      For more information, see Preparing for a routed provider network.

6. Provision networks for the overcloud. This command takes a definition file for overcloud networks as input. You must use the output file in your command to deploy the overcloud:

   (undercloud)$ openstack overcloud network provision \
   --output /home/stack/dcn0/overcloud-networks-deployed.yaml \
   /home/stack/dcn0/network_data.yaml

   Important

   If your network_data.yaml template includes additional networks which were not included when you provisioned networks for the central location, then you must re-run the network provisioning command on the central location:

   (undercloud)$ openstack overcloud network provision \
   --output /home/stack/central/overcloud-networks-deployed.yaml \
   /home/stack/central/network_data.yaml

7. Provision bare metal instances. This command takes a definition file for bare metal nodes as input. You must use the output file in your command to deploy the overcloud:

   (undercloud)$ openstack overcloud node provision \
   --stack dcn0 \
   --network-config \
   -o /home/stack/dcn0/deployed_metal.yaml \
   ~/overcloud-baremetal-deploy.yaml

8. Configure the naming conventions for your site in the site-name.yaml environment file.

   parameter_defaults:
       NovaComputeAvailabilityZone: dcn0
       ControllerExtraConfig:
           nova::availability_zone::default_schedule_zone: dcn0
       NovaCrossAZAttach: false

9. Deploy the stack for the dcn0 edge site:

   openstack overcloud deploy \
   --stack dcn0 \
   --templates /usr/share/openstack-tripleo-heat-templates/ \
   -r /home/stack/dcn0/dcn0_roles.yaml \
   -n /home/stack/network_data.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
   -e /home/stack/overcloud-deploy/central/central-export.yaml \
   -e /home/stack/dcn0/overcloud-networks-deployed.yaml \
   -e /home/stack/dcn0/overcloud-vip-deployed.yaml \
   -e /home/stack/dcn0/deployed_metal.yaml

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc credentials file:

   $ source ~/stackrc

3. Include the NovaImageTypeExcludeList parameter in one of your custom environment files:

   parameter_defaults:
     NovaImageTypeExcludeList:
           - raw

4. Include the environment file that contains the NovaImageTypeExcludeList parameter in the overcloud deployment command, along with any other environment files relevant to your deployment:

   openstack overcloud deploy --templates \
   -n network_data.yaml \
   -r roles_data.yaml \
   -e <environment_files> \
   -e <new_environment_file>

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate an environment file ~/dcn0/dcn0-images-env.yaml:

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file /home/stack/dcn0/dcn0-images-env.yaml

4. Generate the appropriate roles for the dcn0 edge location:

   openstack overcloud roles generate DistributedComputeHCI DistributedComputeHCIScaleOut \
   -o ~/dcn0/dcn0_roles.yaml

5. Provision networks for the overcloud. This command takes a definition file for overcloud networks as input. You must use the output file in your command to deploy the overcloud:

   (undercloud)$ openstack overcloud network provision \
   --output /home/stack/dcn0/overcloud-networks-deployed.yaml \
   /home/stack/network_data.yaml

   Important

   If your network_data.yaml template includes additional networks which were not included when you provisioned networks for the central location, then you must re-run the network provisioning command on the central location:

   (undercloud)$ openstack overcloud network provision \
   --output /home/stack/central/overcloud-networks-deployed.yaml \
   /home/stack/central/network_data.yaml

6. Provision bare metal instances. This command takes a definition file for bare metal nodes as input. You must use the output file in your command to deploy the overcloud:

   (undercloud)$ openstack overcloud node provision \
   --stack dcn0 \
   --network-config \
   -o /home/stack/dcn0/deployed_metal.yaml \
   /home/stack/overcloud-baremetal-deploy.yaml

7. If you are deploying the edge site with hyperconverged storage, you must create an initial-ceph.conf configuration file with the following parameters. For more information, see Configuring the Red Hat Ceph Storage cluster for HCI:

   [osd]
   osd_memory_target_autotune = true
   osd_numa_auto_affinity = true
   [mgr]
   mgr/cephadm/autotune_memory_target_ratio = 0.2

8. Use the deployed_metal.yaml file as input to the openstack overcloud ceph deploy command. The openstack overcloud ceph deploy command outputs a yaml file that describes the deployed Ceph cluster:

   openstack overcloud ceph deploy \
   /home/stack/dcn0/deployed_metal.yaml \
   --stack dcn0 \
   --config ~/dcn0/initial-ceph.conf \
   --output ~/dcn0/deployed_ceph.yaml \
   --container-image-prepare ~/containers.yaml \
   --network-data ~/network-data.yaml \
   --cluster dcn0 \
   --roles-data dcn_roles.yaml

   • Include initial-ceph.conf only when deploying hyperconverged infrastructure.

9. Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone and the Cinder storage availability zone must match:

   parameter_defaults:
       NovaComputeAvailabilityZone: dcn0
       ControllerExtraConfig:
           nova::availability_zone::default_schedule_zone: dcn0
       NovaCrossAZAttach: false
       CinderStorageAvailabilityZone: dcn0
       CinderVolumeCluster: dcn0
       GlanceBackendID: dcn0

10. Configure a glance.yaml template with contents similar to the following:

    parameter_defaults:
        GlanceEnabledImportMethods: web-download,copy-image
        GlanceBackend: rbd
        GlanceStoreDescription: 'dcn0 rbd glance store'
        GlanceBackendID: dcn0
        GlanceMultistoreConfig:
          central:
            GlanceBackend: rbd
            GlanceStoreDescription: 'central rbd glance store'
            CephClusterName: central

11. Deploy the stack for the dcn0 location:[d]

    openstack overcloud deploy \
    --stack dcn0 \
    --templates /usr/share/openstack-tripleo-heat-templates/ \
    -r ~/dcn0/dcn0_roles.yaml \
    -n ~/dcn0/network-data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-storage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
    -e /home/stack/overcloud-deploy/central/central-export.yaml \
    -e /home/stack/dcn0/deployed_ceph.yaml \
    -e /home/stack/dcn-common/central_ceph_external.yaml \
    -e /home/stack/dcn0/overcloud-vip-deployed.yaml \
    -e /home/stack/dcn0/deployed_metal.yaml \
    -e /home/stack/dcn0/overcloud-networks-deployed.yaml \
    -e ~/control-plane/glance.yaml

Procedure

1. Create the following pools in your Ceph cluster. If you are deploying at the central location, include the backups and metrics pools:

   [root@ceph ~]# ceph osd pool create volumes <_PGnum_>
   [root@ceph ~]# ceph osd pool create images <_PGnum_>
   [root@ceph ~]# ceph osd pool create vms <_PGnum_>
   [root@ceph ~]# ceph osd pool create backups <_PGnum_>
   [root@ceph ~]# ceph osd pool create metrics <_PGnum_>

   Replace <_PGnum_> with the number of placement groups. You can use the Ceph Placement Groups (PGs) per Pool Calculator to determine a suitable value.

2. Create the OpenStack client user in Ceph to provide the Red Hat OpenStack Platform environment access to the appropriate pools:

   ceph auth add client.openstack mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images'

   Save the provided Ceph client key that is returned. Use this key as the value for the CephClientKey parameter when you configure the undercloud.

   Note

   If you run this command at the central location and plan to use Cinder backup or telemetry services, add allow rwx pool=backups, allow pool=metrics to the command.

3. Save the file system ID of your Ceph Storage cluster. The value of the fsid parameter in the [global] section of your Ceph configuration file is the file system ID:

   [global]
   fsid = 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
   ...

   Use this value as the value for the CephClusterFSID parameter when you configure the undercloud.

4. On the undercloud, create an environment file to configure your nodes to connect to the unmanaged Ceph cluster. Use a recognizable naming convention, such as ceph-external-<SITE>.yaml where SITE is the location for your deployment, such as ceph-external-central.yaml, ceph-external-dcn1.yaml, and so on.

     parameter_defaults:
       # The cluster FSID
       CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19'
       # The CephX user auth key
       CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ=='
       # The list of IPs or hostnames of the Ceph monitors
       CephExternalMonHost: '172.16.1.7, 172.16.1.8, 172.16.1.9'
       # The desired name of the generated key and conf files
       CephClusterName: dcn1

   1. Use the previously saved values for the CephClusterFSID and CephClientKey parameters.
   2. Use a comma delimited list of ip addresses from the Ceph monitors as the value for the CephExternalMonHost parameter.
   3. You must select a unique value for the CephClusterName parameter amongst edge sites. Reusing a name will result in the configuration file being overwritten.

5. If you deployed Red Hat Ceph storage using Red Hat OpenStack Platform director at the central location, then you can export the ceph configuration to an environment file central_ceph_external.yaml. This environment file connects DCN sites to the central hub Ceph cluster, so the information is specific to the Ceph cluster deployed in the previous steps:

   sudo -E openstack overcloud export ceph \
   --stack central \
   --output-file /home/stack/dcn-common/central_ceph_external.yaml

   If the central location has Red Hat Ceph Storage deployed externally, then you cannot use the openstack overcloud export ceph command to generate the central_ceph_external.yaml file. You must create the central_ceph_external.yaml file manually instead:

   parameter_defaults:
     CephExternalMultiConfig:
       - cluster: "central"
         fsid: "3161a3b4-e5ff-42a0-9f53-860403b29a33"
         external_cluster_mon_ips: "172.16.11.84, 172.16.11.87, 172.16.11.92"
         keys:
           - name: "client.openstack"
             caps:
               mgr: "allow *"
               mon: "profile rbd"
               osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
             key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q=="
             mode: "0600"
         dashboard_enabled: false
         ceph_conf_overrides:
           client:
             keyring: /etc/ceph/central.client.openstack.keyring

6. Create an environment file with similar details about each site with an unmanaged Red Hat Ceph Storage cluster for the central location. The openstack overcloud export ceph command does not work for sites with unmanaged Red Hat Ceph Storage clusters. When you update the central location, this file will allow the central location the storage clusters at your edge sites as secondary locations

   parameter_defaults:
     CephExternalMultiConfig:
   cluster: dcn1
   …
   cluster: dcn2
   …

7. Use the external-ceph.yaml, ceph-external-<SITE>.yaml, and the central_ceph_external.yaml environment files when deploying the overcloud:

   openstack overcloud deploy \
       --stack dcn1 \
       --templates /usr/share/openstack-tripleo-heat-templates/ \
       -r ~/dcn1/roles_data.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-storage.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
       -e /home/stack/dnc1/ceph-external-dcn1.yaml  \
       ....
       -e /home/stack/overcloud-deploy/central/central-export.yaml \
       -e /home/stack/dcn-common/central_ceph_external.yaml \
       -e /home/stack/dcn1/dcn_ceph_keys.yaml \
       -e /home/stack/dcn1/role-counts.yaml \
       -e /home/stack/dcn1/ceph.yaml \
       -e /home/stack/dcn1/site-name.yaml \
       -e /home/stack/dcn1/tuning.yaml \
       -e /home/stack/dcn1/glance.yaml

8. Redeploy the central location after all edge locations have been deployed.

Procedure

1. Create a ~/central/glance_update.yaml file similar to the following. This example includes a configuration for two edge sites, dcn0 and dcn1:

     parameter_defaults:
       GlanceEnabledImportMethods: web-download,copy-image
       GlanceBackend: rbd
       GlanceStoreDescription: 'central rbd glance store'
       CephClusterName: central
       GlanceBackendID: central
       GlanceMultistoreConfig:
         dcn0:
           GlanceBackend: rbd
           GlanceStoreDescription: 'dcn0 rbd glance store'
           CephClientUserName: 'openstack'
           CephClusterName: dcn0
           GlanceBackendID: dcn0
         dcn1:
           GlanceBackend: rbd
           GlanceStoreDescription: 'dcn1 rbd glance store'
           CephClientUserName: 'openstack'
           CephClusterName: dcn1
           GlanceBackendID: dcn1

2. Create the dcn_ceph.yaml file. In the following example, this file configures the glance service at the central site as a client of the Ceph clusters of the edge sites, dcn0 and dcn1.

   openstack overcloud export ceph \
   --stack dcn0,dcn1 \
   --output-file ~/central/dcn_ceph.yaml

3. Redeploy the central site using the original templates and include the newly created dcn_ceph.yaml and glance_update.yaml files.

   Note

   Include deployed_metal.yaml from other edge sites in your overcloud deploy command, if their leaf networks were not initially provided while creating the central stack.

   openstack overcloud deploy \
   --stack central \
   --templates /usr/share/openstack-tripleo-heat-templates/ \
   -r ~/central/central_roles.yaml \
   -n ~/network-data.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-storage.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
   -e /home/stack/central/overcloud-networks-deployed.yaml \
   -e /home/stack/central/overcloud-vip-deployed.yaml \
   -e /home/stack/central/deployed_metal.yaml \
   -e /home/stack/central/deployed_ceph.yaml \
   -e /home/stack/central/dcn_ceph.yaml \
   -e /home/stack/central/glance_update.yaml

4. On a controller at the central location, restart the cinder-volume service. If you deployed the central location with the cinder-backup service, then restart the cinder-backup service too:

   ssh tripleo-admin@controller-0 sudo pcs resource restart openstack-cinder-volume
   ssh tripleo-admin@controller-0 sudo pcs resource restart openstack-cinder-backup

Procedure

1. Log in to Red Hat OpenStack Platform Director as stack.

2. Generate the DistributedComputeHCIDashboard role and any other roles appropriate for your environment:

   openstack overcloud roles generate DistributedComputeHCIDashboard -o ~/dnc0/roles.yaml

3. Include the roles.yaml and the network_data_dashboard.yaml in the overcloud deploy command:

   $ openstack overcloud deploy --templates \
   -r ~/<dcn>/<dcn_site_roles>.yaml \
   -n /usr/share/openstack-tripleo-heat-templates/network_data_dashboard.yaml \
   -e <overcloud_environment_files> \
   ...
   -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-dashboard.yaml \

   Note

   The deployment provides the three ip addresses where the dashboard is enabled on the storage network.

1. Retrieve dashboard admin login credentials specific to the selected stack:

   grep grafana_admin /home/stack/config-download/<stack>/cephadm/cephadm-extra-vars-heat.yml

2. Within the inventory specific to the selected stack, /home/stack/config-download/<stack>/cephadm/inventory.yml, locate the DistributedComputeHCI role hosts list and save all three of the storage_ip values. In the example below the first two dashboard IPs are 172.16.11.84 and 172.16.11.87:

   DistributedComputeHCI:
     hosts:
       dcn1-distributed-compute-hci-0:
         ansible_host: 192.168.24.16
   ...
   storage_hostname: dcn1-distributed-compute-hci-0.storage.localdomain
   storage_ip: 172.16.11.84
   ...
       dcn1-distributed-compute-hci-1:
   ansible_host: 192.168.24.22
   ...
   storage_hostname: dcn1-distributed-compute-hci-1.storage.localdomain
   storage_ip: 172.16.11.87

3. You can check that the Ceph Dashboard is active at one of these IP addresses if they are accessible to you. These IP addresses are on the storage network and are not routed. If these IP addresses are not available, you must configure a load balancer for the three IP addresses that you get from the inventory to obtain a virtual IP address for verification.

Procedure

1. Source your credentials file.

   Example

   $ source ~/centralrc

2. Create a file, octavia-dcn-parameters.yaml, and using the syntax shown below, add the Load-balancing service AZs and their management networks for which you want the Ansible playbook to create the required networking resources.

   The value, octavia_controller_AZ_name, is the name of the AZ in which all of the Load-balancer services run:

   octavia_controller_availability_zone: <octavia_controller_AZ_name>
   octavia_availability_zones:
     <octavia_controller_AZ_name>: $ no cidr needed, it uses the already existing subnet
     <octavia_AZ_n>:
       lb_mgmt_subnet_cidr: <CIDR_address_n>
     <octavia_AZ_n2>:
       lb_mgmt_subnet_cidr: <CIDR_address_n2>

   Important

   The names of the Load-balancing service AZs that you specify must match the names of the pre-existing Compute service AZs. You can obtain the names of the Compute service AZs by running openstack availability zone list --compute.

   The Ansible playbook creates a network, subnet, and router for each AZ, and names them using the AZ names that you specify in octavia-dcn-parameters.yaml following this convention: lb-mgmt-<AZ_name>-net, lb-mgmt-<AZ_name>-subnet, and lb-mgmt-<AZ_name>-router, respectively. The exception is for the network resources for octavia_controller_AZ_name: the playbook uses the existing load-balancing management network and subnet, lb-mgmt-net and lb-mgmt-subnet, respectively, and creates an associated router that it names, lb-mgmt-router.

   In this example, 3 AZs are specified: az-central, az-dcn1, and az-dcn2. The az-central AZ uses the existing load-balancing management network, lb-mgmt-net. The other two AZs use 172.47.0.0/16 and 172.48.0.0/16, respectively:

   Example

   octavia_controller_availability_zone: az-central
   octavia_availability_zones:
     az-central: # no cidr needed; it uses the existing subnet
     az-dcn1:
       lb_mgmt_subnet_cidr: 172.47.0.0/16
     az-dcn2:
       lb_mgmt_subnet_cidr: 172.48.0.0/16

3. Run the Ansible playbook, octavia-dcn-deployment.yaml, and include the AZ definitions that you created in octavia-dcn-parameters.yaml:

   Example

   $ ansible-playbook -i overcloud-deploy/central/config-download/\
   central/tripleo-ansible-inventory.yaml \
   /usr/share/ansible/tripleo-playbooks/octavia-dcn-deployment.yaml \
   -e @octavia-dcn-parameters.yaml -e stack=central -v

Verification

1. Confirm that the required lb-mgmt-* subnets are present.

   $ openstack subnet list -c Name -c Subnet

   Sample output

   +-------------------------+------------------+
   | Name                    | Subnet           |
   +-------------------------+------------------+
   | lb-mgmt-az-dcn2-subnet  | 172.48.0.0/16    |
   | segment5                | 10.0.20.0/24     |
   | segment3                | 10.101.30.0/24   |
   | segment2                | 10.101.20.0/24   |
   | lb-mgmt-az-dcn1-subnet  | 172.47.0.0/16    |
   | heat_tempestconf_subnet | 192.168.199.0/24 |
   | segment4                | 10.0.10.0/24     |
   | lb-mgmt-subnet          | 172.24.0.0/16    |
   | segment1                | 10.101.10.0/24   |
   | lb-mgmt-backbone-subnet | 172.49.0.0/16    |
   | segment6                | 10.0.30.0/24     |
   +-------------------------+------------------+

2. Confirm that the required virtual routers are present.

   $ openstack router list -c Name -c Status

   Sample output

   +------------------------+--------+
   | Name                   | Status |
   +------------------------+--------+
   | lb-mgmt-az-dcn2-router | ACTIVE |
   | lb-mgmt-az-dcn1-router | ACTIVE |
   | lb-mgmt-router         | ACTIVE |
   +------------------------+--------+

Procedure

1. Source your credentials file.

   Example

   $ source ~/centralrc

2. Gather the names of the Compute service AZs that you will use to name your Load-balancing service AZs.

   Important

   The names of the Load-balancing service AZ that you create must match the names of your Compute service AZs.

   $ openstack availability zone list --compute

   Sample output

   +------------+-------------+
   | Zone Name  | Zone Status |
   +------------+-------------+
   | az-central | available   |
   | az-dcn1    | available   |
   | az-dcn2    | available   |
   | internal   | available   |
   +------------+-------------+

3. Gather the IDs for the management networks that you will use to create your Load-balancing service AZs:

   $ openstack network list -c Name -c ID

   Sample output

   +--------------------------------------+--------------------------+
   | ID                                   | Name                     |
   +--------------------------------------+--------------------------+
   | 10458d6b-e7c9-436f-92d9-711677c9d9fd | lb-mgmt-az-dcn2-net      |
   | 662a94f5-51eb-4a4c-86c4-52dcbf471ef9 | lb-mgmt-net              |
   | 6b97ef58-2a25-4ea5-931f-b7c07cd09474 | lb-mgmt-backbone-net     |
   | 99f4215b-fad8-432d-8444-1f894154dc30 | heat_tempestconf_network |
   | a2884aaf-846c-4936-9982-3083f6a71d9b | lb-mgmt-az-dcn1-net      |
   | d7f7de6c-0e84-49e2-9042-697fa85d2532 | public                   |
   | e887a9f9-15f7-4854-a797-033cedbfe5f3 | public2                  |
   +--------------------------------------+--------------------------+

4. Create an AZ profile. Repeat this step to create an AZ profile for each Load-balancing service AZ that you want to create:

   $ openstack loadbalancer availabilityzoneprofile create \
   --name <AZ_profile_name> --provider amphora --availability-zone-data '{"compute_zone": "<compute_AZ_name>","management_network": "<lb_mgmt_AZ_net_UUID>"}'

   Example - create profile for az-central

   In this example, an AZ profile (az_profile_central) is created that uses the management network (lb-mgmt-net) on a Compute node that runs in the Compute AZ (az-central):

   $ openstack loadbalancer availabilityzoneprofile create \
   --name az_profile_central --provider amphora --availability-zone-data \
   '{"compute_zone": "az-central","management_network": \
   "662a94f5-51eb-4a4c-86c4-52dcbf471ef9"}'

5. Repeat step 4 to create an AZ profile for each Load-balancing service AZ that you want to create.

   Example - create profile for az-dcn1

   In this example, an AZ profile (az-profile-dcn1) is created that uses the management network (lb-mgmt-az-dcn1-net) on a Compute node that runs in the Compute AZ (az-dcn1):

   $ openstack loadbalancer availabilityzoneprofile create \
   --name az-profile-dcn1 --provider amphora --availability-zone-data \
   '{"compute-zone": "az-dcn1","management-network": \
   "a2884aaf-846c-4936-9982-3083f6a71d9b"}'

   Example - create profile for az-dcn2

   In this example, an AZ profile (az-profile-dcn2) is created that uses the management network (lb-mgmt-az-dcn2-net) on a Compute node that runs in the Compute AZ (az-dcn2):

   $ openstack loadbalancer availabilityzoneprofile create \
   --name az-profile-dcn2 --provider amphora --availability-zone-data \
   '{"compute-zone": "az-dcn2","management-network": \
   "10458d6b-e7c9-436f-92d9-711677c9d9fd"}'

6. Using the AZ profile, create a Load-balancing service AZ. Repeat this step for any additional AZs, using the appropriate profile for each AZ.

   Example - create AZ: az-central

   In this example, a Load-balancing service AZ (az-central) is created by using the AZ profile (az-profile-central):

   $ openstack loadbalancer availabilityzone create --name az-central \
   --availabilityzoneprofile az-profile-central \
   --description "AZ for Headquarters" --enable

   Example - create AZ: az-dcn1

   In this example, a Load-balancing service AZ (az-dcn1) is created by using the AZ profile (az-profile-az-dcn1):

   $ openstack loadbalancer availabilityzone create --name az-dcn1 \
   --availabilityzoneprofile az-profile-az-dcn1 \
   --description "AZ for South Region" --enable

   Example - create AZ: az-dcn2

   In this example, a Load-balancing service AZ (az-dcn2) is created by using the AZ profile (az-profile-az-dcn2):

   $ openstack loadbalancer availabilityzone create --name az-dcn2 \
   --availabilityzoneprofile az-profile-az-dcn2 \
   --description "AZ for North Region" --enable

Procedure

1. Source your credentials file.

   Example

   $ source ~/centralrc

2. To create a load balancer for a DCN environment, use the loadbalancer create command with the --availability-zone option and specify the appropriate AZ.

   Example

   For example, to create a non-terminated HTTPS load balancer (lb1) on a public subnet (public_subnet) on availability zone (az-central), you would enter the following command:

   $ openstack loadbalancer create --name lb1 --vip-subnet-id \
   public_subnet --availability-zone az-central --wait

3. Continue to create your load balancer by adding a listener, pool, health monitor, and load balancer members.

   For more information, see the Configuring load balancing as a service guide.

Procedure

1. On the undercloud, use SSH to connect to the DistributedComputeHCI node that you want to remove:

   $ ssh tripleo-admin@<dcn-computehci-node>

2. Start a cephadm shell. Use the configuration file and keyring file for the site that the host being removed is in:

   $ sudo cephadm shell --config /etc/ceph/dcn2.conf \
   --keyring /etc/ceph/dcn2.client.admin.keyring

3. Record the OSDs (object storage devices) associated with the DistributedComputeHCI node you are removing for use reference in a later step:

   [ceph: root@dcn2-computehci2-1 ~]# ceph osd tree -c /etc/ceph/dcn2.conf
   …
   -3       0.24399     host dcn2-computehci2-1
    1   hdd 0.04880         osd.1                           up  1.00000 1.00000
    7   hdd 0.04880         osd.7                           up  1.00000 1.00000
   11   hdd 0.04880         osd.11                          up  1.00000 1.00000
   15   hdd 0.04880         osd.15                          up  1.00000 1.00000
   18   hdd 0.04880         osd.18                          up  1.00000 1.00000
   …

4. Use SSH to connect to another node in the same cluster and remove the monitor from the cluster:

   $ sudo cephadm shell --config /etc/ceph/dcn2.conf \
   --keyring /etc/ceph/dcn2.client.admin.keyring
   
   [ceph: root@dcn-computehci2-0]# ceph mon remove dcn2-computehci2-1 -c /etc/ceph/dcn2.conf
   removing mon.dcn2-computehci2-1 at [v2:172.23.3.153:3300/0,v1:172.23.3.153:6789/0], there will be 2 monitors

5. Use SSH to log in again to the node that you are removing from the cluster.

6. Stop and disable the mgr service:

   [tripleo-admin@dcn2-computehci2-1 ~]$ sudo systemctl --type=service | grep ceph
   ceph-crash@dcn2-computehci2-1.service    loaded active     running       Ceph crash dump collector
   ceph-mgr@dcn2-computehci2-1.service      loaded active     running       Ceph Manager
   
   [tripleo-admin@dcn2-computehci2-1 ~]$ sudo systemctl stop ceph-mgr@dcn2-computehci2-1
   
   [tripleo-admin@dcn2-computehci2-1 ~]$ sudo systemctl --type=service | grep ceph
   ceph-crash@dcn2-computehci2-1.service  loaded active running Ceph crash dump collector
   
   [tripleo-admin@dcn2-computehci2-1 ~]$ sudo systemctl disable ceph-mgr@dcn2-computehci2-1
   Removed /etc/systemd/system/multi-user.target.wants/ceph-mgr@dcn2-computehci2-1.service.

7. Start the cephadm shell:

   $ sudo cephadm shell --config /etc/ceph/dcn2.conf \
   --keyring /etc/ceph/dcn2.client.admin.keyring

8. Verify that the mgr service for the node is removed from the cluster:

   [ceph: root@dcn2-computehci2-1 ~]# ceph -s
   
   cluster:
       id:     b9b53581-d590-41ac-8463-2f50aa985001
       health: HEALTH_WARN
               3 pools have too many placement groups
               mons are allowing insecure global_id reclaim
   
     services:
       mon: 2 daemons, quorum dcn2-computehci2-2,dcn2-computehci2-0 (age 2h)
       mgr: dcn2-computehci2-2(active, since 20h), standbys: dcn2-computehci2-0
       osd: 15 osds: 15 up (since 3h), 15 in (since 3h)
   
     data:
       pools:   3 pools, 384 pgs
       objects: 32 objects, 88 MiB
       usage:   16 GiB used, 734 GiB / 750 GiB avail
       pgs:     384 active+clean

   Note

   The node that the mgr service is removed from is no longer listed when the mgr service is successfully removed.

9. Export the Red Hat Ceph Storage specification:

   [ceph: root@dcn2-computehci2-1 ~]# ceph orch ls --export > spec.yml

10. Edit the specifications in the spec.yaml file:

    • Remove all instances of the host <dcn-computehci-node> from spec.yml

    • Remove all instances of the <dcn-computehci-node> entry from the following:

      • service_type: osd
      • service_type: mon
      • service_type: host

11. Reapply the Red Hat Ceph Storage specification:

    [ceph: root@dcn2-computehci2-1 /]# ceph orch apply -i spec.yml

12. Remove the OSDs that you identified using ceph osd tree:

    [ceph: root@dcn2-computehci2-1 /]# ceph orch osd rm --zap 1 7 11 15 18
    Scheduled OSD(s) for removal

13. Verify the status of the OSDs being removed. Do not continue until the following command returns no output:

    [ceph: root@dcn2-computehci2-1 /]# ceph orch osd rm status
    OSD_ID  HOST                    STATE     PG_COUNT  REPLACE  FORCE  DRAIN_STARTED_AT
    1       dcn2-computehci2-1      draining  27        False    False  2021-04-23 21:35:51.215361
    7       dcn2-computehci2-1      draining  8         False    False  2021-04-23 21:35:49.111500
    11      dcn2-computehci2-1      draining  14        False    False  2021-04-23 21:35:50.243762

14. Verify that no daemons remain on the host you are removing:

    [ceph: root@dcn2-computehci2-1 /]# ceph orch ps dcn2-computehci2-1

    If daemons are still present, you can remove them with the following command:

    [ceph: root@dcn2-computehci2-1 /]# ceph orch host drain dcn2-computehci2-1

15. Remove the <dcn-computehci-node> host from the Red Hat Ceph Storage cluster:

    [ceph: root@dcn2-computehci2-1 /]# ceph orch host rm dcn2-computehci2-1
    Removed host ‘dcn2-computehci2-1’

Procedure

1. Identify and disable the cinder-volume service on the node you are removing:

   (central) [stack@site-undercloud-0 ~]$ openstack volume service list --service cinder-volume
   | cinder-volume | dcn2-computehci2-1@tripleo_ceph | az-dcn2    | enabled | up    | 2022-03-23T17:41:43.000000 |
   (central) [stack@site-undercloud-0 ~]$ openstack volume service set --disable dcn2-computehci2-1@tripleo_ceph cinder-volume

2. Log on to a different DistributedComputeHCI node in the stack:

   $ ssh tripleo-admin@dcn2-computehci2-0

3. Remove the cinder-volume service associated with the node that you are removing:

   [root@dcn2-computehci2-0 ~]# podman exec -it cinder_volume cinder-manage service remove cinder-volume dcn2-computehci2-1@tripleo_ceph
   Service cinder-volume on host dcn2-computehci2-1@tripleo_ceph removed.

4. Stop and disable the tripleo_cinder_volume service on the node that you are removing:

   [root@dcn2-computehci2-1 ~]# systemctl stop tripleo_cinder_volume.service
   [root@dcn2-computehci2-1 ~]# systemctl disable tripleo_cinder_volume.service
   Removed /etc/systemd/system/multi-user.target.wants/tripleo_cinder_volume.service

Procedure

1. Copy the baremetal-deployment.yaml file:

   cp /home/stack/dcn2/overcloud-baremetal-deploy.yaml \
   /home/stack/dcn2/baremetal-deployment-scaledown.yaml

2. Edit the baremetal-deployement-scaledown.yaml file. Identify the host you want to remove and set the provisioned parameter to have a value of false:

   instances:
   ...
     - hostname: dcn2-computehci2-1
       provisioned: false

3. Remove the node from the stack:

   openstack overcloud node delete --stack dcn2 --baremetal-deployment /home/stack/dcn2/baremetal_deployment_scaledown.yaml

4. Optional: If you are going to reuse the node, use ironic to clean the disk. This is required if the node will host Ceph OSDs:

   openstack baremetal node manage $UUID
   openstack baremetal node clean $UUID --clean-steps '[{"interface":"deploy", "step": "erase_devices_metadata"}]'
   openstack baremetal provide $UUID

5. Redeploy the central site. Include all templates that you used for the initial configuration:

   openstack overcloud deploy \
   --stack central \
   --templates /usr/share/openstack-tripleo-heat-templates/ \
   -r ~/control-plane/central_roles.yaml \
   -n ~/network-data.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-storage.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
   -e /home/stack/central/overcloud-networks-deployed.yaml \
   -e /home/stack/central/overcloud-vip-deployed.yaml \
   -e /home/stack/central/deployed_metal.yaml \
   -e /home/stack/central/deployed_ceph.yaml \
   -e /home/stack/central/dcn_ceph.yaml \
   -e /home/stack/central/glance_update.yaml

Procedure

1. You must set the EtcdIntialClusterState parameter to existing in one of the templates called by your deploy script:

   parameter_defaults:
     EtcdInitialClusterState: existing

2. Redeploy using the deployment script specific to the stack:

   (undercloud) [stack@site-undercloud-0 ~]$ ./overcloud_deploy_dcn2.sh
   …
   Overcloud Deployed without error

3. Export the Red Hat Ceph Storage data from the stack:

   (undercloud) [stack@site-undercloud-0 ~]$ sudo -E openstack overcloud export ceph --stack dcn1,dcn2 --config-download-dir /var/lib/mistral --output-file ~/central/dcn2_scale_up_ceph_external.yaml

4. Replace dcn_ceph_external.yaml with the newly generated dcn2_scale_up_ceph_external.yaml in the deploy script for the central location.

5. Perform a stack update at central:

   (undercloud) [stack@site-undercloud-0 ~]$ ./overcloud_deploy.sh
   ...
   Overcloud Deployed without error

Procedure

1. Ensure the value of the status field is enabled, and that the value of the State field is up:

   (central) [stack@site-undercloud-0 ~]$ openstack compute service list -c Binary -c Host -c Zone -c Status -c State
   +----------------+-----------------------------------------+------------+---------+-------+
   | Binary         | Host                                    | Zone       | Status  | State |
   +----------------+-----------------------------------------+------------+---------+-------+
   ...
   | nova-compute   | dcn1-compute1-0.redhat.local            | az-dcn1    | enabled | up    |
   | nova-compute   | dcn1-compute1-1.redhat.local            | az-dcn1    | enabled | up    |
   | nova-compute   | dcn2-computehciscaleout2-0.redhat.local | az-dcn2    | enabled | up    |
   | nova-compute   | dcn2-computehci2-0.redhat.local         | az-dcn2    | enabled | up    |
   | nova-compute   | dcn2-computescaleout2-0.redhat.local    | az-dcn2    | enabled | up    |
   | nova-compute   | dcn2-computehci2-2.redhat.local         | az-dcn2    | enabled | up    |
   ...

2. Ensure that all network agents are in the up state:

   (central) [stack@site-undercloud-0 ~]$ openstack network agent list -c "Agent Type" -c Host -c Alive -c State
   +--------------------+-----------------------------------------+-------+-------+
   | Agent Type         | Host                                    | Alive | State |
   +--------------------+-----------------------------------------+-------+-------+
   | DHCP agent         | dcn3-compute3-1.redhat.local            | :-)   | UP    |
   | Open vSwitch agent | central-computehci0-1.redhat.local      | :-)   | UP    |
   | DHCP agent         | dcn3-compute3-0.redhat.local            | :-)   | UP    |
   | DHCP agent         | central-controller0-2.redhat.local      | :-)   | UP    |
   | Open vSwitch agent | dcn3-compute3-1.redhat.local            | :-)   | UP    |
   | Open vSwitch agent | dcn1-compute1-1.redhat.local            | :-)   | UP    |
   | Open vSwitch agent | central-computehci0-0.redhat.local      | :-)   | UP    |
   | DHCP agent         | central-controller0-1.redhat.local      | :-)   | UP    |
   | L3 agent           | central-controller0-2.redhat.local      | :-)   | UP    |
   | Metadata agent     | central-controller0-1.redhat.local      | :-)   | UP    |
   | Open vSwitch agent | dcn2-computescaleout2-0.redhat.local    | :-)   | UP    |
   | Open vSwitch agent | dcn2-computehci2-5.redhat.local         | :-)   | UP    |
   | Open vSwitch agent | central-computehci0-2.redhat.local      | :-)   | UP    |
   | DHCP agent         | central-controller0-0.redhat.local      | :-)   | UP    |
   | Open vSwitch agent | central-controller0-1.redhat.local      | :-)   | UP    |
   | Open vSwitch agent | dcn2-computehci2-0.redhat.local         | :-)   | UP    |
   | Open vSwitch agent | dcn1-compute1-0.redhat.local            | :-)   | UP    |
   ...

3. Verify the status of the Ceph Cluster:

   1. Use SSH to connect to the new DistributedComputeHCI node and check the status of the Ceph cluster:

      [root@dcn2-computehci2-5 ~]# podman exec -it ceph-mon-dcn2-computehci2-5 \
      ceph -s -c /etc/ceph/dcn2.conf

   2. Verify that both the ceph mon and ceph mgr services exist for the new node:

      services:
          mon: 3 daemons, quorum dcn2-computehci2-2,dcn2-computehci2-0,dcn2-computehci2-5 (age 3d)
          mgr: dcn2-computehci2-2(active, since 3d), standbys: dcn2-computehci2-0, dcn2-computehci2-5
          osd: 20 osds: 20 up (since 3d), 20 in (since 3d)

   3. Verify the status of the ceph osds with ‘ceph osd tree’. Ensure all osds for our new node are in STATUS up:

      [root@dcn2-computehci2-5 ~]# podman exec -it ceph-mon-dcn2-computehci2-5 ceph osd tree -c /etc/ceph/dcn2.conf
      ID CLASS WEIGHT  TYPE NAME                           STATUS REWEIGHT PRI-AFF
      -1       0.97595 root default
      -5       0.24399     host dcn2-computehci2-0
       0   hdd 0.04880         osd.0                           up  1.00000 1.00000
       4   hdd 0.04880         osd.4                           up  1.00000 1.00000
       8   hdd 0.04880         osd.8                           up  1.00000 1.00000
      13   hdd 0.04880         osd.13                          up  1.00000 1.00000
      17   hdd 0.04880         osd.17                          up  1.00000 1.00000
      -9       0.24399     host dcn2-computehci2-2
       3   hdd 0.04880         osd.3                           up  1.00000 1.00000
       5   hdd 0.04880         osd.5                           up  1.00000 1.00000
      10   hdd 0.04880         osd.10                          up  1.00000 1.00000
      14   hdd 0.04880         osd.14                          up  1.00000 1.00000
      19   hdd 0.04880         osd.19                          up  1.00000 1.00000
      -3       0.24399     host dcn2-computehci2-5
       1   hdd 0.04880         osd.1                           up  1.00000 1.00000
       7   hdd 0.04880         osd.7                           up  1.00000 1.00000
      11   hdd 0.04880         osd.11                          up  1.00000 1.00000
      15   hdd 0.04880         osd.15                          up  1.00000 1.00000
      18   hdd 0.04880         osd.18                          up  1.00000 1.00000
      -7       0.24399     host dcn2-computehciscaleout2-0
       2   hdd 0.04880         osd.2                           up  1.00000 1.00000
       6   hdd 0.04880         osd.6                           up  1.00000 1.00000
       9   hdd 0.04880         osd.9                           up  1.00000 1.00000
      12   hdd 0.04880         osd.12                          up  1.00000 1.00000
      16   hdd 0.04880         osd.16                          up  1.00000 1.00000

4. Verify the cinder-volume service for the new DistributedComputeHCI node is in Status ‘enabled’ and in State ‘up’:

   (central) [stack@site-undercloud-0 ~]$ openstack volume service list --service cinder-volume -c Binary -c Host -c Zone -c Status -c State
   +---------------+---------------------------------+------------+---------+-------+
   | Binary        | Host                            | Zone       | Status  | State |
   +---------------+---------------------------------+------------+---------+-------+
   | cinder-volume | hostgroup@tripleo_ceph          | az-central | enabled | up    |
   | cinder-volume | dcn1-compute1-1@tripleo_ceph    | az-dcn1    | enabled | up    |
   | cinder-volume | dcn1-compute1-0@tripleo_ceph    | az-dcn1    | enabled | up    |
   | cinder-volume | dcn2-computehci2-0@tripleo_ceph | az-dcn2    | enabled | up    |
   | cinder-volume | dcn2-computehci2-2@tripleo_ceph | az-dcn2    | enabled | up    |
   | cinder-volume | dcn2-computehci2-5@tripleo_ceph | az-dcn2    | enabled | up    |
   +---------------+---------------------------------+------------+---------+-------+

   Note

   If the State of the cinder-volume service is down, then the service has not been started on the node.

5. Use ssh to connect to the new DistributedComputeHCI node and check the status of the Glance services with ‘systemctl’:

   [root@dcn2-computehci2-5 ~]# systemctl --type service | grep glance
     tripleo_glance_api.service                        loaded active     running       glance_api container
     tripleo_glance_api_healthcheck.service            loaded activating start   start glance_api healthcheck
     tripleo_glance_api_tls_proxy.service              loaded active     running       glance_api_tls_proxy container

Procedure

1. Log onto the replacement node and check logs for the etcd service. Check that the logs show the etcd service is reporting a cluster ID mismatch in the /var/log/containers/stdouts/etcd.log log file:

   2022-04-06T18:00:11.834104130+00:00 stderr F 2022-04-06 18:00:11.834045 E | rafthttp: request cluster ID mismatch (got 654f4cf0e2cfb9fd want 918b459b36fe2c0c)

2. Set the EtcdInitialClusterState parameter to the value of existing in your deployment templates and rerun the deployment script.

3. Use SSH to connect to the replacement node and run the following commands as root:

   [root@dcn2-computehci2-4 ~]# systemctl stop tripleo_etcd
   [root@dcn2-computehci2-4 ~]# rm -rf /var/lib/etcd/*
   [root@dcn2-computehci2-4 ~]# systemctl start tripleo_etcd

4. Recheck the /var/log/containers/stdouts/etcd.log log file to verify that the node successfully joined the cluster:

   2022-04-06T18:24:22.130059875+00:00 stderr F 2022-04-06 18:24:22.129395 I | etcdserver/membership: added member 96f61470cd1839e5 [https://dcn2-computehci2-4.internalapi.redhat.local:2380] to cluster 654f4cf0e2cfb9fd

5. Check the state of the cinder-volume service, and confirm it reads up on the replacement node when you run openstack volume service list.

Procedure

1. Create an ansible inventory directory for your overcloud stacks:

   $ mkdir inventories
   
   $ find ~/overcloud-deploy/*/config-download \
     -name tripleo-ansible-inventory.yaml |\
     while read f; do cp $f inventories/$(basename $(dirname $f)).yaml; done

2. Create a list of image IDs that you want to pre-cache:

   1. Retrieve a comprehensive list of available images:

      $ source centralrc
      
      $ openstack image list
      +--------------------------------------+---------+--------+
      | ID                                   | Name    | Status |
      +--------------------------------------+---------+--------+
      | 07bc2424-753b-4f65-9da5-5a99d8383fe6 | image_0 | active |
      | d5187afa-c821-4f22-aa4b-4e76382bef86 | image_1 | active |
      +--------------------------------------+---------+--------+

   2. Create an ansible playbook argument file called nova_cache_args.yml, and add the IDs of the images that you want to pre-cache:

      ---
      tripleo_nova_image_cache_images:
        - id: 07bc2424-753b-4f65-9da5-5a99d8383fe6
        - id: d5187afa-c821-4f22-aa4b-4e76382bef86

3. Run the tripleo_nova_image_cache.yml ansible playbook:

   $ source centralrc
   
   $ ansible-playbook -i inventories \
   --extra-vars "@nova_cache_args.yml" \
   /usr/share/ansible/tripleo-playbooks/tripleo_nova_image_cache.yml

Procedure

1. Create a Ceph key for external access. This key is sensitive. You can generate the key using the following:

   python3 -c 'import os,struct,time,base64; key = os.urandom(16) ; \
   header = struct.pack("<hiih", 1, int(time.time()), 0, len(key)) ; \
   print(base64.b64encode(header + key).decode())'

2. In the directory of the stack you are deploying, create a ceph_keys.yaml environment file with contents like the following, using the output from the previous command for the key:

   parameter_defaults:
     CephExtraKeys:
       - name: "client.external"
         caps:
           mgr: "allow *"
           mon: "profile rbd"
           osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
         key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q=="
         mode: "0600"

3. Include the ceph_keys.yaml environment file in the deployment of the site. For example, to deploy the central site with with the ceph_keys.yaml environment file, run a command like the following:

    overcloud deploy \
            --stack central \
            --templates /usr/share/openstack-tripleo-heat-templates/ \
            ….
            -e ~/central/ceph_keys.yaml