Distributed compute node and storage deployment

Procedure

1. Log in to the undercloud as the stack user.

2. If you do not already have an undercloud.conf file, copy the sample template file:

   [stack@director ~]$ cp /usr/share/python-tripleoclient/undercloud.conf.sample ~/undercloud.conf

3. Edit the undercloud.conf file.

4. Set the following values in the [DEFAULT] section:

   1. Set local_ip to the undercloud IP on leaf0:

      local_ip = 192.168.10.1/24

   2. Set undercloud_public_host to the externally facing IP address of the undercloud:

      undercloud_public_host = 10.1.1.1

   3. Set undercloud_admin_host to the administration IP address of the undercloud. This IP address is usually on leaf0:

      undercloud_admin_host = 192.168.10.2

   4. Set local_interface to the interface to bridge for the local network:

      local_interface = eth1

   5. Set enable_routed_networks to true:

      enable_routed_networks = true

   6. Define your list of subnets using the subnets parameter. Define one subnet for each L2 segment in the routed spine and leaf:

      subnets = leaf0,leaf1,leaf2

   7. Specify the subnet associated with the physical L2 segment local to the undercloud using the local_subnet parameter:

      local_subnet = leaf0

   8. Set the value of undercloud_nameservers.

      undercloud_nameservers = 10.11.5.19,10.11.5.20

      Tip

      You can find the current IP addresses of the DNS servers that are used for the undercloud nameserver by looking in /etc/resolv.conf.

5. Create a new section for each subnet that you define in the subnets parameter:

   [leaf0]
   cidr = 192.168.10.0/24
   dhcp_start = 192.168.10.10
   dhcp_end = 192.168.10.90
   inspection_iprange = 192.168.10.100,192.168.10.190
   gateway = 192.168.10.1
   masquerade = False
   
   [leaf1]
   cidr = 192.168.11.0/24
   dhcp_start = 192.168.11.10
   dhcp_end = 192.168.11.90
   inspection_iprange = 192.168.11.100,192.168.11.190
   gateway = 192.168.11.1
   masquerade = False
   
   [leaf2]
   cidr = 192.168.12.0/24
   dhcp_start = 192.168.12.10
   dhcp_end = 192.168.12.90
   inspection_iprange = 192.168.12.100,192.168.12.190
   gateway = 192.168.12.1
   masquerade = False

6. Save the undercloud.conf file.

7. Run the undercloud installation command:

   [stack@director ~]$ openstack undercloud install

Procedure

1. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

2. Create flavors for each custom role:

   $ ROLES="control compute_leaf0 compute_leaf1 compute_leaf2 ceph-storage_leaf0 ceph-storage_leaf1 ceph-storage_leaf2"
   $ for ROLE in $ROLES; do openstack flavor create --id auto --ram <ram_size_mb> --disk <disk_size_gb> --vcpus <no_vcpus> $ROLE ; done
   $ for ROLE in $ROLES; do openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property resources:DISK_GB='0' --property resources:MEMORY_MB='0' --property resources:VCPU='0' $ROLE ; done

   • Replace <ram_size_mb> with the RAM of the bare metal node, in MB.
   • Replace <disk_size_gb> with the size of the disk on the bare metal node, in GB.
   • Replace <no_vcpus> with the number of CPUs on the bare metal node.

3. Retrieve a list of your nodes to identify their UUIDs:

   (undercloud)$ openstack baremetal node list

4. Tag each bare metal node to its leaf network and role by using a custom resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.LEAF-ROLE <node>

   Replace <node> with the ID of the bare metal node.

   For example, enter the following command to tag a node with UUID 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13 to the Compute role on Leaf2:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.COMPUTE-LEAF2 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13

5. Associate each leaf network role flavor with the custom resource class:

   (undercloud)$ openstack flavor set \
    --property resources:CUSTOM_BAREMETAL_LEAF_ROLE=1 \
   <custom_role>

   To determine the name of a custom resource class that corresponds to a resource class of a Bare Metal Provisioning service node, convert the resource class to uppercase, replace each punctuation mark with an underscore, and prefix with CUSTOM_.

   Note

   A flavor can request only one instance of a bare metal resource class.

6. In the node-info.yaml file, specify the flavor that you want to use for each custom leaf role, and the number of nodes to allocate for each custom leaf role. For example, the following configuration specifies the flavor to use, and the number of nodes to allocate for the custom leaf roles compute_leaf0, compute_leaf1, compute_leaf2, ceph-storage_leaf0, ceph-storage_leaf1, and ceph-storage_leaf2:

   parameter_defaults:
     OvercloudControllerFlavor: control
     OvercloudComputeLeaf0Flavor: compute_leaf0
     OvercloudComputeLeaf1Flavor: compute_leaf1
     OvercloudComputeLeaf2Flavor: compute_leaf2
     OvercloudCephStorageLeaf0Flavor: ceph-storage_leaf0
     OvercloudCephStorageLeaf1Flavor: ceph-storage_leaf1
     OvercloudCephStorageLeaf2Flavor: ceph-storage_leaf2
     ControllerLeaf0Count: 3
     ComputeLeaf0Count: 3
     ComputeLeaf1Count: 3
     ComputeLeaf2Count: 3
     CephStorageLeaf0Count: 3
     CephStorageLeaf1Count: 3
     CephStorageLeaf2Count: 3

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Check the bare metal nodes:

   $ openstack baremetal node list

3. Ensure that the bare metal nodes are either in enroll or manageable state. If the bare metal node is not in one of these states, the command that sets the physical_network property on the baremetal port fails. To set all nodes to manageable state, run the following command:

   $ for node in $(openstack baremetal node list -f value -c Name); do openstack baremetal node manage $node --wait; done

4. Check which baremetal ports are associated with which baremetal node:

   $ openstack baremetal port list --node <node-uuid>

5. Set the physical-network parameter for the ports. In the example below, three subnets are defined in the configuration: leaf0, leaf1, and leaf2. The local_subnet is leaf0. Because the physical network for the local_subnet is always ctlplane, the baremetal port connected to leaf0 uses ctlplane. The remaining ports use the other leaf names:

   $ openstack baremetal port set --physical-network ctlplane <port-uuid>
   $ openstack baremetal port set --physical-network leaf1 <port-uuid>
   $ openstack baremetal port set --physical-network leaf2 <port-uuid>

6. Introspect the nodes before you deploy the overcloud. Include the --all-manageable and --provide options to set the nodes as available for deployment:

   $ openstack overcloud node introspect --all-manageable --provide

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the undercloud credentials file:

   $ source ~/stackrc

3. In the /home/stack/undercloud.conf file, do the following:

   1. Locate the subnets parameter, and add a new subnet for the leaf that you are adding.

      A subnet represents an L2 segment in the routed spine and leaf:

      Example

      In this example, a new subnet (leaf3) is added for the new leaf (leaf3):

      subnets = leaf0,leaf1,leaf2,leaf3

   2. Create a section for the subnet that you added.

      Example

      In this example, the section [leaf3] is added for the new subnet (leaf3):

      [leaf0]
      cidr = 192.168.10.0/24
      dhcp_start = 192.168.10.10
      dhcp_end = 192.168.10.90
      inspection_iprange = 192.168.10.100,192.168.10.190
      gateway = 192.168.10.1
      masquerade = False
      
      [leaf1]
      cidr = 192.168.11.0/24
      dhcp_start = 192.168.11.10
      dhcp_end = 192.168.11.90
      inspection_iprange = 192.168.11.100,192.168.11.190
      gateway = 192.168.11.1
      masquerade = False
      
      [leaf2]
      cidr = 192.168.12.0/24
      dhcp_start = 192.168.12.10
      dhcp_end = 192.168.12.90
      inspection_iprange = 192.168.12.100,192.168.12.190
      gateway = 192.168.12.1
      masquerade = False
      
      [leaf3]
      cidr = 192.168.13.0/24
      dhcp_start = 192.168.13.10
      dhcp_end = 192.168.13.90
      inspection_iprange = 192.168.13.100,192.168.13.190
      gateway = 192.168.13.1
      masquerade = False

4. Save the undercloud.conf file.

5. Reinstall your undercloud:

   $ openstack undercloud install

Procedure

1. Deploy the central stack with ManageNetworks: true or leave unset.
2. Deploy the additional stack with ManageNetworks: false.

Procedure

1. Identify the UUIDs of the related network resources:

   $ openstack network show internal_api -c id -f value
   $ openstack subnet show internal_api_subnet -c id -f value
   $ openstack port show internal_api_virtual_ip -c id -f value

2. Save the values that are shown in the output of the above commands and add them to the network definition for the internal_api network in the network_data.yaml file for the separate stack:

   - name: InternalApi
     external_resource_network_id: 93861871-7814-4dbc-9e6c-7f51496b43af
     external_resource_subnet_id: c85c8670-51c1-4b17-a580-1cfb4344de27
     external_resource_vip_id: 8bb9d96f-72bf-4964-a05c-5d3fed203eb7
     name_lower: internal_api
     vip: true
     ip_subnet: '172.16.2.0/24'
     allocation_pools: [{'start': '172.16.2.4', 'end': '172.16.2.250'}]
     ipv6_subnet: 'fd00:fd00:fd00:2000::/64'
     ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}]
     mtu: 1400

Procedure

1. Define the central heat stack:

   $ mkdir central
   $ touch central/overrides.yaml

2. Extract data from the central heat stack into a common directory for all DCN sites:

   $ mkdir dcn-common
   $ touch dcn-common/overrides.yaml
   $ touch dcn-common/central-export.yaml

   The central-export.yaml file is created later by the openstack overcloud export command. It is in the dcn-common directory because all DCN deployments in this guide must use this file.

3. Define the dcn0 site.

   $ mkdir dcn0
   $ touch dcn0/overrides.yaml

Procedure

1. Add your Registry Service Account credentials to containers.yaml.

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
         ceph_namespace: registry.redhat.io/rhceph
         ceph_image: rhceph-4-rhel8
         ceph_tag: latest
         name_prefix: openstack-
         namespace: registry.redhat.io/rhosp16-rhel8
         tag: latest
     ContainerImageRegistryCredentials:
       # https://access.redhat.com/RegistryAuthentication
       registry.redhat.io:
         registry-service-account-username: registry-service-account-password

2. Generate the environment file as images-env.yaml:

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file images-env.yaml

   The resulting images-env.yaml file is included as part of the overcloud deployment procedure for the stack for which it is generated.

Procedure

1. Log in to the undercloud host as the stack user.

2. Copy the default roles directory:

   cp -r /usr/share/openstack-tripleo-heat-templates/roles ~/.

3. Create a new file named DistributedComputeDpdk.yaml from the DistributedCompute.yaml file:

   cp roles/DistributedCompute.yaml roles/DistributedComputeDpdk.yaml

4. Add DPDK services to the new DistributedComputeDpdk.yaml file. You can identify the parameters that you need to add by identifying the parameters in the ComputeOvsDpdk.yaml file that are not present in the DistributedComputeDpdk.yaml file.

   diff -u roles/DistributedComputeDpdk.yaml roles/ComputeOvsDpdk.yaml

   In the output, the parameters that are preceded by + are present in the ComputeOvsDpdk.yaml file but are not present in the DistributedComputeDpdk.yaml file. Include these parameters in the new DistributedComputeDpdk.yaml file.

5. Use the DistributedComputeDpdk.yaml to create a DistributedComputeDpdk roles file :

   openstack overcloud roles generate --roles-path ~/roles/ -o ~/roles/roles-custom.yaml DistributedComputeDpdk

1. In the home directory, create directories for each stack that you plan to deploy.

   mkdir /home/stack/central
   mkdir /home/stack/dcn0
   mkdir /home/stack/dcn1

2. Create a file called central/overrides.yaml with settings similar to the following:

   parameter_defaults:
     NtpServer:
       - 0.pool.ntp.org
       - 1.pool.ntp.org
     ControllerCount: 3
     ComputeCount: 0
     OvercloudControllerFlavor: baremetal
     OvercloudComputeFlavor: baremetal
     ControllerSchedulerHints:
       'capabilities:node': '0-controller-%index%'
     GlanceBackend: swift

   • ControllerCount: 3 specifies that three nodes will be deployed. These will use swift for glance, lvm for cinder, and host the control-plane services for edge compute nodes.
   • ComputeCount: 0 is an optional parameter to prevent Compute nodes from being deployed with the central Controller nodes.

   • GlanceBackend: swift uses Object Storage (swift) as the Image Service (glance) back end.

     The resulting configuration interacts with the distributed compute nodes (DCNs) in the following ways:

   • The Image service on the DCN creates a cached copy of the image it receives from the central Object Storage back end. The Image service uses HTTP to copy the image from Object Storage to the local disk cache.

     Note

     The central Controller node must be able to connect to the distributed compute node (DCN) site. The central Controller node can use a routed layer 3 connection.

3. Generate roles for the central location using roles appropriate for your environment:

   openstack overcloud roles generate Controller \
   -o ~/central/control_plane_roles.yaml

4. Generate an environment file ~/central/central-images-env.yaml:

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file ~/central/central-images-env.yaml

5. Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone, Cinder storage availability zone must match:

   cat > /home/stack/central/site-name.yaml << EOF
   parameter_defaults:
       NovaComputeAvailabilityZone: central
       ControllerExtraConfig:
           nova::availability_zone::default_schedule_zone: central
       NovaCrossAZAttach: false
       CinderStorageAvailabilityZone: central
   EOF

6. Deploy the central Controller node. For example, you can use a deploy.sh file with the following contents:

   #!/bin/bash
   
   source ~/stackrc
   time openstack overcloud deploy \
   --stack central \
   --templates /usr/share/openstack-tripleo-heat-templates/ \
   -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
   -e ~/central/containers-env-file.yaml \
   -e ~/central/overrides.yaml \
   -e ~/central/site-name.yaml

1. In the home directory, create directories for each stack that you plan to deploy.

   mkdir /home/stack/central
   mkdir /home/stack/dcn0
   mkdir /home/stack/dcn1

2. Set the name of the Ceph cluster, as well as configuration parameters relative to the available hardware. For more information, see Configuring Ceph with Custom Config Settings:

   cat > /home/stack/central/ceph.yaml << EOF
   parameter_defaults:
     CephClusterName: central
     CephAnsibleDisksConfig:
       osd_scenario: lvm
       osd_objectstore: bluestore
       devices:
         - /dev/sda
         - /dev/sdb
     CephPoolDefaultSize: 3
     CephPoolDefaultPgNum: 128
   
   EOF

3. Generate roles for the central location using roles appropriate for your environment:

   openstack overcloud roles generate Compute Controller CephStorage \
   -o ~/central/central_roles.yaml
   
   cat > /home/stack/central/role-counts.yaml << EOF
   parameter_defaults:
     ControllerCount: 3
     ComputeCount: 2
     CephStorage: 3
   EOF

4. Generate an environment file ~/central/central-images-env.yaml

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file ~/central/central-images-env.yaml

5. Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone and the Cinder storage availability zone must match:

   cat > /home/stack/central/site-name.yaml << EOF
   parameter_defaults:
       NovaComputeAvailabilityZone: central
       ControllerExtraConfig:
           nova::availability_zone::default_schedule_zone: central
       NovaCrossAZAttach: false
       CinderStorageAvailabilityZone: central
       GlanceBackendID: central
   EOF

6. Configure a glance.yaml template with contents similar to the following:

   parameter_defaults:
       GlanceEnabledImportMethods: web-download,copy-image
       GlanceBackend: rbd
       GlanceStoreDescription: 'central rbd glance store'
       GlanceBackendID: central
       CephClusterName: central

7. After you prepare all of the other templates, deploy the central stack:

   openstack overcloud deploy \
          --stack central \
          --templates /usr/share/openstack-tripleo-heat-templates/ \
          -r ~/central/central_roles.yaml \
       ...
          -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
          -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
          -e ~/central/central-images-env.yaml \
          -e ~/central/role-counts.yaml \
          -e ~/central/site-name.yaml \
          -e ~/central/ceph.yaml \
          -e ~/central/glance.yaml

1. In the home directory, create directories for each stack that you plan to deploy. Use this to separate templates designed for their respective sites.

   mkdir /home/stack/central
   mkdir /home/stack/dcn0
   mkdir /home/stack/dcn1

2. Generate roles for the central location using roles that Red Hat OpenStack Platform director manages. When integrating with external Ceph, do not use Ceph roles:

   cat > /home/stack/central/role-counts.yaml << EOF
   parameter_defaults:
     ControllerCount: 3
     ComputeCount: 2
   EOF

3. Generate an environment file ~/central/central-images-env.yaml

   sudo openstack tripleo container image prepare \
   -e containers.yaml \
   --output-env-file ~/central/central-images-env.yaml

4. Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone and the Cinder storage availability zone must match:

   cat > /home/stack/central/site-name.yaml << EOF
   parameter_defaults:
       NovaComputeAvailabilityZone: central
       ControllerExtraConfig:
           nova::availability_zone::default_schedule_zone: central
       NovaCrossAZAttach: false
       CinderStorageAvailabilityZone: central
       GlanceBackendID: central
   EOF

5. Configure a glance.yaml template with contents similar to the following:

   parameter_defaults:
       GlanceEnabledImportMethods: web-download,copy-image
       GlanceBackend: rbd
       GlanceStoreDescription: 'central rbd glance store'
       GlanceBackendID: central
       CephClusterName: central

6. When Ceph is deployed without Red Hat OpenStack Platform director, do not use the ceph-ansible.yaml environment file. Use the ceph-ansible-external.yaml environment file instead.

   openstack overcloud deploy \
          --stack central \
          --templates /usr/share/openstack-tripleo-heat-templates/ \
          -r ~/central/central_roles.yaml \
       ...
          -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml \
          -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
          -e ~/central/central-images-env.yaml \
          -e ~/central/role-counts.yaml \
          -e ~/central/site-name.yaml \
          -e ~/central/ceph.yaml \
          -e ~/central/glance.yaml

Procedure

1. Export stack information from the central stack. You must deploy the central stack before running this command:

   openstack overcloud export \
           --config-download-dir /var/lib/mistral/central/ \
           --stack central \
           --output-file ~/dcn-common/central-export.yaml

   Note

   The config-download-dir value defaults to /var/lib/mistral/<stack>/.

2. Create the central_ceph_external.yaml file. This environment file connects DCN sites to the central hub Ceph cluster, so the information is specific to the Ceph cluster deployed in the previous steps.

   sudo -E openstack overcloud export ceph \
   --stack central \
   --config-download-dir /var/lib/mistral \
   --output-file ~/dcn-common/central_ceph_external.yaml

   When Ceph is deployed without Red Hat OpenStack Platform director, you cannot run the openstack overcloud export ceph command. Manually create the central_ceph_external.yaml file:

   parameter_defaults:
     CephExternalMultiConfig:
       - cluster: "central"
         fsid: "3161a3b4-e5ff-42a0-9f53-860403b29a33"
         external_cluster_mon_ips: "172.16.11.84, 172.16.11.87, 172.16.11.92"
         keys:
           - name: "client.openstack"
             caps:
               mgr: "allow *"
               mon: "profile rbd"
               osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
             key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q=="
             mode: "0600"
         dashboard_enabled: false
         ceph_conf_overrides:
           client:
             keyring: /etc/ceph/central.client.openstack.keyring

   • The fsid parameter is the file system ID of your Ceph Storage cluster: This value is specified in the cluster configuration file in the [global] section:

     [global]
     fsid = 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
     ...

   • The key parameter is the ceph client key for the openstack account:

     [root@ceph ~]# ceph auth list
     ...
     [client.openstack]
         key = AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw==
         caps mgr = "allow *"
         caps mon = "profile rbd"
         caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups, profile rbd pool=metrics"
     ...

     For more information on the parameters shown in the sample central_ceph_external.yaml file, see Creating a custom environment file.

3. Create the ~/dcn0/glance.yaml file for Image service configuration overrides:

   parameter_defaults:
     GlanceEnabledImportMethods: web-download,copy-image
     GlanceBackend: rbd
     GlanceStoreDescription: 'dcn0 rbd glance store'
     GlanceBackendID: dcn0
     GlanceMultistoreConfig:
       central:
         GlanceBackend: rbd
         GlanceStoreDescription: 'central rbd glance store'
         CephClientUserName: 'openstack'
         CephClusterName: central

4. Configure the ceph.yaml file with configuration parameters relative to the available hardware.

   cat > /home/stack/dcn0/ceph.yaml << EOF
   parameter_defaults:
     CephClusterName: dcn0
     CephAnsibleDisksConfig:
       osd_scenario: lvm
       osd_objectstore: bluestore
       devices:
         - /dev/sda
         - /dev/sdb
     CephPoolDefaultSize: 3
     CephPoolDefaultPgNum: 128
   EOF

   For more information, see Mapping the Ceph Storage node disk layout.

5. Implement system tuning by using a file that contains the following parameters tuned to the requirements of you environment:

   cat > /home/stack/dcn0/tuning.yaml << EOF
   parameter_defaults:
     CephAnsibleExtraConfig:
       is_hci: true
     CephConfigOverrides:
       osd_recovery_op_priority: 3
       osd_recovery_max_active: 3
       osd_max_backfills: 1
    ## Set relative to your hardware:
     # DistributedComputeHCIParameters:
     #   NovaReservedHostMemory: 181000
     # DistributedComputeHCIExtraConfig:
     #   nova::cpu_allocation_ratio: 8.2
   EOF

   • For more information about setting the values for the parameters CephAnsibleExtraConfig, see Setting ceph-ansible group variables.
   • For more information about setting the values for the parameters CephConfigOverrides, see Customizing the Ceph Storage cluster.

6. Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone and the Cinder storage availability zone must match. The CinderVolumeCluster parameter is included when deploying an edge site with storage. This parameter is used when cinder-volume is deployed as active/active, which is required at edge sites. As a best practice, set the Cinder cluster name to match the availability zone:

   cat > /home/stack/central/site-name.yaml << EOF
   parameter_defaults:
       ...
       NovaComputeAvailabilityZone: dcn0
       NovaCrossAZAttach: false
       CinderStorageAvailabilityZone: dcn0
       CinderVolumeCluster: dcn0

7. Generate the roles.yaml file to be used for the dcn0 deployment, for example:

   openstack overcloud roles generate DistributedComputeHCI DistributedComputeHCIScaleOut -o ~/dcn0/roles_data.yaml

8. Set the number systems in each role by creating the ~/dcn0/roles-counts.yaml file with the desired values for each role.

   When using hyperconverged infrastructure (HCI), you must allocate three nodes to the DistributedComputeHCICount role to satisfy requirements for Ceph Mon and GlanceApiEdge services.

   parameter_defaults:
     ControllerCount: 0
     ComputeCount: 0
     DistributedComputeHCICount: 3
     DistributedComputeHCIScaleOutCount: 1  # Optional
     DistributedComputeScaleOutCount: 1     # Optional

9. Retrieve the container images for the edge site:

   sudo openstack tripleo container image prepare \
   --environment-directory dcn0 \
   -r ~/dcn0/roles_data.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
   ...
   -e /home/stack/dcn-common/central-export.yaml \
   -e /home/stack/containers-prepare-parameter.yaml \
   --output-env-file ~/dcn0/dcn0-images-env.yaml

   Note

   You must include all environment files to be used for the deployment in the openstack tripleo container image prepare command.

10. Deploy the edge site:

    openstack overcloud deploy \
        --stack dcn0 \
        --templates /usr/share/openstack-tripleo-heat-templates/ \
        -r ~/dcn0/roles_data.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
        -e ~/dnc0/dcn0-images-env.yaml \
        ....
        -e ~/dcn-common/central-export.yaml \
        -e ~/dcn-common/central_ceph_external.yaml \
        -e ~/dcn0/dcn_ceph_keys.yaml \
        -e ~/dcn0/role-counts.yaml \
        -e ~/dcn0/ceph.yaml \
        -e ~/dcn0/site-name.yaml \
        -e ~/dcn0/tuning.yaml \
        -e ~/dcn0/glance.yaml

    Note

    You must include heat templates for the configuration of networking in your openstack overcloud deploy command. Designing for edge architecture requires spine and leaf networking. See Spine Leaf Networking for more details.

Procedure

1. As the stack user on the undercloud, create a new directory for dcn9:

   $ cd ~
   $ mkdir dcn9

2. Copy the existing dcn0 templates to the new directory and replace the dcn0 strings with dcn9:

   $ cp dcn0/ceph.yaml dcn9/ceph.yaml
   $ sed s/dcn0/dcn9/g -i dcn9/ceph.yaml
   $ cp dcn0/overrides.yaml dcn9/overrides.yaml
   $ sed s/dcn0/dcn9/g -i dcn9/overrides.yaml
   $ sed s/"0-ceph-%index%"/"9-ceph-%index%"/g -i dcn9/overrides.yaml
   $ cp dcn0/deploy.sh dcn9/deploy.sh
   $ sed s/dcn0/dcn9/g -i dcn9/deploy.sh

3. Review the files in the dcn9 directory to confirm that they suit your requirements.

4. Edit undercloud.conf to add a new leaf. In the following example, leaf9 is added to undercloud.conf:

   [leaf0]
   cidr = 192.168.10.0/24
   dhcp_start = 192.168.10.10
   dhcp_end = 192.168.10.90
   inspection_iprange = 192.168.10.100,192.168.10.190
   gateway = 192.168.10.1
   masquerade = False
   
   …
   [leaf9]
   cidr = 192.168.19.0/24
   dhcp_start = 192.168.19.10
   dhcp_end = 192.168.19.90
   inspection_iprange = 192.168.19.100,192.168.19.190
   gateway = 192.168.10.1
   masquerade = False

5. Rerun the openstack undercloud install command to update the environment configuration.

6. In your overcloud templates, update the value of the NetworkDeploymentActions parameter from a value of ["CREATE"], to a value of ["CREATE", "UPDATE"]. If this parameter is not currently included in your templates, add it to one of your environment files, or create a new environment file.

   cat > /home/stack/central/network-environment.yaml << EOF
   parameter_defaults:
     NetworkDeploymentActions: ["CREATE", "UPDATE"]
   EOF

7. Run the deploy script for the central location. Include all templates that you used when you first deployed the central location, as well as the newly created or edited network-environment.yaml file:

   openstack overcloud deploy \
       --stack central \
       --templates /usr/share/openstack-tripleo-heat-templates/ \
       -r ~/central/roles_data.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
       -e ~/central/dcn9-images-env.yaml \
       ....
       -e ~/dcn-common/central-export.yaml \
       -e ~/dcn-common/central_ceph_external.yaml \
       -e ~/central/dcn_ceph_keys.yaml \
       -e ~/central/role-counts.yaml \
       -e ~/central/ceph.yaml \
       -e ~/central/site-name.yaml \
       -e ~/central/tuning.yaml \
       -e ~/central/glance.yaml

8. Verify that your nodes are available and in Provisioning state:

   $ openstack baremetal node list

9. When your nodes are available, deploy the new edge site with all appropriate templates:

   openstack overcloud deploy \
       --stack dcn9 \
       --templates /usr/share/openstack-tripleo-heat-templates/ \
       -r ~/dcn9/roles_data.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
       -e ~/dnc9/dcn9-images-env.yaml \
       ....
       -e ~/dcn-common/central-export.yaml \
       -e ~/dcn-common/central_ceph_external.yaml \
       -e ~/dcn9/dcn_ceph_keys.yaml \
       -e ~/dcn9/role-counts.yaml \
       -e ~/dcn9/ceph.yaml \
       -e ~/dcn9/site-name.yaml \
       -e ~/dcn9/tuning.yaml \
       -e ~/dcn9/glance.yaml

10. If you’ve deployed the locations with direct edge-to-edge communication, you must redeploy each edge site to update routes and establish communication with the new location.

Procedure

1. Create a ~/central/glance_update.yaml file similar to the following. This example includes a configuration for two edge sites, dcn0 and dcn1:

     parameter_defaults:
       GlanceEnabledImportMethods: web-download,copy-image
       GlanceBackend: rbd
       GlanceStoreDescription: 'central rbd glance store'
       CephClusterName: central
       GlanceBackendID: central
       GlanceMultistoreConfig:
         dcn0:
           GlanceBackend: rbd
           GlanceStoreDescription: 'dcn0 rbd glance store'
           CephClientUserName: 'openstack'
           CephClusterName: dcn0
           GlanceBackendID: dcn0
         dcn1:
           GlanceBackend: rbd
           GlanceStoreDescription: 'dcn1 rbd glance store'
           CephClientUserName: 'openstack'
           CephClusterName: dcn1
           GlanceBackendID: dcn1

2. Create the dcn_ceph.yaml file. In the following example, this file configures the glance service at the central site as a client of the Ceph clusters of the edge sites, dcn0 and dcn1.

   sudo -E openstack overcloud export ceph \
   --stack dcn0,dcn1 \
   --config-download-dir /var/lib/mistral \
   --output-file ~/central/dcn_ceph.yaml

3. Redeploy the central site using the original templates and include the newly created dcn_ceph.yaml and glance_update.yaml files.

   openstack overcloud deploy \
          --stack central \
          --templates /usr/share/openstack-tripleo-heat-templates/ \
          -r ~/central/central_roles.yaml \
       ...
          -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
          -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \
          -e ~/central/central-images-env.yaml \
          -e ~/central/role-counts.yaml \
          -e ~/central/site-name.yaml
          -e ~/central/ceph.yaml \
          -e ~/central/ceph_keys.yaml \
          -e ~/central/glance.yaml \
          -e ~/central/dcn_ceph_external.yaml

4. On a controller at the central location, restart the cinder-volume service. If you deployed the central location with the cinder-backup service, then restart the cinder-backup service too:

   ssh heat-admin@controller-0 sudo pcs resource restart openstack-cinder-volume
   ssh heat-admin@controller-0 sudo pcs resource restart openstack-cinder-backup

Procedure

1. Create an ansible inventory file for the stack. You can specify multiple stacks in a comma delimited list to cache images at more than one site:

   $ source stackrc
   
   $ tripleo-ansible-inventory --plan central,dcn0,dcn1 \
   --static-yaml-inventory inventory.yaml

2. Create a list of image IDs that you want to pre-cache:

   1. Retrieve a comprehensive list of available images:

      $ source centralrc
      
      $ openstack image list
      +--------------------------------------+---------+--------+
      | ID                                   | Name    | Status |
      +--------------------------------------+---------+--------+
      | 07bc2424-753b-4f65-9da5-5a99d8383fe6 | image_0 | active |
      | d5187afa-c821-4f22-aa4b-4e76382bef86 | image_1 | active |
      +--------------------------------------+---------+--------+

   2. Create an ansible playbook argument file called nova_cache_args.yml, and add the IDs of the images that you want to pre-cache:

      ---
      tripleo_nova_image_cache_images:
        - id: 07bc2424-753b-4f65-9da5-5a99d8383fe6
        - id: d5187afa-c821-4f22-aa4b-4e76382bef86

3. Run the tripleo_nova_image_cache.yml ansible playbook:

   $ source centralrc
   
   $ ansible-playbook -i inventory.yaml \
   --extra-vars "@nova_cache_args.yml" \
   /usr/share/ansible/tripleo-playbooks/tripleo_nova_image_cache.yml

1. If you are deploying storage at the edge, include the following parameters in your modified tripleo heat templates for edge stacks:

   TEMPLATES=/usr/share/openstack-tripleo-heat-templates
   
   resource_registry:
     OS::TripleO::Services::IpaClient:
       ${TEMPLATES}/deployment/ipa/ipaservices-baremetal-ansible.yaml
   
   parameter_defaults:
     EnableEtcdInternalTLS: true

Process

1. Create a Ceph key for external access. This key is sensitive. You can generate the key using the following:

   python3 -c 'import os,struct,time,base64; key = os.urandom(16) ; \
   header = struct.pack("<hiih", 1, int(time.time()), 0, len(key)) ; \
   print(base64.b64encode(header + key).decode())'

2. In the directory of the stack you are deploying, create a ceph_keys.yaml environment file with contents like the following, using the output from the previous command for the key:

   parameter_defaults:
     CephExtraKeys:
       - name: "client.external"
         caps:
           mgr: "allow *"
           mon: "profile rbd"
           osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
         key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q=="
         mode: "0600"

3. Include the ceph_keys.yaml environment file in the deployment of the site. For example, to deploy the central site with with the ceph_keys.yaml environment file, run a command like the following:

    overcloud deploy \
            --stack central \
            --templates /usr/share/openstack-tripleo-heat-templates/ \
            ….
            -e ~/central/ceph_keys.yaml