Chapter 16. Deploying routed provider networks
16.1. Advantages of routed provider networks
In Red Hat OpenStack Platform (RHOSP), operators can create routed provider networks. Routed provider networks are typically used in edge deployments, and rely on multiple layer 2 network segments instead of traditional networks that have only one segment.
Routed provider networks simplify the cloud for end users because they see only one network. For cloud operators, routed provider networks deliver scalabilty and fault tolerance. For example, if a major error occurs, only one segment is impacted instead of the entire network failing.
Before routed provider networks, operators typically had to choose from one of the following architectures:
- A single, large layer 2 network
- Multiple, smaller layer 2 networks
Single, large layer 2 networks become complex when scaling and reduce fault tolerance (increase failure domains).
Multiple, smaller layer 2 networks scale better and shrink failure domains, but can introduce complexity for end users.
Starting with RHOSP 16.2 and later, you can deploy routed provider networks using the Modular Layer 2 plug-in with the Open Virtual Network mechanism driver (ML2/OVN). (Routed provider network support for the ML2/Open vSwitch (OVS) and SR-IOV mechanism drivers was introduced in RHOSP 16.1.1.)
Additional resources
16.2. Fundamentals of routed provider networks
A routed provider network is different from other types of networks because of the one-to-one association between a network subnet and a segment. In the past, the Red Hat OpenStack (RHOSP) Networking service has not supported routed provider networks, because the Networking service required that all subnets must either belong to the same segment or to no segment.
With routed provider networks, the IP addresses available to virtual machine (VM) instances depend on the segment of the network available on the particular compute node. The Networking service port can be associated with only one network segment.
Similar to conventional networking, layer 2 (switching) handles transit of traffic between ports on the same network segment and layer 3 (routing) handles transit of traffic between segments.
The Networking service does not provide layer 3 services between segments. Instead, it relies on physical network infrastructure to route subnets. Thus, both the Networking service and physical network infrastructure must contain configuration for routed provider networks, similar to conventional provider networks.
Because the Compute service (nova) scheduler is not network segment aware, when you deploy routed provider networks, you must map each leaf or rack segment or DCN edge site to a Compute service host-aggregate or availability zone.
If you require a DHCP-metadata service, you must define an availability zone for each edge site or network segment, to ensure that the local DHCP agent is deployed.
Additional resources
16.3. Limitations of routed provider networks
Routed provider networks are not supported by all mechanism drivers and there are restrictions with the Compute service scheduler and other software as noted in the following list:
- North-south routing with central SNAT or a floating IP is not supported.
- When using SR-IOV or PCI pass-through, physical network (physnet) names must be the same in central and remote sites or segments. You cannot reuse segment IDs.
The Compute service (nova) scheduler is not segment-aware. (You must map each segment or edge site to a Compute host-aggregate or availability zone.) Currently, there are only two VM instance boot options available:
-
Boot using
port-id
and no IP address, specifying Compute availability zone (segment or edge site). -
Boot using
network-id
, specifying the Compute availability zone (segment or edge site).
-
Boot using
- Cold or live migration works only when you specify the destination Compute availability zone (segment or edge site).
16.4. Preparing for a routed provider network
There are several tasks that you must perform before you can create a routed provider network in Red Hat OpenStack Platform (RHOSP).
Procedure
Within a network, use a unique physical network name for each segment. This enables reuse of the same segmentation details between subnets.
For example, use the same VLAN ID across all segments of a particular provider network.
Implement routing between segments.
Each subnet on a segment must contain the gateway address of the router interface on that particular subnet.
Table 16.1. Sample segments for routing Segment Version Addresses Gateway segment1
4
203.0.113.0/24
203.0.113.1
segment1
6
fd00:203:0:113::/64
fd00:203:0:113::1
segment2
4
198.51.100.0/24
198.51.100.1
segment2
6
fd00:198:51:100::/64
fd00:198:51:100::1
Map segments to Compute nodes.
Routed provider networks imply that Compute nodes reside on different segments. Ensure that every Compute host in a routed provider network has direct connectivity to one of its segments.
Table 16.2. Sample segment to Compute node mappings Host Rack Physical network compute0001
rack 1
segment 1
compute0002
rack 1
segment 1
…
…
…
compute0101
rack 2
segment 2
compute0102
rack 2
segment 2
compute0102
rack 2
segment 2
…
…
…
When you deploy with the Modular Layer 2 plug-in with the Open vSwitch mechanism driver (ML2/OVS), you must deploy at least one DHCP agent per segment.
Unlike conventional provider networks, a DHCP agent cannot support more than one segment within a network. Deploy DHCP agents on the Compute nodes containing the segments rather than on the network nodes to reduce the node count.
Table 16.3. Sample DHCP agent per segment mapping Host Rack Physical network network0001
rack 1
segment 1
network0002
rack 1
segment 1
…
…
…
You deploy a DHCP agent and a RHOSP Networking service (neutron) metadata agent on the Compute nodes by using a custom roles file.
Here is an example:
############################################################################### # Role: ComputeSriov # ############################################################################### - name: ComputeSriov description: | Compute SR-IOV Role CountDefault: 1 networks: External: subnet: external_subnet InternalApi: subnet: internal_api_subnet Tenant: subnet: tenant_subnet Storage: subnet: storage_subnet RoleParametersDefault: TunedProfileName: "cpu-partitioning" update_serial: 25 ServicesDefault: - OS::TripleO::Services::Aide - OS::TripleO::Services::AuditD - OS::TripleO::Services::BootParams - OS::TripleO::Services::CACerts ... - OS::TripleO::Services::NeutronDhcpAgent - OS::TripleO::Services::NeutronMetadataAgent ...
In a custom environment file, add the following key-value pair:
parameter_defaults: .... NeutronEnableIsolatedMetadata: 'True' ....
Additional resources
- Section 16.5, “Creating a routed provider network”
- Composable services and custom roles in the Advanced Overcloud Customization guide
16.5. Creating a routed provider network
Routed provider networks simplify the Red Hat OpenStack Platform (RHOSP) cloud for end users because they see only one network. For cloud operators, routed provider networks deliver scalabilty and fault tolerance.
When you perform this procedure, you create a routed provider network with two network segments. Each segment contains one IPv4 subnet and one IPv6 subnet.
Prerequisites
- Complete the steps in xref:prepare-routed-prov-network_deploy-routed-prov-networks.
Procedure
Create a VLAN provider network that includes a default segment.
In this example, the VLAN provider network is named
multisegment1
and uses a physical network calledprovider1
and a VLAN whose ID is128
:Example
$ openstack network create --share --provider-physical-network provider1 \ --provider-network-type vlan --provider-segment 128 multisegment1
Sample output
+---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | id | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | | ipv4_address_scope | None | | ipv6_address_scope | None | | l2_adjacency | True | | mtu | 1500 | | name | multisegment1 | | port_security_enabled | True | | provider:network_type | vlan | | provider:physical_network | provider1 | | provider:segmentation_id | 128 | | revision_number | 1 | | router:external | Internal | | shared | True | | status | ACTIVE | | subnets | | | tags | [] | +---------------------------+--------------------------------------+
Rename the default network segment to
segment1
.Obtain the segment ID:
$ openstack network segment list --network multisegment1
Sample output
+--------------------------------------+----------+--------------------------------------+--------------+---------+ | ID | Name | Network | Network Type | Segment | +--------------------------------------+----------+--------------------------------------+--------------+---------+ | 43e16869-ad31-48e4-87ce-acf756709e18 | None | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | vlan | 128 | +--------------------------------------+----------+--------------------------------------+--------------+---------+
Using the segment ID, rename the network segment to
segment1
:$ openstack network segment set --name segment1 43e16869-ad31-48e4-87ce-acf756709e18
Create a second segment on the provider network.
In this example, the network segment uses a physical network called
provider2
and a VLAN whose ID is129
:Example
$ openstack network segment create --physical-network provider2 \ --network-type vlan --segment 129 --network multisegment1 segment2
Sample output
+------------------+--------------------------------------+ | Field | Value | +------------------+--------------------------------------+ | description | None | | headers | | | id | 053b7925-9a89-4489-9992-e164c8cc8763 | | name | segment2 | | network_id | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | | network_type | vlan | | physical_network | provider2 | | revision_number | 1 | | segmentation_id | 129 | | tags | [] | +------------------+--------------------------------------+
Verify that the network contains the
segment1
andsegment2
segments:$ openstack network segment list --network multisegment1
Sample output
+--------------------------------------+----------+--------------------------------------+--------------+---------+ | ID | Name | Network | Network Type | Segment | +--------------------------------------+----------+--------------------------------------+--------------+---------+ | 053b7925-9a89-4489-9992-e164c8cc8763 | segment2 | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | vlan | 129 | | 43e16869-ad31-48e4-87ce-acf756709e18 | segment1 | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | vlan | 128 | +--------------------------------------+----------+--------------------------------------+--------------+---------+
Create one IPv4 subnet and one IPv6 subnet on the
segment1
segment.In this example, the IPv4 subnet uses
203.0.113.0/24
:Example
$ openstack subnet create \ --network multisegment1 --network-segment segment1 \ --ip-version 4 --subnet-range 203.0.113.0/24 \ multisegment1-segment1-v4
Sample output
+-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 203.0.113.2-203.0.113.254 | | cidr | 203.0.113.0/24 | | enable_dhcp | True | | gateway_ip | 203.0.113.1 | | id | c428797a-6f8e-4cb1-b394-c404318a2762 | | ip_version | 4 | | name | multisegment1-segment1-v4 | | network_id | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | | revision_number | 1 | | segment_id | 43e16869-ad31-48e4-87ce-acf756709e18 | | tags | [] | +-------------------+--------------------------------------+
In this example, the IPv6 subnet uses
fd00:203:0:113::/64
:Example
$ openstack subnet create \ --network multisegment1 --network-segment segment1 \ --ip-version 6 --subnet-range fd00:203:0:113::/64 \ --ipv6-address-mode slaac multisegment1-segment1-v6
Sample output
+-------------------+------------------------------------------------------+ | Field | Value | +-------------------+------------------------------------------------------+ | allocation_pools | fd00:203:0:113::2-fd00:203:0:113:ffff:ffff:ffff:ffff | | cidr | fd00:203:0:113::/64 | | enable_dhcp | True | | gateway_ip | fd00:203:0:113::1 | | id | e41cb069-9902-4c01-9e1c-268c8252256a | | ip_version | 6 | | ipv6_address_mode | slaac | | ipv6_ra_mode | None | | name | multisegment1-segment1-v6 | | network_id | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | | revision_number | 1 | | segment_id | 43e16869-ad31-48e4-87ce-acf756709e18 | | tags | [] | +-------------------+------------------------------------------------------+
NoteBy default, IPv6 subnets on provider networks rely on physical network infrastructure for stateless address autoconfiguration (SLAAC) and router advertisement.
Create one IPv4 subnet and one IPv6 subnet on the
segment2
segment.In this example, the IPv4 subnet uses
198.51.100.0/24
:Example
$ openstack subnet create \ --network multisegment1 --network-segment segment2 \ --ip-version 4 --subnet-range 198.51.100.0/24 \ multisegment1-segment2-v4
Sample output
+-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 198.51.100.2-198.51.100.254 | | cidr | 198.51.100.0/24 | | enable_dhcp | True | | gateway_ip | 198.51.100.1 | | id | 242755c2-f5fd-4e7d-bd7a-342ca95e50b2 | | ip_version | 4 | | name | multisegment1-segment2-v4 | | network_id | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | | revision_number | 1 | | segment_id | 053b7925-9a89-4489-9992-e164c8cc8763 | | tags | [] | +-------------------+--------------------------------------+
In this example, the IPv6 subnet uses
fd00:198:51:100::/64
:Example
$ openstack subnet create \ --network multisegment1 --network-segment segment2 \ --ip-version 6 --subnet-range fd00:198:51:100::/64 \ --ipv6-address-mode slaac multisegment1-segment2-v6
Sample output
+-------------------+--------------------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------------------+ | allocation_pools | fd00:198:51:100::2-fd00:198:51:100:ffff:ffff:ffff:ffff | | cidr | fd00:198:51:100::/64 | | enable_dhcp | True | | gateway_ip | fd00:198:51:100::1 | | id | b884c40e-9cfe-4d1b-a085-0a15488e9441 | | ip_version | 6 | | ipv6_address_mode | slaac | | ipv6_ra_mode | None | | name | multisegment1-segment2-v6 | | network_id | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | | revision_number | 1 | | segment_id | 053b7925-9a89-4489-9992-e164c8cc8763 | | tags | [] | +-------------------+--------------------------------------------------------+
Verification
Verify that each IPv4 subnet associates with at least one DHCP agent:
$ openstack network agent list --agent-type dhcp --network multisegment1
Sample output
+--------------------------------------+------------+-------------+-------------------+-------+-------+--------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+------------+-------------+-------------------+-------+-------+--------------------+ | c904ed10-922c-4c1a-84fd-d928abaf8f55 | DHCP agent | compute0001 | nova | :-) | UP | neutron-dhcp-agent | | e0b22cc0-d2a6-4f1c-b17c-27558e20b454 | DHCP agent | compute0101 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+------------+-------------+-------------------+-------+-------+--------------------+
Verify that inventories were created for each segment IPv4 subnet in the Compute service placement API.
Run this command for all segment IDs:
$ SEGMENT_ID=053b7925-9a89-4489-9992-e164c8cc8763 $ openstack resource provider inventory list $SEGMENT_ID
Sample output
In this sample output, only one of the segments is shown:
+----------------+------------------+----------+----------+-----------+----------+-------+ | resource_class | allocation_ratio | max_unit | reserved | step_size | min_unit | total | +----------------+------------------+----------+----------+-----------+----------+-------+ | IPV4_ADDRESS | 1.0 | 1 | 2 | 1 | 1 | 30 | +----------------+------------------+----------+----------+-----------+----------+-------+
Verify that host aggregates were created for each segment in the Compute service:
$ openstack aggregate list
Sample output
In this example, only one of the segments is shown:
+----+---------------------------------------------------------+-------------------+ | Id | Name | Availability Zone | +----+---------------------------------------------------------+-------------------+ | 10 | Neutron segment id 053b7925-9a89-4489-9992-e164c8cc8763 | None | +----+---------------------------------------------------------+-------------------+
Launch one or more instances. Each instance obtains IP addresses according to the segment it uses on the particular compute node.
NoteIf a fixed IP is specified by the user in the port create request, that particular IP is allocated immediately to the port. However, creating a port and passing it to an instance yields a different behavior than conventional networks. If the fixed IP is not specified on the port create request, the Networking service defers assignment of IP addresses to the port until the particular compute node becomes apparent. For example, when you run this command:
$ openstack port create --network multisegment1 port1
Sample output
+-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | UP | | binding_vnic_type | normal | | id | 6181fb47-7a74-4add-9b6b-f9837c1c90c4 | | ip_allocation | deferred | | mac_address | fa:16:3e:34:de:9b | | name | port1 | | network_id | 6ab19caa-dda9-4b3d-abc4-5b8f435b98d9 | | port_security_enabled | True | | revision_number | 1 | | security_groups | e4fcef0d-e2c5-40c3-a385-9c33ac9289c5 | | status | DOWN | | tags | [] | +-----------------------+--------------------------------------+
Additional resources
- Section 16.4, “Preparing for a routed provider network”
- network create in the Command Line Interface Reference
- network segment create in the Command Line Interface Reference
- subnet create in the Command Line Interface Reference
- port create in the Command Line Interface Reference
16.6. Migrating a non-routed network to a routed provider network
You can migrate a non-routed network to a routed provider network by associating the subnet of the network with the ID of the network segment.
Prerequisites
The non-routed network you are migrating must contain only one segment and only one subnet.
ImportantIn non-routed provider networks that contain multiple subnets or network segments it is not possible to safely migrate to a routed provider network. In non-routed networks, addresses from the subnet allocation pools are assigned to ports without consideration of the network segment to which the port is bound.
Procedure
For the network that is being migrated, obtain the ID of the current network segment.
Example
$ openstack network segment list --network my_network
Sample output
+--------------------------------------+------+--------------------------------------+--------------+---------+ | ID | Name | Network | Network Type | Segment | +--------------------------------------+------+--------------------------------------+--------------+---------+ | 81e5453d-4c9f-43a5-8ddf-feaf3937e8c7 | None | 45e84575-2918-471c-95c0-018b961a2984 | flat | None | +--------------------------------------+------+--------------------------------------+--------------+---------+
For the network that is being migrated, obtain the ID of the current subnet.
Example
$ openstack network segment list --network my_network
Sample output
+--------------------------------------+-----------+--------------------------------------+---------------+ | ID | Name | Network | Subnet | +--------------------------------------+-----------+--------------------------------------+---------------+ | 71d931d2-0328-46ae-93bc-126caf794307 | my_subnet | 45e84575-2918-471c-95c0-018b961a2984 | 172.24.4.0/24 | +--------------------------------------+-----------+--------------------------------------+---------------+
Verify that the current
segment_id
of the subnet has a value ofNone
.Example
$ openstack subnet show my_subnet --c segment_id
Sample output
+------------+-------+ | Field | Value | +------------+-------+ | segment_id | None | +------------+-------+
Change the value of the subnet
segment_id
to the network segment ID.Here is an example:
$ openstack subnet set --network-segment 81e5453d-4c9f-43a5-8ddf-feaf3937e8c7 my_subnet
Verification
Verify that the subnet is now associated with the desired network segment.
Example
$ openstack subnet show my_subnet --c segment_id
Sample output
+------------+--------------------------------------+ | Field | Value | +------------+--------------------------------------+ | segment_id | 81e5453d-4c9f-43a5-8ddf-feaf3937e8c7 | +------------+--------------------------------------+
Additional resources
- subnet show in the Command Line Interface Reference
- subnet set in the Command Line Interface Reference