Chapter 3. Primary networks
3.1. About-user-defined networks Copy linkLink copied to clipboard!
UserDefinedNetwork is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
User-defined networks (UDNs) extend OVN-Kubernetes to enable custom layer 2 and layer 3 network segments with default isolation, providing enhanced network flexibility, security, and segmentation capabilities for multitenant deployments and custom network architectures.
3.1.1. Overview of user-defined networks Copy linkLink copied to clipboard!
To secure and improve network segmentation and isolation, administrators can create primary or secondary networks at the namespace level by using the UserDefinedNetwork CR.
Before the implementation of user-defined networks (UDN), the OVN-Kubernetes CNI plugin for OpenShift Container Platform only supported a layer 3 topology on the primary or main network. Due to Kubernetes design principles: all pods are attached to the main network, all pods communicate with each other by their IP addresses, and inter-pod traffic is restricted according to network policy.
While the Kubernetes design is useful for simple deployments, this layer 3 topology restricts customization of primary network segment configurations, especially for modern multitenant deployments.
UDN improves the flexibility and segmentation capabilities of the default layer 3 topology for a Kubernetes pod network by enabling custom layer 2 and layer 3 network segments, where all these segments are isolated by default. These segments act as either primary or secondary networks for container pods and virtual machines that use the default OVN-Kubernetes CNI plugin. UDNs enable a wide range of network architectures and topologies, enhancing network flexibility, security, and performance.
The following sections further emphasize the benefits and limitations of user-defined networks,UserDefinedNetwork CR, how to create the CR, and additional configuration details that might be relevant to your deployment.
3.1.2. Benefits of a user-defined network Copy linkLink copied to clipboard!
User-defined networks enable tenant isolation by providing each namespace with its own isolated primary network, reducing cross-tenant traffic risks and simplifying network management by eliminating the need for complex network policies.
User-defined networks offer the following benefits:
Enhanced network isolation for security
- Tenant isolation: Namespaces can have their own isolated primary network, similar to how tenants are isolated in Red Hat OpenStack Platform (RHOSP). This improves security by reducing the risk of cross-tenant traffic.
Network flexibility
- Layer 2 and layer 3 support: Cluster administrators can configure primary networks as layer 2 or layer 3 network types.
Simplified network management
- Reduced network configuration complexity: With user-defined networks, the need for complex network policies are eliminated because isolation can be achieved by grouping workloads in different networks.
Advanced capabilities
- Consistent and selectable IP addressing: Users can specify and reuse IP subnets across different namespaces and clusters, providing a consistent networking environment.
- Support for multiple networks: The user-defined networking feature allows administrators to connect multiple namespaces to a single network, or to create distinct networks for different sets of namespaces.
Simplification of application migration from Red Hat OpenStack Platform (RHOSP)
- Network parity: With user-defined networking, the migration of applications from OpenStack to OpenShift Container Platform is simplified by providing similar network isolation and configuration options.
Developers and administrators can create a user-defined network that is namespace scoped using the custom resource. An overview of the process is as follows:
-
An administrator creates a namespace for a user-defined network with the
k8s.ovn.org/primary-user-defined-networklabel. -
The
UserDefinedNetworkCR is created by either the cluster administrator or the user. - The user creates pods in the namespace.
3.1.3. Limitations for UserDefinedNetwork custom resource Copy linkLink copied to clipboard!
To deploy a successful user-defined networks (UDN), you must consider their limitations including DNS resolution behavior, restricted access to default network services such as the image registry, network policy constraints between isolated networks, and the requirement to create namespaces and networks before pods.
Consider the following limitations before implementing a UDN.
DNS limitations:
- DNS lookups for pods resolve to the pod’s IP address on the cluster default network. Even if a pod is part of a user-defined network, DNS lookups will not resolve to the pod’s IP address on that user-defined network. However, DNS lookups for services and external entities will function as expected.
- When a pod is assigned to a primary UDN, it can access the Kubernetes API (KAPI) and DNS services on the cluster’s default network.
- Initial network assignment: You must create the namespace and network before creating pods. Assigning a namespace with pods to a new network or creating a UDN in an existing namespace will not be accepted by OVN-Kubernetes.
- Health check limitations: Kubelet health checks are performed by the cluster default network, which does not confirm the network connectivity of the primary interface on the pod. Consequently, scenarios where a pod appears healthy by the default network, but has broken connectivity on the primary interface, are possible with user-defined networks.
- Network policy limitations: Network policies that enable traffic between namespaces connected to different user-defined primary networks are not effective. These traffic policies do not take effect because there is no connectivity between these isolated networks.
-
Creation and modification limitation: The
ClusterUserDefinedNetworkCR and theUserDefinedNetworkCR cannot be modified after being created.
3.1.4. Layer 2 and layer 3 topologies Copy linkLink copied to clipboard!
A layer 2 topology creates a distributed virtual switch across cluster nodes, this network topology provides a smooth live migration of virtual machine (VM) within the same subnet. A layer 3 topology creates unique segments per node with routing between them, this network topology effectively manages large broadcast domains.
In a flat layer 2 topology, virtual machines and pods connect to the virtual switch so that all these components can communicate with each other within the same subnet. This topology is useful for the live migration of VMs across nodes in the cluster. The following diagram shows a flat layer 2 topology with two nodes that use the virtual switch for live migration purposes:
Figure 3.1. A flat layer 2 topology that uses a virtual switch for component communication
If you decide not to specify a layer 2 subnet, then you must manually configure IP addresses for each pod in your cluster. When you do not specify a layer 2 subnet, port security is limited to preventing Media Access Control (MAC) spoofing only, and does not include IP spoofing. A layer 2 topology creates a single broadcast domain that can be challenging in large network environments, where the topology might cause a broadcast storm that can degrade network performance.
To access more configurable options for your network, you can integrate a layer 2 topology with a user-defined network (UDN). The following diagram shows two nodes that use a UDN with a layer 2 topology that includes pods that exist on each node. Each node includes two interfaces:
- A node interface, which is a compute node that connects networking components to the node.
-
An Open vSwitch (OVS) bridge such as
br-ex, which creates an layer 2 OVN switch so that pods can communicate with each other and share resources.
An external switch connects these two interfaces, while the gateway or router handles routing traffic between the external switch and the layer 2 OVN switch. VMs and pods in a node can use the UDN to communicate with each other. The layer 2 OVN switch handles node traffic over a UDN so that live migrate of a VM from one node to another is possible.
Figure 3.2. A user-defined network (UDN) that uses a layer 2 topology
A layer 3 topology creates a unique layer 2 segment for each node in a cluster. The layer 3 routing mechanism interconnects these segments so that virtual machines and pods that are hosted on different nodes can communicate with each other. A layer 3 topology can effectively manage large broadcast domains by assigning each domain to a specific node, so that broadcast traffic has a reduced scope. To configure a layer 3 topology, you must configure cidr and hostSubnet parameters.
3.1.5. Best practices for UserDefinedNetwork Copy linkLink copied to clipboard!
To deploy a successful instance of the UserDefinedNetwork (UDN) CR, you must follow masquerade IP address requirements, avoid default and openshift-* namespaces, set a proper namespace selector configuration, and ensure physical network parameter matching.
The following details provide a best practice for designing a UDN CR:
-
openshift-*namespaces should not be used to set up a UDN. 2 masquerade IP addresses are required for user defined networks. You must reconfigure your masquerade subnet to be large enough to hold the required number of networks.
Important-
For OpenShift Container Platform 4.17 and later, clusters use
169.254.0.0/17for IPv4 andfd69::/112for IPv6 as the default masquerade subnet. These ranges should be avoided by users. For updated clusters, there is no change to the default masquerade subnet. - Changing the cluster’s masquerade subnet is unsupported after a user-defined network has been configured for a project. Attempting to modify the masquerade subnet after a UDN has been set up can disrupt the network connectivity and cause configuration issues.
-
For OpenShift Container Platform 4.17 and later, clusters use
-
Ensure tenants are using the
UserDefinedNetworkresource and not theNetworkAttachmentDefinition(NAD) resource. This can create security risks between tenants. - When creating network segmentation, you should only use the NAD resource if user-defined network segmentation cannot be completed using the UDN resource.
-
The cluster subnet and services CIDR for a UDN cannot overlap with the default cluster subnet CIDR. OVN-Kubernetes network plugin uses
100.64.0.0/16as the default network’s join subnet, you must not use that value to configure a UDNjoinSubnetsfield. If the default address values are used anywhere in the network for the cluster, you must override it by setting thejoinSubnetsfield. For more information, see "Additional configuration details for a UserDefinedNetworks CR". -
The cluster subnet and services CIDR for a UDN cannot overlap with the default cluster subnet CIDR. OVN-Kubernetes network plugin uses
100.64.0.0/16as the default join subnet for the network, you must not use that value to configure a UDNjoinSubnetsfield. If the default address values are used anywhere in the network for the cluster you must override the default values by setting thejoinSubnetsfield. For more information, see "Additional configuration details for a UserDefinedNetworks CR".
3.1.6. Creating a UserDefinedNetwork custom resource Copy linkLink copied to clipboard!
Create a UserDefinedNetwork CR by using the CLI to enable namespace-scoped network segmentation and isolation, allowing you to define custom Layer 2 or Layer 3 network topologies for pods within specific namespaces.
The following procedure creates a UserDefinedNetwork CR that is namespace scoped. Based upon your use case, create your request by using either the my-layer-two-udn.yaml example for a Layer2 topology type or the my-layer-three-udn.yaml example for a Layer3 topology type.
Prerequisites
-
You have logged in with
cluster-adminprivileges, or you haveviewandeditrole-based access control (RBAC).
Procedure
Optional: For a
UserDefinedNetworkCR that uses a primary network, create a namespace with thek8s.ovn.org/primary-user-defined-networklabel by entering the following command:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a request for either a
Layer2orLayer3topology type user-defined network:Create a YAML file, such as
my-layer-two-udn.yaml, to define your request for aLayer2topology as in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
name-
Name of your
UserDefinedNetworkresource. This should not bedefaultor duplicate any global namespaces created by the Cluster Network Operator (CNO). topology-
Specifies the network configuration; accepted values are
Layer2andLayer3. Specifying aLayer2topology type creates one logical switch that is shared by all nodes. role-
Specifies a
PrimaryorSecondaryrole. subnetsFor
Layer2topology types the following specifies config details for thesubnetfield:- The subnets field is optional.
-
The subnets field is of type
stringand accepts standard CIDR formats for both IPv4 and IPv6. -
The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of
10.100.0.0/16and2001:db8::/64. -
Layer2subnets can be omitted. If omitted, users must configure IP addresses for the pods. As a consequence, port security only prevents MAC spoofing. -
The
Layer2subnetsfield is mandatory when theipamLifecyclefield is specified.
Create a YAML file, such as
my-layer-three-udn.yaml, to define your request for aLayer3topology as in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
name-
Name of your
UserDefinedNetworkresource. This should not bedefaultor duplicate any global namespaces created by the Cluster Network Operator (CNO). topology-
Specifies the network configuration; accepted values are
Layer2andLayer3. Specifying aLayer2topology type creates one logical switch that is shared by all nodes. role-
Specifies a
PrimaryorSecondaryrole. subnetsFor
Layer3topology types the following specifies config details for thesubnetfield:-
The
subnetsfield is mandatory. The type for the
subnetsfield iscidrandhostSubnet:-
cidris equivalent to theclusterNetworkconfiguration settings of a cluster. The IP addresses in the CIDR are distributed to pods in the user defined network. This parameter accepts a string value. -
hostSubnetdefines the per-node subnet prefix. -
For IPv6, only a
/64length is supported forhostSubnet.
-
-
The
Apply your request by running the following command:
oc apply -f <my_layer_two_udn.yaml>
$ oc apply -f <my_layer_two_udn.yaml>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Where
<my_layer_two_udn.yaml>is the name of yourLayer2orLayer3configuration file.Verify that your request is successful by running the following command:
oc get userdefinednetworks udn-1 -n <some_custom_namespace> -o yaml
$ oc get userdefinednetworks udn-1 -n <some_custom_namespace> -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Where
some_custom_namespaceis the namespace you created for your user-defined network.Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.1.6.1. Additional configuration details for a UserDefinedNetworks CR Copy linkLink copied to clipboard!
Configure optional advanced settings for ClusterUserDefinedNetwork and UserDefinedNetwork CRs when default values conflict with your network topology or when you need persistent IP addresses, custom gateways, or specific subnet configurations.
It is not recommended to set these fields without explicit need and understanding of OVN-Kubernetes network topology.
- Optional configurations for user-defined networks
| CUDN field | UDN field | Type | Description |
|
|
| object |
When omitted, the platform sets default values for the
The |
|
|
| object |
The
Setting a value of Persistent is only supported when |
|
|
| object |
The |
|
|
| integer |
The maximum transmission units (MTU). The default value is |
where:
<topology>-
Is one of
layer2orlayer3.
3.2. Creating primary networks using a NetworkAttachmentDefinition Copy linkLink copied to clipboard!
Use the NetworkAttachmentDefinition (NAD) resource to create primary networks when you need to use CNI plugins other than OVN-Kubernetes, such as IPVLAN or MACVLAN, or when you require direct control over the Container Network Interface (CNI) configuration for advanced networking scenarios.
3.2.1. Approaches to managing a primary network Copy linkLink copied to clipboard!
You can manage the life cycle of a primary network created by a NAD CR through the Cluster Network Operator (CNO) or a YAML manifest. Using the CNO provides automated management of the network resource, while applying a YAML manifest allows for direct control over the network configuration.
- Modifying the Cluster Network Operator (CNO) configuration
-
With this method, the CNO automatically creates and manages the
NetworkAttachmentDefinitionobject. In addition to managing the object lifecycle, the CNO ensures that a DHCP is available for a primary network that uses a DHCP assigned IP address. - Applying a YAML manifest
-
With this method, you can manage the primary network directly by creating an
NetworkAttachmentDefinitionobject. This approach allows for the invocation of multiple CNI plugins in order to attach primary network interfaces in a pod.
Each approach is mutually exclusive and you can only use one approach for managing a primary network at a time. For either approach, the primary network is managed by a Container Network Interface (CNI) plugin that you configure.
When deploying OpenShift Container Platform nodes with multiple network interfaces on Red Hat OpenStack Platform (RHOSP) with OVN SDN, DNS configuration of the secondary interface might take precedence over the DNS configuration of the primary interface. In this case, remove the DNS nameservers for the subnet ID that is attached to the secondary interface by running the following command:
openstack subnet set --dns-nameserver 0.0.0.0 <subnet_id>
$ openstack subnet set --dns-nameserver 0.0.0.0 <subnet_id>
3.2.2. Creating a primary network attachment with the Cluster Network Operator Copy linkLink copied to clipboard!
When you specify a primary network to create by using the Cluster Network Operator (CNO), the (CNO) creates the NetworkAttachmentDefinition custom resource definition (CRD) automatically and manages it.
Do not edit the NetworkAttachmentDefinition CRDs that the Cluster Network Operator manages. Doing so might disrupt network traffic on your primary network.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Log in as a user with
cluster-adminprivileges.
Procedure
Optional: Create the namespace for the primary networks:
oc create namespace <namespace_name>
$ oc create namespace <namespace_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow To edit the CNO configuration, enter the following command:
oc edit networks.operator.openshift.io cluster
$ oc edit networks.operator.openshift.io clusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow Modify the CR that you are creating by adding the configuration for the primary network that you are creating, as in the following example CR.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Save your changes and quit the text editor to commit your changes.
Verification
Confirm that the CNO created the
NetworkAttachmentDefinitionCRD by running the following command. A delay might exist before the CNO creates the CRD. The expected output shows the name of the NAD CRD and the creation age in minutes.oc get network-attachment-definitions -n <namespace>
$ oc get network-attachment-definitions -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<namespace>- Specifies the namespace for the network attachment that you added to the CNO configuration.
3.2.3. Configuration for a primary network attachment Copy linkLink copied to clipboard!
You configure a primary network by using the NetworkAttachmentDefinition API in the k8s.cni.cncf.io API group.
The configuration for the API is described in the following table:
| Field | Type | Description |
|---|---|---|
|
|
| The name for the primary network. |
|
|
| The namespace that the object is associated with. |
|
|
| The CNI plugin configuration in JSON format. |
3.2.4. Creating a primary network attachment by applying a YAML manifest Copy linkLink copied to clipboard!
Create a primary network attachment by directly applying a NetworkAttachmentDefinition YAML manifest. This gives you full control over the network configuration without relying on the Cluster Network Operator to manage the resource automatically.
Prerequisites
-
You have installed the OpenShift CLI (
oc). -
You have logged in as a user with
cluster-adminprivileges. - You are working in the namespace where the NAD is to be deployed.
Procedure
Create a YAML file with your primary network configuration, such as in the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Optional: You can specify a namespace to which the NAD is applied. If you are working in the namespace where the NAD is to be deployed, the
namespacespecification is not necessary.
-
Optional: You can specify a namespace to which the NAD is applied. If you are working in the namespace where the NAD is to be deployed, the
To create the primary network, enter the following command:
oc apply -f <file>.yaml
$ oc apply -f <file>.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<file>- Specifies the name of the file contained the YAML manifest.