Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 4. Secondary networks
4.1. Creating secondary networks on OVN-Kubernetes Copier lienLien copié sur presse-papiers!
As a cluster administrator, you can configure a secondary network for your cluster by using the NetworkAttachmentDefinition (NAD) resource.
4.1.1. Configuration for an OVN-Kubernetes secondary network Copier lienLien copié sur presse-papiers!
The Red Hat OpenShift Networking OVN-Kubernetes network plugin allows the configuration of secondary network interfaces for pods. To configure secondary network interfaces, you must define the configurations in the NetworkAttachmentDefinition custom resource definition (CRD).
Pod and multi-network policy creation might remain in a pending state until the OVN-Kubernetes control plane agent in the nodes processes the associated network-attachment-definition CRD.
You can configure an OVN-Kubernetes secondary network in layer 2, layer 3, or localnet topologies. For more information about features supported on these topologies, see "UserDefinedNetwork and NetworkAttachmentDefinition support matrix".
The following sections provide example configurations for each of the topologies that OVN-Kubernetes currently allows for secondary networks.
Networks names must be unique. For example, creating multiple NetworkAttachmentDefinition CRDs with different configurations that reference the same network is unsupported.
4.1.1.1. Supported platforms for OVN-Kubernetes secondary network Copier lienLien copié sur presse-papiers!
You can use an OVN-Kubernetes secondary network with the following supported platforms:
- Bare metal
- IBM Power®
- IBM Z®
- IBM® LinuxONE
- VMware vSphere
- Red Hat OpenStack Platform (RHOSP)
4.1.1.2. OVN-Kubernetes network plugin JSON configuration table Copier lienLien copié sur presse-papiers!
The OVN-Kubernetes network plugin JSON configuration object describes the configuration parameters for the OVN-Kubernetes CNI network plugin. The following table details these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. The required value is |
|
|
|
The name of the network. These networks are not namespaced. For example, a network named |
|
|
|
The name of the CNI plugin to configure. This value must be set to |
|
|
|
The topological configuration for the network. Must be one of |
|
|
| The subnet to use for the network across the cluster.
For When omitted, the logical switch implementing the network only provides layer 2 communication, and users must configure IP addresses for the pods. Port security only prevents MAC spoofing. |
|
|
| The maximum transmission unit (MTU). If you do not set a value, the Cluster Network Operator (CNO) sets a default MTU value by calculating the difference among the underlay MTU of the primary network interface, the overlay MTU of the pod network, such as the Geneve (Generic Network Virtualization Encapsulation), and byte capacity of any enabled features, such as IPsec. |
|
|
|
The metadata |
|
|
| A comma-separated list of CIDRs and IP addresses. IP addresses are removed from the assignable IP address pool and are never passed to the pods. |
|
|
|
If topology is set to |
|
|
|
If topology is set to |
4.1.1.3. Compatibility with multi-network policy Copier lienLien copié sur presse-papiers!
When defining a network policy, the network policy rules that can be used depend on whether the OVN-Kubernetes secondary network defines the subnets field.
The multi-network policy API, which is provided by the MultiNetworkPolicy custom resource definition (CRD) in the k8s.cni.cncf.io API group, is compatible with an OVN-Kubernetes secondary network.
Refer to the following table that details supported multi-network policy selectors that are based on a subnets CNI configuration:
subnets field specified | Allowed multi-network policy selectors |
|---|---|
| Yes |
|
| No |
|
You can use the k8s.v1.cni.cncf.io/policy-for annotation on a MultiNetworkPolicy object to point to a NetworkAttachmentDefinition (NAD) custom resource (CR). The NAD CR defines the network to which the policy applies. The following example multi-network policy that uses a pod selector is valid only if the subnets field is defined in the secondary network CNI configuration for the secondary network named blue2:
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
name: allow-same-namespace
annotations:
k8s.v1.cni.cncf.io/policy-for: blue2
spec:
podSelector:
ingress:
- from:
- podSelector: {}
The following example uses the ipBlock network multi-network policy that is always valid for an OVN-Kubernetes secondary network:
Example multi-network policy that uses an IP block selector
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
name: ingress-ipblock
annotations:
k8s.v1.cni.cncf.io/policy-for: default/flatl2net
spec:
podSelector:
matchLabels:
name: access-control
policyTypes:
- Ingress
ingress:
- from:
- ipBlock:
cidr: 10.200.0.0/30
4.1.1.4. Configuration for a localnet switched topology Copier lienLien copié sur presse-papiers!
The switched localnet topology interconnects the workloads created as Network Attachment Definitions (NADs) through a cluster-wide logical switch to a physical network.
You must map a secondary network to the OVS bridge to use it as an OVN-Kubernetes secondary network. Bridge mappings allow network traffic to reach the physical network. A bridge mapping associates a physical network name, also known as an interface label, to a bridge created with Open vSwitch (OVS).
You can create an NodeNetworkConfigurationPolicy (NNCP) object, part of the nmstate.io/v1 API group, to declaratively create the mapping. This API is provided by the NMState Operator. By using this API you can apply the bridge mapping to nodes that match your specified nodeSelector expression, such as node-role.kubernetes.io/worker: ''. With this declarative approach, the NMState Operator applies secondary network configuration to all nodes specified by the node selector automatically and transparently.
When attaching a secondary network, you can either use the existing br-ex bridge or create a new bridge. Which approach to use depends on your specific network infrastructure. Consider the following approaches:
-
If your nodes include only a single network interface, you must use the existing bridge. This network interface is owned and managed by OVN-Kubernetes and you must not remove it from the
br-exbridge or alter the interface configuration. If you remove or alter the network interface, your cluster network stops working correctly. - If your nodes include several network interfaces, you can attach a different network interface to a new bridge, and use that for your secondary network. This approach provides for traffic isolation from your primary cluster network.
You cannot make configuration changes to the br-ex bridge or its underlying interfaces in the NodeNetworkConfigurationPolicy (NNCP) resource as a postinstallation task. As a workaround, use a secondary network interface connected to your host or switch.
The localnet1 network is mapped to the br-ex bridge in the following sharing-a-bridge example:
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: mapping
spec:
nodeSelector:
node-role.kubernetes.io/worker: ''
desiredState:
ovn:
bridge-mappings:
- localnet: localnet1
bridge: br-ex
state: present
+ where:
+ metadata.name:: The name for the configuration object. spec.nodeSelector.node-role.kubernetes.io/worker:: A node selector that specifies the nodes to apply the node network configuration policy to. spec.desiredState.ovn.bridge-mappings.localnet:: The name for the secondary network from which traffic is forwarded to the OVS bridge. This secondary network must match the name of the spec.config.name field of the NetworkAttachmentDefinition CRD that defines the OVN-Kubernetes secondary network. spec.desiredState.ovn.bridge-mappings.bridge:: The name of the OVS bridge on the node. This value is required only if you specify state: present. spec.desiredState.ovn.bridge-mappings.state:: The state for the mapping. Must be either present to add the bridge or absent to remove the bridge. The default value is present.
+ The following JSON example configures a localnet secondary network that is named localnet1. Note that the value for the mtu parameter must match the MTU value that was set for the secondary network interface that is mapped to the br-ex bridge interface.
{
"cniVersion": "0.3.1",
"name": "localnet1",
"type": "ovn-k8s-cni-overlay",
"topology":"localnet",
"physicalNetworkName": "localnet1",
"subnets": "202.10.130.112/28",
"vlanID": 33,
"mtu": 1500,
"netAttachDefName": "ns1/localnet-network",
"excludeSubnets": "10.100.200.0/29"
}
In the following multiple interfaces example, the localnet2 network interface is attached to the ovs-br1 bridge. Through this attachment, the network interface is available to the OVN-Kubernetes network plugin as a secondary network.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: ovs-br1-multiple-networks
spec:
nodeSelector:
node-role.kubernetes.io/worker: ''
desiredState:
interfaces:
- name: ovs-br1
description: |-
A dedicated OVS bridge with eth1 as a port
allowing all VLANs and untagged traffic
type: ovs-bridge
state: up
bridge:
allow-extra-patch-ports: true
options:
stp: false
mcast-snooping-enable: true
port:
- name: eth1
ovn:
bridge-mappings:
- localnet: localnet2
bridge: ovs-br1
state: present
+ where:
+ metadata.name:: Specifies the name of the configuration object. node-role.kubernetes.io/worker:: Specifies a node selector that identifies the nodes to which the node network configuration policy applies. desiredState.interfaces.name:: Specifies a new OVS bridge that operates separately from the default bridge used by OVN-Kubernetes for cluster traffic. options.mcast-snooping-enable:: Specifies whether to enable multicast snooping. When enabled, multicast snooping prevents network devices from flooding multicast traffic to all network members. By default, an OVS bridge does not enable multicast snooping. The default value is false. bridge.port.name:: Specifies the network device on the host system to associate with the new OVS bridge. ovn.bridge-mappings.localnet:: Specifies the name of the secondary network that forwards traffic to the OVS bridge. This name must match the value of the spec.config.name field in the NetworkAttachmentDefinition CRD that defines the OVN-Kubernetes secondary network. ovn.bridge-mappings.bridge:: Specifies the name of the OVS bridge on the node. The value is required only when state: present is set. ovn.bridge-mappings.state:: Specifies the state of the mapping. Valid values are present to add the bridge or absent to remove the bridge. The default value is present.
+ The following JSON example configures a localnet secondary network that is named localnet2. Note that the value for the mtu parameter must match the MTU value that was set for the eth1 secondary network interface.
{
"cniVersion": "0.3.1",
"name": "localnet2",
"type": "ovn-k8s-cni-overlay",
"topology":"localnet",
"physicalNetworkName": "localnet2",
"subnets": "202.10.130.112/28",
"vlanID": 33,
"mtu": 1500,
"netAttachDefName": "ns1/localnet-network",
"excludeSubnets": "10.100.200.0/29"
}
4.1.1.4.1. Configuration for a layer 2 switched topology Copier lienLien copié sur presse-papiers!
The switched (layer 2) topology networks interconnect the workloads through a cluster-wide logical switch. This configuration can be used for IPv6 and dual-stack deployments.
Layer 2 switched topology networks only allow for the transfer of data packets between pods within a cluster.
The following JSON example configures a switched secondary network:
{
"cniVersion": "0.3.1",
"name": "l2-network",
"type": "ovn-k8s-cni-overlay",
"topology":"layer2",
"subnets": "10.100.200.0/24",
"mtu": 1300,
"netAttachDefName": "ns1/l2-network",
"excludeSubnets": "10.100.200.0/29"
}
4.1.1.5. Configuring pods for secondary networks Copier lienLien copié sur presse-papiers!
You must specify the secondary network attachments through the k8s.v1.cni.cncf.io/networks annotation.
The following example provisions a pod with two secondary attachments, one for each of the attachment configurations presented in this guide:
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: l2-network
name: tinypod
namespace: ns1
spec:
containers:
- args:
- pause
image: k8s.gcr.io/e2e-test-images/agnhost:2.36
imagePullPolicy: IfNotPresent
name: agnhost-container
4.1.1.6. Configuring pods with a static IP address Copier lienLien copié sur presse-papiers!
You can configure pods with a static IP address. The example in the procedure provisions a pod with a static IP address.
- You can specify the IP address for the secondary network attachment of a pod only when the secondary network attachment, a namespaced-scoped object, uses a layer 2 or localnet topology.
- Specifying a static IP address for the pod is only possible when the attachment configuration does not feature subnets.
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: '[
{
"name": "l2-network",
"mac": "02:03:04:05:06:07",
"interface": "myiface1",
"ips": [
"192.0.2.20/24"
]
}
]'
name: tinypod
namespace: ns1
spec:
containers:
- args:
- pause
image: k8s.gcr.io/e2e-test-images/agnhost:2.36
imagePullPolicy: IfNotPresent
name: agnhost-container
where:
k8s.v1.cni.cncf.io/networks.name-
The name of the network. This value must be unique across all
NetworkAttachmentDefinitionCRDs. k8s.v1.cni.cncf.io/networks.mac- The MAC address to be assigned for the interface.
k8s.v1.cni.cncf.io/networks.interface- The name of the network interface to be created for the pod.
k8s.v1.cni.cncf.io/networks.ips- The IP addresses to be assigned to the network interface.
4.2. Creating secondary networks with other CNI plugins Copier lienLien copié sur presse-papiers!
The specific configuration fields for secondary networks are described in the following sections.
4.2.1. Configuration for a bridge secondary network Copier lienLien copié sur presse-papiers!
The bridge CNI plugin JSON configuration object describes the configuration parameters for the Bridge CNI plugin. The following table details these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. A minimum version of |
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The name of the CNI plugin to configure: |
|
|
| The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. |
|
|
|
Optional: Specify the name of the virtual bridge to use. If the bridge interface does not exist on the host, the bridge interface gets created. The default value is |
|
|
|
Optional: Set to |
|
|
|
Optional: Set to |
|
|
|
Optional: Set to |
|
|
|
Optional: Set to |
|
|
|
Optional: Set to |
|
|
|
Optional: Set to |
|
|
| Optional: Specify a virtual LAN (VLAN) tag as an integer value. By default, no VLAN tag is assigned. |
|
|
|
Optional: Indicates whether the default vlan must be preserved on the |
|
|
|
Optional: Assign a VLAN trunk tag. The default value is |
|
|
| Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel. |
|
|
|
Optional: Enables duplicate address detection for the container side |
|
|
|
Optional: Enables mac spoof check, limiting the traffic originating from the container to the mac address of the interface. The default value is |
The VLAN parameter configures the VLAN tag on the host end of the veth and also enables the vlan_filtering feature on the bridge interface.
To configure an uplink for an L2 network, you must allow the VLAN on the uplink interface by using the following command:
$ bridge vlan add vid VLAN_ID dev DEV
4.2.1.1. Bridge CNI plugin configuration example Copier lienLien copié sur presse-papiers!
The following example configures a secondary network named bridge-net:
{
"cniVersion": "0.3.1",
"name": "bridge-net",
"type": "bridge",
"isGateway": true,
"vlan": 2,
"ipam": {
"type": "dhcp"
}
}
4.2.2. Configuration for a Bond CNI secondary network Copier lienLien copié sur presse-papiers!
The Bond Container Network Interface (Bond CNI) enables the aggregation of multiple network interfaces into a single logical bonded interface within a container, which enhanches network redundancy and fault tolerance. Only SR-IOV Virtual Functions (VFs) are supported for bonding with this plugin.
The following table describes the configuration parameters for the Bond CNI plugin:
| Field | Type | Description |
|---|---|---|
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The CNI specification version. A minimum version of |
|
|
|
Specifies the name of the CNI plugin to configure: |
|
|
| Specifies the address resolution protocol (ARP) link monitoring frequency in milliseconds. This parameter defines how often the bond interface sends ARP requests to check the availability of its aggregated interfaces. |
|
|
|
Optional: Specifies the maximum transmission unit (MTU) of the bond. The default is |
|
|
|
Optional: Specifies the |
|
|
| Specifies the bonding policy. |
|
|
|
Specifies the transmit hash policy for load balancing across the aggregated interfaces. This parameter defaults to |
|
|
|
Optional: Specifies whether the network interfaces intended for bonding are expected to be created and available directly within the network namespace of the container when the bond starts. If |
|
|
| Specifies the interfaces to be bonded. |
|
|
| The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. |
xmitHashPolicy is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
4.2.2.1. Bond CNI plugin configuration example Copier lienLien copié sur presse-papiers!
The following example configures a secondary network named bond-net1:
{
"type": "bond",
"cniVersion": "0.3.1",
"name": "bond-net1",
"mode": "active-backup",
"failOverMac": 1,
"linksInContainer": true,
"miimon": "100",
"mtu": 1500,
"links": [
{"name": "net1"},
{"name": "net2"}
],
"ipam": {
"type": "host-local",
"subnet": "10.56.217.0/24",
"routes": [{
"dst": "0.0.0.0/0"
}],
"gateway": "10.56.217.1"
}
}
The following example configures a secondary network named bond-tlb-net with the xmitHashPolicy feature enabled:
{
"type": "bond",
"cniVersion": "0.3.1",
"name": "bond-tlb-net",
"mode": "tlb",
"xmitHashPolicy": "layer2+3",
"failOverMac": 0,
"linksInContainer": true,
"miimon": "100",
"mtu": 1500,
"links": [
{"name": "net1"},
{"name": "net2"}
],
"ipam": {
"type": "host-local",
"subnet": "10.57.218.0/24",
"routes": [{
"dst": "0.0.0.0/0"
}],
"gateway": "10.57.218.1"
}
}
-
xmitHashPolicy: This parameter dictates how outgoing network traffic is distributed across thenet1andnet2active member interfaces within the bond. The hashing algorithm combines layer 2 information, specifically source and destination MAC addresses, with layer 3 information, which includes source and destination IP addresses.
4.2.3. Configuration for a host device secondary network Copier lienLien copié sur presse-papiers!
The host device CNI plugin JSON configuration object describes the configuration parameters for the host-device CNI plugin.
Specify your network device by setting only one of the following parameters: device,hwaddr, kernelpath, or pciBusID.
The following table details the configuration parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. A minimum version of |
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The name of the CNI plugin to configure: |
|
|
|
Optional: The name of the device, such as |
|
|
| Optional: The device hardware MAC address. |
|
|
|
Optional: The Linux kernel device path, such as |
|
|
|
Optional: The PCI address of the network device, such as |
4.2.3.1. host-device configuration example Copier lienLien copié sur presse-papiers!
The following example configures a secondary network named hostdev-net:
{
"cniVersion": "0.3.1",
"name": "hostdev-net",
"type": "host-device",
"device": "eth1"
}
4.2.4. Configuration for a dummy device additional network Copier lienLien copié sur presse-papiers!
The dummy CNI plugin functions like a loopback device. The plugin is a virtual interface, and you can use the plugin to route the packets to a designated IP address. Unlike a loopback device, the IP address is arbitrary and is not restricted to the 127.0.0.0/8 address range.
The dummy device CNI plugin JSON configuration object describes the configuration parameters for the dummy CNI plugin. The following table details these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. A minimum version of |
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The name of the CNI plugin that you want to configure. The required value is |
|
|
| The configuration object for the IPAM CNI plugin. The plugin manages the IP address assignment for the attachment definition. |
4.2.4.1. dummy configuration example Copier lienLien copié sur presse-papiers!
The following example configures an additional network named hostdev-net:
{
"cniVersion": "0.3.1",
"name": "dummy-net",
"type": "dummy",
"ipam": {
"type": "host-local",
"subnet": "10.1.1.0/24"
}
}
4.2.5. Configuration for a VLAN secondary network Copier lienLien copié sur presse-papiers!
The VLAN CNI plugin JSON configuration object describes the configuration parameters for the VLAN, vlan, CNI plugin. The following table details these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. A minimum version of |
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The name of the CNI plugin to configure: |
|
|
|
The Ethernet interface to associate with the network attachment. If a |
|
|
|
Set the ID of the |
|
|
| The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. |
|
|
| Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel. |
|
|
| Optional: DNS information to return. For example, a priority-ordered list of DNS nameservers. |
|
|
|
Optional: Specifies whether the |
A NetworkAttachmentDefinition custom resource definition (CRD) with a vlan configuration can be used only on a single pod in a node because the CNI plugin cannot create multiple vlan subinterfaces with the same vlanId on the same master interface.
4.2.5.1. VLAN configuration example Copier lienLien copié sur presse-papiers!
The following example demonstrates a vlan configuration with a secondary network that is named vlan-net:
{
"name": "vlan-net",
"cniVersion": "0.3.1",
"type": "vlan",
"master": "eth0",
"mtu": 1500,
"vlanId": 5,
"linkInContainer": false,
"ipam": {
"type": "host-local",
"subnet": "10.1.1.0/24"
},
"dns": {
"nameservers": [ "10.1.1.1", "8.8.8.8" ]
}
}
-
ipam.type.host-local: Allocates IPv4 and IPv6 IP addresses from a specified set of address ranges. IPAM plugin stores the IP addresses locally on the host filesystem so that the addresses remain unique to the host.
4.2.6. Configuration for an IPVLAN secondary network Copier lienLien copié sur presse-papiers!
The IPVLAN CNI plugin JSON configuration object describes the configuration parameters for the IPVLAN, ipvlan, CNI plugin. The following table details these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. A minimum version of |
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The name of the CNI plugin to configure: |
|
|
| The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. This is required unless the plugin is chained. |
|
|
|
Optional: The operating mode for the virtual network. The value must be |
|
|
|
Optional: The Ethernet interface to associate with the network attachment. If a |
|
|
| Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel. |
|
|
|
Optional: Specifies whether the |
-
The
ipvlanobject does not allow virtual interfaces to communicate with themasterinterface. Therefore the container is not able to reach the host by using theipvlaninterface. Be sure that the container joins a network that provides connectivity to the host, such as a network supporting the Precision Time Protocol (PTP). -
A single
masterinterface cannot simultaneously be configured to use bothmacvlanandipvlan. -
For IP allocation schemes that cannot be interface agnostic, the
ipvlanplugin can be chained with an earlier plugin that handles this logic. If themasteris omitted, then the previous result must contain a single interface name for theipvlanplugin to enslave. Ifipamis omitted, then the previous result is used to configure theipvlaninterface.
4.2.6.1. IPVLAN CNI plugin configuration example Copier lienLien copié sur presse-papiers!
The following example configures a secondary network named ipvlan-net:
{
"cniVersion": "0.3.1",
"name": "ipvlan-net",
"type": "ipvlan",
"master": "eth1",
"linkInContainer": false,
"mode": "l3",
"ipam": {
"type": "static",
"addresses": [
{
"address": "192.168.10.10/24"
}
]
}
}
4.2.7. Configuration for a MACVLAN secondary network Copier lienLien copié sur presse-papiers!
The MACVLAN CNI plugin JSON configuration object describes the configuration parameters for the MAC Virtual LAN (MACVLAN) Container Network Interface (CNI) plugin. The following table describes these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. A minimum version of |
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The name of the CNI plugin to configure: |
|
|
| The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. |
|
|
|
Optional: Configures traffic visibility on the virtual network. Must be either |
|
|
| Optional: The host network interface to associate with the newly created macvlan interface. If a value is not specified, then the default route interface is used. |
|
|
| Optional: The maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel. |
|
|
|
Optional: Specifies whether the |
If you specify the master key for the plugin configuration, use a different physical network interface than the one that is associated with your primary network plugin to avoid possible conflicts.
4.2.7.1. MACVLAN CNI plugin configuration example Copier lienLien copié sur presse-papiers!
The following example configures a secondary network named macvlan-net:
{
"cniVersion": "0.3.1",
"name": "macvlan-net",
"type": "macvlan",
"master": "eth1",
"linkInContainer": false,
"mode": "bridge",
"ipam": {
"type": "dhcp"
}
}
4.2.8. Configuration for a TAP secondary network Copier lienLien copié sur presse-papiers!
The TAP CNI plugin JSON configuration object describes the configuration parameters for the TAP CNI plugin. The following table describes these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The CNI specification version. A minimum version of |
|
|
| The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations. |
|
|
|
The name of the CNI plugin to configure: |
|
|
| Optional: Request the specified MAC address for the interface. |
|
|
| Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel. |
|
|
| Optional: The SELinux context to associate with the tap device. Note
The value |
|
|
|
Optional: Set to |
|
|
| Optional: The user owning the tap device. |
|
|
| Optional: The group owning the tap device. |
|
|
| Optional: Set the tap device as a port of an already existing bridge. |
4.2.8.1. Tap configuration example Copier lienLien copié sur presse-papiers!
The following example configures a secondary network named mynet:
{
"name": "mynet",
"cniVersion": "0.3.1",
"type": "tap",
"mac": "00:11:22:33:44:55",
"mtu": 1500,
"selinuxcontext": "system_u:system_r:container_t:s0",
"multiQueue": true,
"owner": 0,
"group": 0
"bridge": "br1"
}
4.2.9. Setting SELinux boolean for the TAP CNI plugin Copier lienLien copié sur presse-papiers!
To create the tap device with the container_t SELinux context, enable the container_use_devices boolean on the host by using the Machine Config Operator (MCO).
Prerequisites
-
You have installed the OpenShift CLI (
oc).
Procedure
Create a new YAML file with the following details:
Example
setsebool-container-use-devices.yamlapiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 99-worker-setsebool spec: config: ignition: version: 3.2.0 systemd: units: - enabled: true name: setsebool.service contents: | [Unit] Description=Set SELinux boolean for the TAP CNI plugin Before=kubelet.service [Service] Type=oneshot ExecStart=/usr/sbin/setsebool container_use_devices=on RemainAfterExit=true [Install] WantedBy=multi-user.target graphical.targetCreate the new
MachineConfigobject by running the following command:$ oc apply -f setsebool-container-use-devices.yamlNoteApplying any changes to the
MachineConfigobject causes all affected nodes to gracefully reboot after the change is applied. The MCO might take some time to apply the update.
Verification
Verify that the change is applied by running the following command:
$ oc get machineconfigpoolsNAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-e5e0c8e8be9194e7c5a882e047379cfa True False False 3 3 3 0 7d2h worker rendered-worker-d6c9ca107fba6cd76cdcbfcedcafa0f2 True False False 3 3 3 0 7dNoteAll nodes should be in the
UpdatedandReadystate.
4.2.10. Configuring routes using the route-override plugin on a secondary network Copier lienLien copié sur presse-papiers!
The Route override CNI plugin JSON configuration object describes the configuration parameters for the route-override CNI plugin. The following table details these parameters:
| Field | Type | Description |
|---|---|---|
|
|
|
The name of the CNI plugin to configure: |
|
|
|
Optional: Set to |
|
|
|
Optional: Set to |
|
|
| Optional: Specify the list of routes to delete from the container namespace. |
|
|
|
Optional: Specify the list of routes to add to the container namespace. Each route is a dictionary with |
|
|
|
Optional: Set this to |
4.2.10.1. Route-override plugin configuration example Copier lienLien copié sur presse-papiers!
The route-override CNI is a type of CNI that is designed to be used when chained with a parent CNI. The CNI type does not operate independently, but relies on the parent CNI to first create the network interface and assign IP addresses before the CNI type can modify the routing rules.
The following example configures a secondary network named mymacvlan. The parent CNI creates a network interface attached to eth1 and assigns an IP address in the 192.168.1.0/24 range by using host-local IPAM. The route-override CNI is then chained to the parent CNI and modifies the routing rules by flushing existing routes, deleting the route to 192.168.0.0/24, and adding a new route for 192.168.0.0/24 with a custom gateway.
{
"cniVersion": "0.3.0",
"name": "mymacvlan",
"plugins": [
{
"type": "macvlan",
"master": "eth1",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.1.0/24"
}
},
{
"type": "route-override",
"flushroutes": true,
"delroutes": [
{
"dst": "192.168.0.0/24"
}
],
"addroutes": [
{
"dst": "192.168.0.0/24",
"gw": "10.1.254.254"
}
]
}
]
}
where:
"type": "macvlan"-
The parent CNI creates a network interface attached to
eth1. "type": "route-override"-
The chained
route-overrideCNI modifies the routing rules.
4.3. Attaching a pod to a secondary network Copier lienLien copié sur presse-papiers!
To enable a pod to use additional network interfaces beyond the primary cluster network in OpenShift Container Platform, you can attach the pod to a secondary network. Secondary networks provide additional connectivity options for your workloads.
4.3.1. Adding a pod to a secondary network Copier lienLien copié sur presse-papiers!
To enable a pod to use additional network interfaces in OpenShift Container Platform, you can attach the pod to a secondary network. The pod continues to send normal cluster-related network traffic over the default network.
When a pod is created, a secondary network is attached to the pod. However, if a pod already exists, you cannot attach a secondary network to it.
The pod must be in the same namespace as the secondary network.
Prerequisites
-
Install the OpenShift CLI (
oc). - Log in to the cluster.
Procedure
Add an annotation to the
Podobject. Only one of the following annotation formats can be used:To attach a secondary network without any customization, add an annotation with the following format:
metadata: annotations: k8s.v1.cni.cncf.io/networks: <network>[,<network>,...]where:
k8s.v1.cni.cncf.io/networks- Specifies the name of the secondary network to associate with the pod. To specify more than one secondary network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same secondary network multiple times, that pod will have multiple network interfaces attached to that network.
To attach a secondary network with customizations, add an annotation with the following format:
metadata: annotations: k8s.v1.cni.cncf.io/networks: |- [ { "name": "<network>", "namespace": "<namespace>", "default-route": ["<default_route>"] } ]where:
<network>-
Specifies the name of the secondary network defined by a
NetworkAttachmentDefinitionobject. <namespace>-
Specifies the namespace where the
NetworkAttachmentDefinitionobject is defined. <default-route>-
Optional parameter. Specifies an override for the default route, such as
192.168.17.1.
Create the pod by entering the following command.
$ oc create -f <name>.yamlReplace
<name>with the name of the pod.Optional: Confirm that the annotation exists in the
podCR by entering the following command. Replace<name>with the name of the pod.$ oc get pod <name> -o yamlIn the following example, the
example-podpod is attached to thenet1secondary network:$ oc get pod example-pod -o yaml apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/networks: macvlan-bridge k8s.v1.cni.cncf.io/network-status: |- [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.128.2.14" ], "default": true, "dns": {} },{ "name": "macvlan-bridge", "interface": "net1", "ips": [ "20.2.2.100" ], "mac": "22:2f:60:a5:f8:00", "dns": {} }] name: example-pod namespace: default spec: ... status: ...where:
k8s.v1.cni.cncf.io/network-status- Specifies a JSON array of objects. Each object describes the status of a secondary network attached to the pod. The annotation value is stored as a plain text value.
4.3.1.1. Specifying pod-specific addressing and routing options Copier lienLien copié sur presse-papiers!
To set static IP addresses, MAC addresses, and default routes for a pod in OpenShift Container Platform, you can configure pod-specific addressing and routing options using JSON-formatted annotations. With these annotations, you can customize network behavior for individual pods on secondary networks.
Prerequisites
- The pod must be in the same namespace as the secondary network.
-
Install the OpenShift CLI (
oc). - You must log in to the cluster.
Procedure
Edit the
Podresource definition. If you are editing an existingPodresource, run the following command to edit its definition in the default editor. Replace<name>with the name of thePodresource to edit.$ oc edit pod <name>In the
Podresource definition, add thek8s.v1.cni.cncf.io/networksparameter to the podmetadatamapping. Thek8s.v1.cni.cncf.io/networksaccepts a JSON string of a list of objects that reference the name ofNetworkAttachmentDefinitioncustom resource (CR) names in addition to specifying additional properties.metadata: annotations: k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]' # ...where:
<network>- Replace with a JSON object as shown in the following examples. The single quotes are required.
In the following example the annotation specifies which network attachment will have the default route, using the
default-routeparameter.apiVersion: v1 kind: Pod metadata: name: example-pod annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "net1" }, { "name": "net2", "default-route": ["192.0.2.1"] }]' spec: containers: - name: example-pod command: ["/bin/bash", "-c", "sleep 2000000000000"] image: centos/toolswhere:
net1,net2-
Specifies the name of the
NetworkAttachmentDefinitionresource that defines the secondary network to associate with the pod. 192.0.2.1-
Specifies a value of a gateway for traffic to be routed over if no other routing entry is present in the routing table. If more than one
default-routekey is specified, this will cause the pod to fail to become active.
The default route will cause any traffic that is not specified in other routes to be routed to the gateway.
ImportantSetting the default route to an interface other than the default network interface for OpenShift Container Platform may cause traffic that is anticipated for pod-to-pod traffic to be routed over another interface.
To verify the routing properties of a pod, the
occommand may be used to execute theipcommand within a pod.$ oc exec -it <pod_name> -- ip routeNoteYou may also reference the pod’s
k8s.v1.cni.cncf.io/network-statusto see which secondary network has been assigned the default route, by the presence of thedefault-routekey in the JSON-formatted list of objects.To set a static IP address or MAC address for a pod you can use the JSON formatted annotations. This requires you create networks that specifically allow for this functionality. This can be specified in a rawCNIConfig for the CNO.
Edit the CNO CR by running the following command:
$ oc edit networks.operator.openshift.io clusterThe following YAML describes the configuration parameters for the CNO:
Cluster Network Operator YAML configuration
name: <name> namespace: <namespace> rawCNIConfig: '{ ... }' type: Rawwhere:
name-
Specifies a name for the secondary network attachment that you are creating. The name must be unique within the specified
namespace. namespace-
Specifies the namespace to create the network attachment in. If you do not specify a value, then the
defaultnamespace is used. rawCNIConfig- Specifies the CNI plugin configuration in JSON format, which is based on the following template.
The following object describes the configuration parameters for utilizing static MAC address and IP address using the macvlan CNI plugin:
macvlan CNI plugin JSON configuration object using static IP and MAC address
{ "cniVersion": "0.3.1", "name": "<name>", "plugins": [{ "type": "macvlan", "capabilities": { "ips": true }, "master": "eth0", "mode": "bridge", "ipam": { "type": "static" } }, { "capabilities": { "mac": true }, "type": "tuning" }] }where:
name-
Specifies the name for the secondary network attachment to create. The name must be unique within the specified
namespace. plugins- Specifies an array of CNI plugin configurations. The first object specifies a macvlan plugin configuration and the second object specifies a tuning plugin configuration.
ips- Specifies that a request is made to enable the static IP address functionality of the CNI plugin runtime configuration capabilities.
master- Specifies the interface that the macvlan plugin uses.
mac- Specifies that a request is made to enable the static MAC address functionality of a CNI plugin.
The above network attachment can be referenced in a JSON formatted annotation, along with keys to specify which static IP and MAC address will be assigned to a given pod.
Edit the pod by entering the following command:
$ oc edit pod <name>macvlan CNI plugin JSON configuration object using static IP and MAC address
apiVersion: v1 kind: Pod metadata: name: example-pod annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "<name>", "ips": [ "192.0.2.205/24" ], "mac": "CA:FE:C0:FF:EE:00" } ]'where:
metadata.name-
Specifies the name for the secondary network attachment to create. The name must be unique within the specified
namespace. metadata.annotations.k8s.v1.cni.cncf.io/ips- Specifies an IP address including the subnet mask.
metadata.annotations.k8s.v1.cni.cncf.io/mac- Specifies the MAC address.
NoteStatic IP addresses and MAC addresses do not have to be used at the same time. You can use them individually, or together.
4.4. Configuring multi-network policy Copier lienLien copié sur presse-papiers!
As an administrator, you can use the MultiNetworkPolicy API to create multiple network policies that manage traffic for pods that are attached to secondary networks. For example, you can create policies that allow or deny traffic based on specific ports, IPs and ranges, or labels.
Multi-network policies can be used to manage traffic on secondary networks in the cluster. These policies cannot manage the default cluster network or primary network of user-defined networks.
As a cluster administrator, you can configure a multi-network policy for any of the following network types:
- Single-Root I/O Virtualization (SR-IOV)
- MAC Virtual Local Area Network (MacVLAN)
- IP Virtual Local Area Network (IPVLAN)
- Bond Container Network Interface (CNI) over SR-IOV
- OVN-Kubernetes secondary networks
Support for configuring multi-network policies for SR-IOV secondary networks is only supported with kernel network interface controllers (NICs). SR-IOV is not supported for Data Plane Development Kit (DPDK) applications.
4.4.1. Differences between multi-network policy and network policy Copier lienLien copié sur presse-papiers!
Although the MultiNetworkPolicy API implements the NetworkPolicy API, ensure that you understand the following key differences between the two policies:
You must use the
MultiNetworkPolicyAPI, as demonstrated in the following example configuration:apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy # ...-
You must use the
multi-networkpolicyresource name when using the CLI to interact with multi-network policies. For example, you can view a multi-network policy object with theoc get multi-networkpolicy <name>command where<name>is the name of a multi-network policy. You can use the
k8s.v1.cni.cncf.io/policy-forannotation on aMultiNetworkPolicyobject to point to aNetworkAttachmentDefinition(NAD) custom resource (CR). The NAD CR defines the network to which the policy applies. The following example multi-network policy includes thek8s.v1.cni.cncf.io/policy-forannotation:apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> # ...where:
<namespace_name>- Specifies the namespace name.
<network_name>- Specifies the name of a network attachment definition.
4.4.2. Enabling multi-network policy for the cluster Copier lienLien copié sur presse-papiers!
As a cluster administrator, you can enable multi-network policy support on your cluster.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Log in to the cluster with a user with
cluster-adminprivileges.
Procedure
Create the
multinetwork-enable-patch.yamlfile with the following YAML:apiVersion: operator.openshift.io/v1 kind: Network metadata: name: cluster spec: useMultiNetworkPolicy: true # ...Configure the cluster to enable multi-network policy. Successful output lists the name of the policy object and the
patchedstatus.$ oc patch network.operator.openshift.io cluster --type=merge --patch-file=multinetwork-enable-patch.yaml
4.4.3. Supporting multi-network policies in IPv6 networks Copier lienLien copié sur presse-papiers!
The ICMPv6 Neighbor Discovery Protocol (NDP) is a set of messages and processes that enable devices to discover and maintain information about neighboring nodes. NDP plays a crucial role in IPv6 networks, facilitating the interaction between devices on the same link.
The Cluster Network Operator (CNO) deploys the iptables implementation of multi-network policy when the useMultiNetworkPolicy parameter is set to true.
To support multi-network policies in IPv6 networks the Cluster Network Operator deploys the following set of custom rules in every pod affected by a multi-network policy:
kind: ConfigMap
apiVersion: v1
metadata:
name: multi-networkpolicy-custom-rules
namespace: openshift-multus
data:
custom-v6-rules.txt: |
# accept NDP
-p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT
-p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT
# accept RA/RS
-p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
-p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
where:
icmpv6-type neighbor-solicitation- This rule allows incoming ICMPv6 neighbor solicitation messages, which are part of the neighbor discovery protocol (NDP). These messages help determine the link-layer addresses of neighboring nodes.
icmpv6-type neighbor-advertisement- This rule allows incoming ICMPv6 neighbor advertisement messages, which are part of NDP and provide information about the link-layer address of the sender.
icmpv6-type router-solicitation- This rule permits incoming ICMPv6 router solicitation messages. Hosts use these messages to request router configuration information.
icmpv6-type router-advertisement- This rule allows incoming ICMPv6 router advertisement messages, which give configuration information to hosts.
You cannot edit the predefined rules.
The rules collectively enable essential ICMPv6 traffic for correct network functioning, including address resolution and router communication in an IPv6 environment. With these rules in place and a multi-network policy denying traffic, applications are not expected to experience connectivity issues.
4.4.4. Working with multi-network policy Copier lienLien copié sur presse-papiers!
To manage network traffic isolation and security for pods on secondary networks, you can create, edit, view, and delete multi-network policies. Before you work with multi-network policies, you must enable multi-network policy support for your cluster.
4.4.4.1. Creating a multi-network policy using the CLI Copier lienLien copié sur presse-papiers!
To define granular rules describing ingress or egress network traffic allowed for namespaces in your cluster, you can create a multi-network policy.
Prerequisites
-
Your cluster uses a network plugin that supports
NetworkPolicyobjects, such as the OVN-Kubernetes network plugin, withmode: NetworkPolicyset. -
You installed the OpenShift CLI (
oc). -
You logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace that the multi-network policy applies to.
Procedure
Create a policy rule.
Create a
<policy_name>.yamlfile:$ touch <policy_name>.yamlwhere:
<policy_name>- Specifies the multi-network policy file name.
Define a multi-network policy in the created file. The following example denies ingress traffic from all pods in all namespaces. This is a fundamental policy, blocking all cross-pod networking other than cross-pod traffic allowed by the configuration of other Network Policies.
apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy endif:: multi[] metadata: name: deny-by-default annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: podSelector: {} policyTypes: - Ingress ingress: []where:
<network_name>Specifies the name of a network attachment definition.
The following example configuration allows ingress traffic from all pods in the same namespace:
apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: name: allow-same-namespace annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: podSelector: ingress: - from: - podSelector: {} # ...where:
<network_name>Specifies the name of a network attachment definition.
The following example allows ingress traffic to one pod from a particular namespace. This policy allows traffic to pods that have the
pod-alabel from pods running innamespace-y.apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: name: allow-traffic-pod annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: podSelector: matchLabels: pod: pod-a policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: namespace-y # ...where:
<network_name>Specifies the name of a network attachment definition.
The following example configuration restricts traffic to a service. This policy when applied ensures every pod with both labels
app=bookstoreandrole=apican only be accessed by pods with labelapp=bookstore. In this example the application could be a REST API server, marked with labelsapp=bookstoreandrole=api.This example configuration addresses the following use cases:
- Restricting the traffic to a service to only the other microservices that need to use it.
Restricting the connections to a database to only permit the application using it.
apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: name: api-allow annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: podSelector: matchLabels: app: bookstore role: api ingress: - from: - podSelector: matchLabels: app: bookstore # ...where:
<network_name>- Specifies the name of a network attachment definition.
To create the multi-network policy object, enter the following command. Successful output lists the name of the policy object and the
createdstatus.$ oc apply -f <policy_name>.yaml -n <namespace>where:
<policy_name>- Specifies the multi-network policy file name.
<namespace>- Optional parameter. If you defined the object in a different namespace than the current namespace, the parameter specifices the namespace.
Successful output lists the name of the policy object and the
createdstatus.NoteIf you log in to the web console with
cluster-adminprivileges, you have a choice of creating a network policy in any namespace in the cluster directly in YAML or from a form in the web console.
4.4.4.2. Editing a multi-network policy Copier lienLien copié sur presse-papiers!
To modify existing policy configurations, you can edit a multi-network policy in a namespace. Edit policies by modifying the policy file and applying it with oc apply, or by using the oc edit command directly.
If you log in with cluster-admin privileges, you can edit network policies in any namespace in the cluster. In the web console, you can edit policies directly in YAML or by using the Actions menu.
Prerequisites
-
Your cluster uses a network plugin that supports
NetworkPolicyobjects, such as the OVN-Kubernetes network plugin, withmode: NetworkPolicyset. -
You installed the OpenShift CLI (
oc). -
You are logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace where the multi-network policy exists.
Procedure
Optional: To list the multi-network policy objects in a namespace, enter the following command:
$ oc get multi-network policy -n <namespace>where:
<namespace>- Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
Edit the multi-network policy object.
If you saved the multi-network policy definition in a file, edit the file and make any necessary changes, and then enter the following command.
$ oc apply -n <namespace> -f <policy_file>.yamlwhere:
<namespace>- Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
<policy_file>- Specifies the name of the file containing the network policy.
If you need to update the multi-network policy object directly, enter the following command:
$ oc edit multi-network policy <policy_name> -n <namespace>where:
<policy_name>- Specifies the name of the network policy.
<namespace>- Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
Confirm that the multi-network policy object is updated.
$ oc describe multi-networkpolicy <policy_name> -n <namespace>where:
<policy_name>- Specifies the name of the multi-network policy.
<namespace>- Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
4.4.4.3. Viewing multi-network policies using the CLI Copier lienLien copié sur presse-papiers!
You can examine the multi-network policies in a namespace.
If you log in with cluster-admin privileges, you can edit network policies in any namespace in the cluster. In the web console, you can edit policies directly in YAML or by using the Actions menu.
Prerequisites
-
You installed the OpenShift CLI (
oc). -
You are logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace where the multi-network policy exists.
Procedure
List multi-network policies in a namespace.
To view multi-network policy objects defined in a namespace enter the following command:
$ oc get multi-networkpolicyOptional: To examine a specific multi-network policy enter the following command:
$ oc describe multi-networkpolicy <policy_name> -n <namespace>where:
<policy_name>- Specifies the name of the multi-network policy to inspect.
<namespace>- Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
4.4.4.4. Deleting a multi-network policy using the CLI Copier lienLien copié sur presse-papiers!
You can delete a multi-network policy in a namespace.
If you log in with cluster-admin privileges, you can delete network policies in any namespace in the cluster. In the web console, you can delete policies directly in YAML or by using the Actions menu.
Prerequisites
-
Your cluster uses a network plugin that supports
NetworkPolicyobjects, such as the OVN-Kubernetes network plugin, withmode: NetworkPolicyset. -
You installed the OpenShift CLI (
oc). -
You logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace where the multi-network policy exists.
Procedure
To delete a multi-network policy object, enter the following command. Successful output lists the name of the policy object and the
deletedstatus.$ oc delete multi-networkpolicy <policy_name> -n <namespace>where:
<policy_name>- Specifies the name of the multi-network policy.
<namespace>- Optional parameter. If you defined the object in a different namespace than the current namespace, the parameter specifices the namespace.
4.4.4.5. Creating a default deny all multi-network policy Copier lienLien copié sur presse-papiers!
The default deny all multi-network policy blocks all cross-pod networking other than network traffic allowed by the configuration of other deployed network policies and traffic between host-networked pods. This procedure enforces a strong deny policy by applying a deny-by-default policy in the my-project namespace.
Without configuring a NetworkPolicy custom resource (CR) that allows traffic communication, the following policy might cause communication problems across your cluster.
Prerequisites
-
Your cluster uses a network plugin that supports
NetworkPolicyobjects, such as the OVN-Kubernetes network plugin, withmode: NetworkPolicyset. -
You installed the OpenShift CLI (
oc). -
You logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace that the multi-network policy applies to.
Procedure
Create the following YAML that defines a
deny-by-defaultpolicy to deny ingress from all pods in all namespaces. Save the YAML in thedeny-by-default.yamlfile:apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: name: deny-by-default namespace: my-project annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: podSelector: {} policyTypes: - Ingress ingress: []where:
namespace-
Specifies the namespace in which to deploy the policy. For example, the
my-projectnamespace. annotations- Specifies the name of namespace project followed by the network attachment definition name.
podSelector-
If this field is empty, the configuration matches all the pods. Therefore, the policy applies to all pods in the
my-projectnamespace. policyTypes-
Specifies a list of rule types that the
NetworkPolicyrelates to. - Ingress-
Specifies
IngressonlypolicyTypes. ingress- Specifies ingress rules. If not specified, all incoming traffic is dropped to all pods.
Apply the policy by entering the following command. Successful output lists the name of the policy object and the
createdstatus.$ oc apply -f deny-by-default.yaml
4.4.4.6. Creating a multi-network policy to allow traffic from external clients Copier lienLien copié sur presse-papiers!
With the deny-by-default policy in place you can proceed to configure a policy that allows traffic from external clients to a pod with the label app=web.
If you log in with a user with the cluster-admin role, then you can create a network policy in any namespace in the cluster.
Follow this procedure to configure a policy that allows external service from the public Internet directly or by using a Load Balancer to access the pod. Traffic is only allowed to a pod with the label app=web.
Prerequisites
-
Your cluster uses a network plugin that supports
NetworkPolicyobjects, such as the OVN-Kubernetes network plugin, withmode: NetworkPolicyset. -
You installed the OpenShift CLI (
oc). -
You logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace that the multi-network policy applies to.
Procedure
Create a policy that allows traffic from the public Internet directly or by using a load balancer to access the pod. Save the YAML in the
web-allow-external.yamlfile:apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: name: web-allow-external namespace: default annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: policyTypes: - Ingress podSelector: matchLabels: app: web ingress: - {}Apply the policy by entering the following command. Successful output lists the name of the policy object and the
createdstatus.$ oc apply -f web-allow-external.yamlThis policy allows traffic from all resources, including external traffic as illustrated in the following diagram:
4.4.4.7. Creating a multi-network policy allowing traffic to an application from all namespaces Copier lienLien copié sur presse-papiers!
You can configure a policy that allows traffic from all pods in all namespaces to a particular application.
If you log in with a user with the cluster-admin role, then you can create a network policy in any namespace in the cluster.
Prerequisites
-
Your cluster uses a network plugin that supports
NetworkPolicyobjects, such as the OVN-Kubernetes network plugin, withmode: NetworkPolicyset. -
You installed the OpenShift CLI (
oc). -
You logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace that the multi-network policy applies to.
Procedure
Create a policy that allows traffic from all pods in all namespaces to a particular application. Save the YAML in the
web-allow-all-namespaces.yamlfile:apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: name: web-allow-all-namespaces namespace: default annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: podSelector: matchLabels: app: web policyTypes: - Ingress ingress: - from: - namespaceSelector: {}where:
app-
Applies the policy only to
app:webpods in default namespace. namespaceSelectorSelects all pods in all namespaces.
NoteBy default, if you do not specify a
namespaceSelectorparameter in the policy object, no namespaces get selected. This means the policy allows traffic only from the namespace where the network policy deployes.
Apply the policy by entering the following command. Successful output lists the name of the policy object and the
createdstatus.$ oc apply -f web-allow-all-namespaces.yaml
Verification
Start a web service in the
defaultnamespace by entering the following command:$ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80Run the following command to deploy an
alpineimage in thesecondarynamespace and to start a shell:$ oc run test-$RANDOM --namespace=secondary --rm -i -t --image=alpine -- shRun the following command in the shell and observe that the service allows the request:
# wget -qO- --timeout=2 http://web.default<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
4.4.4.8. Creating a multi-network policy allowing traffic to an application from a namespace Copier lienLien copié sur presse-papiers!
You can configure a policy that allows traffic to a pod with the label app=web from a particular namespace. This configuration is useful in the following use cases:
- Restrict traffic to a production database only to namespaces that have production workloads deployed.
- Enable monitoring tools deployed to a particular namespace to scrape metrics from the current namespace.
If you log in with a user with the cluster-admin role, then you can create a network policy in any namespace in the cluster.
Prerequisites
-
Your cluster uses a network plugin that supports
NetworkPolicyobjects, such as the OVN-Kubernetes network plugin, withmode: NetworkPolicyset. -
You installed the OpenShift CLI (
oc). -
You logged in to the cluster with a user with
cluster-adminprivileges. - You are working in the namespace that the multi-network policy applies to.
Do not apply the network.openshift.io/policy-group: ingress label to custom namespace or projects. This label is Operator-managed and reserved for OpenShift Container Platform networking functions. It should not be altered on system-created namespaces.
Using this label can result in intermittent network connectivity drops, unintended application of system NetworkPolicies resource, or configuration drift as the operator attempts to reconcile the state. For custom traffic grouping, always use unique, user-defined labels as shown in the following procedure.
Procedure
Create a policy that allows traffic from all pods in a particular namespaces with a label
purpose=production. Save the YAML in theweb-allow-prod.yamlfile:apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: name: web-allow-prod namespace: default annotations: k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name> spec: podSelector: matchLabels: app: web policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: purpose: productionwhere:
app-
Applies the policy only to
app:webpods in the default namespace. purpose-
Restricts traffic to only pods in namespaces that have the label
purpose=production.
Apply the policy by entering the following command. Successful output lists the name of the policy object and the
createdstatus.$ oc apply -f web-allow-prod.yaml
Verification
Start a web service in the
defaultnamespace by entering the following command:$ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80Run the following command to create the
prodnamespace:$ oc create namespace prodRun the following command to label the
prodnamespace:$ oc label namespace/prod purpose=productionRun the following command to create the
devnamespace:$ oc create namespace devRun the following command to label the
devnamespace:$ oc label namespace/dev purpose=testingRun the following command to deploy an
alpineimage in thedevnamespace and to start a shell:$ oc run test-$RANDOM --namespace=dev --rm -i -t --image=alpine -- shRun the following command in the shell and observe the reason for the blocked request. For example, expected output states
wget: download timed out.# wget -qO- --timeout=2 http://web.defaultRun the following command to deploy an
alpineimage in theprodnamespace and start a shell:$ oc run test-$RANDOM --namespace=prod --rm -i -t --image=alpine -- shRun the following command in the shell and observe that the request is allowed:
# wget -qO- --timeout=2 http://web.default<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
4.5. Removing a pod from a secondary network Copier lienLien copié sur presse-papiers!
To disconnect a pod from specific network configurations in OpenShift Container Platform, you can remove the pod from a secondary network. Delete the pod to remove its connection to the secondary network.
4.5.1. Removing a pod from a secondary network Copier lienLien copié sur presse-papiers!
To disconnect a pod from specific network configurations in OpenShift Container Platform, you can remove the pod from a secondary network. Delete the pod using the oc delete pod command to remove its connection to the secondary network.
Prerequisites
- A secondary network is attached to the pod.
-
Install the OpenShift CLI (
oc). - Log in to the cluster.
Procedure
Delete the pod by entering the following command:
$ oc delete pod <name> -n <namespace>where:
<name>- Specifies the name of the pod.
<namespace>- Specifies the namespace that contains the pod.
4.6. Editing a secondary network Copier lienLien copié sur presse-papiers!
To update network settings or change network parameters for a secondary network in OpenShift Container Platform, you can modify the configuration for an existing secondary network. Edit the NetworkAttachmentDefinition custom resource to apply your changes.
4.6.1. Modifying a NetworkAttachmentDefinition custom resource Copier lienLien copié sur presse-papiers!
To update network settings or change network parameters for a secondary network in OpenShift Container Platform, you can modify the NetworkAttachmentDefinition custom resource. Edit the Cluster Network Operator CR to apply your changes.
Prerequisites
- You have configured a secondary network for your cluster.
-
Install the OpenShift CLI (
oc). -
Log in as a user with
cluster-adminprivileges.
Procedure
Edit the Cluster Network Operator (CNO) CR in your default text editor by running the following command:
$ oc edit networks.operator.openshift.io cluster-
In the
additionalNetworkscollection, update the secondary network with your changes. - Save your changes and quit the text editor to commit your changes.
Optional: Confirm that the CNO updated the
NetworkAttachmentDefinitionobject by running the following command. Replace<network_name>with the name of the secondary network to display. There might be a delay before the CNO updates theNetworkAttachmentDefinitionobject to reflect your changes.$ oc get network-attachment-definitions <network_name> -o yamlFor example, the following console output displays a
NetworkAttachmentDefinitionobject that is namednet1:$ oc get network-attachment-definitions net1 -o go-template='{{printf "%s\n" .spec.config}}' { "cniVersion": "0.3.1", "type": "macvlan", "master": "ens5", "mode": "bridge", "ipam": {"type":"static","routes":[{"dst":"0.0.0.0/0","gw":"10.128.2.1"}],"addresses":[{"address":"10.128.2.100/23","gateway":"10.128.2.1"}],"dns":{"nameservers":["172.30.0.10"],"domain":"us-west-2.compute.internal","search":["us-west-2.compute.internal"]}} }
4.6.2. Using an OVN-Kubernetes localnet topology to map VLANs to a secondary interface Copier lienLien copié sur presse-papiers!
You can use OVN-Kubernetes localnet topology in a NetworkAttachmentDefinition (NAD) to map a specific VLAN ID from the physical network to the secondary interface of a pod.
To provide multiple VLANs for cluster workloads in OpenShift Container Platform, define additional VLANs in the NetworkAttachmentDefinition custom resource (CR). Configuring trunk ports ensures that the physical network associates correctly with your virtual infrastructure for reliable traffic management.
The example in the procedure demonstrates the following configurations:
- Physical switch ports connect to OpenShift Container Platform nodes by using VLAN trunking. The trunk carries tagged traffic for the VLANs you define in NADs.
-
The
br-exacts as the OVS bridge that connects virtual workloads to the physical workloads. -
Multiple NADs with specific VLAN tags get created by using the
localnettopology. This configuration defines specific VLAN IDs for traffic isolation. - Pods or virtual machines (VMs) attach to the NAD CRs for the purposes of improved network connectivity.
Prerequisites
-
You installed the OpenShift CLI (
oc). -
You logged in as a user with
cluster-adminprivileges. - You installed the NMState Operator.
-
You configured the
br-exbridge interface during cluster installation.
Procedure
Create an
NetworkAttachmentDefinitionCR for each VLAN, such asnad-cvlan100.yaml. OVN-Kubernetes uses the NAD files to tag and untag Ethernet frames for pods or VMs.Example configuration
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vlan-100 namespace: default spec: config: |- { "cniVersion": "0.4.0", "name": "localnet-vlan-100", "type": "ovn-k8s-cni-overlay", "physicalNetworkName": "physnet", "topology": "localnet", "vlanID": 100, "mtu": 1500, "netAttachDefName": "default/vlan-100" } # ...Attach pods or VMs to the VLANs by referencing the NAD in the configuration for the pod or VM:
Example pod configuration
apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/networks: vlan-100 # ...Example VM configuration
apiVersion: kubevirt.io/v1 kind: VirtualMachine spec: template: spec: networks: - multus: networkName: vlan-100 name: secondary-vlan # ...
4.7. Configuring IP address assignment on secondary networks Copier lienLien copié sur presse-papiers!
You can configure IP address assignments for secondary networks so that pods can connect to the secondary networks.
4.7.1. Configuration of IP address assignment for a network attachment Copier lienLien copié sur presse-papiers!
For secondary networks, you can assign IP addresses by using an IP Address Management (IPAM) CNI plugin, which supports various assignment methods, including Dynamic Host Configuration Protocol (DHCP) and static assignment.
The DHCP IPAM CNI plugin responsible for dynamic assignment of IP addresses operates with two distinct components:
- CNI Plugin: Responsible for integrating with the Kubernetes networking stack to request and release IP addresses.
- DHCP IPAM CNI Daemon: A listener for DHCP events that coordinates with existing DHCP servers in the environment to handle IP address assignment requests. This daemon is not a DHCP server itself.
For networks requiring type: dhcp in their IPAM configuration, ensure the DHCP server meets the following conditions:
- A DHCP server is available and running in the environment.
- The DHCP server is external to the cluster and you expect the server to form part of the existing network infrastructure for the customer.
- The DHCP server is appropriately configured to serve IP addresses to the nodes.
In cases where a DHCP server is unavailable in the environment, consider using the Whereabouts IPAM CNI plugin. The Whereabouts CNI provides similar IP address management capabilities without the need for an external DHCP server.
Use the Whereabouts CNI plugin when no external DHCP server exists or where static IP address management is preferred. The Whereabouts plugin includes a reconciler daemon to manage stale IP address allocations.
Ensure the periodic renewal of a DHCP lease throughout the lifetime of a container by including a separate daemon, the DHCP IPAM CNI Daemon. To deploy the DHCP IPAM CNI daemon, change the Cluster Network Operator (CNO) configuration to trigger the deployment of this daemon as part of the secondary network setup.
4.7.1.1. Static IP address assignment configuration Copier lienLien copié sur presse-papiers!
The following table describes the configuration for static IP address assignment:
| Field | Type | Description |
|---|---|---|
|
|
|
The IPAM address type. The value |
|
|
| An array of objects specifying IP addresses to assign to the virtual interface. Both IPv4 and IPv6 IP addresses are supported. |
|
|
| An array of objects specifying routes to configure inside the pod. |
|
|
| Optional: An array of objects specifying the DNS configuration. |
The addresses array requires objects with the following fields:
| Field | Type | Description |
|---|---|---|
|
|
|
An IP address and network prefix that you specify. For example, if you specify |
|
|
| The default gateway to route egress network traffic to. |
| Field | Type | Description |
|---|---|---|
|
|
|
The IP address range in CIDR format, such as |
|
|
| The gateway that routes network traffic. |
| Field | Type | Description |
|---|---|---|
|
|
| An array of one or more IP addresses where DNS queries get sent. |
|
|
|
The default domain to append to a hostname. For example, if the domain is set to |
|
|
|
An array of domain names to append to an unqualified hostname, such as |
Static IP address assignment configuration example
{
"ipam": {
"type": "static",
"addresses": [
{
"address": "191.168.1.7/24"
}
]
}
}
4.7.1.2. Dynamic IP address (DHCP) assignment configuration Copier lienLien copié sur presse-papiers!
A pod obtains its original DHCP lease when the pod gets created. The lease must be periodically renewed by a minimal DHCP server deployment running on the cluster.
For an Ethernet network attachment, the SR-IOV Network Operator does not create a DHCP server deployment; the Cluster Network Operator is responsible for creating the minimal DHCP server deployment.
To trigger the deployment of the DHCP server, you must create a shim network attachment by editing the Cluster Network Operator configuration, as in the following example:
Example shim network attachment definition
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
name: cluster
spec:
additionalNetworks:
- name: dhcp-shim
namespace: default
type: Raw
rawCNIConfig: |-
{
"name": "dhcp-shim",
"cniVersion": "0.3.1",
"type": "bridge",
"ipam": {
"type": "dhcp"
}
}
# ...
where:
type- Specifies dynamic IP address assignment for the cluster.
4.7.1.3. Dynamic IP address assignment configuration with Whereabouts Copier lienLien copié sur presse-papiers!
The Whereabouts CNI plugin helps the dynamic assignment of an IP address to a secondary network without the use of a DHCP server.
The Whereabouts CNI plugin also supports overlapping IP address ranges and configuration of the same CIDR range multiple times within separate NetworkAttachmentDefinition CRDs. This provides greater flexibility and management capabilities in multitenant environments.
4.7.1.3.1. Dynamic IP address configuration parameters Copier lienLien copié sur presse-papiers!
The following table describes the configuration objects for dynamic IP address assignment with Whereabouts:
| Field | Type | Description |
|---|---|---|
|
|
|
The IPAM address type. The value |
|
|
| An IP address and range in CIDR notation. IP addresses are assigned from within this range of addresses. |
|
|
| Optional: A list of zero or more IP addresses and ranges in CIDR notation. IP addresses within an excluded address range are not assigned. |
|
|
| Optional: Helps ensure that each group or domain of pods gets its own set of IP addresses, even if they share the same range of IP addresses. Setting this field is important for keeping networks separate and organized, notably in multi-tenant environments. |
4.7.1.3.2. Dynamic IP address assignment configuration with Whereabouts that excludes IP address ranges Copier lienLien copié sur presse-papiers!
The following example shows a dynamic address assignment configuration in a NAD file that uses Whereabouts:
Whereabouts dynamic IP address assignment that excludes specific IP address ranges
{
"ipam": {
"type": "whereabouts",
"range": "192.0.2.192/27",
"exclude": [
"192.0.2.192/30",
"192.0.2.196/32"
]
}
}
4.7.1.3.3. Dynamic IP address assignment that uses Whereabouts with overlapping IP address ranges Copier lienLien copié sur presse-papiers!
The following example shows a dynamic IP address assignment that uses overlapping IP address ranges for multitenant networks.
NetworkAttachmentDefinition 1
{
"ipam": {
"type": "whereabouts",
"range": "192.0.2.192/29",
"network_name": "example_net_common",
}
}
where:
network_name-
Optional parameter. If set, must match the
network_nameofNetworkAttachmentDefinition 2.
NetworkAttachmentDefinition 2
{
"ipam": {
"type": "whereabouts",
"range": "192.0.2.192/24",
"network_name": "example_net_common",
}
}
where:
network_name-
Optional parameter. If set, must match the
network_nameofNetworkAttachmentDefinition 1.
4.7.1.4. Creating a whereabouts-reconciler daemon set Copier lienLien copié sur presse-papiers!
The Whereabouts reconciler is responsible for managing dynamic IP address assignments for the pods within a cluster by using the Whereabouts IP Address Management (IPAM) solution. The Whereabouts reconciler ensures that each pod gets a unique IP address from the specified IP address range. The Whereabouts reconciler also handles IP address releases when pods are deleted or scaled down.
You can also use a NetworkAttachmentDefinition custom resource definition (CRD) for dynamic IP address assignment.
The whereabouts-reconciler daemon set is automatically created when you configure a secondary network through the Cluster Network Operator. The whereabouts-reconciler DaemonSet does not get automatically created when you configure a secondary network from a YAML manifest.
To trigger the deployment of the whereabouts-reconciler daemon set, you must manually create a whereabouts-shim network attachment by editing the Cluster Network Operator custom resource (CR) file.
Procedure
Edit the
Network.operator.openshift.ioCR by running the following command:$ oc edit network.operator.openshift.io clusterInclude the
additionalNetworkssection shown in this example YAML extract within thespecdefinition of the CR:apiVersion: operator.openshift.io/v1 kind: Network metadata: name: cluster # ... spec: additionalNetworks: - name: whereabouts-shim namespace: default rawCNIConfig: |- { "name": "whereabouts-shim", "cniVersion": "0.3.1", "type": "bridge", "ipam": { "type": "whereabouts" } } type: Raw # ...- Save the file and exit the text editor.
Verify that the
whereabouts-reconcilerdaemon set deployed successfully by running the following command:$ oc get all -n openshift-multus | grep whereabouts-reconcilerpod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s
4.7.1.5. Configuring the Whereabouts IP reconciler schedule Copier lienLien copié sur presse-papiers!
The Whereabouts IPAM CNI plugin runs the IP address reconciler daily. This process cleans up any stranded IP address allocations that might result in exhausting IP addresses and therefore prevent new pods from getting a stranded IP address allocated to them.
Use this procedure to change the frequency at which the IP reconciler runs.
Prerequisites
-
You installed the OpenShift CLI (
oc). -
You have access to the cluster as a user with the
cluster-adminrole. -
You have deployed the
whereabouts-reconcilerdaemon set, and thewhereabouts-reconcilerpods are up and running.
Procedure
Run the following command to create a
ConfigMapobject namedwhereabouts-configin theopenshift-multusnamespace with a specific cron expression for the IP reconciler:$ oc create configmap whereabouts-config -n openshift-multus --from-literal=reconciler_cron_expression="*/15 * * * *"This cron expression indicates the IP reconciler runs every 15 minutes. Adjust the expression based on your specific requirements.
NoteThe
whereabouts-reconcilerdaemon set can only consume a cron expression pattern that includes five asterisks. Red Hat does not support the sixth asterisk, which is used to denote seconds.Retrieve information about resources related to the
whereabouts-reconcilerdaemon set and pods within theopenshift-multusnamespace by running the following command:$ oc get all -n openshift-multus | grep whereabouts-reconcilerpod/whereabouts-reconciler-2p7hw 1/1 Running 0 4m14s pod/whereabouts-reconciler-76jk7 1/1 Running 0 4m14s daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 4m16sRun the following command to verify that the
whereabouts-reconcilerpod runs the IP reconciler with the configured interval:$ oc -n openshift-multus logs whereabouts-reconciler-2p7hw2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CREATE 2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CHMOD 2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..data_tmp": RENAME 2024-02-02T16:33:54Z [verbose] using expression: */15 * * * * 2024-02-02T16:33:54Z [verbose] configuration updated to file "/cron-schedule/..data". New cron expression: */15 * * * * 2024-02-02T16:33:54Z [verbose] successfully updated CRON configuration id "00c2d1c9-631d-403f-bb86-73ad104a6817" - new cron expression: */15 * * * * 2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/config": CREATE 2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_26_17.3874177937": REMOVE 2024-02-02T16:45:00Z [verbose] starting reconciler run 2024-02-02T16:45:00Z [debug] NewReconcileLooper - inferred connection data 2024-02-02T16:45:00Z [debug] listing IP pools 2024-02-02T16:45:00Z [debug] no IP addresses to cleanup 2024-02-02T16:45:00Z [verbose] reconciler success
4.7.1.6. Fast IPAM configuration for the Whereabouts IPAM CNI plugin Copier lienLien copié sur presse-papiers!
Wherabouts is an IP Address Management (IPAM) Container Network Interface (CNI) plugin that assigns IP addresses at a cluster-wide level. Whereabouts does not require a Dynamic Host Configuration Protocol (DHCP) server.
A typical Wherabouts workflow is described as follows:
-
Whereabouts takes an address range in classless inter-domain routing (CIDR) notation, such as
192.168.2.0/24, and assigns IP addresses within that range, such as192.168.2.1to192.168.2.254. - Whereabouts assigns an IP address, the lowest value address in a CIDR range, to a pod and tracks the IP address in a data store for the lifetime of that pod.
- When the pod is removed, Whereabouts frees the address from the pod so that the address is available for assignment.
To improve the performance of Whereabouts, especially if nodes in your cluster run a high amount of pods, you can enable the Fast IPAM feature.
Fast IPAM is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
The Fast IPAM feature uses nodeslicepools, which are managed by the Whereabouts Controller, to optimize IP allocation for nodes.
Prerequisites
-
You added the
whereabouts-shimconfiguration to theNetwork.operator.openshift.iocustom resource (CR), so that the Cluster Network Operator (CNO) can deploy the Whereabouts Controller. See "Creating a Whereabouts reconciler daemon set". -
For the Fast IPAM feature to work, ensure that the
NetworkAttachmentDefinition(NAD) and the pod exist in the sameopenshift-multusnamespace.
Procedure
Confirm that the Whereabouts Controller is running by entering the following command.
$ oc get pods -n openshift-multus | grep whereabouts-controllerwhereabouts-controller-5cbfd6c475-fr7d7 1/1 Running 0 22s ...ImportantIf the Whereabouts Controller is not running, the Fast IPAM does not work.
Create a NAD file for your cluster and add the Fast IPAM details to the file as demonstrated in the following example configuration:
apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: wb-ipam namespace: openshift-multus spec: config: '{ "cniVersion": "0.3.0", "name": "wb-ipam-cni-name", "type": "bridge", "bridge": "cni0", "ipam": { "type": "whereabouts", "range": "10.5.0.0/20", "node_slice_size": "/24" } }' # ...where:
namespace- The namespace where CNO deploys the NAD.
name- The name of the Whereabouts IPAM CNI plugin.
type-
The type of IPAM CNI plugin, such as
whereabouts. range- The IP address range for the IP pool that the Whereabouts IPAM CNI plugin uses for allocating IP addresses to pods.
node_slice_size- Sets the slice size of IP addresses available to each node.
Add the Whereabouts IPAM CNI plugin annotation details to the YAML file for the pod:
apiVersion: v1 kind: Pod metadata: name: samplepod annotations: k8s.v1.cni.cncf.io/networks: openshift-multus/wb-ipam spec: containers: - name: samplecontainer command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"] image: registry.redhat.io/ubi9/ubi-minimal # ...where:
name- The name of the pod.
k8s.v1.cni.cncf.io/networks-
The annotation details that references the Whereabouts IPAM CNI plugin name that exists in the
openshift-multusnamespace. - name- The name of the container for the pod.
command- Defines the entry point for the container and controls the behavior of the container in the Whereabouts IPAM CNI plugin.
Apply the NAD file configuration to pods that exist on nodes that run in your cluster:
$ oc create -f <NAD_file_name>.yaml
Verification
Show the IP address details of the pod by entering the following command:
$ oc describe pod <pod_name>... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.128.3.174" ], "mac": "0a:58:0a:80:03:ae", "default": true, "dns": {} },{ "name": "openshift-multus/wb-ipam", "interface": "net1", "ips": [ "10.5.0.1" ], "mac": "1a:04:6f:a4:15:3c", "dns": {} }] k8s.v1.cni.cncf.io/networks: openshift-multus/wb-ipam ...Access the pod and confirm its interfaces by entering the following command:
$ oc exec <pod_name> -- ip a... 3: net1@if439: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 1a:04:6f:a4:15:3c brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.5.0.1/20 brd 10.5.15.255 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::1804:6fff:fea4:153c/64 scope link valid_lft forever preferred_lft forever ...where:
inet: Pod is attached to the10.5.0.1IP address on thenet1interface as expected.Check that the node selector pool exists in the
openshift-multusnamespace by entering the following command. The expected output shows the name of the node selector pool, such asnodeslicepool, and the creation age in minutes, such as `32m.$ oc get nodeslicepool -n openshift-multusExample output
NAME AGE wb-ipam-cni-name 32m
4.7.1.7. Creating a configuration for assignment of dual-stack IP addresses dynamically Copier lienLien copié sur presse-papiers!
You can dynamically assign dual-stack IP addresses to a secondary network so that pods can communicate over both IPv4 and IPv6 addresses.
You can configure the following IP address assignment types in the ipRanges parameter:
- IPv4 addresses
- IPv6 addresses
- multiple IP address assignment
Procedure
-
Set
typetowhereabouts. Use
ipRangesto allocate IP addresses as shown in the following example:cniVersion: operator.openshift.io/v1 kind: Network metadata: name: cluster spec: additionalNetworks: - name: whereabouts-shim namespace: default type: Raw rawCNIConfig: |- { "name": "whereabouts-dual-stack", "cniVersion": "0.3.1, "type": "bridge", "ipam": { "type": "whereabouts", "ipRanges": [ {"range": "192.168.10.0/24"}, {"range": "2001:db8::/64"} ] } }- Attach the secondary network to a pod. For more information, see "Adding a pod to a secondary network".
Verification
Verify that all IP addresses got assigned to the network interfaces within the network namespace of a pod by entering the following command:
$ oc exec -it <pod_name> -- ip awhere:
<podname>- The name of the pod.
4.8. Configuring the master interface in the container network namespace Copier lienLien copié sur presse-papiers!
You can create and manage a MAC-VLAN, IP-VLAN, and VLAN subinterface based on a master interface.
4.8.1. About configuring the master interface in the container network namespace Copier lienLien copié sur presse-papiers!
You can create a MAC-VLAN, an IP-VLAN, or a VLAN subinterface that is based on a master interface that exists in a container namespace. You can also create a master interface as part of the pod network configuration in a separate network attachment definition CRD.
To use a container namespace master interface, you must specify true for the linkInContainer parameter that exists in the subinterface configuration of the NetworkAttachmentDefinition CRD.
4.8.1.1. Creating multiple VLANs on SR-IOV VFs Copier lienLien copié sur presse-papiers!
You can create multiple VLANs based on SR-IOV VFs. For this configuration, create an SR-IOV network and then define the network attachments for the VLAN interfaces.
The following diagram shows the setup process for creating multiple VLANs on SR-IOV VFs.
Prerequisites
-
You installed the OpenShift CLI (
oc). -
You have access to the cluster as a user with the
cluster-adminrole. - You have installed the SR-IOV Network Operator.
Procedure
Create a dedicated container namespace where you want to deploy your pod by using the following command:
$ oc new-project test-namespaceCreate an SR-IOV node policy.
Create an
SriovNetworkNodePolicyobject, and then save the YAML in thesriov-node-network-policy.yamlfile:apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: sriovnic namespace: openshift-sriov-network-operator spec: deviceType: netdevice isRdma: false needVhostNet: true nicSelector: vendor: "15b3" deviceID: "101b" rootDevices: ["00:05.0"] numVfs: 10 priority: 99 resourceName: sriovnic nodeSelector: feature.node.kubernetes.io/network-sriov.capable: "true"where:
vendor-
The vendor hexadecimal code of the SR-IOV network device. The value
15b3associates with a Mellanox NIC. deviceIDThe device hexadecimal code of the SR-IOV network device.
NoteThe SR-IOV network node policy configuration example, with the setting
deviceType: netdevice, is tailored specifically for Mellanox Network Interface Cards (NICs).
Apply the YAML configuration by running the following command:
$ oc apply -f sriov-node-network-policy.yamlNoteApplying the YAML configuration might take time because of a node reboot operation.
Create an SR-IOV network:
Create the
SriovNetworkcustom resource (CR) for the additional secondary SR-IOV network attachment as demonstrated in the following example CR. Save the YAML as asriov-network-attachment.yamlfile:apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetwork metadata: name: sriov-network namespace: openshift-sriov-network-operator spec: networkNamespace: test-namespace resourceName: sriovnic spoofChk: "off" trust: "on"Apply the YAML by running the following command:
$ oc apply -f sriov-network-attachment.yaml
Create the VLAN secondary network.
Using the following YAML example, create a file named
vlan100-additional-network-configuration.yaml:apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vlan-100 namespace: test-namespace spec: config: | { "cniVersion": "0.4.0", "name": "vlan-100", "plugins": [ { "type": "vlan", "master": "ext0", "mtu": 1500, "vlanId": 100, "linkInContainer": true, "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]} } ] }where:
master-
The VLAN configuration needs to specify the
mastername. You can specify the name in the networks annotation of a pod. linkInContainer-
The
linkInContainerparameter must be specified.
Apply the YAML file by running the following command:
$ oc apply -f vlan100-additional-network-configuration.yaml
Create a pod definition by using the earlier specified networks.
Using the following YAML configuration example, create a file named
pod-a.yamlfile:NoteThe manifest example includes the following resources:
- Namespace with security labels
- Pod definition with appropriate network annotation
apiVersion: v1 kind: Namespace metadata: name: test-namespace labels: pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/warn: privileged security.openshift.io/scc.podSecurityLabelSync: "false" --- apiVersion: v1 kind: Pod metadata: name: nginx-pod namespace: test-namespace annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "interface": "ext0.100" } ]' spec: securityContext: runAsNonRoot: true containers: - name: nginx-container image: nginxinc/nginx-unprivileged:latest securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] ports: - containerPort: 80 seccompProfile: type: "RuntimeDefault"where:
interface-
The name to be used as the
masterinterface for the VLAN interface.
Apply the YAML file by running the following command:
$ oc apply -f pod-a.yaml
Get detailed information about the
nginx-podwithin thetest-namespaceby running the following command:$ oc describe pods nginx-pod -n test-namespaceName: nginx-pod Namespace: test-namespace Priority: 0 Node: worker-1/10.46.186.105 Start Time: Mon, 14 Aug 2023 16:23:13 -0400 Labels: <none> Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.26" ], "mac": "0a:58:0a:83:00:1a", "default": true, "dns": {} },{ "name": "test-namespace/sriov-network", "interface": "ext0", "mac": "6e:a7:5e:3f:49:1b", "dns": {}, "device-info": { "type": "pci", "version": "1.0.0", "pci": { "pci-address": "0000:d8:00.2" } } },{ "name": "test-namespace/vlan-100", "interface": "ext0.100", "ips": [ "1.1.1.1" ], "mac": "6e:a7:5e:3f:49:1b", "dns": {} }] k8s.v1.cni.cncf.io/networks: [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i... openshift.io/scc: privileged Status: Running IP: 10.131.0.26 IPs: IP: 10.131.0.26
4.8.1.2. Creating a subinterface based on a bridge master interface in a container namespace Copier lienLien copié sur presse-papiers!
You can create a subinterface based on a bridge master interface that exists in a container namespace. Creating a subinterface can be applied to other types of interfaces.
Prerequisites
-
You have installed the OpenShift CLI (
oc). -
You are logged in to the OpenShift Container Platform cluster as a user with
cluster-adminprivileges.
Procedure
Create a dedicated container namespace where you want to deploy your pod by entering the following command:
$ oc new-project test-namespaceUsing the following YAML configuration example, create a bridge
NetworkAttachmentDefinitioncustom resource definition (CRD) file namedbridge-nad.yaml:apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: bridge-network spec: config: '{ "cniVersion": "0.4.0", "name": "bridge-network", "type": "bridge", "bridge": "br-001", "isGateway": true, "ipMasq": true, "hairpinMode": true, "ipam": { "type": "host-local", "subnet": "10.0.0.0/24", "routes": [{"dst": "0.0.0.0/0"}] } }'Run the following command to apply the
NetworkAttachmentDefinitionCRD to your OpenShift Container Platform cluster:$ oc apply -f bridge-nad.yamlVerify that you successfully created a
NetworkAttachmentDefinitionCRD by entering the following command. The expected output shows the name of the NAD CRD and the creation age in minutes.$ oc get network-attachment-definitionsUsing the following YAML example, create a file named
ipvlan-additional-network-configuration.yamlfor the IPVLAN secondary network configuration:apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ipvlan-net namespace: test-namespace spec: config: '{ "cniVersion": "0.3.1", "name": "ipvlan-net", "type": "ipvlan", "master": "net1", "mode": "l3", "linkInContainer": true, "ipam": {"type": "whereabouts", "ipRanges": [{"range": "10.0.0.0/24"}]} }'where:
master- Specifies the ethernet interface to associate with the network attachment. The ethernet interface is subsequently configured in the pod networks annotation.
linkInContainer-
Specifies that the
masterinterface exists in the container network namespace.
Apply the YAML file by running the following command:
$ oc apply -f ipvlan-additional-network-configuration.yamlVerify that the
NetworkAttachmentDefinitionCRD has been created successfully by running the following command. The expected output shows the name of the NAD CRD and the creation age in minutes.$ oc get network-attachment-definitionsUsing the following YAML configuration example, create a file named
pod-a.yamlfor the pod definition:apiVersion: v1 kind: Pod metadata: name: pod-a namespace: test-namespace annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "bridge-network", "interface": "net1"1 }, { "name": "ipvlan-net", "interface": "net2" } ]' spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: test-pod image: quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL]where:
k8s.v1.cni.cncf.io/networks,interface-
Specifies the name to be used as the
masterfor the IPVLAN interface.
Apply the YAML file by running the following command:
$ oc apply -f pod-a.yaml
Verification
Verify that the pod is running by using the following command:
$ oc get pod -n test-namespaceNAME READY STATUS RESTARTS AGE pod-a 1/1 Running 0 2m36sShow network interface information about the
pod-aresource within thetest-namespaceby running the following command:$ oc exec -n test-namespace pod-a -- ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if105: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 0a:58:0a:d9:00:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.217.0.93/23 brd 10.217.1.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::488b:91ff:fe84:a94b/64 scope link valid_lft forever preferred_lft forever 4: net1@if107: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.0.0.2/24 brd 10.0.0.255 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::bcda:bdff:fe7e:f437/64 scope link valid_lft forever preferred_lft forever 5: net2@net1: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff inet 10.0.0.1/24 brd 10.0.0.255 scope global net2 valid_lft forever preferred_lft forever inet6 fe80::beda:bd00:17e:f437/64 scope link valid_lft forever preferred_lft foreverThis output shows that the network interface
net2associates with the physical interfacenet1.
4.9. Removing an additional network Copier lienLien copié sur presse-papiers!
To clean up unused network configurations or free up network resources in OpenShift Container Platform, you can remove an additional network attachment. Delete the NetworkAttachmentDefinition custom resource to remove the secondary network from your cluster.
4.9.1. Removing a secondary NetworkAttachmentDefinition custom resource Copier lienLien copié sur presse-papiers!
To clean up unused network configurations or free up network resources in OpenShift Container Platform, you can remove a secondary NetworkAttachmentDefinition CR. Edit the Cluster Network Operator CR and delete the NetworkAttachmentDefinition CR to remove the secondary network from your cluster.
When a secondary network is removed from the cluster, it is not removed from any pods that it is attached to.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Log in as a user with
cluster-adminprivileges.
Procedure
Edit the Cluster Network Operator (CNO) in your default text editor by running the following command:
$ oc edit networks.operator.openshift.io clusterModify the custom resource (CR) by removing the configuration that the CNO created from the
additionalNetworkscollection for the secondary network that you want to remove.apiVersion: operator.openshift.io/v1 kind: Network metadata: name: cluster spec: additionalNetworks: []where:
spec.additionalNetworks-
Specifies the secondary network attachment definition that you want to remove from the
additionalNetworkscollection. If you are removing the configuration mapping for the only secondary network attachment definition in theadditionalNetworkscollection, you must specify an empty collection.
Remove the
NetworkAttachmentDefinitionCR from the network of your cluster by entering the following command:$ oc delete net-attach-def <name_of_network_attachment_definition>Replace
<name_of_network_attachment_definition>with the name of theNetworkAttachmentDefinitionCR that you want to remove.- Save your changes and quit the text editor to commit your changes.
Optional: Confirm that the secondary network CR was deleted by running the following command:
$ oc get network-attachment-definition --all-namespaces
4.10. Enabling multi-networking for advanced use cases with CNI plugin chaining Copier lienLien copié sur presse-papiers!
You can use Container Network Interface (CNI) plugin chaining to enable advanced multi-networking use cases for your pods.
4.10.1. About CNI chaining Copier lienLien copié sur presse-papiers!
CNI plugin chaining allows pods to use multiple network interfaces. This enables advanced configurations such as traffic isolation and prioritized routing through granular traffic policies.
By using CNI plugin chaining, different types of traffic can be isolated to meet performance, security, and compliance requirements, providing greater flexibility in network design and traffic management.
Some scenarios where this might be useful include:
- Multi-Network topologies: Enables you to attach pods to multiple networks, each with its own traffic policy, where relevant.
- Traffic isolation: Provides separate networks for management, storage, and application traffic to ensure each has the appropriate security and QoS settings.
- Custom routing rules: Ensures that specific traffic, for example SIP traffic, always uses a designated network interface, while other traffic follows the default network.
- Enhanced network performance: Allows you to prioritize certain traffic types or manage congestion by directing them through dedicated network interfaces.
4.10.2. Configuring plugin chaining with the route-override CNI plugin Copier lienLien copié sur presse-papiers!
Plugin chaining allows you to configure multiple CNI plugins to be applied sequentially to the same network interface, where each plugin in the chain processes the interface in order.
When you define a NetworkAttachmentDefinition (NAD) with a plugins array, the first plugin can create the interface, and a second plugin can modify its routing configuration.
The route-override CNI plugin is commonly used as the second plugin in a chain to modify the routing configuration of an interface created by the first plugin. It supports the following operations:
-
addroutes:Add static routes to direct traffic for specific destination networks through the interface. -
delroutes:Remove specific routes from the interface. -
flushroutes:Remove all routes from the interface. -
flushgateway:Remove the default gateway route from the interface.
The following example demonstrates plugin chaining by configuring a pod with two additional network interfaces, each on a separate VLAN with custom routing:
-
eth1on the192.168.100.0/24network (VLAN 100), with a static route directing10.0.0.0/8traffic through this interface. -
eth2on the192.168.200.0/24network (VLAN 200), with a static route directing172.16.0.0/12traffic through this interface.
Each interface uses a chain of two plugins: macvlan to create the interface on a VLAN, and route-override to add static routes that direct specific traffic through that interface.
Prerequisites
-
Install the OpenShift CLI (
oc). -
An account with
cluster-adminprivileges.
Procedure
Create a namespace for the example by running the following command:
$ oc create namespace chain-exampleCreate the first NetworkAttachmentDefinition (NAD) with a chained plugin configuration.
Create a YAML file, such as
management.yaml, to define a NAD that configures a new interface,eth1, on VLAN 100 with the following configuration:apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: management-net namespace: chain-example spec: config: '{ "cniVersion": "1.0.0", "name": "management-net", "plugins": [ { "type": "macvlan", "master": "br-ex", "vlan": 100, "mode": "bridge", "ipam": { "type": "static", "addresses": [ { "address": "192.168.100.10/24", "gateway": "192.168.100.1" } ] } }, { "type": "route-override", "addroutes": [ { "dst": "10.0.0.0/8", "gw": "192.168.100.1" } ] } ] }'
Create the NAD by running the following command:
$ oc apply -f management.yamlCreate the second NAD with a chained plugin configuration.
Create a YAML file, such as
sip.yaml, to define a NAD that configures a new interface,eth2, on VLAN 200 with the following configuration:apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: sip-net namespace: chain-example spec: config: '{ "cniVersion": "1.0.0", "name": "sip-net", "plugins": [ { "type": "macvlan", "master": "br-ex", "vlan": 200, "mode": "bridge", "ipam": { "type": "static", "addresses": [ { "address": "192.168.200.10/24", "gateway": "192.168.200.1" } ] } }, { "type": "route-override", "addroutes": [ { "dst": "172.16.0.0/12", "gw": "192.168.200.1" } ] } ] }'
Create the NAD by running the following command:
$ oc apply -f sip.yamlAttach the
NetworkAttachmentDefinitionresources to a pod by creating a pod definition file, such aspod.yaml, with the following configuration:apiVersion: v1 kind: Pod metadata: name: chain-test-pod namespace: chain-example labels: app: chain-test annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "management-net", "interface": "eth1" }, { "name": "sip-net", "interface": "eth2" } ]' spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: test-container image: registry.access.redhat.com/ubi9/ubi:latest command: ["sleep", "infinity"] securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"]Create the pod by running the following command:
$ oc apply -f pod.yamlVerify the pod is running with the following command:
$ oc wait --for=condition=Ready pod/chain-test-pod -n chain-example --timeout=120sExample output:
pod/chain-test-pod condition met
Verification
Run the following command to list all network interfaces and their assigned IP addresses inside the pod. This verifies that the pod has the additional interfaces configured by plugin chaining:
$ oc exec chain-test-pod -n chain-example -- ip aExample output:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc noqueue state UP link/ether 0a:58:0a:83:02:19 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.131.2.25/23 brd 10.131.3.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::858:aff:fe83:219/64 scope link valid_lft forever preferred_lft forever 3: eth1@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP qlen 1000 link/ether aa:25:73:ff:a7:00 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.100.10/24 brd 192.168.100.255 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::a825:73ff:feff:a700/64 scope link valid_lft forever preferred_lft forever 4: eth2@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP qlen 1000 link/ether aa:a4:6c:4e:e8:97 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.200.10/24 brd 192.168.200.255 scope global eth2 valid_lft forever preferred_lft forever inet6 fe80::a8a4:6cff:fe4e:e897/64 scope link valid_lft forever preferred_lft foreverThis output shows the pod has three network interfaces:
-
eth0:The default interface, connected to the cluster network. -
eth1:The first additional interface frommanagement-net, with IP192.168.100.10. -
eth2:The second additional interface fromsip-net, with IP192.168.200.10.
-
Run the following command to verify that the
route-overrideplugin added the expected static routes:$ oc exec chain-test-pod -n chain-example -- ip routeExample output:
default via 10.132.0.1 dev eth0 10.0.0.0/8 via 192.168.100.1 dev eth1 10.132.0.0/23 dev eth0 proto kernel scope link src 10.132.1.97 10.132.0.0/14 via 10.132.0.1 dev eth0 100.64.0.0/16 via 10.132.0.1 dev eth0 169.254.0.5 via 10.132.0.1 dev eth0 172.16.0.0/12 via 192.168.200.1 dev eth2 172.30.0.0/16 via 10.132.0.1 dev eth0 192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.10 192.168.200.0/24 dev eth2 proto kernel scope link src 192.168.200.10This output confirms that the
route-overrideplugin in each chain added the expected static routes:-
For
10.0.0.0/8 via 192.168.100.1 dev eth1, traffic destined for10.0.0.0/8is routed througheth1via themanagement-netgateway. This route was added by theroute-overrideplugin in themanagement-netchain. -
For
172.16.0.0/12 via 192.168.200.1 dev eth2, traffic destined for172.16.0.0/12is routed througheth2via thesip-netgateway. This route was added by theroute-overrideplugin in thesip-netchain. -
The connected subnet routes (
192.168.100.0/24and192.168.200.0/24) were created by themacvlanplugin, while the default route useseth0, the cluster network interface.
-
For