Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Multiple networks

OpenShift Container Platform 4.17

Configuring and managing multiple network interfaces and virtual routing in OpenShift Container Platform

Red Hat OpenShift Documentation Team

Abstract

This document explains how to set up and manage primary and secondary networks, including virtual routing and forwarding, in OpenShift Container Platform.

Chapter 1. Understanding multiple networks
Copy link

By default, OVN-Kubernetes serves as the Container Network Interface (CNI) of an OpenShift Container Platform cluster. With OVN-Kubernetes as the default CNI of a cluster, OpenShift Container Platform administrators or users can leverage user-defined networks (UDNs) or NetworkAttachmentDefinition (NADs) to create one, or multiple, default networks that handle all ordinary network traffic of the cluster. Both user-defined networks and Network Attachment Definitions can serve as the following network types:

  • Primary networks: Act as the primary network for the pod. By default, all traffic passes through the primary network unless a pod route is configured to send traffic through other networks.
  • Secondary networks: Act as secondary, non-default networks for a pod. Secondary networks provide separate interfaces dedicated to specific traffic types or purposes. Only pod traffic that is explicitly configured to use a secondary network is routed through its interface.

However, during cluster installation, OpenShift Container Platform administrators can configure alternative default secondary pod networks by leveraging the Multus CNI plugin. With Multus, multiple CNI plugins such as ipvlan, macvlan, or Network Attachment Definitions can be used together to serve as secondary networks for pods.

Note

User-defined networks are only available when OVN-Kubernetes is used as the CNI. They are not supported for use with other CNIs.

You can define an secondary network based on the available CNI plugins and attach one or more of these networks to your pods. You can define more than one secondary network for your cluster depending on your needs. This gives you flexibility when you configure pods that deliver network functionality, such as switching or routing.

For a complete list of supported CNI plugins, see "Secondary networks in OpenShift Container Platform".

For information about user-defined networks, see About user-defined networks (UDNs).

For information about Network Attachment Definitions, see Creating primary networks using a NetworkAttachmentDefinition.

1.1. Usage scenarios for a secondary network
Copy link

You can use a secondary network, also known as a secondary network, in situations where network isolation is needed, including data plane and control plane separation. Isolating network traffic is useful for the following performance and security reasons:

  1. Performance

    Traffic management: You can send traffic on two different planes to manage how much traffic is along each plane.

  2. Security

    Network isolation: You can send sensitive traffic onto a network plane that is managed specifically for security considerations, and you can separate private data that must not be shared between tenants or customers.

All of the pods in the cluster still use the cluster-wide default network to maintain connectivity across the cluster. Every pod has an eth0 interface that is attached to the cluster-wide pod network. You can view the interfaces for a pod by using the oc exec -it <pod_name> -- ip a command. If you add secondary network interfaces that use Multus CNI, they are named net1, net2, …​, netN.

To attach secondary network interfaces to a pod, you must create configurations that define how the interfaces are attached. You specify each interface by using either a UserDefinedNetwork custom resource (CR) or a NetworkAttachmentDefinition CR. A CNI configuration inside each of these CRs defines how that interface is created.

For more information about creating a UserDefinedNetwork CR, see About user-defined networks.

For more information about creating a NetworkAttachmentDefinition CR, see Creating primary networks using a NetworkAttachmentDefinition.

OpenShift Container Platform provides the following CNI plugins for creating secondary networks in your cluster:

Chapter 2. Primary networks
Copy link

2.1. About user-defined networks
Copy link

Important

UserDefinedNetwork is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Before the implementation of user-defined networks (UDN), the OVN-Kubernetes CNI plugin for OpenShift Container Platform only supported a Layer 3 topology on the primary or main network. Due to Kubernetes design principles: all pods are attached to the main network, all pods communicate with each other by their IP addresses, and inter-pod traffic is restricted according to network policy.

UDN improves the flexibility and segmentation capabilities of the default Layer 3 topology for a Kubernetes pod network by enabling custom Layer 2 and Layer 3 network segments, where all these segments are isolated by default. These segments act as either primary or secondary networks for container pods and virtual machines that use the default OVN-Kubernetes CNI plugin. UDNs enable a wide range of network architectures and topologies, enhancing network flexibility, security, and performance.

Note

Support for the Localnet topology on both primary and secondary networks will be added in a future version of OpenShift Container Platform.

A cluster administrator can use a UDN to create and define primary or secondary networks that span multiple namespaces at the cluster level by leveraging the ClusterUserDefinedNetwork custom resource (CR). Additionally, a cluster administrator or a cluster user can use a UDN to define secondary networks at the namespace level with the UserDefinedNetwork CR.

The following diagram shows four cluster namespaces, where each namespace has a single assigned user-defined network (UDN), and each UDN has an assigned custom subnet for its pod IP allocations. The OVN-Kubernetes handles any overlapping UDN subnets. Without using the Kubernetes network policy, a pod attached to a UDN can communicate with other pods in that UDN. By default, these pods are isolated from communicating with pods that exist in other UDNs. For microsegmentation, you can apply network policy within a UDN. You can assign one or more UDNs to a namespace, with a limitation of only one primary UDN to a namespace, and one or more namespaces to a UDN.

Figure 2.1. Namespace isolation using a UserDefinedNetwork CR

The namespace isolation concept in a user-defined network (UDN)
View larger image
The namespace isolation concept in a user-defined network (UDN)

The following sections further emphasize the benefits and limitations of user-defined networks, the best practices when creating a UserDefinedNetwork custom resource, how to create the custom resource, and additional configuration details that might be relevant to your deployment.

2.1.1. Benefits of a user-defined network
Copy link

User-defined networks provide the following benefits:

  1. Enhanced network isolation for security

    • Tenant isolation: Namespaces can have their own isolated primary network, similar to how tenants are isolated in Red Hat OpenStack Platform (RHOSP). This improves security by reducing the risk of cross-tenant traffic.

  2. Network flexibility

    • Layer 2 and layer 3 support: Cluster administrators can configure primary networks as layer 2 or layer 3 network types.

  3. Simplified network management

    • Reduced network configuration complexity: With user-defined networks, the need for complex network policies are eliminated because isolation can be achieved by grouping workloads in different networks.

  4. Advanced capabilities

    • Consistent and selectable IP addressing: Users can specify and reuse IP subnets across different namespaces and clusters, providing a consistent networking environment.
    • Support for multiple networks: The user-defined networking feature allows administrators to connect multiple namespaces to a single network, or to create distinct networks for different sets of namespaces.

  5. Simplification of application migration from Red Hat OpenStack Platform (RHOSP)

    • Network parity: With user-defined networking, the migration of applications from OpenStack to OpenShift Container Platform is simplified by providing similar network isolation and configuration options.

Developers and administrators can create a user-defined network that is namespace scoped using the custom resource. An overview of the process is as follows:

  1. An administrator creates a namespace for a user-defined network with the k8s.ovn.org/primary-user-defined-network label.
  2. The UserDefinedNetwork CR is created by either the cluster administrator or the user.
  3. The user creates pods in the namespace.

While user-defined networks (UDN) offer highly customizable network configuration options, there are limitations that cluster administrators and developers should be aware of when implementing and managing these networks. Consider the following limitations before implementing a user-defined network.

  • DNS limitations:

    • DNS lookups for pods resolve to the pod’s IP address on the cluster default network. Even if a pod is part of a user-defined network, DNS lookups will not resolve to the pod’s IP address on that user-defined network. However, DNS lookups for services and external entities will function as expected.
    • When a pod is assigned to a primary UDN, it can access the Kubernetes API (KAPI) and DNS services on the cluster’s default network.
  • Initial network assignment: You must create the namespace and network before creating pods. Assigning a namespace with pods to a new network or creating a UDN in an existing namespace will not be accepted by OVN-Kubernetes.
  • Health check limitations: Kubelet health checks are performed by the cluster default network, which does not confirm the network connectivity of the primary interface on the pod. Consequently, scenarios where a pod appears healthy by the default network, but has broken connectivity on the primary interface, are possible with user-defined networks.
  • Network policy limitations: Network policies that enable traffic between namespaces connected to different user-defined primary networks are not effective. These traffic policies do not take effect because there is no connectivity between these isolated networks.
  • Creation and modification limitation: The ClusterUserDefinedNetwork CR and the UserDefinedNetwork CR cannot be modified after being created.

2.1.3. Layer 2 and layer 3 topologies
Copy link

A flat layer 2 topology creates a virtual switch that is distributed across all nodes in a cluster. Virtual machines and pods connect to this virtual switch so that all these components can communicate with each other within the same subnet. A flat layer 2 topology is useful for live migration of virtual machines across nodes that exist in a cluster. The following diagram shows a flat layer 2 topology with two nodes that use the virtual switch for live migration purposes:

Figure 2.2. A flat layer 2 topology that uses a virtual switch for component communication

A flat layer 2 topology with a virtual switch so that virtual machines in node-1 to node-2 can communicate with each other
View larger image
A flat layer 2 topology with a virtual switch so that virtual machines in node-1 to node-2 can communicate with each other

If you decide not to specify a layer 2 subnet, then you must manually configure IP addresses for each pod in your cluster. When you do not specify a layer 2 subnet, port security is limited to preventing Media Access Control (MAC) spoofing only, and does not include IP spoofing. A layer 2 topology creates a single broadcast domain that can be challenging in large network environments, where the topology might cause a broadcast storm that can degrade network performance.

To access more configurable options for your network, you can integrate a layer 2 topology with a user-defined network (UDN). The following diagram shows two nodes that use a UDN with a layer 2 topology that includes pods that exist on each node. Each node includes two interfaces:

  • A node interface, which is a compute node that connects networking components to the node.
  • An Open vSwitch (OVS) bridge such as br-ex, which creates an layer 2 OVN switch so that pods can communicate with each other and share resources.

An external switch connects these two interfaces, while the gateway or router handles routing traffic between the external switch and the layer 2 OVN switch. VMs and pods in a node can use the UDN to communicate with each other. The layer 2 OVN switch handles node traffic over a UDN so that live migrate of a VM from one node to another is possible.

Figure 2.3. A user-defined network (UDN) that uses a layer 2 topology

A UDN that uses a layer 2 topology for migrating a VM from node-1 to node-2
View larger image
A UDN that uses a layer 2 topology for migrating a VM from node-1 to node-2

A layer 3 topology creates a unique layer 2 segment for each node in a cluster. The layer 3 routing mechanism interconnects these segments so that virtual machines and pods that are hosted on different nodes can communicate with each other. A layer 3 topology can effectively manage large broadcast domains by assigning each domain to a specific node, so that broadcast traffic has a reduced scope. To configure a layer 3 topology, you must configure cidr and hostSubnet parameters.

2.1.4. Best practices for UserDefinedNetwork
Copy link

Before setting up a UserDefinedNetwork (UDN) resource, users should consider the following information:

  • openshift-* namespaces should not be used to set up a UDN.

  • 2 masquerade IP addresses are required for user defined networks. You must reconfigure your masquerade subnet to be large enough to hold the required number of networks.

    Important
    • For OpenShift Container Platform 4.17 and later, clusters use 169.254.0.0/17 for IPv4 and fd69::/112 for IPv6 as the default masquerade subnet. These ranges should be avoided by users. For updated clusters, there is no change to the default masquerade subnet.
    • Changing the cluster’s masquerade subnet is unsupported after a user-defined network has been configured for a project. Attempting to modify the masquerade subnet after a UDN has been set up can disrupt the network connectivity and cause configuration issues.
  • Ensure tenants are using the UserDefinedNetwork resource and not the NetworkAttachmentDefinition (NAD) resource. This can create security risks between tenants.
  • When creating network segmentation, you should only use the NAD resource if user-defined network segmentation cannot be completed using the UDN resource.
  • The cluster subnet and services CIDR for a UDN cannot overlap with the default cluster subnet CIDR. OVN-Kubernetes network plugin uses 100.64.0.0/16 as the default network’s join subnet, you must not use that value to configure a UDN joinSubnets field. If the default address values are used anywhere in the network for the cluster, you must override it by setting the joinSubnets field. For more information, see "Additional configuration details for a UserDefinedNetworks CR".
  • The cluster subnet and services CIDR for a UDN cannot overlap with the default cluster subnet CIDR. OVN-Kubernetes network plugin uses 100.64.0.0/16 as the default join subnet for the network, you must not use that value to configure a UDN joinSubnets field. If the default address values are used anywhere in the network for the cluster you must override the default values by setting the joinSubnets field. For more information, see "Additional configuration details for a UserDefinedNetworks CR".

The following procedure creates a user-defined network that is namespace scoped. Based upon your use case, create your request using either the my-layer-two-udn.yaml example for a Layer2 topology type or the my-layer-three-udn.yaml example for a Layer3 topology type.

Perquisites

  • You have logged in with cluster-admin privileges, or you have view and edit role-based access control (RBAC).

Procedure

  1. Optional: For a UserDefinedNetwork CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network label by entering the following command: 

    $ cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: <udn_namespace_name>
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
EOF
    Copy to Clipboard Toggle word wrap

  2. Create a request for either a Layer2 or Layer3 topology type user-defined network:

    1. Create a YAML file, such as my-layer-two-udn.yaml, to define your request for a Layer2 topology as in the following example: 

      apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: udn-1
      1 
      
  namespace: <some_custom_namespace>
spec:
  topology: Layer2
      2 
      
  layer2:
      3 
      
    role: Primary
      4 
      
    subnets:
      - "10.0.0.0/24"
      - "2001:db8::/60"
      5
      Copy to Clipboard Toggle word wrap
      1
      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).
      2
      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes.
      3
      This field specifies the topology configuration. It can be layer2 or layer3.
      4
      Specifies a Primary or Secondary role.
      5
      For Layer2 topology types the following specifies config details for the subnet field:
      • The subnets field is optional.
      • The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
      • The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
      • Layer2 subnets can be omitted. If omitted, users must configure IP addresses for the pods. As a consequence, port security only prevents MAC spoofing.
      • The Layer2 subnets field is mandatory when the ipamLifecycle field is specified.

    2. Create a YAML file, such as my-layer-three-udn.yaml, to define your request for a Layer3 topology as in the following example: 

      apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: udn-2-primary
      1 
      
  namespace: <some_custom_namespace>
spec:
  topology: Layer3
      2 
      
  layer3:
      3 
      
    role: Primary
      4 
      
    subnets:
      5 
      
      - cidr: 10.150.0.0/16
        hostSubnet: 24
      - cidr: 2001:db8::/60
        hostSubnet: 64
# ...
      Copy to Clipboard Toggle word wrap
      1
      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).
      2
      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets.
      3
      This field specifies the topology configuration. Valid values are layer2 or layer3.
      4
      Specifies a Primary or Secondary role.
      5
      For Layer3 topology types the following specifies config details for the subnet field:
      • The subnets field is mandatory.

      • The type for the subnets field is cidr and hostSubnet:

        • cidr is equivalent to the clusterNetwork configuration settings of a cluster. The IP addresses in the CIDR are distributed to pods in the user defined network. This parameter accepts a string value.
        • hostSubnet defines the per-node subnet prefix.
        • For IPv6, only a /64 length is supported for hostSubnet.

  3. Apply your request by running the following command: 

    $ oc apply -f <my_layer_two_udn.yaml>
    Copy to Clipboard Toggle word wrap

    Where <my_layer_two_udn.yaml> is the name of your Layer2 or Layer3 configuration file.

  4. Verify that your request is successful by running the following command: 

    $ oc get userdefinednetworks udn-1 -n <some_custom_namespace> -o yaml
    Copy to Clipboard Toggle word wrap

    Where some_custom_namespace is the namespace you created for your user-defined network.

    Example output

    apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  creationTimestamp: "2024-08-28T17:18:47Z"
  finalizers:
  - k8s.ovn.org/user-defined-network-protection
  generation: 1
  name: udn-1
  namespace: some-custom-namespace
  resourceVersion: "53313"
  uid: f483626d-6846-48a1-b88e-6bbeb8bcde8c
spec:
  layer2:
    role: Primary
    subnets:
    - 10.0.0.0/24
    - 2001:db8::/60
  topology: Layer2
status:
  conditions:
  - lastTransitionTime: "2024-08-28T17:18:47Z"
    message: NetworkAttachmentDefinition has been created
    reason: NetworkAttachmentDefinitionReady
    status: "True"
    type: NetworkReady
    Copy to Clipboard Toggle word wrap

The following table explains additional configurations for UDN that are optional. It is not recommended to set these fields without explicit need and understanding of OVN-Kubernetes network topology.

  1. Optional configurations for user-defined networks
Expand

CUDN field

UDN field

Type

Description

spec.network.<topology>.joinSubnets

spec.<topology>.joinSubnets

object

When omitted, the platform sets default values for the joinSubnets field of 100.65.0.0/16 for IPv4 and fd99::/64 for IPv6. If the default address values are used anywhere in the cluster’s network you must override it by setting the joinSubnets field. If you choose to set this field, ensure it does not conflict with other subnets in the cluster such as the cluster subnet, the default network cluster subnet, and the masquerade subnet.

The joinSubnets field configures the routing between different segments within a user-defined network. Dual-stack clusters can set 2 subnets, one for each IP family; otherwise, only 1 subnet is allowed. This field is only allowed for the Primary network.

spec.network.<topology>.ipam.lifecycle

spec.<topology>.ipam.lifecycle

object

The spec.ipam.lifecycle field configures the IP address management system (IPAM). You might use this field for virtual workloads to ensure persistent IP addresses. The only allowed value is Persistent, which ensures that your virtual workloads have persistent IP addresses across reboots and migration. These are assigned by the container network interface (CNI) and used by OVN-Kubernetes to program pod IP addresses. You must not change this for pod annotations.

Setting a value of Persistent is only supported when ipam.mode parameter is set to Enabled.

spec.network.<topology>.ipam.mode

spec.network.<topology>.ipam.mode

object

The mode parameter controls how much of the IP configuration is managed by OVN-Kubernetes. The following options are available:

Enabled:
When enabled, OVN-Kubernetes applies the IP configuration to the SDN infrastructure and assigns IP addresses from the selected subnet to the individual pods. This is the default setting. When set to Enabled, the subnets field must be defined. Enabled is the default configuration.

Disabled:
When disabled, OVN-Kubernetes only assigns MAC addresses and provides layer 2 communication, which allows users to configure IP addresses. Disabled is only available for layer 2 (secondary) networks. By disabling IPAM, features that rely on selecting pods by IP, for example, network policy, services, and so on, no longer function. Additionally, IP port security is also disabled for interfaces attached to this network. The subnets field must be empty when spec.ipam.mode is set to Disabled.

spec.network.<topology>.mtu

spec.<topology>.mtu

integer

The maximum transmission units (MTU). The default value is 1400. The boundary for IPv4 is 576, and for IPv6 it is 1280.

where:

<topology>
Is one of layer2 or layer3.

The following sections explain how to create and manage primary networks using the NetworkAttachmentDefinition (NAD) resource.

2.2.1. Approaches to managing a primary network
Copy link

You can manage the life cycle of a primary network created by NAD with one of the following two approaches:

  • By modifying the Cluster Network Operator (CNO) configuration. With this method, the CNO automatically creates and manages the NetworkAttachmentDefinition object. In addition to managing the object lifecycle, the CNO ensures that a DHCP is available for a primary network that uses a DHCP assigned IP address.
  • By applying a YAML manifest. With this method, you can manage the primary network directly by creating an NetworkAttachmentDefinition object. This approach allows for the invocation of multiple CNI plugins in order to attach primary network interfaces in a pod.

Each approach is mutually exclusive and you can only use one approach for managing a primary network at a time. For either approach, the primary network is managed by a Container Network Interface (CNI) plugin that you configure.

Note

When deploying OpenShift Container Platform nodes with multiple network interfaces on Red Hat OpenStack Platform (RHOSP) with OVN SDN, DNS configuration of the secondary interface might take precedence over the DNS configuration of the primary interface. In this case, remove the DNS nameservers for the subnet ID that is attached to the secondary interface by running the following command: 

$ openstack subnet set --dns-nameserver 0.0.0.0 <subnet_id>
Copy to Clipboard Toggle word wrap

The Cluster Network Operator (CNO) manages additional network definitions. When you specify a primary network to create, the CNO creates the NetworkAttachmentDefinition custom resource definition (CRD) automatically.

Important

Do not edit the NetworkAttachmentDefinition CRDs that the Cluster Network Operator manages. Doing so might disrupt network traffic on your primary network.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

  1. Optional: Create the namespace for the primary networks: 

    $ oc create namespace <namespace_name>
    Copy to Clipboard Toggle word wrap

  2. To edit the CNO configuration, enter the following command: 

    $ oc edit networks.operator.openshift.io cluster
    Copy to Clipboard Toggle word wrap

  3. Modify the CR that you are creating by adding the configuration for the primary network that you are creating, as in the following example CR. 

    apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  # ...
  additionalNetworks:
  - name: tertiary-net
    namespace: namespace2
    type: Raw
    rawCNIConfig: |-
      {
        "cniVersion": "0.3.1",
        "name": "tertiary-net",
        "type": "ipvlan",
        "master": "eth1",
        "mode": "l2",
        "ipam": {
          "type": "static",
          "addresses": [
            {
              "address": "192.168.1.23/24"
            }
          ]
        }
      }
    Copy to Clipboard Toggle word wrap
  4. Save your changes and quit the text editor to commit your changes.

Verification

  • Confirm that the CNO created the NetworkAttachmentDefinition CRD by running the following command. A delay might exist before the CNO creates the CRD. The expected output shows the name of the NAD CRD and the creation age in minutes. 

    $ oc get network-attachment-definitions -n <namespace>
    Copy to Clipboard Toggle word wrap

    where:

    <namespace>
    Specifies the namespace for the network attachment that you added to the CNO configuration.

A primary network is configured by using the NetworkAttachmentDefinition API in the k8s.cni.cncf.io API group.

The configuration for the API is described in the following table:

Expand
Table 2.1. NetworkAttachmentDefinition API fields
FieldTypeDescription

metadata.name

string

The name for the primary network.

metadata.namespace

string

The namespace that the object is associated with.

spec.config

string

The CNI plugin configuration in JSON format.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You have logged in as a user with cluster-admin privileges.
  • You are working in the namespace where the NAD is to be deployed.

Procedure

  1. Create a YAML file with your primary network configuration, such as in the following example: 

    apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: next-net
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "work-network",
      "namespace": "namespace2",
    1 
    
      "type": "host-device",
      "device": "eth1",
      "ipam": {
        "type": "dhcp"
      }
    }
    Copy to Clipboard Toggle word wrap
    1
    Optional: You can specify a namespace to which the NAD is applied. If you are working in the namespace where the NAD is to be deployed, this spec is not necessary.

  2. To create the primary network, enter the following command: 

    $ oc apply -f <file>.yaml
    Copy to Clipboard Toggle word wrap

    where:

    <file>
    Specifies the name of the file contained the YAML manifest.

Chapter 3. Secondary networks
Copy link

3.1. Creating secondary networks on OVN-Kubernetes
Copy link

As a cluster administrator, you can configure a secondary network for your cluster using the NetworkAttachmentDefinition (NAD) resource.

Note

Support for user-defined networks as a secondary network will be added in a future version of OpenShift Container Platform.

The Red Hat OpenShift Networking OVN-Kubernetes network plugin allows the configuration of secondary network interfaces for pods. To configure secondary network interfaces, you must define the configurations in the NetworkAttachmentDefinition custom resource definition (CRD).

Note

Pod and multi-network policy creation might remain in a pending state until the OVN-Kubernetes control plane agent in the nodes processes the associated network-attachment-definition CRD.

You can configure an OVN-Kubernetes secondary network in layer 2, layer 3, or localnet topologies. For more information about features supported on these topologies, see "UserDefinedNetwork and NetworkAttachmentDefinition support matrix".

The following sections provide example configurations for each of the topologies that OVN-Kubernetes currently allows for secondary networks.

Note

Networks names must be unique. For example, creating multiple NetworkAttachmentDefinition CRDs with different configurations that reference the same network is unsupported.

You can use an OVN-Kubernetes secondary network with the following supported platforms:

  • Bare metal
  • IBM Power®
  • IBM Z®
  • IBM® LinuxONE
  • VMware vSphere
  • Red Hat OpenStack Platform (RHOSP)

The following table describes the configuration parameters for the OVN-Kubernetes CNI network plugin:

Expand
Table 3.1. OVN-Kubernetes network plugin JSON configuration table
FieldTypeDescription

cniVersion

string

The CNI specification version. The required value is 0.3.1.

name

string

The name of the network. These networks are not namespaced. For example, a network named l2-network can be referenced by NetworkAttachmentDefinition custom resources (CRs) that exist in different namespaces. This configuration allows pods that use the NetworkAttachmentDefinition CR in different namespaces to communicate over the same secondary network. However, the NetworkAttachmentDefinition CRs must share the same network-specific parameters, such as topology, subnets, mtu, excludeSubnets, and vlanID. The vlanID parameter applies only when the topology field is set to localnet.

type

string

The name of the CNI plugin to configure. This value must be set to ovn-k8s-cni-overlay.

topology

string

The topological configuration for the network. Must be one of layer2 or localnet.

subnets

string

The subnet to use for the network across the cluster.

For "topology":"layer2" deployments, IPv6 (2001:DBB::/64) and dual-stack (192.168.100.0/24,2001:DBB::/64) subnets are supported.

When omitted, the logical switch implementing the network only provides layer 2 communication, and users must configure IP addresses for the pods. Port security only prevents MAC spoofing.

mtu

string

The maximum transmission unit (MTU). If you do not set a value, the Cluster Network Operator (CNO) sets a default MTU value by calculating the difference among the underlay MTU of the primary network interface, the overlay MTU of the pod network, such as the Geneve (Generic Network Virtualization Encapsulation), and byte capacity of any enabled features, such as IPsec.

netAttachDefName

string

The metadata namespace and name of the network attachment definition CRD where this configuration is included. For example, if this configuration is defined in a NetworkAttachmentDefinition CRD in namespace ns1 named l2-network, this should be set to ns1/l2-network.

excludeSubnets

string

A comma-separated list of CIDRs and IP addresses. IP addresses are removed from the assignable IP address pool and are never passed to the pods.

vlanID

integer

If topology is set to localnet, the specified VLAN tag is assigned to traffic from this secondary network. The default is to not assign a VLAN tag.

3.1.1.3. Compatibility with multi-network policy
Copy link

The multi-network policy API, which is provided by the MultiNetworkPolicy custom resource definition (CRD) in the k8s.cni.cncf.io API group, is compatible with an OVN-Kubernetes secondary network. When defining a network policy, the network policy rules that can be used depend on whether the OVN-Kubernetes secondary network defines the subnets field. Refer to the following table for details:

Expand
Table 3.2. Supported multi-network policy selectors based on subnets CNI configuration
subnets field specifiedAllowed multi-network policy selectors

Yes

  • podSelector and namespaceSelector
  • ipBlock

No

  • ipBlock

You can use the k8s.v1.cni.cncf.io/policy-for annotation on a MultiNetworkPolicy object to point to a NetworkAttachmentDefinition (NAD) custom resource (CR). The NAD CR defines the network to which the policy applies. The following example multi-network policy is valid only if the subnets field is defined in the secondary network CNI configuration for the secondary network named blue2:

Example multi-network policy that uses a pod selector

apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-same-namespace
  annotations:
    k8s.v1.cni.cncf.io/policy-for: blue2
1 

spec:
  podSelector:
  ingress:
  - from:
    - podSelector: {}
Copy to Clipboard Toggle word wrap

The following example uses the ipBlock network policy selector, which is always valid for an OVN-Kubernetes secondary network:

Example multi-network policy that uses an IP block selector

apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name:  ingress-ipblock
  annotations:
    k8s.v1.cni.cncf.io/policy-for: default/flatl2net
spec:
  podSelector:
    matchLabels:
      name: access-control
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 10.200.0.0/30
Copy to Clipboard Toggle word wrap

The switched localnet topology interconnects the workloads created as Network Attachment Definitions (NADs) through a cluster-wide logical switch to a physical network.

You must map a secondary network to the OVN bridge to use it as an OVN-Kubernetes secondary network. Bridge mappings allow network traffic to reach the physical network. A bridge mapping associates a physical network name, also known as an interface label, to a bridge created with Open vSwitch (OVS).

You can create an NodeNetworkConfigurationPolicy (NNCP) object, part of the nmstate.io/v1 API group, to declaratively create the mapping. This API is provided by the NMState Operator. By using this API you can apply the bridge mapping to nodes that match your specified nodeSelector expression, such as node-role.kubernetes.io/worker: ''. With this declarative approach, the NMState Operator applies secondary network configuration to all nodes specified by the node selector automatically and transparently.

When attaching a secondary network, you can either use the existing br-ex bridge or create a new bridge. Which approach to use depends on your specific network infrastructure. Consider the following approaches:

  • If your nodes include only a single network interface, you must use the existing bridge. This network interface is owned and managed by OVN-Kubernetes and you must not remove it from the br-ex bridge or alter the interface configuration. If you remove or alter the network interface, your cluster network will stop working correctly.
  • If your nodes include several network interfaces, you can attach a different network interface to a new bridge, and use that for your secondary network. This approach provides for traffic isolation from your primary cluster network.

The localnet1 network is mapped to the br-ex bridge in the following example:

Example mapping for sharing a bridge

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: mapping
1 

spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
2 

  desiredState:
    ovn:
      bridge-mappings:
      - localnet: localnet1
3 

        bridge: br-ex
4 

        state: present
5
Copy to Clipboard Toggle word wrap

1 1
The name for the configuration object.
2
A node selector that specifies the nodes to apply the node network configuration policy to.
3
The name for the secondary network from which traffic is forwarded to the OVS bridge. This secondary network must match the name of the spec.config.name field of the NetworkAttachmentDefinition CRD that defines the OVN-Kubernetes secondary network.
4
The name of the OVS bridge on the node. This value is required only if you specify state: present.
5
The state for the mapping. Must be either present to add the bridge or absent to remove the bridge. The default value is present.

The following JSON example configures a localnet secondary network that is named localnet1. Note that the value for the mtu parameter must match the MTU value that was set for the secondary network interface that is mapped to the br-ex bridge interface. 

{
  "cniVersion": "0.3.1",
  "name": "localnet1",
  "type": "ovn-k8s-cni-overlay",
  "topology":"localnet",
  "physicalNetworkName": "localnet1",
  "subnets": "202.10.130.112/28",
  "vlanID": 33,
  "mtu": 1500,
  "netAttachDefName": "ns1/localnet-network",
  "excludeSubnets": "10.100.200.0/29"
}
Copy to Clipboard Toggle word wrap

In the following example, the localnet2 network interface is attached to the ovs-br1 bridge. Through this attachment, the network interface is available to the OVN-Kubernetes network plugin as a secondary network.

Example mapping for nodes with multiple interfaces

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ovs-br1-multiple-networks
1 

spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
2 

  desiredState:
    interfaces:
    - name: ovs-br1
3 

      description: |-
        A dedicated OVS bridge with eth1 as a port
        allowing all VLANs and untagged traffic
      type: ovs-bridge
      state: up
      bridge:
        allow-extra-patch-ports: true
        options:
          stp: false
          mcast-snooping-enable: true
4 

        port:
        - name: eth1
5 

    ovn:
      bridge-mappings:
      - localnet: localnet2
6 

        bridge: ovs-br1
7 

        state: present
8
Copy to Clipboard Toggle word wrap

1
Specifies the name of the configuration object.
2
Specifies a node selector that identifies the nodes to which the node network configuration policy applies.
3
Specifies a new OVS bridge that operates separately from the default bridge used by OVN-Kubernetes for cluster traffic.
4
Specifies whether to enable multicast snooping. When enabled, multicast snooping prevents network devices from flooding multicast traffic to all network members. By default, an OVS bridge does not enable multicast snooping. The default value is false.
5
Specifies the network device on the host system to associate with the new OVS bridge.
6
Specifies the name of the secondary network that forwards traffic to the OVS bridge. This name must match the value of the spec.config.name field in the NetworkAttachmentDefinition CRD that defines the OVN-Kubernetes secondary network.
7
Specifies the name of the OVS bridge on the node. The value is required only when state: present is set.
8
Specifies the state of the mapping. Valid values are present to add the bridge or absent to remove the bridge. The default value is present.

The following JSON example configures a localnet secondary network that is named localnet2. Note that the value for the mtu parameter must match the MTU value that was set for the eth1 secondary network interface. 

{
  "cniVersion": "0.3.1",
  "name": "localnet2",
  "type": "ovn-k8s-cni-overlay",
  "topology":"localnet",
  "physicalNetworkName": "localnet2",
  "subnets": "202.10.130.112/28",
  "vlanID": 33,
  "mtu": 1500,
  "netAttachDefName": "ns1/localnet-network",
  "excludeSubnets": "10.100.200.0/29"
}
Copy to Clipboard Toggle word wrap

The switched (layer 2) topology networks interconnect the workloads through a cluster-wide logical switch. This configuration can be used for IPv6 and dual-stack deployments.

Note

Layer 2 switched topology networks only allow for the transfer of data packets between pods within a cluster.

The following JSON example configures a switched secondary network: 

{
  "cniVersion": "0.3.1",
  "name": "l2-network",
  "type": "ovn-k8s-cni-overlay",
  "topology":"layer2",
  "subnets": "10.100.200.0/24",
  "mtu": 1300,
  "netAttachDefName": "ns1/l2-network",
  "excludeSubnets": "10.100.200.0/29"
}
Copy to Clipboard Toggle word wrap
3.1.1.5. Configuring pods for secondary networks
Copy link

You must specify the secondary network attachments through the k8s.v1.cni.cncf.io/networks annotation.

The following example provisions a pod with two secondary attachments, one for each of the attachment configurations presented in this guide. 

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: l2-network
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container
Copy to Clipboard Toggle word wrap
3.1.1.6. Configuring pods with a static IP address
Copy link

The following example provisions a pod with a static IP address.

Note
  • You can specify the IP address for the secondary network attachment of a pod only when the secondary network attachment, a namespaced-scoped object, uses a layer 2 or localnet topology.
  • Specifying a static IP address for the pod is only possible when the attachment configuration does not feature subnets. 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "l2-network",
1 

        "mac": "02:03:04:05:06:07",
2 

        "interface": "myiface1",
3 

        "ips": [
          "192.0.2.20/24"
          ]
4 

      }
    ]'
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container
Copy to Clipboard Toggle word wrap
1
The name of the network. This value must be unique across all NetworkAttachmentDefinition CRDs.
2
The MAC address to be assigned for the interface.
3
The name of the network interface to be created for the pod.
4
The IP addresses to be assigned to the network interface.

The specific configuration fields for secondary networks are described in the following sections.

The following object describes the configuration parameters for the Bridge CNI plugin:

Expand
Table 3.3. Bridge CNI plugin JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: bridge.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

bridge

string

Optional: Specify the name of the virtual bridge to use. If the bridge interface does not exist on the host, it is created. The default value is cni0.

ipMasq

boolean

Optional: Set to true to enable IP masquerading for traffic that leaves the virtual network. The source IP address for all traffic is rewritten to the bridge’s IP address. If the bridge does not have an IP address, this setting has no effect. The default value is false.

isGateway

boolean

Optional: Set to true to assign an IP address to the bridge. The default value is false.

isDefaultGateway

boolean

Optional: Set to true to configure the bridge as the default gateway for the virtual network. The default value is false. If isDefaultGateway is set to true, then isGateway is also set to true automatically.

forceAddress

boolean

Optional: Set to true to allow assignment of a previously assigned IP address to the virtual bridge. When set to false, if an IPv4 address or an IPv6 address from overlapping subsets is assigned to the virtual bridge, an error occurs. The default value is false.

hairpinMode

boolean

Optional: Set to true to allow the virtual bridge to send an Ethernet frame back through the virtual port it was received on. This mode is also known as reflective relay. The default value is false.

promiscMode

boolean

Optional: Set to true to enable promiscuous mode on the bridge. The default value is false.

vlan

string

Optional: Specify a virtual LAN (VLAN) tag as an integer value. By default, no VLAN tag is assigned.

preserveDefaultVlan

string

Optional: Indicates whether the default vlan must be preserved on the veth end connected to the bridge. Defaults to true.

vlanTrunk

list

Optional: Assign a VLAN trunk tag. The default value is none.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

enabledad

boolean

Optional: Enables duplicate address detection for the container side veth. The default value is false.

macspoofchk

boolean

Optional: Enables mac spoof check, limiting the traffic originating from the container to the mac address of the interface. The default value is false.

Note

The VLAN parameter configures the VLAN tag on the host end of the veth and also enables the vlan_filtering feature on the bridge interface.

Note

To configure an uplink for an L2 network, you must allow the VLAN on the uplink interface by using the following command: 

$  bridge vlan add vid VLAN_ID dev DEV
Copy to Clipboard Toggle word wrap
3.2.1.1. Bridge CNI plugin configuration example
Copy link

The following example configures a secondary network named bridge-net: 

{
  "cniVersion": "0.3.1",
  "name": "bridge-net",
  "type": "bridge",
  "isGateway": true,
  "vlan": 2,
  "ipam": {
    "type": "dhcp"
    }
}
Copy to Clipboard Toggle word wrap

The Bond Container Network Interface (Bond CNI) enables the aggregation of multiple network interfaces into a single logical "bonded" interface within a container, enhancing network redundancy and fault tolerance. Only SR-IOV Virtual Functions (VFs) are supported for bonding with this plugin.

The following table describes the configuration parameters for the Bond CNI plugin:

Expand
Table 3.4. Bond CNI plugin JSON configuration object
FieldTypeDescription

name

string

Specifies the name given to this CNI network attachment definition. This name is used to identify and reference the interface within the container.

cniVersion

string

The CNI specification version.

type

string

Specifies the name of the CNI plugin to configure: bond.

miimon

string

Specifies the address resolution protocol (ARP) link monitoring frequency in milliseconds. This parameter defines how often the bond interface sends ARP requests to check the availability of its aggregated interfaces.

mtu

integer

Optional: Specifies the maximum transmission unit (MTU) of the bond. The default is 1500.

failOverMac

integer

Optional: Specifies the failOverMac setting for the bond. Default is 0.

mode

string

Specifies the bonding policy.

linksInContainer

boolean

Optional: Specifies whether the network interfaces intended for bonding are expected to be created and available directly within the container’s network namespace when the bond starts. If false which is the default, the CNI plugin looks for these interfaces on the host system first before attempting to form the bond.

links

object

Specifies the interfaces to be bonded.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

3.2.2.1. Bond CNI plugin configuration example
Copy link

The following example configures a secondary network named bond-net1: 

{
 "type": "bond",
 "cniVersion": "0.3.1",
 "name": "bond-net1",
 "mode": "active-backup",
 "failOverMac": 1,
 "linksInContainer": true,
 "miimon": "100",
 "mtu": 1500,
 "links": [
       {"name": "net1"},
       {"name": "net2"}
   ],
  "ipam": {
        "type": "host-local",
        "subnet": "10.56.217.0/24",
        "routes": [{
        "dst": "0.0.0.0/0"
        }],
        "gateway": "10.56.217.1"
    }
}
Copy to Clipboard Toggle word wrap
Note

Specify your network device by setting only one of the following parameters: device,hwaddr, kernelpath, or pciBusID.

The following object describes the configuration parameters for the host-device CNI plugin:

Expand
Table 3.5. Host device CNI plugin JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: host-device.

device

string

Optional: The name of the device, such as eth0.

hwaddr

string

Optional: The device hardware MAC address.

kernelpath

string

Optional: The Linux kernel device path, such as /sys/devices/pci0000:00/0000:00:1f.6.

pciBusID

string

Optional: The PCI address of the network device, such as 0000:00:1f.6.

3.2.3.1. host-device configuration example
Copy link

The following example configures a secondary network named hostdev-net: 

{
  "cniVersion": "0.3.1",
  "name": "hostdev-net",
  "type": "host-device",
  "device": "eth1"
}
Copy to Clipboard Toggle word wrap

3.2.4. Configuration for a VLAN secondary network
Copy link

The following object describes the configuration parameters for the VLAN, vlan, CNI plugin:

Expand
Table 3.6. VLAN CNI plugin JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: vlan.

master

string

The Ethernet interface to associate with the network attachment. If a master is not specified, the interface for the default network route is used.

vlanId

integer

Set the ID of the vlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

dns

integer

Optional: DNS information to return. For example, a priority-ordered list of DNS nameservers.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

Important

A NetworkAttachmentDefinition custom resource definition (CRD) with a vlan configuration can be used only on a single pod in a node because the CNI plugin cannot create multiple vlan subinterfaces with the same vlanId on the same master interface.

3.2.4.1. VLAN configuration example
Copy link

The following example demonstrates a vlan configuration with a secondary network that is named vlan-net: 

{
  "name": "vlan-net",
  "cniVersion": "0.3.1",
  "type": "vlan",
  "master": "eth0",
  "mtu": 1500,
  "vlanId": 5,
  "linkInContainer": false,
  "ipam": {
      "type": "host-local",
      "subnet": "10.1.1.0/24"
  },
  "dns": {
      "nameservers": [ "10.1.1.1", "8.8.8.8" ]
  }
}
Copy to Clipboard Toggle word wrap

The following object describes the configuration parameters for the IPVLAN, ipvlan, CNI plugin:

Expand
Table 3.7. IPVLAN CNI plugin JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: ipvlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. This is required unless the plugin is chained.

mode

string

Optional: The operating mode for the virtual network. The value must be l2, l3, or l3s. The default value is l2.

master

string

Optional: The Ethernet interface to associate with the network attachment. If a master is not specified, the interface for the default network route is used.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

Important
  • The ipvlan object does not allow virtual interfaces to communicate with the master interface. Therefore the container is not able to reach the host by using the ipvlan interface. Be sure that the container joins a network that provides connectivity to the host, such as a network supporting the Precision Time Protocol (PTP).
  • A single master interface cannot simultaneously be configured to use both macvlan and ipvlan.
  • For IP allocation schemes that cannot be interface agnostic, the ipvlan plugin can be chained with an earlier plugin that handles this logic. If the master is omitted, then the previous result must contain a single interface name for the ipvlan plugin to enslave. If ipam is omitted, then the previous result is used to configure the ipvlan interface.
3.2.5.1. IPVLAN CNI plugin configuration example
Copy link

The following example configures a secondary network named ipvlan-net: 

{
  "cniVersion": "0.3.1",
  "name": "ipvlan-net",
  "type": "ipvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "l3",
  "ipam": {
    "type": "static",
    "addresses": [
       {
         "address": "192.168.10.10/24"
       }
    ]
  }
}
Copy to Clipboard Toggle word wrap

The following object describes the configuration parameters for the MAC Virtual LAN (MACVLAN) Container Network Interface (CNI) plugin:

Expand
Table 3.8. MACVLAN CNI plugin JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: macvlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

mode

string

Optional: Configures traffic visibility on the virtual network. Must be either bridge, passthru, private, or vepa. If a value is not provided, the default value is bridge.

master

string

Optional: The host network interface to associate with the newly created macvlan interface. If a value is not specified, then the default route interface is used.

mtu

integer

Optional: The maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

Note

If you specify the master key for the plugin configuration, use a different physical network interface than the one that is associated with your primary network plugin to avoid possible conflicts.

3.2.6.1. MACVLAN CNI plugin configuration example
Copy link

The following example configures a secondary network named macvlan-net: 

{
  "cniVersion": "0.3.1",
  "name": "macvlan-net",
  "type": "macvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "bridge",
  "ipam": {
    "type": "dhcp"
    }
}
Copy to Clipboard Toggle word wrap

3.2.7. Configuration for a TAP secondary network
Copy link

The following object describes the configuration parameters for the TAP CNI plugin:

Expand
Table 3.9. TAP CNI plugin JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: tap.

mac

string

Optional: Request the specified MAC address for the interface.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

selinuxcontext

string

Optional: The SELinux context to associate with the tap device.

Note

The value system_u:system_r:container_t:s0 is required for OpenShift Container Platform.

multiQueue

boolean

Optional: Set to true to enable multi-queue.

owner

integer

Optional: The user owning the tap device.

group

integer

Optional: The group owning the tap device.

bridge

string

Optional: Set the tap device as a port of an already existing bridge.

3.2.7.1. Tap configuration example
Copy link

The following example configures a secondary network named mynet: 

{
 "name": "mynet",
 "cniVersion": "0.3.1",
 "type": "tap",
 "mac": "00:11:22:33:44:55",
 "mtu": 1500,
 "selinuxcontext": "system_u:system_r:container_t:s0",
 "multiQueue": true,
 "owner": 0,
 "group": 0
 "bridge": "br1"
}
Copy to Clipboard Toggle word wrap

To create the tap device with the container_t SELinux context, enable the container_use_devices boolean on the host by using the Machine Config Operator (MCO).

Prerequisites

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a new YAML file named, such as setsebool-container-use-devices.yaml, with the following details: 

    apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-setsebool
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
      - enabled: true
        name: setsebool.service
        contents: |
          [Unit]
          Description=Set SELinux boolean for the TAP CNI plugin
          Before=kubelet.service

          [Service]
          Type=oneshot
          ExecStart=/usr/sbin/setsebool container_use_devices=on
          RemainAfterExit=true

          [Install]
          WantedBy=multi-user.target graphical.target
    Copy to Clipboard Toggle word wrap

  2. Create the new MachineConfig object by running the following command: 

    $ oc apply -f setsebool-container-use-devices.yaml
    Copy to Clipboard Toggle word wrap
    Note

    Applying any changes to the MachineConfig object causes all affected nodes to gracefully reboot after the change is applied. This update can take some time to be applied.

  3. Verify the change is applied by running the following command: 

    $ oc get machineconfigpools
    Copy to Clipboard Toggle word wrap

    Expected output

    NAME        CONFIG                                                UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master      rendered-master-e5e0c8e8be9194e7c5a882e047379cfa      True      False      False      3              3                   3                     0                      7d2h
worker      rendered-worker-d6c9ca107fba6cd76cdcbfcedcafa0f2      True      False      False      3              3                   3                     0                      7d
    Copy to Clipboard Toggle word wrap

    Note

    All nodes should be in the updated and ready state.

The following object describes the configuration parameters for the route-override CNI plugin:

Expand
Table 3.10. Route override CNI plugin JSON configuration object
FieldTypeDescription

type

string

The name of the CNI plugin to configure: route-override.

flushroutes

boolean

Optional: Set to true to flush any existing routes.

flushgateway

boolean

Optional: Set to true to flush the default route namely the gateway route.

delroutes

object

Optional: Specify the list of routes to delete from the container namespace.

addroutes

object

Optional: Specify the list of routes to add to the container namespace. Each route is a dictionary with dst and optional gw fields. If gw is omitted, the plugin uses the default gateway value.

skipcheck

boolean

Optional: Set this to true to skip the check command. By default, CNI plugins verify the network setup during the container lifecycle. When modifying routes dynamically with route-override, skipping this check ensures the final configuration reflects the updated routes.

The route-override CNI is a type of CNI that it is designed to be used when chained with a parent CNI. It does not operate independently, but relies on the parent CNI to first create the network interface and assign IP addresses before it can modify the routing rules.

The following example configures a secondary network named mymacvlan. The parent CNI creates a network interface attached to eth1 and assigns an IP address in the 192.168.1.0/24 range using host-local IPAM. The route-override CNI is then chained to the parent CNI and modifies the routing rules by flushing existing routes, deleting the route to 192.168.0.0/24, and adding a new route for 192.168.0.0/24 with a custom gateway. 

{
    "cniVersion": "0.3.0",
    "name": "mymacvlan",
    "plugins": [
        {
            "type": "macvlan",
1 

            "master": "eth1",
            "mode": "bridge",
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.1.0/24"
            }
        },
        {
            "type": "route-override",
2 

            "flushroutes": true,
            "delroutes": [
                {
                    "dst": "192.168.0.0/24"
                }
            ],
            "addroutes": [
                {
                    "dst": "192.168.0.0/24",
                    "gw": "10.1.254.254"
                }
            ]
        }
    ]
}
Copy to Clipboard Toggle word wrap
1
The parent CNI creates a network interface attached to eth1.
2
The chained route-override CNI modifies the routing rules.

3.3. Attaching a pod to a secondary network
Copy link

As a cluster user you can attach a pod to a secondary network.

3.3.1. Adding a pod to a secondary network
Copy link

You can add a pod to a secondary network. The pod continues to send normal cluster-related network traffic over the default network.

When a pod is created, a secondary networks is attached to the pod. However, if a pod already exists, you cannot attach a secondary network to it.

The pod must be in the same namespace as the secondary network.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in to the cluster.

Procedure

  1. Add an annotation to the Pod object. Only one of the following annotation formats can be used:

    1. To attach a secondary network without any customization, add an annotation with the following format. Replace <network> with the name of the secondary network to associate with the pod: 

      metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: <network>[,<network>,...]
      1
      Copy to Clipboard Toggle word wrap
      1
      To specify more than one secondary network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same secondary network multiple times, that pod will have multiple network interfaces attached to that network.

    2. To attach a secondary network with customizations, add an annotation with the following format: 

      metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: |-
      [
        {
          "name": "<network>",
      1 
      
          "namespace": "<namespace>",
      2 
      
          "default-route": ["<default-route>"]
      3 
      
        }
      ]
      Copy to Clipboard Toggle word wrap
      1
      Specify the name of the secondary network defined by a NetworkAttachmentDefinition object.
      2
      Specify the namespace where the NetworkAttachmentDefinition object is defined.
      3
      Optional: Specify an override for the default route, such as 192.168.17.1.

  2. To create the pod, enter the following command. Replace <name> with the name of the pod. 

    $ oc create -f <name>.yaml
    Copy to Clipboard Toggle word wrap

  3. Optional: To Confirm that the annotation exists in the Pod CR, enter the following command, replacing <name> with the name of the pod. 

    $ oc get pod <name> -o yaml
    Copy to Clipboard Toggle word wrap

    In the following example, the example-pod pod is attached to the net1 secondary network: 

    $ oc get pod example-pod -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-bridge
    k8s.v1.cni.cncf.io/network-status: |-
    1 
    
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.128.2.14"
          ],
          "default": true,
          "dns": {}
      },{
          "name": "macvlan-bridge",
          "interface": "net1",
          "ips": [
              "20.2.2.100"
          ],
          "mac": "22:2f:60:a5:f8:00",
          "dns": {}
      }]
  name: example-pod
  namespace: default
spec:
  ...
status:
  ...
    Copy to Clipboard Toggle word wrap
    1
    The k8s.v1.cni.cncf.io/network-status parameter is a JSON array of objects. Each object describes the status of a secondary network attached to the pod. The annotation value is stored as a plain text value.

When attaching a pod to a secondary network, you may want to specify further properties about that network in a particular pod. This allows you to change some aspects of routing, as well as specify static IP addresses and MAC addresses. To accomplish this, you can use the JSON formatted annotations.

Prerequisites

  • The pod must be in the same namespace as the secondary network.
  • Install the OpenShift CLI (oc).
  • You must log in to the cluster.

Procedure

To add a pod to a secondary network while specifying addressing and/or routing options, complete the following steps:

  1. Edit the Pod resource definition. If you are editing an existing Pod resource, run the following command to edit its definition in the default editor. Replace <name> with the name of the Pod resource to edit. 

    $ oc edit pod <name>
    Copy to Clipboard Toggle word wrap

  2. In the Pod resource definition, add the k8s.v1.cni.cncf.io/networks parameter to the pod metadata mapping. The k8s.v1.cni.cncf.io/networks accepts a JSON string of a list of objects that reference the name of NetworkAttachmentDefinition custom resource (CR) names in addition to specifying additional properties. 

    metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]'
    1
    Copy to Clipboard Toggle word wrap
    1
    Replace <network> with a JSON object as shown in the following examples. The single quotes are required.

  3. In the following example the annotation specifies which network attachment will have the default route, using the default-route parameter. 

    apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
    {
      "name": "net1"
    },
    {
      "name": "net2",
    1 
    
      "default-route": ["192.0.2.1"]
    2 
    
    }]'
spec:
  containers:
  - name: example-pod
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: centos/tools
    Copy to Clipboard Toggle word wrap
    1
    The name key is the name of the secondary network to associate with the pod.
    2
    The default-route key specifies a value of a gateway for traffic to be routed over if no other routing entry is present in the routing table. If more than one default-route key is specified, this will cause the pod to fail to become active.

The default route will cause any traffic that is not specified in other routes to be routed to the gateway.

Important

Setting the default route to an interface other than the default network interface for OpenShift Container Platform may cause traffic that is anticipated for pod-to-pod traffic to be routed over another interface.

To verify the routing properties of a pod, the oc command may be used to execute the ip command within a pod. 

$ oc exec -it <pod_name> -- ip route
Copy to Clipboard Toggle word wrap
Note

You may also reference the pod’s k8s.v1.cni.cncf.io/network-status to see which secondary network has been assigned the default route, by the presence of the default-route key in the JSON-formatted list of objects.

To set a static IP address or MAC address for a pod you can use the JSON formatted annotations. This requires you create networks that specifically allow for this functionality. This can be specified in a rawCNIConfig for the CNO.

  1. Edit the CNO CR by running the following command: 

    $ oc edit networks.operator.openshift.io cluster
    Copy to Clipboard Toggle word wrap

The following YAML describes the configuration parameters for the CNO:

Cluster Network Operator YAML configuration

name: <name>
1 

namespace: <namespace>
2 

rawCNIConfig: '{
3 

  ...
}'
type: Raw
Copy to Clipboard Toggle word wrap

1
Specify a name for the secondary network attachment that you are creating. The name must be unique within the specified namespace.
2
Specify the namespace to create the network attachment in. If you do not specify a value, then the default namespace is used.
3
Specify the CNI plugin configuration in JSON format, which is based on the following template.

The following object describes the configuration parameters for utilizing static MAC address and IP address using the macvlan CNI plugin:

macvlan CNI plugin JSON configuration object using static IP and MAC address

{
  "cniVersion": "0.3.1",
  "name": "<name>",
1 

  "plugins": [{
2 

      "type": "macvlan",
      "capabilities": { "ips": true },
3 

      "master": "eth0",
4 

      "mode": "bridge",
      "ipam": {
        "type": "static"
      }
    }, {
      "capabilities": { "mac": true },
5 

      "type": "tuning"
    }]
}
Copy to Clipboard Toggle word wrap

1
Specifies the name for the secondary network attachment to create. The name must be unique within the specified namespace.
2
Specifies an array of CNI plugin configurations. The first object specifies a macvlan plugin configuration and the second object specifies a tuning plugin configuration.
3
Specifies that a request is made to enable the static IP address functionality of the CNI plugin runtime configuration capabilities.
4
Specifies the interface that the macvlan plugin uses.
5
Specifies that a request is made to enable the static MAC address functionality of a CNI plugin.

The above network attachment can be referenced in a JSON formatted annotation, along with keys to specify which static IP and MAC address will be assigned to a given pod.

Edit the pod with: 

$ oc edit pod <name>
Copy to Clipboard Toggle word wrap

macvlan CNI plugin JSON configuration object using static IP and MAC address

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "<name>",
1 

        "ips": [ "192.0.2.205/24" ],
2 

        "mac": "CA:FE:C0:FF:EE:00"
3 

      }
    ]'
Copy to Clipboard Toggle word wrap

1
Use the <name> as provided when creating the rawCNIConfig above.
2
Provide an IP address including the subnet mask.
3
Provide the MAC address.
Note

Static IP addresses and MAC addresses do not have to be used at the same time, you may use them individually, or together.

To verify the IP address and MAC properties of a pod with secondary networks, use the oc command to execute the ip command within a pod. 

$ oc exec -it <pod_name> -- ip a
Copy to Clipboard Toggle word wrap

3.4. Configuring multi-network policy
Copy link

Administrators can use the MultiNetworkPolicy API to create multiple network policies that manage traffic for pods attached to secondary networks. For example, you can create policies that allow or deny traffic based on specific ports, IPs/ranges, or labels.

Multi-network policies can be used to manage traffic on secondary networks in the cluster. These policies cannot manage the default cluster network or primary network of user-defined networks.

As a cluster administrator, you can configure a multi-network policy for any of the following network types:

  • Single-Root I/O Virtualization (SR-IOV)
  • MAC Virtual Local Area Network (MacVLAN)
  • IP Virtual Local Area Network (IPVLAN)
  • Bond Container Network Interface (CNI) over SR-IOV
  • OVN-Kubernetes secondary networks
Note

Support for configuring multi-network policies for SR-IOV secondary networks is only supported with kernel network interface controllers (NICs). SR-IOV is not supported for Data Plane Development Kit (DPDK) applications.

Although the MultiNetworkPolicy API implements the NetworkPolicy API, there are several important differences:

  • You must use the MultiNetworkPolicy API: 

    apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
    Copy to Clipboard Toggle word wrap
  • You must use the multi-networkpolicy resource name when using the CLI to interact with multi-network policies. For example, you can view a multi-network policy object with the oc get multi-networkpolicy <name> command where <name> is the name of a multi-network policy.

  • You can use the k8s.v1.cni.cncf.io/policy-for annotation on a MultiNetworkPolicy object to point to a NetworkAttachmentDefinition (NAD) custom resource (CR). The NAD CR defines the network to which the policy applies.

    Example multi-network policy that includes the k8s.v1.cni.cncf.io/policy-for annotation

    apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
    Copy to Clipboard Toggle word wrap

    where:

    <namespace_name>
    Specifies the namespace name.
    <network_name>
    Specifies the name of a network attachment definition.

As a cluster administrator, you can enable multi-network policy support on your cluster.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in to the cluster with a user with cluster-admin privileges.

Procedure

  1. Create the multinetwork-enable-patch.yaml file with the following YAML: 

    apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  useMultiNetworkPolicy: true
    Copy to Clipboard Toggle word wrap

  2. Configure the cluster to enable multi-network policy. Successful output lists the name of the policy object and the patched status. 

    $ oc patch network.operator.openshift.io cluster --type=merge --patch-file=multinetwork-enable-patch.yaml
    Copy to Clipboard Toggle word wrap

The ICMPv6 Neighbor Discovery Protocol (NDP) is a set of messages and processes that enable devices to discover and maintain information about neighboring nodes. NDP plays a crucial role in IPv6 networks, facilitating the interaction between devices on the same link.

The Cluster Network Operator (CNO) deploys the iptables implementation of multi-network policy when the useMultiNetworkPolicy parameter is set to true.

To support multi-network policies in IPv6 networks the Cluster Network Operator deploys the following set of rules in every pod affected by a multi-network policy:

Multi-network policy custom rules

kind: ConfigMap
apiVersion: v1
metadata:
  name: multi-networkpolicy-custom-rules
  namespace: openshift-multus
data:

  custom-v6-rules.txt: |
    # accept NDP
    -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT
1 

    -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT
2 

    # accept RA/RS
    -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
3 

    -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
4
Copy to Clipboard Toggle word wrap

1
This rule allows incoming ICMPv6 neighbor solicitation messages, which are part of the neighbor discovery protocol (NDP). These messages help determine the link-layer addresses of neighboring nodes.
2
This rule allows incoming ICMPv6 neighbor advertisement messages, which are part of NDP and provide information about the link-layer address of the sender.
3
This rule permits incoming ICMPv6 router solicitation messages. Hosts use these messages to request router configuration information.
4
This rule allows incoming ICMPv6 router advertisement messages, which give configuration information to hosts.
Note

You cannot edit these predefined rules.

These rules collectively enable essential ICMPv6 traffic for correct network functioning, including address resolution and router communication in an IPv6 environment. With these rules in place and a multi-network policy denying traffic, applications are not expected to experience connectivity issues.

3.4.4. Working with multi-network policy
Copy link

As a cluster administrator, you can create, edit, view, and delete multi-network policies.

3.4.4.1. Prerequisites
Copy link
  • You have enabled multi-network policy support for your cluster.

To define granular rules describing ingress or egress network traffic allowed for namespaces in your cluster, you can create a multi-network policy.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin, with mode: NetworkPolicy set.
  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace that the multi-network policy applies to.

Procedure

  1. Create a policy rule:

    1. Create a <policy_name>.yaml file: 

      $ touch <policy_name>.yaml
      Copy to Clipboard Toggle word wrap

      where:

      <policy_name>
      Specifies the multi-network policy file name.

    2. Define a multi-network policy in the file that you just created, such as in the following examples:

      Deny ingress from all pods in all namespaces

      This is a fundamental policy, blocking all cross-pod networking other than cross-pod traffic allowed by the configuration of other Network Policies. 

      apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: deny-by-default
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress: []
      Copy to Clipboard Toggle word wrap

      where:

      <network_name>
      Specifies the name of a network attachment definition.

      Allow ingress from all pods in the same namespace

      apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-same-namespace
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
spec:
  podSelector:
  ingress:
  - from:
    - podSelector: {}
      Copy to Clipboard Toggle word wrap

      where:

      <network_name>
      Specifies the name of a network attachment definition.

      Allow ingress traffic to one pod from a particular namespace

      This policy allows traffic to pods that have the pod-a label from pods running in namespace-y. 

      apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-traffic-pod
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
spec:
  podSelector:
   matchLabels:
      pod: pod-a
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
           kubernetes.io/metadata.name: namespace-y
      Copy to Clipboard Toggle word wrap

      where:

      <network_name>
      Specifies the name of a network attachment definition.

      Restrict traffic to a service

      This policy when applied ensures every pod with both labels app=bookstore and role=api can only be accessed by pods with label app=bookstore. In this example the application could be a REST API server, marked with labels app=bookstore and role=api.

      This example addresses the following use cases:

      • Restricting the traffic to a service to only the other microservices that need to use it.

      • Restricting the connections to a database to only permit the application using it. 

        apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: api-allow
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
spec:
  podSelector:
    matchLabels:
      app: bookstore
      role: api
  ingress:
  - from:
      - podSelector:
          matchLabels:
            app: bookstore
        Copy to Clipboard Toggle word wrap

        where:

        <network_name>
        Specifies the name of a network attachment definition.

  2. To create the multi-network policy object, enter the following command. Successful output lists the name of the policy object and the created status. 

    $ oc apply -f <policy_name>.yaml -n <namespace>
    Copy to Clipboard Toggle word wrap

    where:

    <policy_name>
    Specifies the multi-network policy file name.
    <namespace>
    Optional parameter. If you defined the object in a different namespace than the current namespace, the parameter specifices the namespace.

    Successful output lists the name of the policy object and the created status.

Note

If you log in to the web console with cluster-admin privileges, you have a choice of creating a network policy in any namespace in the cluster directly in YAML or from a form in the web console.

3.4.4.3. Editing a multi-network policy
Copy link

You can edit a multi-network policy in a namespace.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin, with mode: NetworkPolicy set.
  • You installed the OpenShift CLI (oc).
  • You are logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace where the multi-network policy exists.

Procedure

  1. Optional: To list the multi-network policy objects in a namespace, enter the following command: 

    $ oc get multi-networkpolicy
    Copy to Clipboard Toggle word wrap

    where:

    <namespace>
    Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.

  2. Edit the multi-network policy object.

    • If you saved the multi-network policy definition in a file, edit the file and make any necessary changes, and then enter the following command. 

      $ oc apply -n <namespace> -f <policy_file>.yaml
      Copy to Clipboard Toggle word wrap

      where:

      <namespace>
      Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
      <policy_file>
      Specifies the name of the file containing the network policy.

    • If you need to update the multi-network policy object directly, enter the following command: 

      $ oc edit multi-networkpolicy <policy_name> -n <namespace>
      Copy to Clipboard Toggle word wrap

      where:

      <policy_name>
      Specifies the name of the network policy.
      <namespace>
      Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.

  3. Confirm that the multi-network policy object is updated. 

    $ oc describe multi-networkpolicy <policy_name> -n <namespace>
    Copy to Clipboard Toggle word wrap

    where:

    <policy_name>
    Specifies the name of the multi-network policy.
    <namespace>
    Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
Note

If you log in to the web console with cluster-admin privileges, you have a choice of editing a network policy in any namespace in the cluster directly in YAML or from the policy in the web console through the Actions menu.

You can examine the multi-network policies in a namespace.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You are logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace where the multi-network policy exists.

Procedure

  • List multi-network policies in a namespace:

    • To view multi-network policy objects defined in a namespace, enter the following command: 

      $ oc get multi-networkpolicy
      Copy to Clipboard Toggle word wrap

    • Optional: To examine a specific multi-network policy, enter the following command: 

      $ oc describe multi-networkpolicy <policy_name> -n <namespace>
      Copy to Clipboard Toggle word wrap

      where:

      <policy_name>
      Specifies the name of the multi-network policy to inspect.
      <namespace>
      Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.
Note

If you log in to the web console with cluster-admin privileges, you have a choice of viewing a network policy in any namespace in the cluster directly in YAML or from a form in the web console.

You can delete a multi-network policy in a namespace.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin, with mode: NetworkPolicy set.
  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace where the multi-network policy exists.

Procedure

  • To delete a multi-network policy object, enter the following command. Successful output lists the name of the policy object and the deleted status. 

    $ oc delete multi-networkpolicy <policy_name> -n <namespace>
    Copy to Clipboard Toggle word wrap

    where:

    <policy_name>
    Specifies the name of the multi-network policy.
    <namespace>
    Optional parameter. If you defined the object in a different namespace than the current namespace, the parameter specifices the namespace.

    Successful output lists the name of the policy object and the deleted status.

Note

If you log in to the web console with cluster-admin privileges, you have a choice of deleting a network policy in any namespace in the cluster directly in YAML or from the policy in the web console through the Actions menu.

This policy blocks all cross-pod networking other than network traffic allowed by the configuration of other deployed network policies and traffic between host-networked pods. This procedure enforces a strong deny policy by applying a deny-by-default policy in the my-project namespace.

Warning

Without configuring a NetworkPolicy custom resource (CR) that allows traffic communication, the following policy might cause communication problems across your cluster.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin, with mode: NetworkPolicy set.
  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace that the multi-network policy applies to.

Procedure

  1. Create the following YAML that defines a deny-by-default policy to deny ingress from all pods in all namespaces. Save the YAML in the deny-by-default.yaml file: 

    apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: deny-by-default
  namespace: my-project
    1 
    
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
    2 
    
spec:
  podSelector: {}
    3 
    
  policyTypes:
    4 
    
  - Ingress
    5 
    
  ingress: []
    6
    Copy to Clipboard Toggle word wrap
    1
    Specifies the namespace in which to deploy the policy. For example, the my-project namespace.
    2
    Specifies the name of namespace project followed by the network attachment definition name.
    3
    If this field is empty, the configuration matches all the pods. Therefore, the policy applies to all pods in the my-project namespace.
    4
    Specifies a list of rule types that the NetworkPolicy relates to.
    5
    Specifies Ingress only policyTypes.
    6
    Specifies ingress rules. If not specified, all incoming traffic is dropped to all pods.

  2. Apply the policy by entering the following command. Successful output lists the name of the policy object and the created status. 

    $ oc apply -f deny-by-default.yaml
    Copy to Clipboard Toggle word wrap

    Successful output lists the name of the policy object and the created status.

With the deny-by-default policy in place you can proceed to configure a policy that allows traffic from external clients to a pod with the label app=web.

Note

If you log in with a user with the cluster-admin role, then you can create a network policy in any namespace in the cluster.

Follow this procedure to configure a policy that allows external service from the public Internet directly or by using a Load Balancer to access the pod. Traffic is only allowed to a pod with the label app=web.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin, with mode: NetworkPolicy set.
  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace that the multi-network policy applies to.

Procedure

  1. Create a policy that allows traffic from the public Internet directly or by using a load balancer to access the pod. Save the YAML in the web-allow-external.yaml file: 

    apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: web-allow-external
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
spec:
  policyTypes:
  - Ingress
  podSelector:
    matchLabels:
      app: web
  ingress:
    - {}
    Copy to Clipboard Toggle word wrap

  2. Apply the policy by entering the following command. Successful output lists the name of the policy object and the created status. 

    $ oc apply -f web-allow-external.yaml
    Copy to Clipboard Toggle word wrap

    Successful output lists the name of the policy object and the created status. This policy allows traffic from all resources, including external traffic as illustrated in the following diagram:

Allow traffic from external clients
View larger image
Allow traffic from external clients
Note

If you log in with a user with the cluster-admin role, then you can create a network policy in any namespace in the cluster.

Follow this procedure to configure a policy that allows traffic from all pods in all namespaces to a particular application.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin, with mode: NetworkPolicy set.
  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace that the multi-network policy applies to.

Procedure

  1. Create a policy that allows traffic from all pods in all namespaces to a particular application. Save the YAML in the web-allow-all-namespaces.yaml file: 

    apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: web-allow-all-namespaces
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
spec:
  podSelector:
    matchLabels:
      app: web
    1 
    
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector: {}
    2
    Copy to Clipboard Toggle word wrap
    1
    Applies the policy only to app:web pods in default namespace.
    2
    Selects all pods in all namespaces.
    Note

    By default, if you do not specify a namespaceSelector parameter in the policy object, no namespaces get selected. This means the policy allows traffic only from the namespace where the network policy deployes.

  2. Apply the policy by entering the following command. Successful output lists the name of the policy object and the created status. 

    $ oc apply -f web-allow-all-namespaces.yaml
    Copy to Clipboard Toggle word wrap

    Successful output lists the name of the policy object and the created status.

Verification

  1. Start a web service in the default namespace by entering the following command: 

    $ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80
    Copy to Clipboard Toggle word wrap

  2. Run the following command to deploy an alpine image in the secondary namespace and to start a shell: 

    $ oc run test-$RANDOM --namespace=secondary --rm -i -t --image=alpine -- sh
    Copy to Clipboard Toggle word wrap

  3. Run the following command in the shell and observe that the service allows the request: 

    # wget -qO- --timeout=2 http://web.default
    Copy to Clipboard Toggle word wrap

    Expected output

    <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
    Copy to Clipboard Toggle word wrap

Note

If you log in with a user with the cluster-admin role, then you can create a network policy in any namespace in the cluster.

Follow this procedure to configure a policy that allows traffic to a pod with the label app=web from a particular namespace. You might want to do this to:

  • Restrict traffic to a production database only to namespaces that have production workloads deployed.
  • Enable monitoring tools deployed to a particular namespace to scrape metrics from the current namespace.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin, with mode: NetworkPolicy set.
  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster with a user with cluster-admin privileges.
  • You are working in the namespace that the multi-network policy applies to.

Procedure

  1. Create a policy that allows traffic from all pods in a particular namespaces with a label purpose=production. Save the YAML in the web-allow-prod.yaml file: 

    apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: web-allow-prod
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/policy-for:<namespace_name>/<network_name>
spec:
  podSelector:
    matchLabels:
      app: web
    1 
    
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          purpose: production
    2
    Copy to Clipboard Toggle word wrap
    1
    Applies the policy only to app:web pods in the default namespace.
    2
    Restricts traffic to only pods in namespaces that have the label purpose=production.

  2. Apply the policy by entering the following command. Successful output lists the name of the policy object and the created status. 

    $ oc apply -f web-allow-prod.yaml
    Copy to Clipboard Toggle word wrap

    Successful output lists the name of the policy object and the created status.

Verification

  1. Start a web service in the default namespace by entering the following command: 

    $ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80
    Copy to Clipboard Toggle word wrap

  2. Run the following command to create the prod namespace: 

    $ oc create namespace prod
    Copy to Clipboard Toggle word wrap

  3. Run the following command to label the prod namespace: 

    $ oc label namespace/prod purpose=production
    Copy to Clipboard Toggle word wrap

  4. Run the following command to create the dev namespace: 

    $ oc create namespace dev
    Copy to Clipboard Toggle word wrap

  5. Run the following command to label the dev namespace: 

    $ oc label namespace/dev purpose=testing
    Copy to Clipboard Toggle word wrap

  6. Run the following command to deploy an alpine image in the dev namespace and to start a shell: 

    $ oc run test-$RANDOM --namespace=dev --rm -i -t --image=alpine -- sh
    Copy to Clipboard Toggle word wrap

  7. Run the following command in the shell and observe the reason for the blocked request. For example, expected output states wget: download timed out. 

    # wget -qO- --timeout=2 http://web.default
    Copy to Clipboard Toggle word wrap

  8. Run the following command to deploy an alpine image in the prod namespace and start a shell: 

    $ oc run test-$RANDOM --namespace=prod --rm -i -t --image=alpine -- sh
    Copy to Clipboard Toggle word wrap

  9. Run the following command in the shell and observe that the request is allowed: 

    # wget -qO- --timeout=2 http://web.default
    Copy to Clipboard Toggle word wrap

    Expected output

    <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
    Copy to Clipboard Toggle word wrap

3.5. Removing a pod from a secondary network
Copy link

As a cluster user you can remove a pod from a secondary network.

3.5.1. Removing a pod from a secondary network
Copy link

You can remove a pod from a secondary network only by deleting the pod.

Prerequisites

  • A secondary network is attached to the pod.
  • Install the OpenShift CLI (oc).
  • Log in to the cluster.

Procedure

  • To delete the pod, enter the following command: 

    $ oc delete pod <name> -n <namespace>
    Copy to Clipboard Toggle word wrap
    • <name> is the name of the pod.
    • <namespace> is the namespace that contains the pod.

3.6. Editing a secondary network
Copy link

As a cluster administrator you can modify the configuration for an existing secondary network.

As a cluster administrator, you can make changes to an existing secondary network. Any existing pods attached to the secondary network will not be updated.

Prerequisites

  • You have configured a secondary network for your cluster.
  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

To edit a secondary network for your cluster, complete the following steps:

  1. Run the following command to edit the Cluster Network Operator (CNO) CR in your default text editor: 

    $ oc edit networks.operator.openshift.io cluster
    Copy to Clipboard Toggle word wrap
  2. In the additionalNetworks collection, update the secondary network with your changes.
  3. Save your changes and quit the text editor to commit your changes.

  4. Optional: Confirm that the CNO updated the NetworkAttachmentDefinition object by running the following command. Replace <network-name> with the name of the secondary network to display. There might be a delay before the CNO updates the NetworkAttachmentDefinition object to reflect your changes. 

    $ oc get network-attachment-definitions <network-name> -o yaml
    Copy to Clipboard Toggle word wrap

    For example, the following console output displays a NetworkAttachmentDefinition object that is named net1: 

    $ oc get network-attachment-definitions net1 -o go-template='{{printf "%s\n" .spec.config}}'
{ "cniVersion": "0.3.1", "type": "macvlan",
"master": "ens5",
"mode": "bridge",
"ipam":       {"type":"static","routes":[{"dst":"0.0.0.0/0","gw":"10.128.2.1"}],"addresses":[{"address":"10.128.2.100/23","gateway":"10.128.2.1"}],"dns":{"nameservers":["172.30.0.10"],"domain":"us-west-2.compute.internal","search":["us-west-2.compute.internal"]}} }
    Copy to Clipboard Toggle word wrap

The following sections give instructions and information for how to configure IP address assignments for secondary networks.

For secondary networks, IP addresses can be assigned using an IP Address Management (IPAM) CNI plugin, which supports various assignment methods, including Dynamic Host Configuration Protocol (DHCP) and static assignment.

The DHCP IPAM CNI plugin responsible for dynamic assignment of IP addresses operates with two distinct components:

  • CNI Plugin: Responsible for integrating with the Kubernetes networking stack to request and release IP addresses.
  • DHCP IPAM CNI Daemon: A listener for DHCP events that coordinates with existing DHCP servers in the environment to handle IP address assignment requests. This daemon is not a DHCP server itself.

For networks requiring type: dhcp in their IPAM configuration, ensure the following:

  • A DHCP server is available and running in the environment. The DHCP server is external to the cluster and is expected to be part of the customer’s existing network infrastructure.
  • The DHCP server is appropriately configured to serve IP addresses to the nodes.

In cases where a DHCP server is unavailable in the environment, it is recommended to use the Whereabouts IPAM CNI plugin instead. The Whereabouts CNI provides similar IP address management capabilities without the need for an external DHCP server.

Note

Use the Whereabouts CNI plugin when there is no external DHCP server or where static IP address management is preferred. The Whereabouts plugin includes a reconciler daemon to manage stale IP address allocations.

A DHCP lease must be periodically renewed throughout the container’s lifetime, so a separate daemon, the DHCP IPAM CNI Daemon, is required. To deploy the DHCP IPAM CNI daemon, modify the Cluster Network Operator (CNO) configuration to trigger the deployment of this daemon as part of the secondary network setup.

The following table describes the configuration for static IP address assignment:

Expand
Table 3.11. ipam static configuration object
FieldTypeDescription

type

string

The IPAM address type. The value static is required.

addresses

array

An array of objects specifying IP addresses to assign to the virtual interface. Both IPv4 and IPv6 IP addresses are supported.

routes

array

An array of objects specifying routes to configure inside the pod.

dns

array

Optional: An array of objects specifying the DNS configuration.

The addresses array requires objects with the following fields:

Expand
Table 3.12. ipam.addresses[] array
FieldTypeDescription

address

string

An IP address and network prefix that you specify. For example, if you specify 10.10.21.10/24, then the secondary network is assigned an IP address of 10.10.21.10 and the netmask is 255.255.255.0.

gateway

string

The default gateway to route egress network traffic to.

Expand
Table 3.13. ipam.routes[] array
FieldTypeDescription

dst

string

The IP address range in CIDR format, such as 192.168.17.0/24 or 0.0.0.0/0 for the default route.

gw

string

The gateway where network traffic is routed.

Expand
Table 3.14. ipam.dns object
FieldTypeDescription

nameservers

array

An array of one or more IP addresses for to send DNS queries to.

domain

array

The default domain to append to a hostname. For example, if the domain is set to example.com, a DNS lookup query for example-host is rewritten as example-host.example.com.

search

array

An array of domain names to append to an unqualified hostname, such as example-host, during a DNS lookup query.

Static IP address assignment configuration example

{
  "ipam": {
    "type": "static",
      "addresses": [
        {
          "address": "191.168.1.7/24"
        }
      ]
  }
}
Copy to Clipboard Toggle word wrap

A pod obtains its original DHCP lease when it is created. The lease must be periodically renewed by a minimal DHCP server deployment running on the cluster.

Important

For an Ethernet network attachment, the SR-IOV Network Operator does not create a DHCP server deployment; the Cluster Network Operator is responsible for creating the minimal DHCP server deployment.

To trigger the deployment of the DHCP server, you must create a shim network attachment by editing the Cluster Network Operator configuration, as in the following example:

Example shim network attachment definition

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  additionalNetworks:
  - name: dhcp-shim
    namespace: default
    type: Raw
    rawCNIConfig: |-
      {
        "name": "dhcp-shim",
        "cniVersion": "0.3.1",
        "type": "bridge",
        "ipam": {
          "type": "dhcp"
        }
      }
  # ...
Copy to Clipboard Toggle word wrap

The following table describes the configuration parameters for dynamic IP address address assignment with DHCP.

Expand
Table 3.15. ipam DHCP configuration object
FieldTypeDescription

type

string

The IPAM address type. The value dhcp is required.

The following JSON example describes the configuration p for dynamic IP address address assignment with DHCP.

Dynamic IP address (DHCP) assignment configuration example

{
  "ipam": {
    "type": "dhcp"
  }
}
Copy to Clipboard Toggle word wrap

The Whereabouts CNI plugin allows the dynamic assignment of an IP address to a secondary network without the use of a DHCP server.

The Whereabouts CNI plugin also supports overlapping IP address ranges and configuration of the same CIDR range multiple times within separate NetworkAttachmentDefinition CRDs. This provides greater flexibility and management capabilities in multi-tenant environments.

The following table describes the configuration objects for dynamic IP address assignment with Whereabouts:

Expand
Table 3.16. ipam whereabouts configuration object
FieldTypeDescription

type

string

The IPAM address type. The value whereabouts is required.

range

string

An IP address and range in CIDR notation. IP addresses are assigned from within this range of addresses.

exclude

array

Optional: A list of zero or more IP addresses and ranges in CIDR notation. IP addresses within an excluded address range are not assigned.

network_name

string

Optional: Helps ensure that each group or domain of pods gets its own set of IP addresses, even if they share the same range of IP addresses. Setting this field is important for keeping networks separate and organized, notably in multi-tenant environments.

The following example shows a dynamic address assignment configuration that uses Whereabouts:

Whereabouts dynamic IP address assignment

{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/27",
    "exclude": [
       "192.0.2.192/30",
       "192.0.2.196/32"
    ]
  }
}
Copy to Clipboard Toggle word wrap

The following example shows a dynamic IP address assignment that uses overlapping IP address ranges for multi-tenant networks.

NetworkAttachmentDefinition 1

{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/29",
    "network_name": "example_net_common",
1 

  }
}
Copy to Clipboard Toggle word wrap

1
Optional. If set, must match the network_name of NetworkAttachmentDefinition 2.

NetworkAttachmentDefinition 2

{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/24",
    "network_name": "example_net_common",
1 

  }
}
Copy to Clipboard Toggle word wrap

1
Optional. If set, must match the network_name of NetworkAttachmentDefinition 1.

The Whereabouts reconciler is responsible for managing dynamic IP address assignments for the pods within a cluster by using the Whereabouts IP Address Management (IPAM) solution. It ensures that each pod gets a unique IP address from the specified IP address range. It also handles IP address releases when pods are deleted or scaled down.

Note

You can also use a NetworkAttachmentDefinition custom resource definition (CRD) for dynamic IP address assignment.

The whereabouts-reconciler daemon set is automatically created when you configure a secondary network through the Cluster Network Operator. It is not automatically created when you configure a secondary network from a YAML manifest.

To trigger the deployment of the whereabouts-reconciler daemon set, you must manually create a whereabouts-shim network attachment by editing the Cluster Network Operator custom resource (CR) file.

Use the following procedure to deploy the whereabouts-reconciler daemon set.

Procedure

  1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command: 

    $ oc edit network.operator.openshift.io cluster
    Copy to Clipboard Toggle word wrap

  2. Include the additionalNetworks section shown in this example YAML extract within the spec definition of the custom resource (CR): 

    apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
# ...
spec:
  additionalNetworks:
  - name: whereabouts-shim
    namespace: default
    rawCNIConfig: |-
      {
       "name": "whereabouts-shim",
       "cniVersion": "0.3.1",
       "type": "bridge",
       "ipam": {
         "type": "whereabouts"
       }
      }
    type: Raw
# ...
    Copy to Clipboard Toggle word wrap
  3. Save the file and exit the text editor.

  4. Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command: 

    $ oc get all -n openshift-multus | grep whereabouts-reconciler
    Copy to Clipboard Toggle word wrap

    Example output

    pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s
    Copy to Clipboard Toggle word wrap

The Whereabouts IPAM CNI plugin runs the IP reconciler daily. This process cleans up any stranded IP allocations that might result in exhausting IPs and therefore prevent new pods from getting an IP allocated to them.

Use this procedure to change the frequency at which the IP reconciler runs.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have deployed the whereabouts-reconciler daemon set, and the whereabouts-reconciler pods are up and running.

Procedure

  1. Run the following command to create a ConfigMap object named whereabouts-config in the openshift-multus namespace with a specific cron expression for the IP reconciler: 

    $ oc create configmap whereabouts-config -n openshift-multus --from-literal=reconciler_cron_expression="*/15 * * * *"
    Copy to Clipboard Toggle word wrap

    This cron expression indicates the IP reconciler runs every 15 minutes. Adjust the expression based on your specific requirements.

    Note

    The whereabouts-reconciler daemon set can only consume a cron expression pattern that includes five asterisks. The sixth, which is used to denote seconds, is currently not supported.

  2. Retrieve information about resources related to the whereabouts-reconciler daemon set and pods within the openshift-multus namespace by running the following command: 

    $ oc get all -n openshift-multus | grep whereabouts-reconciler
    Copy to Clipboard Toggle word wrap

    Example output

    pod/whereabouts-reconciler-2p7hw                   1/1     Running   0             4m14s
pod/whereabouts-reconciler-76jk7                   1/1     Running   0             4m14s
daemonset.apps/whereabouts-reconciler          6         6         6       6            6           kubernetes.io/os=linux   4m16s
    Copy to Clipboard Toggle word wrap

  3. Run the following command to verify that the whereabouts-reconciler pod runs the IP reconciler with the configured interval: 

    $ oc -n openshift-multus logs whereabouts-reconciler-2p7hw
    Copy to Clipboard Toggle word wrap

    Example output

    2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CREATE
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CHMOD
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..data_tmp": RENAME
2024-02-02T16:33:54Z [verbose] using expression: */15 * * * *
2024-02-02T16:33:54Z [verbose] configuration updated to file "/cron-schedule/..data". New cron expression: */15 * * * *
2024-02-02T16:33:54Z [verbose] successfully updated CRON configuration id "00c2d1c9-631d-403f-bb86-73ad104a6817" - new cron expression: */15 * * * *
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/config": CREATE
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_26_17.3874177937": REMOVE
2024-02-02T16:45:00Z [verbose] starting reconciler run
2024-02-02T16:45:00Z [debug] NewReconcileLooper - inferred connection data
2024-02-02T16:45:00Z [debug] listing IP pools
2024-02-02T16:45:00Z [debug] no IP addresses to cleanup
2024-02-02T16:45:00Z [verbose] reconciler success
    Copy to Clipboard Toggle word wrap

Dual-stack IP address assignment can be configured with the ipRanges parameter for:

  • IPv4 addresses
  • IPv6 addresses
  • multiple IP address assignment

Procedure

  1. Set type to whereabouts.

  2. Use ipRanges to allocate IP addresses as shown in the following example: 

    cniVersion: operator.openshift.io/v1
kind: Network
=metadata:
  name: cluster
spec:
  additionalNetworks:
  - name: whereabouts-shim
    namespace: default
    type: Raw
    rawCNIConfig: |-
      {
       "name": "whereabouts-dual-stack",
       "cniVersion": "0.3.1,
       "type": "bridge",
       "ipam": {
         "type": "whereabouts",
         "ipRanges": [
                  {"range": "192.168.10.0/24"},
                  {"range": "2001:db8::/64"}
              ]
       }
      }
    Copy to Clipboard Toggle word wrap
  3. Attach network to a pod. For more information, see "Adding a pod to a secondary network".
  4. Verify that all IP addresses are assigned.

  5. Run the following command to ensure the IP addresses are assigned as metadata. 

    $ oc exec -it mypod -- ip a
    Copy to Clipboard Toggle word wrap

The following section provides instructions and information for how to create and manage a MAC-VLAN, IP-VLAN, and VLAN subinterface based on a master interface.

You can create a MAC-VLAN, an IP-VLAN, or a VLAN subinterface that is based on a master interface that exists in a container namespace. You can also create a master interface as part of the pod network configuration in a separate network attachment definition CRD.

To use a container namespace master interface, you must specify true for the linkInContainer parameter that exists in the subinterface configuration of the NetworkAttachmentDefinition CRD.

3.8.1.1. Creating multiple VLANs on SR-IOV VFs
Copy link

An example use case for utilizing this feature is to create multiple VLANs based on SR-IOV VFs. To do so, begin by creating an SR-IOV network and then define the network attachments for the VLAN interfaces.

The following example shows how to configure the setup illustrated in this diagram.

Figure 3.1. Creating VLANs

Creating VLANs
View larger image
Creating VLANs

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the SR-IOV Network Operator.

Procedure

  1. Create a dedicated container namespace where you want to deploy your pod by using the following command: 

    $ oc new-project test-namespace
    Copy to Clipboard Toggle word wrap

  2. Create an SR-IOV node policy:

    1. Create an SriovNetworkNodePolicy object, and then save the YAML in the sriov-node-network-policy.yaml file: 

      apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
 name: sriovnic
 namespace: openshift-sriov-network-operator
spec:
 deviceType: netdevice
 isRdma: false
 needVhostNet: true
 nicSelector:
   vendor: "15b3"
      1 
      
   deviceID: "101b"
      2 
      
   rootDevices: ["00:05.0"]
 numVfs: 10
 priority: 99
 resourceName: sriovnic
 nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
      Copy to Clipboard Toggle word wrap
      Note

      The SR-IOV network node policy configuration example, with the setting deviceType: netdevice, is tailored specifically for Mellanox Network Interface Cards (NICs).

      1
      The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.
      2
      The device hexadecimal code of the SR-IOV network device.

    2. Apply the YAML by running the following command: 

      $ oc apply -f sriov-node-network-policy.yaml
      Copy to Clipboard Toggle word wrap
      Note

      Applying this might take some time due to the node requiring a reboot.

  3. Create an SR-IOV network:

    1. Create the SriovNetwork custom resource (CR) for the additional secondary SR-IOV network attachment as in the following example CR. Save the YAML as the file sriov-network-attachment.yaml: 

      apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
 name: sriov-network
 namespace: openshift-sriov-network-operator
spec:
 networkNamespace: test-namespace
 resourceName: sriovnic
 spoofChk: "off"
 trust: "on"
      Copy to Clipboard Toggle word wrap

    2. Apply the YAML by running the following command: 

      $ oc apply -f sriov-network-attachment.yaml
      Copy to Clipboard Toggle word wrap

  4. Create the VLAN secondary network:

    1. Using the following YAML example, create a file named vlan100-additional-network-configuration.yaml: 

      apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vlan-100
  namespace: test-namespace
spec:
  config: |
    {
      "cniVersion": "0.4.0",
      "name": "vlan-100",
      "plugins": [
        {
          "type": "vlan",
          "master": "ext0",
      1 
      
          "mtu": 1500,
          "vlanId": 100,
          "linkInContainer": true,
      2 
      
          "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]}
        }
      ]
    }
      Copy to Clipboard Toggle word wrap
      1
      The VLAN configuration needs to specify the master name. This can be configured in the pod networks annotation.
      2
      The linkInContainer parameter must be specified.

    2. Apply the YAML file by running the following command: 

      $ oc apply -f vlan100-additional-network-configuration.yaml
      Copy to Clipboard Toggle word wrap

  5. Create a pod definition by using the earlier specified networks:

    1. Using the following YAML example, create a file named pod-a.yaml file:

      Note

      The manifest below includes 2 resources:

      • Namespace with security labels
      • Pod definition with appropriate network annotation 
      apiVersion: v1
kind: Namespace
metadata:
  name: test-namespace
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
    security.openshift.io/scc.podSecurityLabelSync: "false"
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  namespace: test-namespace
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "sriov-network",
        "namespace": "test-namespace",
        "interface": "ext0"
      1 
      
      },
      {
        "name": "vlan-100",
        "namespace": "test-namespace",
        "interface": "ext0.100"
      }
    ]'
spec:
  securityContext:
    runAsNonRoot: true
  containers:
    - name: nginx-container
      image: nginxinc/nginx-unprivileged:latest
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      ports:
        - containerPort: 80
      seccompProfile:
        type: "RuntimeDefault"
      Copy to Clipboard Toggle word wrap
      1
      The name to be used as the master for the VLAN interface.

    2. Apply the YAML file by running the following command: 

      $ oc apply -f pod-a.yaml
      Copy to Clipboard Toggle word wrap

  6. Get detailed information about the nginx-pod within the test-namespace by running the following command: 

    $ oc describe pods nginx-pod -n test-namespace
    Copy to Clipboard Toggle word wrap

    Example output

    Name:         nginx-pod
Namespace:    test-namespace
Priority:     0
Node:         worker-1/10.46.186.105
Start Time:   Mon, 14 Aug 2023 16:23:13 -0400
Labels:       <none>
Annotations:  k8s.ovn.org/pod-networks:
                {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ovn-kubernetes",
                    "interface": "eth0",
                    "ips": [
                        "10.131.0.26"
                    ],
                    "mac": "0a:58:0a:83:00:1a",
                    "default": true,
                    "dns": {}
                },{
                    "name": "test-namespace/sriov-network",
                    "interface": "ext0",
                    "mac": "6e:a7:5e:3f:49:1b",
                    "dns": {},
                    "device-info": {
                        "type": "pci",
                        "version": "1.0.0",
                        "pci": {
                            "pci-address": "0000:d8:00.2"
                        }
                    }
                },{
                    "name": "test-namespace/vlan-100",
                    "interface": "ext0.100",
                    "ips": [
                        "1.1.1.1"
                    ],
                    "mac": "6e:a7:5e:3f:49:1b",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks:
                [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i...
              openshift.io/scc: privileged
Status:       Running
IP:           10.131.0.26
IPs:
  IP:  10.131.0.26
    Copy to Clipboard Toggle word wrap

You can create a subinterface based on a bridge master interface that exists in a container namespace. Creating a subinterface can be applied to other types of interfaces.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You are logged in to the OpenShift Container Platform cluster as a user with cluster-admin privileges.

Procedure

  1. Create a dedicated container namespace where you want to deploy your pod by entering the following command: 

    $ oc new-project test-namespace
    Copy to Clipboard Toggle word wrap

  2. Using the following YAML example, create a bridge NetworkAttachmentDefinition custom resource definition (CRD) file named bridge-nad.yaml: 

    apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bridge-network
spec:
  config: '{
    "cniVersion": "0.4.0",
    "name": "bridge-network",
    "type": "bridge",
    "bridge": "br-001",
    "isGateway": true,
    "ipMasq": true,
    "hairpinMode": true,
    "ipam": {
      "type": "host-local",
      "subnet": "10.0.0.0/24",
      "routes": [{"dst": "0.0.0.0/0"}]
    }
  }'
    Copy to Clipboard Toggle word wrap

  3. Run the following command to apply the NetworkAttachmentDefinition CRD to your OpenShift Container Platform cluster: 

    $ oc apply -f bridge-nad.yaml
    Copy to Clipboard Toggle word wrap

  4. Verify that you successfully created a NetworkAttachmentDefinition CRD by entering the following command. The expected output shows the name of the NAD CRD and the creation age in minutes. 

    $ oc get network-attachment-definitions
    Copy to Clipboard Toggle word wrap

  5. Using the following YAML example, create a file named ipvlan-additional-network-configuration.yaml for the IPVLAN secondary network configuration: 

    apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ipvlan-net
  namespace: test-namespace
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "ipvlan-net",
    "type": "ipvlan",
    "master": "net1",
    1 
    
    "mode": "l3",
    "linkInContainer": true,
    2 
    
    "ipam": {"type": "whereabouts", "ipRanges": [{"range": "10.0.0.0/24"}]}
  }'
    Copy to Clipboard Toggle word wrap
    1
    Specifies the ethernet interface to associate with the network attachment. This is subsequently configured in the pod networks annotation.
    2
    Specifies that the master interface is in the container network namespace.

  6. Apply the YAML file by running the following command: 

    $ oc apply -f ipvlan-additional-network-configuration.yaml
    Copy to Clipboard Toggle word wrap

  7. Verify that the NetworkAttachmentDefinition CRD has been created successfully by running the following command. The expected output shows the name of the NAD CRD and the creation age in minutes. 

    $ oc get network-attachment-definitions
    Copy to Clipboard Toggle word wrap

  8. Using the following YAML example, create a file named pod-a.yaml for the pod definition: 

    apiVersion: v1
kind: Pod
metadata:
  name: pod-a
  namespace: test-namespace
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "bridge-network",
        "interface": "net1"
    1 
    
      },
      {
        "name": "ipvlan-net",
        "interface": "net2"
      }
    ]'
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: test-pod
    image: quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]
    Copy to Clipboard Toggle word wrap
    1
    Specifies the name to be used as the master for the IPVLAN interface.

  9. Apply the YAML file by running the following command: 

    $ oc apply -f pod-a.yaml
    Copy to Clipboard Toggle word wrap

  10. Verify that the pod is running by using the following command: 

    $ oc get pod -n test-namespace
    Copy to Clipboard Toggle word wrap

    Example output

    NAME    READY   STATUS    RESTARTS   AGE
pod-a   1/1     Running   0          2m36s
    Copy to Clipboard Toggle word wrap

  11. Show network interface information about the pod-a resource within the test-namespace by running the following command: 

    $ oc exec -n test-namespace pod-a -- ip a
    Copy to Clipboard Toggle word wrap

    Example output

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if105: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 0a:58:0a:d9:00:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.217.0.93/23 brd 10.217.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::488b:91ff:fe84:a94b/64 scope link
       valid_lft forever preferred_lft forever
4: net1@if107: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.2/24 brd 10.0.0.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::bcda:bdff:fe7e:f437/64 scope link
       valid_lft forever preferred_lft forever
5: net2@net1: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global net2
       valid_lft forever preferred_lft forever
    inet6 fe80::beda:bd00:17e:f437/64 scope link
       valid_lft forever preferred_lft forever
    Copy to Clipboard Toggle word wrap

    This output shows that the network interface net2 is associated with the physical interface net1.

3.9. Removing an additional network
Copy link

As a cluster administrator you can remove an additional network attachment.

As a cluster administrator, you can remove a secondary network from your OpenShift Container Platform cluster. The secondary network is not removed from any pods it is attached to.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

To remove a secondary network from your cluster, complete the following steps:

  1. Edit the Cluster Network Operator (CNO) in your default text editor by running the following command: 

    $ oc edit networks.operator.openshift.io cluster
    Copy to Clipboard Toggle word wrap

  2. Modify the CR by removing the configuration that the CNO created from the additionalNetworks collection for the secondary network that you want to remove. 

    apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  additionalNetworks: []
    1
    Copy to Clipboard Toggle word wrap
    1
    If you are removing the configuration mapping for the only secondary network attachment definition in the additionalNetworks collection, you must specify an empty collection.

  3. To remove a network attachment definition from the network of your cluster, enter the following command: 

    $ oc delete net-attach-def <name_of_NAD>
    1
    Copy to Clipboard Toggle word wrap
    1
    Replace <name_of_NAD> with the name of your network attachment definition.
  4. Save your changes and quit the text editor to commit your changes.

  5. Optional: Confirm that the secondary network CR was deleted by running the following command: 

    $ oc get network-attachment-definition --all-namespaces
    Copy to Clipboard Toggle word wrap

Chapter 4. Virtual routing and forwarding
Copy link

4.1. About virtual routing and forwarding
Copy link

Virtual routing and forwarding (VRF) devices combined with IP rules provide the ability to create virtual routing and forwarding domains. VRF reduces the number of permissions needed by CNF, and provides increased visibility of the network topology of secondary networks. VRF is used to provide multi-tenancy functionality, for example, where each tenant has its own unique routing tables and requires different default gateways.

Processes can bind a socket to the VRF device. Packets through the binded socket use the routing table associated with the VRF device. An important feature of VRF is that it impacts only OSI model layer 3 traffic and above so L2 tools, such as LLDP, are not affected. This allows higher priority IP rules such as policy based routing to take precedence over the VRF device rules directing specific traffic.

In telecommunications use cases, each CNF can potentially be connected to multiple different networks sharing the same address space. These secondary networks can potentially conflict with the cluster’s main network CIDR. Using the CNI VRF plugin, network functions can be connected to different customers' infrastructure using the same IP address, keeping different customers isolated. IP addresses are overlapped with OpenShift Container Platform IP space. The CNI VRF plugin also reduces the number of permissions needed by CNF and increases the visibility of network topologies of secondary networks.

Chapter 5. Assigning a secondary network to a VRF
Copy link

As a cluster administrator, you can configure a secondary network for a virtual routing and forwarding (VRF) domain by using the CNI VRF plugin. The virtual network that this plugin creates is associated with the physical interface that you specify.

Using a secondary network with a VRF instance has the following advantages:

Workload isolation
Isolate workload traffic by configuring a VRF instance for the secondary network.
Improved security
Enable improved security through isolated network paths in the VRF domain.
Multi-tenancy support
Support multi-tenancy through network segmentation with a unique routing table in the VRF domain for each tenant.
Note

Applications that use VRFs must bind to a specific device. The common usage is to use the SO_BINDTODEVICE option for a socket. The SO_BINDTODEVICE option binds the socket to the device that is specified in the passed interface name, for example, eth1. To use the SO_BINDTODEVICE option, the application must have CAP_NET_RAW capabilities.

Using a VRF through the ip vrf exec command is not supported in OpenShift Container Platform pods. To use VRF, bind applications directly to the VRF interface.

The Cluster Network Operator (CNO) manages secondary network definitions. When you specify a secondary network to create, the CNO creates the NetworkAttachmentDefinition custom resource (CR) automatically.

Note

Do not edit the NetworkAttachmentDefinition CRs that the Cluster Network Operator manages. Doing so might disrupt network traffic on your secondary network.

To create a secondary network attachment with the CNI VRF plugin, perform the following procedure.

Prerequisites

  • Install the OpenShift Container Platform CLI (oc).
  • Log in to the OpenShift cluster as a user with cluster-admin privileges.

Procedure

  1. Create the Network custom resource (CR) for the additional network attachment and insert the rawCNIConfig configuration for the secondary network, as in the following example CR. Save the YAML as the file additional-network-attachment.yaml. 

    apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  additionalNetworks:
    - name: test-network-1
      namespace: additional-network-1
      type: Raw
      rawCNIConfig: '{
        "cniVersion": "0.3.1",
        "name": "macvlan-vrf",
        "plugins": [
    1 
    
        {
          "type": "macvlan",
          "master": "eth1",
          "ipam": {
              "type": "static",
              "addresses": [
              {
                  "address": "191.168.1.23/24"
              }
              ]
          }
        },
        {
          "type": "vrf",
    2 
    
          "vrfname": "vrf-1",
    3 
    
          "table": 1001
    4 
    
        }]
      }'
    Copy to Clipboard Toggle word wrap
    1
    plugins must be a list. The first item in the list must be the secondary network underpinning the VRF network. The second item in the list is the VRF plugin configuration.
    2
    type must be set to vrf.
    3
    vrfname is the name of the VRF that the interface is assigned to. If it does not exist in the pod, it is created.
    4
    Optional. table is the routing table ID. By default, the tableid parameter is used. If it is not specified, the CNI assigns a free routing table ID to the VRF.
    Note

    VRF functions correctly only when the resource is of type netdevice.

  2. Create the Network resource: 

    $ oc create -f additional-network-attachment.yaml
    Copy to Clipboard Toggle word wrap

  3. Confirm that the CNO created the NetworkAttachmentDefinition CR by running the following command. Replace <namespace> with the namespace that you specified when configuring the network attachment, for example, additional-network-1. The expected output shows the name of the NAD CR and the creation age in minutes. 

    $ oc get network-attachment-definitions -n <namespace>
    Copy to Clipboard Toggle word wrap
    Note

    There might be a delay before the CNO creates the CR.

Verification

  1. Create a pod and assign it to the secondary network with the VRF instance:

    1. Create a YAML file that defines the Pod resource:

      Example pod-additional-net.yaml file

      apiVersion: v1
kind: Pod
metadata:
 name: pod-additional-net
 annotations:
   k8s.v1.cni.cncf.io/networks: '[
       {
               "name": "test-network-1"
      1 
      
       }
 ]'
spec:
 containers:
 - name: example-pod-1
   command: ["/bin/bash", "-c", "sleep 9000000"]
   image: centos:8
      Copy to Clipboard Toggle word wrap

      1
      Specify the name of the secondary network with the VRF instance.

    2. Create the Pod resource by running the following command. The expected output shows the name of the Pod resource and the creation age in minutes. 

      $ oc create -f pod-additional-net.yaml
      Copy to Clipboard Toggle word wrap

  2. Verify that the pod network attachment is connected to the VRF secondary network. Start a remote session with the pod and run the following command. The expected output shows the name of the VRF interface and its unique ID in the routing table. 

    $ ip vrf show
    Copy to Clipboard Toggle word wrap

  3. Confirm that the VRF interface is the controller for the secondary interface: 

    $ ip link
    Copy to Clipboard Toggle word wrap

    Example output

    5: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master red state UP mode
    Copy to Clipboard Toggle word wrap

Legal Notice
Copy link

Copyright © 2025 Red Hat

OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).

Modified versions must remove all Red Hat trademarks.

Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.

Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat