Chapter 4. Multiple networks

Procedure

1. Optional: For a ClusterUserDefinedNetwork CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network label by entering the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: <cudn_namespace_name>
     labels:
       k8s.ovn.org/primary-user-defined-network: ""
   EOF

   Copy to Clipboard

2. Create a request for either a Layer2 or Layer3 topology type cluster-wide user-defined network:

   1. Create a YAML file, such as cluster-layer-two-udn.yaml, to define your request for a Layer2 topology as in the following example:

      apiVersion: k8s.ovn.org/v1
      kind: ClusterUserDefinedNetwork
      metadata:
        name: <cudn_name> 

      1

      
      spec:
        namespaceSelector: 

      2

      
          matchLabels: 

      3

      
            "<label_1_key>": "<label_1_value>" 

      4

      
            "<label_2_key>": "<label_2_value>" 

      5

      
        network: 

      6

      
          topology: Layer2 

      7

      
          layer2: 

      8

      
            role: Primary 

      9

      
            subnets:
              - "2001:db8::/64"
              - "10.100.0.0/16" 

      10

      Copy to Clipboard

      1

      Name of your ClusterUserDefinedNetwork CR.

      2

      A label query over the set of namespaces that the cluster UDN CR applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces.

      3

      Uses the matchLabels selector type, where terms are evaluated with an AND relationship.

      4 5

      Because the matchLabels selector type is used, provisions namespaces that contain both <label_1_key>=<label_1_value> and <label_2_key>=<label_2_value> labels.

      6

      Describes the network configuration.

      7

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes.

      8

      This field specifies the topology configuration. It can be layer2 or layer3.

      9

      Specifies Primary or Secondary. Primary is the only role specification supported in 4.19.

      10

      For Layer2 topology types the following specifies config details for the subnet field:

      • The subnets field is optional.
      • The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
      • The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
      • Layer2 subnets can be omitted. If omitted, users must configure static IP addresses for the pods. As a consequence, port security only prevents MAC spoofing. For more information, see "Configuring pods with a static IP address".

   2. Create a YAML file, such as cluster-layer-three-udn.yaml, to define your request for a Layer3 topology as in the following example:

      apiVersion: k8s.ovn.org/v1
      kind: ClusterUserDefinedNetwork
      metadata:
        name: <cudn_name> 

      1

      
      spec:
        namespaceSelector: 

      2

      
          matchExpressions: 

      3

      
          - key: kubernetes.io/metadata.name 

      4

      
            operator: In 

      5

      
            values: ["<example_namespace_one>", "<example_namespace_two>"] 

      6

      
        network: 

      7

      
          topology: Layer3 

      8

      
          layer3: 

      9

      
            role: Primary 

      10

      
            subnets: 

      11

      
              - cidr: 10.100.0.0/16
                hostSubnet: 24

      Copy to Clipboard

      1

      Name of your ClusterUserDefinedNetwork CR.

      2

      A label query over the set of namespaces that the cluster UDN applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces.

      3

      Uses the matchExpressions selector type, where terms are evaluated with an OR relationship.

      4

      Specifies the label key to match.

      5

      Specifies the operator. Valid values include: In, NotIn, Exists, and DoesNotExist.

      6

      Because the matchExpressions type is used, provisions namespaces matching either <example_namespace_one> or <example_namespace_two>.

      7

      Describes the network configuration.

      8

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets.

      9

      This field specifies the topology configuration. Valid values are layer2 or layer3.

      10

      Specifies a Primary or Secondary role. Primary is the only role specification supported in 4.19.

      11

      For Layer3 topology types the following specifies config details for the subnet field:

      • The subnets field is mandatory.

      • The type for the subnets field is cidr and hostSubnet:

        • cidr is the cluster subnet and accepts a string value.
        • hostSubnet specifies the nodes subnet prefix that the cluster subnet is split to.
        • For IPv6, only a /64 length is supported for hostSubnet.

3. Apply your request by running the following command:

   $ oc create --validate=true -f <example_cluster_udn>.yaml

   Copy to Clipboard

   Where <example_cluster_udn>.yaml is the name of your Layer2 or Layer3 configuration file.

4. Verify that your request is successful by running the following command:

   $ oc get clusteruserdefinednetwork <cudn_name> -o yaml

   Copy to Clipboard

   Where <cudn_name> is the name you created of your cluster-wide user-defined network.

   Example output

   apiVersion: k8s.ovn.org/v1
   kind: ClusterUserDefinedNetwork
   metadata:
     creationTimestamp: "2024-12-05T15:53:00Z"
     finalizers:
     - k8s.ovn.org/user-defined-network-protection
     generation: 1
     name: my-cudn
     resourceVersion: "47985"
     uid: 16ee0fcf-74d1-4826-a6b7-25c737c1a634
   spec:
     namespaceSelector:
       matchExpressions:
       - key: custom.network.selector
         operator: In
         values:
         - example-namespace-1
         - example-namespace-2
         - example-namespace-3
     network:
       layer3:
         role: Primary
         subnets:
         - cidr: 10.100.0.0/16
       topology: Layer3
   status:
     conditions:
     - lastTransitionTime: "2024-11-19T16:46:34Z"
       message: 'NetworkAttachmentDefinition has been created in following namespaces:
         [example-namespace-1, example-namespace-2, example-namespace-3]'
       reason: NetworkAttachmentDefinitionReady
       status: "True"
       type: NetworkCreated

   Copy to Clipboard

Procedure

1. Create a cluster-wide user-defined network with a Localnet topology:

   1. Create a YAML file, such as cluster-udn-localnet.yaml, to define your request for a Localnet topology as in the following example:

      apiVersion: k8s.ovn.org/v1
      kind: ClusterUserDefinedNetwork
      metadata:
        name: <cudn_name> 

      1

      
      spec:
        namespaceSelector: 

      2

      
          matchLabels: 

      3

      
            "<label_1_key>": "<label_1_value>" 

      4

      
            "<label_2_key>": "<label_2_value>" 

      5

      
        network: 

      6

      
          topology: Localnet 

      7

      
          localnet: 

      8

      
            role: Secondary 

      9

      
            physicalNetworkName: test
            ipam: {lifecycle: Persistent}
            subnets: ["192.168.0.0/16", "2001:dbb::/64"] 

      10

      Copy to Clipboard

      1

      Name of your ClusterUserDefinedNetwork (CUDN) CR.

      2

      A label query over the set of namespaces that the cluster CUDN CR applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default, openshift-*, or any other system namespaces.

      3

      Uses the matchLabels selector type, where terms are evaluated with an AND relationship.

      4 5

      In this example, the CUDN CR is deployed to namespaces that contain both <label_1_key>=<label_1_value> and <label_2_key>=<label_2_value> labels.

      6

      Describes the network configuration.

      7

      Specifying a Localnet topology type creates one logical switch that is directly bridged to one provider network.

      8

      This field specifies the localnet topology.

      9

      Specifies the role for the network configuration. Secondary is the only role specification supported for the localnet topology.

      10

      For Localnet topology types the following specifies config details for the subnet field:

      • The subnets field is optional.
      • The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
      • The subnets field accepts one or two items. For two items, they must be of a different IP family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
      • localnet subnets can be omitted. If omitted, users must configure static IP addresses for the pods. As a consequence, port security only prevents MAC spoofing. For more information, see "Configuring pods with a static IP address".

2. Apply your request by running the following command:

   $ oc create --validate=true -f <example_cluster_udn>.yaml

   Copy to Clipboard

   where:

   <example_cluster_udn>.yaml

   Is the name of your Localnet configuration file.

3. Verify that your request is successful by running the following command:

   $ oc get clusteruserdefinednetwork <cudn_name> -o yaml

   Copy to Clipboard

   where:

   <cudn_name>

   Is the name you created of your cluster-wide user-defined network.

Procedure

1. From the Administrator perspective, click Networking UserDefinedNetworks.
2. Click ClusterUserDefinedNetwork.
3. In the Name field, specify a name for the cluster-scoped UDN.
4. Specify a value in the Subnet field.
5. In the Project(s) Match Labels field, add the appropriate labels to select namespaces that the cluster UDN applies to.
6. Click Create. The cluster-scoped UDN serves as the default primary network for pods located in namespaces that contain the labels that you specified in step 5.

Procedure

1. Optional: For a UserDefinedNetwork CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network label by entering the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: <udn_namespace_name>
     labels:
       k8s.ovn.org/primary-user-defined-network: ""
   EOF

   Copy to Clipboard

2. Create a request for either a Layer2 or Layer3 topology type user-defined network:

   1. Create a YAML file, such as my-layer-two-udn.yaml, to define your request for a Layer2 topology as in the following example:

      apiVersion: k8s.ovn.org/v1
      kind: UserDefinedNetwork
      metadata:
        name: udn-1 

      1

      
        namespace: <some_custom_namespace>
      spec:
        topology: Layer2 

      2

      
        layer2: 

      3

      
          role: Primary 

      4

      
          subnets:
            - "10.0.0.0/24"
            - "2001:db8::/60" 

      5

      Copy to Clipboard

      1

      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).

      2

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes.

      3

      This field specifies the topology configuration. It can be layer2 or layer3.

      4

      Specifies a Primary or Secondary role.

      5

      For Layer2 topology types the following specifies config details for the subnet field:

      • The subnets field is optional.
      • The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
      • The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
      • Layer2 subnets can be omitted. If omitted, users must configure IP addresses for the pods. As a consequence, port security only prevents MAC spoofing.
      • The Layer2 subnets field is mandatory when the ipamLifecycle field is specified.

   2. Create a YAML file, such as my-layer-three-udn.yaml, to define your request for a Layer3 topology as in the following example:

      apiVersion: k8s.ovn.org/v1
      kind: UserDefinedNetwork
      metadata:
        name: udn-2-primary 

      1

      
        namespace: <some_custom_namespace>
      spec:
        topology: Layer3 

      2

      
        layer3: 

      3

      
          role: Primary 

      4

      
          subnets: 

      5

      
            - cidr: 10.150.0.0/16
              hostSubnet: 24
            - cidr: 2001:db8::/60
              hostSubnet: 64
      # ...

      Copy to Clipboard

      1

      Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO).

      2

      The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets.

      3

      This field specifies the topology configuration. Valid values are layer2 or layer3.

      4

      Specifies a Primary or Secondary role.

      5

      For Layer3 topology types the following specifies config details for the subnet field:

      • The subnets field is mandatory.

      • The type for the subnets field is cidr and hostSubnet:

        • cidr is equivalent to the clusterNetwork configuration settings of a cluster. The IP addresses in the CIDR are distributed to pods in the user defined network. This parameter accepts a string value.
        • hostSubnet defines the per-node subnet prefix.
        • For IPv6, only a /64 length is supported for hostSubnet.

3. Apply your request by running the following command:

   $ oc apply -f <my_layer_two_udn>.yaml

   Copy to Clipboard

   Where <my_layer_two_udn>.yaml is the name of your Layer2 or Layer3 configuration file.

4. Verify that your request is successful by running the following command:

   $ oc get userdefinednetworks udn-1 -n <some_custom_namespace> -o yaml

   Copy to Clipboard

   Where some_custom_namespace is the namespace you created for your user-defined network.

   Example output

   apiVersion: k8s.ovn.org/v1
   kind: UserDefinedNetwork
   metadata:
     creationTimestamp: "2024-08-28T17:18:47Z"
     finalizers:
     - k8s.ovn.org/user-defined-network-protection
     generation: 1
     name: udn-1
     namespace: some-custom-namespace
     resourceVersion: "53313"
     uid: f483626d-6846-48a1-b88e-6bbeb8bcde8c
   spec:
     layer2:
       role: Primary
       subnets:
       - 10.0.0.0/24
       - 2001:db8::/60
     topology: Layer2
   status:
     conditions:
     - lastTransitionTime: "2024-08-28T17:18:47Z"
       message: NetworkAttachmentDefinition has been created
       reason: NetworkAttachmentDefinitionReady
       status: "True"
       type: NetworkCreated

   Copy to Clipboard

Procedure

1. From the Administrator perspective, click Networking UserDefinedNetworks.
2. Click Create UserDefinedNetwork.
3. From the Project name list, select the namespace that you previously created.
4. Specify a value in the Subnet field.
5. Click Create. The user-defined network serves as the default primary network for pods that you create in this namespace.

Procedure

1. Create a new YAML file named, such as setsebool-container-use-devices.yaml, with the following details:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-worker-setsebool
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: setsebool.service
           contents: |
             [Unit]
             Description=Set SELinux boolean for the TAP CNI plugin
             Before=kubelet.service
   
             [Service]
             Type=oneshot
             ExecStart=/usr/sbin/setsebool container_use_devices=on
             RemainAfterExit=true
   
             [Install]
             WantedBy=multi-user.target graphical.target

   Copy to Clipboard

2. Create the new MachineConfig object by running the following command:

   $ oc apply -f setsebool-container-use-devices.yaml

   Copy to Clipboard
   Note

   Applying any changes to the MachineConfig object causes all affected nodes to gracefully reboot after the change is applied. This update can take some time to be applied.

3. Verify the change is applied by running the following command:

   $ oc get machineconfigpools

   Copy to Clipboard

   Expected output

   NAME        CONFIG                                                UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master      rendered-master-e5e0c8e8be9194e7c5a882e047379cfa      True      False      False      3              3                   3                     0                      7d2h
   worker      rendered-worker-d6c9ca107fba6cd76cdcbfcedcafa0f2      True      False      False      3              3                   3                     0                      7d

   Copy to Clipboard

   Note

   All nodes should be in the updated and ready state.

1. Edit the Pod resource definition. If you are editing an existing Pod resource, run the following command to edit its definition in the default editor. Replace <name> with the name of the Pod resource to edit.

   $ oc edit pod <name>

   Copy to Clipboard

2. In the Pod resource definition, add the k8s.v1.cni.cncf.io/networks parameter to the pod metadata mapping. The k8s.v1.cni.cncf.io/networks accepts a JSON string of a list of objects that reference the name of NetworkAttachmentDefinition custom resource (CR) names in addition to specifying additional properties.

   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]' 

   1

   Copy to Clipboard

   1

   Replace <network> with a JSON object as shown in the following examples. The single quotes are required.

3. In the following example the annotation specifies which network attachment will have the default route, using the default-route parameter.

   apiVersion: v1
   kind: Pod
   metadata:
     name: example-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
       {
         "name": "net1"
       },
       {
         "name": "net2", 

   1

   
         "default-route": ["192.0.2.1"] 

   2

   
       }]'
   spec:
     containers:
     - name: example-pod
       command: ["/bin/bash", "-c", "sleep 2000000000000"]
       image: centos/tools

   Copy to Clipboard

   1

   The name key is the name of the secondary network to associate with the pod.

   2

   The default-route key specifies a value of a gateway for traffic to be routed over if no other routing entry is present in the routing table. If more than one default-route key is specified, this will cause the pod to fail to become active.

1. Edit the CNO CR by running the following command:

   $ oc edit networks.operator.openshift.io cluster

   Copy to Clipboard