7.5. Troubleshooting network issues

Procedure

1. Add a hint file to your /etc/default/nodeip-configuration file, for example:

   NODEIP_HINT=192.0.2.1

   重要
   • Do not use the exact IP address of a node as a hint, for example, 192.0.2.5. Using the exact IP address of a node causes the node using the hint IP address to fail to configure correctly.
   • The IP address in the hint file is only used to determine the correct subnet. It will not receive traffic as a result of appearing in the hint file.

2. Generate the base-64 encoded content by running the following command:

   $ echo -n 'NODEIP_HINT=192.0.2.1' | base64 -w0

   Example output

   Tk9ERUlQX0hJTlQ9MTkyLjAuMCxxxx==

3. Activate the hint by creating a machine config manifest for both master and worker roles before deploying the cluster:

   99-nodeip-hint-master.yaml

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 99-nodeip-hint-master
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,<encoded_content>
           mode: 0644
           overwrite: true
           path: /etc/default/nodeip-configuration

   where, spec.config.storage.files.contents.source.<encoded_content>

   Replace this placeholder with the base64-encoded content of the /etc/default/nodeip-configuration file, for example, Tk9ERUlQX0hJTlQ9MTkyLjAuMCxxxx==. Note that a space is not acceptable after the comma and before the encoded content.

1. Save the manifest to the directory where you store your cluster configuration, for example, ~/clusterconfigs.
2. Deploy the cluster.

1. Drains nodes in singular order based on the selected machine configuration pool.
2. Injects Ignition configuration files into each node, so that each node receives the additional br-ex1 bridge network configuration.
3. Verify that the br-ex MAC address matches the MAC address for the interface that br-ex uses for the network connection.
4. Executes the configure-ovs.sh shell script that references the new interface definition.
5. Adds br-ex and br-ex1 to the host node.
6. Uncordons the nodes.

Procedure

1. Optional: Create an interface connection that your additional bridge, br-ex1, can use by completing the following steps. The example steps show the creation of a new bond and its dependent interfaces that are all defined in a machine configuration manifest file. The additional bridge uses the MachineConfig object to form a additional bond interface.

   重要

   Do not use the Kubernetes NMState Operator or a NodeNetworkConfigurationPolicy (NNCP) manifest file to define the additional interface. Ensure that the additional interface or sub-interfaces when defining a bond interface are not used by an existing br-ex OVN Kubernetes network deployment.

   You cannot make configuration changes to the br-ex bridge or its underlying interfaces as a postinstallation task. As a workaround, use a secondary network interface connected to your host or switch.

   1. Create the following interface definition files. These files get added to a machine configuration manifest file so that host nodes can access the definition files.

      Example of the first interface definition file that is named eno1.config

      [connection]
      id=eno1
      type=ethernet
      interface-name=eno1
      master=bond1
      slave-type=bond
      autoconnect=true
      autoconnect-priority=20

      Example of the second interface definition file that is named eno2.config

      [connection]
      id=eno2
      type=ethernet
      interface-name=eno2
      master=bond1
      slave-type=bond
      autoconnect=true
      autoconnect-priority=20

      Example of the second bond interface definition file that is named bond1.config

      [connection]
      id=bond1
      type=bond
      interface-name=bond1
      autoconnect=true
      connection.autoconnect-slaves=1
      autoconnect-priority=20
      
      [bond]
      mode=802.3ad
      miimon=100
      xmit_hash_policy="layer3+4"
      
      [ipv4]
      method=auto

   2. Convert the definition files to Base64 encoded strings by running the following command:

      $ base64 <directory_path>/en01.config

      $ base64 <directory_path>/eno2.config

      $ base64 <directory_path>/bond1.config

2. Prepare the environment variables. Replace <machine_role> with the node role, such as worker, and replace <interface_name> with the name of your additional br-ex bridge name.

   $ export ROLE=<machine_role>

3. Define each interface definition in a machine configuration manifest file:

   Example of a machine configuration file with definitions added for bond1, eno1, and en02

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: ${worker}
     name: 12-${ROLE}-sec-bridge-cni
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:;base64,<base-64-encoded-contents-for-bond1.conf>
           path: /etc/NetworkManager/system-connections/bond1.nmconnection
           filesystem: root
           mode: 0600
         - contents:
             source: data:;base64,<base-64-encoded-contents-for-eno1.conf>
           path: /etc/NetworkManager/system-connections/eno1.nmconnection
           filesystem: root
           mode: 0600
         - contents:
             source: data:;base64,<base-64-encoded-contents-for-eno2.conf>
           path: /etc/NetworkManager/system-connections/eno2.nmconnection
           filesystem: root
           mode: 0600
   # ...

4. Create a machine configuration manifest file for configuring the network plugin by entering the following command in your terminal:

   $ oc create -f <machine_config_file_name>

5. Create an Open vSwitch (OVS) bridge, br-ex1, on nodes by using the OVN-Kubernetes network plugin to create an extra_bridge file`. Ensure that you save the file in the /etc/ovnk/extra_bridge path of the host. The file must state the interface name that supports the additional bridge and not the default interface that supports br-ex, which holds the primary IP address of the node.

   Example configuration for the extra_bridge file, /etc/ovnk/extra_bridge, that references a additional interface

   bond1

6. Create a machine configuration manifest file that defines the existing static interface that hosts br-ex1 on any nodes restarted on your cluster:

   Example of a machine configuration file that defines bond1 as the interface for hosting br-ex1

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: ${worker}
     name: 12-worker-extra-bridge
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/ovnk/extra_bridge
             mode: 0420
             overwrite: true
             contents:
               source: data:text/plain;charset=utf-8,bond1
             filesystem: root

7. Apply the machine-configuration to your selected nodes:

   $ oc create -f <machine_config_file_name>

8. Optional: You can override the br-ex selection logic for nodes by creating a machine configuration file that in turn creates a /var/lib/ovnk/iface_default_hint resource.

   注意

   The resource lists the name of the interface that br-ex selects for your cluster. By default, br-ex selects the primary interface for a node based on boot order and the IP address subnet in the machine network. Certain machine network configurations might require that br-ex continues to select the default interfaces or bonds for a host node.

   1. Create a machine configuration file on the host node to override the default interface.

      重要

      Only create this machine configuration file for the purposes of changing the br-ex selection logic. Using this file to change the IP addresses of existing nodes in your cluster is not supported.

      Example of a machine configuration file that overrides the default interface

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: ${worker}
        name: 12-worker-br-ex-override
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
              - path: /var/lib/ovnk/iface_default_hint
                mode: 0420
                overwrite: true
                contents:
                  source: data:text/plain;charset=utf-8,bond0
                filesystem: root

      where:

      • Ensure bond0 exists on the node before you apply the machine configuration file to the node.
   2. Before you apply the configuration to all new nodes in your cluster, reboot the host node to verify that br-ex selects the intended interface and does not conflict with the new interfaces that you defined on br-ex1.

   3. Apply the machine configuration file to all new nodes in your cluster:

      $ oc create -f <machine_config_file_name>

Verification

1. Identify the IP addresses of nodes with the exgw-ip-addresses label in your cluster to verify that the nodes use the additional bridge instead of the default bridge:

   $ oc get nodes -o json | grep --color exgw-ip-addresses

   Example output

   "k8s.ovn.org/l3-gateway-config":
      \"exgw-ip-address\":\"172.xx.xx.yy/24\",\"next-hops\":[\"xx.xx.xx.xx\"],

2. Observe that the additional bridge exists on target nodes by reviewing the network interface names on the host node:

   $ oc debug node/<node_name> -- chroot /host sh -c "ip a | grep mtu | grep br-ex"

   Example output

   Starting pod/worker-1-debug ...
   To use host binaries, run `chroot /host`
   # ...
   5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
   6: br-ex1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000

3. Optional: If you use /var/lib/ovnk/iface_default_hint, check that the MAC address of br-ex matches the MAC address of the primary selected interface:

   $ oc debug node/<node_name> -- chroot /host sh -c "ip a | grep -A1 -E 'br-ex|bond0'

   Example output that shows the primary interface for br-ex as bond0

   Starting pod/worker-1-debug ...
   To use host binaries, run `chroot /host`
   # ...
   sh-5.1# ip a | grep -A1 -E 'br-ex|bond0'
   2: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
       link/ether fa:16:3e:47:99:98 brd ff:ff:ff:ff:ff:ff
   --
   5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
       link/ether fa:16:3e:47:99:98 brd ff:ff:ff:ff:ff:ff
       inet 10.xx.xx.xx/21 brd 10.xx.xx.255 scope global dynamic noprefixroute br-ex