Chapter 11. Node networking

Procedure

1. List all the NodeNetworkState objects in the cluster:

   $ oc get nns

   Copy to Clipboard Toggle word wrap

2. Inspect a NodeNetworkState to view the network on that node. The output in this example has been redacted for clarity:

   $ oc get nns node01 -o yaml

   Copy to Clipboard Toggle word wrap

   Example output

   apiVersion: nmstate.io/v1alpha1
   kind: NodeNetworkState
   metadata:
     name: node01 

   1

   
   status:
     currentState: 

   2

   
       dns-resolver:
   ...
       interfaces:
   ...
       route-rules:
   ...
       routes:
   ...
     lastSuccessfulUpdateTime: "2020-01-31T12:14:00Z" 

   3

   Copy to Clipboard Toggle word wrap

   1

   The name of the NodeNetworkState is taken from the node.

   2

   The currentState contains the complete network configuration for the node, including DNS, interfaces, and routes.

   3

   Timestamp of the last successful update. This is updated periodically as long as the node is reachable and can be used to evalute the freshness of the report.

Procedure

1. Create the NodeNetworkConfigurationPolicy manifest. The following example configures a Linux bridge on all worker nodes:

   apiVersion: nmstate.io/v1alpha1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: <br1-eth1-policy> 

   1

   
   spec:
     nodeSelector: 

   2

   
       node-role.kubernetes.io/worker: "" 

   3

   
     desiredState:
       interfaces:
         - name: br1
           description: Linux bridge with eth1 as a port 

   4

   
           type: linux-bridge
           state: up
           ipv4:
             dhcp: true
             enabled: true
           bridge:
             options:
               stp:
                 enabled: false
             port:
               - name: eth1

   Copy to Clipboard Toggle word wrap

   1

   Name of the Policy.

   2

   Optional: If you do not include the nodeSelector, the Policy applies to all nodes in the cluster.

   3

   This example uses the node-role.kubernetes.io/worker: "" node selector to select all worker nodes in the cluster.

   4

   Optional: Human-readable description for the interface.

2. Create the Policy:

   $ oc apply -f <br1-eth1-policy.yaml> 

   1

   Copy to Clipboard Toggle word wrap

   1

   File name of the Policy manifest.

Procedure

1. To confirm that a Policy has been applied to the cluster, list the Policies and their status:

   $ oc get nncp

   Copy to Clipboard Toggle word wrap

2. Optional: If a Policy is taking longer than expected to successfully configure, you can inspect the requested state and status conditions of a particular Policy:

   $ oc get nncp <policy> -o yaml

   Copy to Clipboard Toggle word wrap

3. Optional: If a policy is taking longer than expected to successfully configure on all nodes, you can list the status of the Enactments on the cluster:

   $ oc get nnce

   Copy to Clipboard Toggle word wrap

4. Optional: To view the configuration of a particular Enactment, including any error reporting for a failed configuration:

   $ oc get nnce <node>.<policy> -o yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Update the NodeNetworkConfigurationPolicy manifest used to create the interface. The following example removes a Linux bridge:

   apiVersion: nmstate.io/v1alpha1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: <br1-eth1-policy> 

   1

   
   spec:
     nodeSelector: 

   2

   
       node-role.kubernetes.io/worker: "" 

   3

   
     desiredState:
       interfaces:
         - name: br1
           type: linux-bridge
           state: absent 

   4

   Copy to Clipboard Toggle word wrap

   1

   Name of the Policy.

   2

   Optional: If you do not include the nodeSelector, the Policy applies to all nodes in the cluster.

   3

   This example uses the node-role.kubernetes.io/worker: "" node selector to select all worker nodes in the cluster.

   4

   Changing the state to absent removes the interface.

2. Update the Policy on the node and remove the interface:

   $ oc apply -f <br1-eth1-policy.yaml> 

   1

   Copy to Clipboard Toggle word wrap

   1

   File name of the Policy manifest.

Procedure

1. Create a NodeNetworkConfigurationPolicy manifest that specifies the NIC and the desired state of up:

   apiVersion: nmstate.io/v1alpha1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: eth1
   spec:
     desiredState:
       interfaces:
       - name: eth1
         type: ethernet
         state: up
         ipv4:
           dhcp: true
           enabled: true

   Copy to Clipboard Toggle word wrap

2. Apply the manifest to the cluster:

   $ oc apply -f <eth1.yaml> 

   1

   Copy to Clipboard Toggle word wrap

   1

   File name of the Policy manifest.

Procedure

1. Create a Policy and apply it to your cluster. The following example creates a simple bridge on the ens01 interface:

   apiVersion: nmstate.io/v1alpha1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: ens01-bridge-testfail
   spec:
     desiredState:
       interfaces:
         - name: br1
           description: Linux bridge with the wrong port
           type: linux-bridge
           state: up
           ipv4:
             dhcp: true
             enabled: true
           bridge:
             options:
               stp:
                 enabled: false
             port:
               - name: ens01

   Copy to Clipboard Toggle word wrap

   $ oc apply -f ens01-bridge-testfail.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   nodenetworkconfigurationpolicy.nmstate.io/ens01-bridge-testfail created

   Copy to Clipboard Toggle word wrap

2. Verify the status of the Policy by running the following command:

   $ oc get nncp

   Copy to Clipboard Toggle word wrap

   The output shows that the Policy failed:

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   FailedToConfigure

   Copy to Clipboard Toggle word wrap

   However the Policy status alone does not indicate if it failed on all nodes or a subset of nodes.

3. List the Enactments to see if the Policy was successful on any of the nodes. If the Policy failed for only a subset it suggests the problem is with specific node configuration; if the Policy failed on all nodes it suggest the problem is with the Policy.

   $ oc get nnce

   Copy to Clipboard Toggle word wrap

   The output shows that the Policy failed on all nodes:

   Example output

   NAME                                   STATUS
   master-1.ens01-bridge-testfail         FailedToConfigure
   master-2.ens01-bridge-testfail         FailedToConfigure
   master-3.ens01-bridge-testfail         FailedToConfigure
   worker-1.ens01-bridge-testfail         FailedToConfigure
   worker-2.ens01-bridge-testfail         FailedToConfigure
   worker-3.ens01-bridge-testfail         FailedToConfigure

   Copy to Clipboard Toggle word wrap

4. View one of the failed Enactments and look at the traceback. The following command uses the output tool jsonpath to filter the output:

   $ oc get nnce worker-1.ens01-bridge-testfail -o jsonpath='{.status.conditions[?(@.type=="Failing")].message}'

   Copy to Clipboard Toggle word wrap

   This command returns a large traceback that has been edited for brevity:

   Example output

   error reconciling NodeNetworkConfigurationPolicy at desired state apply: , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' ''
   ...
   libnmstate.error.NmstateVerificationError:
   desired
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port:
     - name: ens01
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   current
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port: []
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   difference
   ==========
   --- desired
   +++ current
   @@ -13,8 +13,7 @@
          hello-time: 2
          max-age: 20
          priority: 32768
   -  port:
   -  - name: ens01
   +  port: []
    description: Linux bridge with the wrong port
    ipv4:
      address: []
     line 651, in _assert_interfaces_equal\n    current_state.interfaces[ifname],\nlibnmstate.error.NmstateVerificationError:

   Copy to Clipboard Toggle word wrap

   The NmstateVerificationError lists the desired Policy configuration, the current configuration of the Policy on the node, and the difference highlighting the parameters that do not match. In this example, the port is included in the difference, which suggests that the problem is the port configuration in the Policy.

5. To ensure that the Policy is configured properly, view the network configuration for one or all of the nodes by requesting the NodeNetworkState. The following command returns the network configuration for the master-1 node:

   $ oc get nns master-1 -o yaml

   Copy to Clipboard Toggle word wrap

   The output shows that the interface name on the nodes is ens1 but the failed Policy incorrectly uses ens01:

   Example output

      - ipv4:
    ...
         name: ens1
         state: up
         type: ethernet

   Copy to Clipboard Toggle word wrap

6. Correct the error by editing the existing Policy:

   $ oc edit nncp ens01-bridge-testfail

   Copy to Clipboard Toggle word wrap

   ...
             port:
               - name: ens1

   Copy to Clipboard Toggle word wrap

   Save the Policy to apply the correction.

7. Check the status of the Policy to ensure it updated successfully:

   $ oc get nncp

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   SuccessfullyConfigured

   Copy to Clipboard Toggle word wrap