11.3. Troubleshooting node network configuration

Procedure

1. Create a Policy and apply it to your cluster. The following example creates a simple bridge on the ens01 interface:

   apiVersion: nmstate.io/v1alpha1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: ens01-bridge-testfail
   spec:
     desiredState:
       interfaces:
         - name: br1
           description: Linux bridge with the wrong port
           type: linux-bridge
           state: up
           ipv4:
             dhcp: true
             enabled: true
           bridge:
             options:
               stp:
                 enabled: false
             port:
               - name: ens01

   Copy to Clipboard Toggle word wrap

   $ oc apply -f ens01-bridge-testfail.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   nodenetworkconfigurationpolicy.nmstate.io/ens01-bridge-testfail created

   Copy to Clipboard Toggle word wrap

2. Verify the status of the Policy by running the following command:

   $ oc get nncp

   Copy to Clipboard Toggle word wrap

   The output shows that the Policy failed:

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   FailedToConfigure

   Copy to Clipboard Toggle word wrap

   However the Policy status alone does not indicate if it failed on all nodes or a subset of nodes.

3. List the Enactments to see if the Policy was successful on any of the nodes. If the Policy failed for only a subset it suggests the problem is with specific node configuration; if the Policy failed on all nodes it suggest the problem is with the Policy.

   $ oc get nnce

   Copy to Clipboard Toggle word wrap

   The output shows that the Policy failed on all nodes:

   Example output

   NAME                                   STATUS
   master-1.ens01-bridge-testfail         FailedToConfigure
   master-2.ens01-bridge-testfail         FailedToConfigure
   master-3.ens01-bridge-testfail         FailedToConfigure
   worker-1.ens01-bridge-testfail         FailedToConfigure
   worker-2.ens01-bridge-testfail         FailedToConfigure
   worker-3.ens01-bridge-testfail         FailedToConfigure

   Copy to Clipboard Toggle word wrap

4. View one of the failed Enactments and look at the traceback. The following command uses the output tool jsonpath to filter the output:

   $ oc get nnce worker-1.ens01-bridge-testfail -o jsonpath='{.status.conditions[?(@.type=="Failing")].message}'

   Copy to Clipboard Toggle word wrap

   This command returns a large traceback that has been edited for brevity:

   Example output

   error reconciling NodeNetworkConfigurationPolicy at desired state apply: , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' ''
   ...
   libnmstate.error.NmstateVerificationError:
   desired
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port:
     - name: ens01
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   current
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port: []
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   difference
   ==========
   --- desired
   +++ current
   @@ -13,8 +13,7 @@
          hello-time: 2
          max-age: 20
          priority: 32768
   -  port:
   -  - name: ens01
   +  port: []
    description: Linux bridge with the wrong port
    ipv4:
      address: []
     line 651, in _assert_interfaces_equal\n    current_state.interfaces[ifname],\nlibnmstate.error.NmstateVerificationError:

   Copy to Clipboard Toggle word wrap

   The NmstateVerificationError lists the desired Policy configuration, the current configuration of the Policy on the node, and the difference highlighting the parameters that do not match. In this example, the port is included in the difference, which suggests that the problem is the port configuration in the Policy.

5. To ensure that the Policy is configured properly, view the network configuration for one or all of the nodes by requesting the NodeNetworkState. The following command returns the network configuration for the master-1 node:

   $ oc get nns master-1 -o yaml

   Copy to Clipboard Toggle word wrap

   The output shows that the interface name on the nodes is ens1 but the failed Policy incorrectly uses ens01:

   Example output

      - ipv4:
    ...
         name: ens1
         state: up
         type: ethernet

   Copy to Clipboard Toggle word wrap

6. Correct the error by editing the existing Policy:

   $ oc edit nncp ens01-bridge-testfail

   Copy to Clipboard Toggle word wrap

   ...
             port:
               - name: ens1

   Copy to Clipboard Toggle word wrap

   Save the Policy to apply the correction.

7. Check the status of the Policy to ensure it updated successfully:

   $ oc get nncp

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   SuccessfullyConfigured

   Copy to Clipboard Toggle word wrap