Chapter 22. Kubernetes NMState

Procedure

1. List all the NodeNetworkState objects in the cluster:

   $ oc get nns

2. Inspect a NodeNetworkState object to view the network on that node. The output in this example has been redacted for clarity:

   $ oc get nns node01 -o yaml

   Example output

   apiVersion: nmstate.io/v1
   kind: NodeNetworkState
   metadata:
     name: node01 1
   status:
     currentState: 2
       dns-resolver:
   ...
       interfaces:
   ...
       route-rules:
   ...
       routes:
   ...
     lastSuccessfulUpdateTime: "2020-01-31T12:14:00Z" 3

   1

   The name of the NodeNetworkState object is taken from the node.

   2

   The currentState contains the complete network configuration for the node, including DNS, interfaces, and routes.

   3

   Timestamp of the last successful update. This is updated periodically as long as the node is reachable and can be used to evalute the freshness of the report.

Procedure

1. Create the NodeNetworkConfigurationPolicy manifest. The following example configures a Linux bridge on all worker nodes and configures the DNS resolver:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: br1-eth1-policy 1
   spec:
     nodeSelector: 2
       node-role.kubernetes.io/worker: "" 3
     maxUnavailable: 3 4
     desiredState:
       interfaces:
         - name: br1
           description: Linux bridge with eth1 as a port 5
           type: linux-bridge
           state: up
           ipv4:
             dhcp: true
             enabled: true
             auto-dns: false
           bridge:
             options:
               stp:
                 enabled: false
             port:
               - name: eth1
       dns-resolver: 6
         config:
           search:
           - example.com
           - example.org
           server:
           - 8.8.8.8

   1

   Name of the policy.

   2

   Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster.

   3

   This example uses the node-role.kubernetes.io/worker: "" node selector to select all worker nodes in the cluster.

   4

   Optional: Specifies the maximum number of nmstate-enabled nodes that the policy configuration can be applied to concurrently. This parameter can be set to either a percentage value (string), for example, "10%", or an absolute value (number), such as 3.

   5

   Optional: Human-readable description for the interface.

   6

   Optional: Specifies the search and server settings for the DNS server.

2. Create the node network policy:

   $ oc apply -f br1-eth1-policy.yaml 1

   1

   File name of the node network configuration policy manifest.

Procedure

1. To confirm that a policy has been applied to the cluster, list the policies and their status:

   $ oc get nncp

2. Optional: If a policy is taking longer than expected to successfully configure, you can inspect the requested state and status conditions of a particular policy:

   $ oc get nncp <policy> -o yaml

3. Optional: If a policy is taking longer than expected to successfully configure on all nodes, you can list the status of the enactments on the cluster:

   $ oc get nnce

4. Optional: To view the configuration of a particular enactment, including any error reporting for a failed configuration:

   $ oc get nnce <node>.<policy> -o yaml

Procedure

1. Update the NodeNetworkConfigurationPolicy manifest used to create the interface. The following example removes a Linux bridge and configures the eth1 NIC with DHCP to avoid losing connectivity:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: <br1-eth1-policy> 1
   spec:
     nodeSelector: 2
       node-role.kubernetes.io/worker: "" 3
     desiredState:
       interfaces:
       - name: br1
         type: linux-bridge
         state: absent 4
       - name: eth1 5
         type: ethernet 6
         state: up 7
         ipv4:
           dhcp: true 8
           enabled: true 9

   1

   Name of the policy.

   2

   Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster.

   3

   This example uses the node-role.kubernetes.io/worker: "" node selector to select all worker nodes in the cluster.

   4

   Changing the state to absent removes the interface.

   5

   The name of the interface that is to be unattached from the bridge interface.

   6

   The type of interface. This example creates an Ethernet networking interface.

   7

   The requested state for the interface.

   8

   Optional: If you do not use dhcp, you can either set a static IP or leave the interface without an IP address.

   9

   Enables ipv4 in this example.

2. Update the policy on the node and remove the interface:

   $ oc apply -f <br1-eth1-policy.yaml> 1

   1

   File name of the policy manifest.

Procedure

1. Create a policy and apply it to your cluster. The following example creates a simple bridge on the ens01 interface:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: ens01-bridge-testfail
   spec:
     desiredState:
       interfaces:
         - name: br1
           description: Linux bridge with the wrong port
           type: linux-bridge
           state: up
           ipv4:
             dhcp: true
             enabled: true
           bridge:
             options:
               stp:
                 enabled: false
             port:
               - name: ens01

   $ oc apply -f ens01-bridge-testfail.yaml

   Example output

   nodenetworkconfigurationpolicy.nmstate.io/ens01-bridge-testfail created

2. Verify the status of the policy by running the following command:

   $ oc get nncp

   The output shows that the policy failed:

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   FailedToConfigure

   However, the policy status alone does not indicate if it failed on all nodes or a subset of nodes.

3. List the node network configuration enactments to see if the policy was successful on any of the nodes. If the policy failed for only a subset of nodes, it suggests that the problem is with a specific node configuration. If the policy failed on all nodes, it suggests that the problem is with the policy.

   $ oc get nnce

   The output shows that the policy failed on all nodes:

   Example output

   NAME                                   STATUS
   control-plane-1.ens01-bridge-testfail        FailedToConfigure
   control-plane-2.ens01-bridge-testfail        FailedToConfigure
   control-plane-3.ens01-bridge-testfail        FailedToConfigure
   compute-1.ens01-bridge-testfail              FailedToConfigure
   compute-2.ens01-bridge-testfail              FailedToConfigure
   compute-3.ens01-bridge-testfail              FailedToConfigure

4. View one of the failed enactments and look at the traceback. The following command uses the output tool jsonpath to filter the output:

   $ oc get nnce compute-1.ens01-bridge-testfail -o jsonpath='{.status.conditions[?(@.type=="Failing")].message}'

   This command returns a large traceback that has been edited for brevity:

   Example output

   error reconciling NodeNetworkConfigurationPolicy at desired state apply: , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' ''
   ...
   libnmstate.error.NmstateVerificationError:
   desired
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port:
     - name: ens01
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   current
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port: []
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   difference
   ==========
   --- desired
   +++ current
   @@ -13,8 +13,7 @@
          hello-time: 2
          max-age: 20
          priority: 32768
   -  port:
   -  - name: ens01
   +  port: []
    description: Linux bridge with the wrong port
    ipv4:
      address: []
     line 651, in _assert_interfaces_equal\n    current_state.interfaces[ifname],\nlibnmstate.error.NmstateVerificationError:

   The NmstateVerificationError lists the desired policy configuration, the current configuration of the policy on the node, and the difference highlighting the parameters that do not match. In this example, the port is included in the difference, which suggests that the problem is the port configuration in the policy.

5. To ensure that the policy is configured properly, view the network configuration for one or all of the nodes by requesting the NodeNetworkState object. The following command returns the network configuration for the control-plane-1 node:

   $ oc get nns control-plane-1 -o yaml

   The output shows that the interface name on the nodes is ens1 but the failed policy incorrectly uses ens01:

   Example output

      - ipv4:
    ...
         name: ens1
         state: up
         type: ethernet

6. Correct the error by editing the existing policy:

   $ oc edit nncp ens01-bridge-testfail

   ...
             port:
               - name: ens1

   Save the policy to apply the correction.

7. Check the status of the policy to ensure it updated successfully:

   $ oc get nncp

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   SuccessfullyConfigured