Chapter 27. Kubernetes NMState

Procedure

1. Unsubscribe the Kubernetes NMState Operator from the

   Subcription

   resource by running the following command:

   $ oc delete --namespace openshift-nmstate subscription kubernetes-nmstate-operator

2. Find the

   ClusterServiceVersion

   (CSV) resource that associates with the Kubernetes NMState Operator:

   $ oc get --namespace openshift-nmstate clusterserviceversion

   Example output that lists a CSV resource

   NAME                              	  DISPLAY                   	VERSION   REPLACES     PHASE
   kubernetes-nmstate-operator.v4.18.0   Kubernetes NMState Operator   4.18.0           	   Succeeded

3. Delete the CSV resource. After you delete the file, OLM deletes certain resources, such as

   RBAC

   , that it created for the Operator.

   $ oc delete --namespace openshift-nmstate clusterserviceversion kubernetes-nmstate-operator.v4.18.0

4. Delete the

   nmstate

   CR and any associated

   Deployment

   resources by running the following commands:

   $ oc -n openshift-nmstate delete nmstate nmstate

   $ oc delete --all deployments --namespace=openshift-nmstate

5. Delete all the custom resource definition (CRD), such as

   nmstates

   , that exist in the

   nmstate.io

   namespace by running the following commands:

   $ oc delete crd nmstates.nmstate.io

   $ oc delete crd nodenetworkconfigurationenactments.nmstate.io

   $ oc delete crd nodenetworkstates.nmstate.io

   $ oc delete crd nodenetworkconfigurationpolicies.nmstate.io

6. Delete the namespace:

   $ oc delete namespace kubernetes-nmstate

Procedure

1. List all the

   NodeNetworkState

   objects in the cluster:

   $ oc get nns

2. Inspect a

   NodeNetworkState

   object to view the network on that node. The output in this example has been redacted for clarity:

   $ oc get nns node01 -o yaml

   Example output

   apiVersion: nmstate.io/v1
   kind: NodeNetworkState
   metadata:
     name: node01 

   1

   
   status:
     currentState: 

   2

   
       dns-resolver:
   ...
       interfaces:
   ...
       route-rules:
   ...
       routes:
   ...
     lastSuccessfulUpdateTime: "2020-01-31T12:14:00Z" 

   3

   1

   The name of the NodeNetworkState object is taken from the node.

   2

   The currentState contains the complete network configuration for the node, including DNS, interfaces, and routes.

   3

   Timestamp of the last successful update. This is updated periodically as long as the node is reachable and can be used to evalute the freshness of the report.

1. Edit the NNCP and remove interface details from the file. Ensure that you do not remove

   name

   ,

   state

   , and

   type

   parameters from the file.
2. Add

   state: absent

   under the

   interfaces.state

   section of the NNCP.
3. Run

   oc apply -f <nncp_file_name>

   . After the Kubernetes NMState Operator applies the node network policy to each node in your cluster, any interface that exists on each node is now marked as absent.
4. Run

   oc delete nncp

   to delete the NNCP.

Procedure

1. To confirm that a policy has been applied to the cluster, list the policies and their status:

   $ oc get nncp

2. Optional: If a policy is taking longer than expected to successfully configure, you can inspect the requested state and status conditions of a particular policy:

   $ oc get nncp <policy> -o yaml

3. Optional: If a policy is taking longer than expected to successfully configure on all nodes, you can list the status of the enactments on the cluster:

   $ oc get nnce

4. Optional: To view the configuration of a particular enactment, including any error reporting for a failed configuration:

   $ oc get nnce <node>.<policy> -o yaml

Procedure

1. Update the

   NodeNetworkConfigurationPolicy

   manifest used to create the interface. The following example removes a Linux bridge and configures the

   eth1

   NIC with DHCP to avoid losing connectivity:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: <br1-eth1-policy>
   spec:
     nodeSelector:
       node-role.kubernetes.io/worker: ""
     desiredState:
       interfaces:
       - name: br1
         type: linux-bridge
         state: absent
       - name: eth1
         type: ethernet
         state: up
         ipv4:
           dhcp: true
           enabled: true

   • metadata.name

     defines the name of the policy.

   • spec.nodeSelector

     defines the

     nodeSelector

     parameter. This parameter is optional. If you do not include the

     nodeSelector

     parameter, the policy applies to all nodes in the cluster. This example uses the

     node-role.kubernetes.io/worker: ""

     node selector to select all worker nodes in the cluster.

   • spec.desiredState.interfaces

     defines the name, type, and desired state of an interface. This example creates both Linux bridge and Ethernet networking interfaces. Setting

     state: absent

     removes the interface.

   • spec.desiredState.interfaces.ipv4

     defines

     ipv4

     settings for the interface. These settings are optional. If you do not use

     dhcp

     , you can either set a static IP or leave the interface without an IP address. Setting

     enabled: true

     enables

     ipv4

     in this example.

2. Update the policy on the node and remove the interface:

   $ oc apply -f <filename.yaml>

   Where

   <filename.yaml>

   is the filename of the policy manifest.

Procedure

1. Create a policy and apply it to your cluster. The following example creates a simple bridge on the

   ens01

   interface:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: ens01-bridge-testfail
   spec:
     desiredState:
       interfaces:
         - name: br1
           description: Linux bridge with the wrong port
           type: linux-bridge
           state: up
           ipv4:
             dhcp: true
             enabled: true
           bridge:
             options:
               stp:
                 enabled: false
             port:
               - name: ens01

   $ oc apply -f ens01-bridge-testfail.yaml

   Example output

   nodenetworkconfigurationpolicy.nmstate.io/ens01-bridge-testfail created

2. Verify the status of the policy by running the following command:

   $ oc get nncp

   The output shows that the policy failed:

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   FailedToConfigure

   However, the policy status alone does not indicate if it failed on all nodes or a subset of nodes.

3. List the node network configuration enactments to see if the policy was successful on any of the nodes. If the policy failed for only a subset of nodes, it suggests that the problem is with a specific node configuration. If the policy failed on all nodes, it suggests that the problem is with the policy.

   $ oc get nnce

   The output shows that the policy failed on all nodes:

   Example output

   NAME                                         STATUS
   control-plane-1.ens01-bridge-testfail        FailedToConfigure
   control-plane-2.ens01-bridge-testfail        FailedToConfigure
   control-plane-3.ens01-bridge-testfail        FailedToConfigure
   compute-1.ens01-bridge-testfail              FailedToConfigure
   compute-2.ens01-bridge-testfail              FailedToConfigure
   compute-3.ens01-bridge-testfail              FailedToConfigure

4. View one of the failed enactments and look at the traceback. The following command uses the output tool

   jsonpath

   to filter the output:

   $ oc get nnce compute-1.ens01-bridge-testfail -o jsonpath='{.status.conditions[?(@.type=="Failing")].message}'

   This command returns a large traceback that has been edited for brevity:

   Example output

   error reconciling NodeNetworkConfigurationPolicy at desired state apply: , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' ''
   ...
   libnmstate.error.NmstateVerificationError:
   desired
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port:
     - name: ens01
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   current
   =======
   ---
   name: br1
   type: linux-bridge
   state: up
   bridge:
     options:
       group-forward-mask: 0
       mac-ageing-time: 300
       multicast-snooping: true
       stp:
         enabled: false
         forward-delay: 15
         hello-time: 2
         max-age: 20
         priority: 32768
     port: []
   description: Linux bridge with the wrong port
   ipv4:
     address: []
     auto-dns: true
     auto-gateway: true
     auto-routes: true
     dhcp: true
     enabled: true
   ipv6:
     enabled: false
   mac-address: 01-23-45-67-89-AB
   mtu: 1500
   
   difference
   ==========
   --- desired
   +++ current
   @@ -13,8 +13,7 @@
          hello-time: 2
          max-age: 20
          priority: 32768
   -  port:
   -  - name: ens01
   +  port: []
    description: Linux bridge with the wrong port
    ipv4:
      address: []
     line 651, in _assert_interfaces_equal\n    current_state.interfaces[ifname],\nlibnmstate.error.NmstateVerificationError:

   The

   NmstateVerificationError

   lists the

   desired

   policy configuration, the

   current

   configuration of the policy on the node, and the

   difference

   highlighting the parameters that do not match. In this example, the

   port

   is included in the

   difference

   , which suggests that the problem is the port configuration in the policy.

5. To ensure that the policy is configured properly, view the network configuration for one or all of the nodes by requesting the

   NodeNetworkState

   object. The following command returns the network configuration for the

   control-plane-1

   node:

   $ oc get nns control-plane-1 -o yaml

   The output shows that the interface name on the nodes is

   ens1

   but the failed policy incorrectly uses

   ens01

   :

   Example output

      - ipv4:
   ...
         name: ens1
         state: up
         type: ethernet

6. Correct the error by editing the existing policy:

   $ oc edit nncp ens01-bridge-testfail

   ...
             port:
               - name: ens1

   Save the policy to apply the correction.

7. Check the status of the policy to ensure it updated successfully:

   $ oc get nncp

   Example output

   NAME                    STATUS
   ens01-bridge-testfail   SuccessfullyConfigured