Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.
Chapter 28. Kubernetes NMState
28.1. About the Kubernetes NMState Operator Link kopierenLink in die Zwischenablage kopiert!
The Kubernetes NMState Operator provides a Kubernetes API for performing state-driven network configuration across the OpenShift Container Platform cluster’s nodes with NMState. The Kubernetes NMState Operator provides users with functionality to configure various network interface types, DNS, and routing on cluster nodes. Additionally, the daemons on the cluster nodes periodically report on the state of each node’s network interfaces to the API server.
Red Hat supports the Kubernetes NMState Operator in production environments on bare-metal, IBM Power®, IBM Z®, IBM® LinuxONE, VMware vSphere, and OpenStack installations.
Before you can use NMState with OpenShift Container Platform, you must install the Kubernetes NMState Operator.
The Kubernetes NMState Operator updates the network configuration of a secondary NIC. It cannot update the network configuration of the primary NIC or the
br-ex
OpenShift Container Platform uses nmstate to report on and configure the state of the node network. This makes it possible to modify the network policy configuration, such as by creating a Linux bridge on all nodes, by applying a single configuration manifest to the cluster.
Node networking is monitored and updated by the following objects:
NodeNetworkState- Reports the state of the network on that node.
NodeNetworkConfigurationPolicy-
Describes the requested network configuration on nodes. You update the node network configuration, including adding and removing interfaces, by applying a
NodeNetworkConfigurationPolicymanifest to the cluster. NodeNetworkConfigurationEnactment- Reports the network policies enacted upon each node.
28.1.1. Installing the Kubernetes NMState Operator Link kopierenLink in die Zwischenablage kopiert!
You can install the Kubernetes NMState Operator by using the web console or the CLI.
28.1.1.1. Installing the Kubernetes NMState Operator by using the web console Link kopierenLink in die Zwischenablage kopiert!
You can install the Kubernetes NMState Operator by using the web console. After it is installed, the Operator can deploy the NMState State Controller as a daemon set across all of the cluster nodes.
Prerequisites
-
You are logged in as a user with privileges.
cluster-admin
Procedure
-
Select Operators
OperatorHub. -
In the search field below All Items, enter and click Enter to search for the Kubernetes NMState Operator.
nmstate - Click on the Kubernetes NMState Operator search result.
- Click on Install to open the Install Operator window.
- Click Install to install the Operator.
- After the Operator finishes installing, click View Operator.
-
Under Provided APIs, click Create Instance to open the dialog box for creating an instance of .
kubernetes-nmstate In the Name field of the dialog box, ensure the name of the instance is
nmstate.NoteThe name restriction is a known issue. The instance is a singleton for the entire cluster.
- Accept the default settings and click Create to create the instance.
Summary
Once complete, the Operator has deployed the NMState State Controller as a daemon set across all of the cluster nodes.
28.1.1.2. Installing the Kubernetes NMState Operator by using the CLI Link kopierenLink in die Zwischenablage kopiert!
You can install the Kubernetes NMState Operator by using the OpenShift CLI (
oc)
Prerequisites
-
You have installed the OpenShift CLI ().
oc -
You are logged in as a user with privileges.
cluster-admin
Procedure
Create the
Operator namespace:nmstate$ cat << EOF | oc apply -f - apiVersion: v1 kind: Namespace metadata: name: openshift-nmstate spec: finalizers: - kubernetes EOFCreate the
:OperatorGroup$ cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-nmstate namespace: openshift-nmstate spec: targetNamespaces: - openshift-nmstate EOFSubscribe to the
Operator:nmstate$ cat << EOF| oc apply -f - apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: kubernetes-nmstate-operator namespace: openshift-nmstate spec: channel: stable installPlanApproval: Automatic name: kubernetes-nmstate-operator source: redhat-operators sourceNamespace: openshift-marketplace EOFConfirm the
(CSV) status for theClusterServiceVersionOperator deployment equalsnmstate:Succeeded$ oc get clusterserviceversion -n openshift-nmstate \ -o custom-columns=Name:.metadata.name,Phase:.status.phaseExample output
Name Phase kubernetes-nmstate-operator.4.14.0-202210210157 SucceededCreate an instance of the
Operator:nmstate$ cat << EOF | oc apply -f - apiVersion: nmstate.io/v1 kind: NMState metadata: name: nmstate EOFVerify that all pods for the NMState Operator are in a
state:Running$ oc get pod -n openshift-nmstateExample output
Name Ready Status Restarts Age pod/nmstate-handler-wn55p 1/1 Running 0 77s pod/nmstate-operator-f6bb869b6-v5m92 1/1 Running 0 4m51s ...
28.1.2. Uninstalling the Kubernetes NMState Operator Link kopierenLink in die Zwischenablage kopiert!
You can use the Operator Lifecycle Manager (OLM) to uninstall the Kubernetes NMState Operator, but by design OLM does not delete any associated custom resource definitions (CRDs), custom resources (CRs), or API Services.
Before you uninstall the Kubernetes NMState Operator from the
Subcription
If you need to reinstall the Kubernetes NMState Operator, see "Installing the Kubernetes NMState Operator by using the CLI" or "Installing the Kubernetes NMState Operator by using the web console".
Prerequisites
-
You have installed the OpenShift CLI ().
oc -
You have installed the CLI tool.
jq -
You are logged in as a user with privileges.
cluster-admin
Procedure
Unsubscribe the Kubernetes NMState Operator from the
resource by running the following command:Subcription$ oc delete --namespace openshift-nmstate subscription kubernetes-nmstate-operatorFind the
(CSV) resource that associates with the Kubernetes NMState Operator:ClusterServiceVersion$ oc get --namespace openshift-nmstate clusterserviceversionExample output that lists a CSV resource
NAME DISPLAY VERSION REPLACES PHASE kubernetes-nmstate-operator.v4.18.0 Kubernetes NMState Operator 4.18.0 SucceededDelete the CSV resource. After you delete the file, OLM deletes certain resources, such as
, that it created for the Operator.RBAC$ oc delete --namespace openshift-nmstate clusterserviceversion kubernetes-nmstate-operator.v4.18.0Delete the
CR and any associatednmstateresources by running the following commands:Deployment$ oc -n openshift-nmstate delete nmstate nmstate$ oc delete --all deployments --namespace=openshift-nmstateAfter you deleted the
CR, remove thenmstateconsole plugin name from thenmstate-console-pluginCR.console.operator.openshift.io/clusterStore the position of the
entry that exists among the list of enable plugins by running the following command. The following command uses thenmstate-console-pluginCLI tool to store the index of the entry in an environment variable namedjq:INDEXINDEX=$(oc get console.operator.openshift.io cluster -o json | jq -r '.spec.plugins | to_entries[] | select(.value == "nmstate-console-plugin") | .key')Remove the
entry from thenmstate-console-pluginCR by running the following patch command:console.operator.openshift.io/cluster$ oc patch console.operator.openshift.io cluster --type=json -p "[{\"op\": \"remove\", \"path\": \"/spec/plugins/$INDEX\"}]"1 - 1
INDEXis an auxiliary variable. You can specify a different name for this variable.
Delete all the custom resource definitions (CRDs), such as
, by running the following commands:nmstates.nmstate.io$ oc delete crd nmstates.nmstate.io$ oc delete crd nodenetworkconfigurationenactments.nmstate.io$ oc delete crd nodenetworkstates.nmstate.io$ oc delete crd nodenetworkconfigurationpolicies.nmstate.ioDelete the namespace:
$ oc delete namespace kubernetes-nmstate
28.2. Observing and updating the node network state and configuration Link kopierenLink in die Zwischenablage kopiert!
For more information about how to install the NMState Operator, see Kubernetes NMState Operator.
28.2.1. Viewing the network state of a node by using the CLI Link kopierenLink in die Zwischenablage kopiert!
Node network state is the network configuration for all nodes in the cluster. A
NodeNetworkState
Procedure
List all the
objects in the cluster:NodeNetworkState$ oc get nnsInspect a
object to view the network on that node. The output in this example has been redacted for clarity:NodeNetworkState$ oc get nns node01 -o yamlExample output
apiVersion: nmstate.io/v1 kind: NodeNetworkState metadata: name: node01 status: currentState: dns-resolver: # ... interfaces: # ... route-rules: # ... routes: # ... lastSuccessfulUpdateTime: "2020-01-31T12:14:00Z"metadata.name-
The name of the
NodeNetworkStateobject is taken from the node. status.currentState-
The
currentStatecontains the complete network configuration for the node, including DNS, interfaces, and routes. status.lastSuccessfulUpdateTime- Timestamp of the last successful update. This is updated periodically if the node is reachable and can be used to evaluate the freshness of the report.
28.2.2. Viewing the network state of a node from the web console Link kopierenLink in die Zwischenablage kopiert!
As an administrator, you can use the OpenShift Container Platform web console to observe
NodeNetworkState
Procedure
Navigate to Networking
NodeNetworkState. In the NodeNetworkState page, you can view the list of
resources and the corresponding interfaces that are created on the nodes. You can use Filter based on Interface state, Interface type, and IP, or the search bar based on criteria Name or Label, to narrow down the displayedNodeNetworkStateresources.NodeNetworkState-
To access the detailed information about resource, click the
NodeNetworkStateresource name listed in the Name column .NodeNetworkState -
to expand and view the Network Details section for the resource, click the > icon . Alternatively, you can click on each interface type under the Network interface column to view the network details.
NodeNetworkState
28.2.3. The NodeNetworkConfigurationPolicy manifest file Link kopierenLink in die Zwischenablage kopiert!
A
NodeNetworkConfigurationPolicy
If you want to apply multiple NNCP CRs to a node, you must create the NNCPs in a logical order that is based on the alphanumeric sorting of the policy names. The Kubernetes NMState Operator continuously checks for a newly created NNCP CR so that the Operator can instantly apply the CR to node. Consider the following logical order issue example:
-
You create NNCP 1 for defining the bridge interface that listens on a VLAN port, such as .
eth1.1000 -
You create NNCP 2 for defining the VLAN interface and specify the port for this interface, such as .
eth1.1000 - You apply NNCP 1 before you apply NNCP 2 to the node.
The node experiences a node connectivity issue because port
eth1.1000
After you apply a node network policy to a node, the Kubernetes NMState Operator configures the networking configuration for nodes according to the node network policy details.
You can create an NNCP by using either the OpenShift CLI (
oc
Before you create an NNCP, ensure that you read the "Example policy configurations for different interfaces" document.
If you want to delete an NNCP, you can use the
oc delete nncp
Deleting the node network policy that added an interface to a node does not change the configuration of the policy on the node. Similarly, removing an interface does not delete the policy, because the Kubernetes NMState Operator re-adds the removed interface whenever a pod or a node is restarted.
To effectively delete the NNCP, the node network policy, and any interfaces would typically require the following actions:
-
Edit the NNCP and remove interface details from the file. Ensure that you do not remove ,
name, andstateparameters from the file.type -
Add under the
state: absentsection of the NNCP.interfaces.state -
Run . After the Kubernetes NMState Operator applies the node network policy to each node in your cluster, any interface that exists on each node is now marked as absent.
oc apply -f <nncp_file_name> -
Run to delete the NNCP.
oc delete nncp
28.2.5. Managing policy from the web console Link kopierenLink in die Zwischenablage kopiert!
You can update the node network configuration, such as adding or removing interfaces from nodes, by applying
NodeNetworkConfigurationPolicy
28.2.5.1. Monitoring the policy status Link kopierenLink in die Zwischenablage kopiert!
You can monitor the policy status from the NodeNetworkConfigurationPolicy page. This page displays all the policies created in the cluster in a tabular format, with the following columns:
- Name
- The name of the policy created.
- Matched nodes
- The count of nodes where the policies are applied. This could be either a subset of nodes based on the node selector or all the nodes on the cluster.
- Node network state
- The enactment state of the matched nodes. You can click on the enactment state and view detailed information on the status.
To find the desired policy, you can filter the list either based on enactment state by using the Filter option, or by using the search option.
28.2.5.2. Creating a policy Link kopierenLink in die Zwischenablage kopiert!
You can create a policy by using either a form or YAML in the web console.
Procedure
-
Navigate to Networking
NodeNetworkConfigurationPolicy. In the NodeNetworkConfigurationPolicy page, click Create, and select From Form option.
In case there are no existing policies, you can alternatively click Create NodeNetworkConfigurationPolicy to createa policy using form.
NoteTo create policy using YAML, click Create, and select With YAML option. The following steps are applicable to create a policy only by using form.
- Optional: Check the Apply this NodeNetworkConfigurationPolicy only to specific subsets of nodes using the node selector checkbox to specify the nodes where the policy must be applied.
- Enter the policy name in the Policy name field.
- Optional: Enter the description of the policy in the Description field.
Optional: In the Policy Interface(s) section, a bridge interface is added by default with preset values in editable fields. Edit the values by executing the following steps:
- Enter the name of the interface in Interface name field.
- Select the network state from Network state dropdown. The default selected value is Up.
Select the type of interface from Type dropdown. The available values are Bridge, Bonding, and Ethernet. The default selected value is Bridge.
NoteAddition of a VLAN interface by using the form is not supported. To add a VLAN interface, you must use YAML to create the policy. Once added, you cannot edit the policy by using form.
Optional: In the IP configuration section, check IPv4 checkbox to assign an IPv4 address to the interface, and configure the IP address assignment details:
- Click IP address to configure the interface with a static IP address, or DHCP to auto-assign an IP address.
If you have selected IP address option, enter the IPv4 address in IPV4 address field, and enter the prefix length in Prefix length field.
If you have selected DHCP option, uncheck the options that you want to disable. The available options are Auto-DNS, Auto-routes, and Auto-gateway. All the options are selected by default.
- Optional: Enter the port number in Port field.
- Optional: Check the checkbox Enable STP to enable STP.
- Optional: To add an interface to the policy, click Add another interface to the policy.
- Optional: To remove an interface from the policy, click icon next to the interface.
NoteAlternatively, you can click Edit YAML on the top of the page to continue editing the form using YAML.
- Click Create to complete policy creation.
28.2.5.3. Updating the policy Link kopierenLink in die Zwischenablage kopiert!
28.2.5.3.1. Updating the policy by using form Link kopierenLink in die Zwischenablage kopiert!
Procedure
-
Navigate to Networking
NodeNetworkConfigurationPolicy. -
In the NodeNetworkConfigurationPolicy page, click the
icon placed next to the policy you want to edit, and click Edit.
- Edit the fields that you want to update.
- Click Save.
Addition of a VLAN interface using the form is not supported. To add a VLAN interface, you must use YAML to create the policy. Once added, you cannot edit the policy using form.
28.2.5.3.2. Updating the policy by using YAML Link kopierenLink in die Zwischenablage kopiert!
Procedure
-
Navigate to Networking
NodeNetworkConfigurationPolicy. - In the NodeNetworkConfigurationPolicy page, click the policy name under the Name column for the policy you want to edit.
- Click the YAML tab, and edit the YAML.
- Click Save.
28.2.5.4. Deleting the policy Link kopierenLink in die Zwischenablage kopiert!
Procedure
-
Navigate to Networking
NodeNetworkConfigurationPolicy. -
In the NodeNetworkConfigurationPolicy page, click the
icon placed next to the policy you want to delete, and click Delete.
- In the pop-up window, enter the policy name to confirm deletion, and click Delete.
28.2.6. Managing policy by using the CLI Link kopierenLink in die Zwischenablage kopiert!
28.2.6.1. Creating an interface on nodes Link kopierenLink in die Zwischenablage kopiert!
Create an interface on nodes in the cluster by applying a
NodeNetworkConfigurationPolicy
By default, the manifest applies to all nodes in the cluster. To add the interface to specific nodes, add the
spec: nodeSelector
<key>:<value>
You can configure multiple nmstate-enabled nodes concurrently. The configuration applies to 50% of the nodes in parallel. This strategy prevents the entire cluster from being unavailable if the network connection fails. To apply the policy configuration in parallel to a specific portion of the cluster, use the
maxUnavailable
NodeNetworkConfigurationPolicy
If you have two nodes and you apply an NNCP manifest with the
maxUnavailable
50%
maxUnavailable
50%
Procedure
Create the
manifest. The following example configures a Linux bridge on all worker nodes and configures the DNS resolver:NodeNetworkConfigurationPolicyapiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: br1-eth1-policy1 spec: nodeSelector:2 node-role.kubernetes.io/worker: ""3 maxUnavailable: 34 desiredState: interfaces: - name: br1 description: Linux bridge with eth1 as a port5 type: linux-bridge state: up ipv4: dhcp: true enabled: true auto-dns: false bridge: options: stp: enabled: false port: - name: eth1 dns-resolver:6 config: search: - example.com - example.org server: - 8.8.8.8- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example uses the
node-role.kubernetes.io/worker: ""node selector to select all worker nodes in the cluster. - 4
- Optional: Specifies the maximum number of nmstate-enabled nodes that the policy configuration can be applied to concurrently. This parameter can be set to either a percentage value (string), for example,
"10%", or an absolute value (number), such as3. - 5
- Optional: Human-readable description for the interface.
- 6
- Optional: Specifies the search and server settings for the DNS server.
Create the node network policy:
$ oc apply -f br1-eth1-policy.yaml1 - 1
- File name of the node network configuration policy manifest.
28.2.8. Example policy configurations for different interfaces Link kopierenLink in die Zwischenablage kopiert!
Before you read the different example
NodeNetworkConfigurationPolicy
- If you want to apply multiple NNCP CRs to a node, you must create the NNCPs in a logical order that is based on the alphanumeric sorting of the policy names. The Kubernetes NMState Operator continuously checks for a newly created NNCP CR so that the Operator can instantly apply the CR to node.
-
When you need to apply a policy to many nodes but you only want to create a single NNCP for all the nodes, the Kubernetes NMState Operator applies the policy to each node in sequence. You can set the speed and coverage of policy application for target nodes with the parameter in the cluster’s configuration file. By setting a lower percentage value for the parameter, you can reduce the risk of a cluster-wide outage if the outage impacts the small percentage of nodes that are receiving the policy application.
maxUnavailable -
If you set the parameter to
maxUnavailablein two NNCP manifests, the policy configuration coverage applies to 100% of the nodes in your cluster.50% - When a node restarts, the Kubernetes NMState Operator cannot control the order to which it applies policies to nodes. The Kubernetes NMState Operator might apply interdependent policies in a sequence that results in a degraded network object.
- Consider specifying all related network configurations in a single policy.
28.2.8.1. Example: Linux bridge interface node network configuration policy Link kopierenLink in die Zwischenablage kopiert!
Create a Linux bridge interface on nodes in the cluster by applying a
NodeNetworkConfigurationPolicy
The following YAML file is an example of a manifest for a Linux bridge interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-eth1-policy
spec:
nodeSelector:
kubernetes.io/hostname: <node01>
desiredState:
interfaces:
- name: br1
description: Linux bridge with eth1 as a port
type: linux-bridge
state: up
ipv4:
dhcp: true
enabled: true
bridge:
options:
stp:
enabled: false
port:
- name: eth1
- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example uses a
hostnamenode selector. - 4
- Name of the interface.
- 5
- Optional: Human-readable description of the interface.
- 6
- The type of interface. This example creates a bridge.
- 7
- The requested state for the interface after creation.
- 8
- Optional: If you do not use
dhcp, you can either set a static IP or leave the interface without an IP address. - 9
- Enables
ipv4in this example. - 10
- Disables
stpin this example. - 11
- The node NIC to which the bridge attaches.
28.2.8.2. Example: VLAN interface node network configuration policy Link kopierenLink in die Zwischenablage kopiert!
Create a VLAN interface on nodes in the cluster by applying a
NodeNetworkConfigurationPolicy
Define all related configurations for the VLAN interface of a node in a single
NodeNetworkConfigurationPolicy
NodeNetworkConfigurationPolicy
When a node restarts, the Kubernetes NMState Operator cannot control the order in which policies are applied. Therefore, if you use separate policies for related network configurations, the Kubernetes NMState Operator might apply these policies in a sequence that results in a degraded network object.
The following YAML file is an example of a manifest for a VLAN interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: vlan-eth1-policy
spec:
nodeSelector:
kubernetes.io/hostname: <node01>
desiredState:
interfaces:
- name: eth1.102
description: VLAN using eth1
type: vlan
state: up
vlan:
base-iface: eth1
id: 102
- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example uses a
hostnamenode selector. - 4
- Name of the interface. When deploying on bare metal, only the
<interface_name>.<vlan_number>VLAN format is supported. - 5
- Optional: Human-readable description of the interface.
- 6
- The type of interface. This example creates a VLAN.
- 7
- The requested state for the interface after creation.
- 8
- The node NIC to which the VLAN is attached.
- 9
- The VLAN tag.
28.2.8.3. Example: Node network configuration policy for virtual functions (Technology Preview) Link kopierenLink in die Zwischenablage kopiert!
Update host network settings for Single Root I/O Virtualization (SR-IOV) network virtual functions (VF) in an existing cluster by applying a
NodeNetworkConfigurationPolicy
Updating host network settings for SR-IOV network VFs is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You can apply a
NodeNetworkConfigurationPolicy
- Configure QoS or MTU host network settings for VFs to optimize performance.
- Add, remove, or update VFs for a network interface.
- Manage VF bonding configurations.
To update host network settings for SR-IOV VFs by using NMState on physical functions that are also managed through the SR-IOV Network Operator, you must set the
externallyManaged
SriovNetworkNodePolicy
true
The following YAML file is an example of a manifest that defines QoS policies for a VF. This file includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: qos
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""
desiredState:
interfaces:
- name: ens1f0
description: Change QOS on VF0
type: ethernet
state: up
ethernet:
sr-iov:
total-vfs: 3
vfs:
- id: 0
max-tx-rate: 200
- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example applies to all nodes with the
workerrole. - 4
- Name of the physical function (PF) network interface.
- 5
- Optional: Human-readable description of the interface.
- 6
- The type of interface.
- 7
- The requested state for the interface after configuration.
- 8
- The total number of VFs.
- 9
- Identifies the VF with an ID of
0. - 10
- Sets a maximum transmission rate, in Mbps, for the VF. This sample value sets a rate of 200 Mbps.
The following YAML file is an example of a manifest that creates a VLAN interface on top of a VF and adds it to a bonded network interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: addvf
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""
maxUnavailable: 3
desiredState:
interfaces:
- name: ens1f0v1
type: ethernet
state: up
- name: ens1f0v1.477
type: vlan
state: up
vlan:
base-iface: ens1f0v1
id: 477
- name: bond0
description: Add vf
type: bond
state: up
link-aggregation:
mode: active-backup
options:
primary: ens1f1v0.477
port:
- ens1f1v0.477
- ens1f0v0.477
- ens1f0v1.477
- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example applies to all nodes with the
workerrole. - 4
- Name of the VF network interface.
- 5
- Name of the VLAN network interface.
- 6
- The VF network interface to which the VLAN interface is attached.
- 7
- Name of the bonding network interface.
- 8
- Optional: Human-readable description of the interface.
- 9
- The type of interface.
- 10
- The requested state for the interface after configuration.
- 11
- The bonding policy for the bond.
- 12
- The primary attached bonding port.
- 13
- The ports for the bonded network interface.
- 14
- In this example, this VLAN network interface is added as an additional interface to the bonded network interface.
28.2.8.4. Example: Bond interface node network configuration policy Link kopierenLink in die Zwischenablage kopiert!
Create a bond interface on nodes in the cluster by applying a
NodeNetworkConfigurationPolicy
OpenShift Container Platform only supports the following bond modes:
-
active-backup
-
balance-xor
-
802.3ad
Other bond modes are not supported.
The
balance-xor
802.3ad
active-backup
The following YAML file is an example of a manifest for a bond interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bond0-eth1-eth2-policy
spec:
nodeSelector:
kubernetes.io/hostname: <node01>
desiredState:
interfaces:
- name: bond0
description: Bond with ports eth1 and eth2
type: bond
state: up
ipv4:
dhcp: true
enabled: true
link-aggregation:
mode: active-backup
options:
miimon: '140'
port:
- eth1
- eth2
mtu: 1450
- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example uses a
hostnamenode selector. - 4
- Name of the interface.
- 5
- Optional: Human-readable description of the interface.
- 6
- The type of interface. This example creates a bond.
- 7
- The requested state for the interface after creation.
- 8
- Optional: If you do not use
dhcp, you can either set a static IP or leave the interface without an IP address. - 9
- Enables
ipv4in this example. - 10
- The driver mode for the bond. This example uses
active backup. - 11
- Optional: This example uses miimon to inspect the bond link every 140ms.
- 12
- The subordinate node NICs in the bond.
- 13
- Optional: The maximum transmission unit (MTU) for the bond. If not specified, this value is set to
1500by default.
28.2.8.5. Example: Ethernet interface node network configuration policy Link kopierenLink in die Zwischenablage kopiert!
Configure an Ethernet interface on nodes in the cluster by applying a
NodeNetworkConfigurationPolicy
The following YAML file is an example of a manifest for an Ethernet interface. It includes sample values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: eth1-policy
spec:
nodeSelector:
kubernetes.io/hostname: <node01>
desiredState:
interfaces:
- name: eth1
description: Configuring eth1 on node01
type: ethernet
state: up
ipv4:
dhcp: true
enabled: true
- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example uses a
hostnamenode selector. - 4
- Name of the interface.
- 5
- Optional: Human-readable description of the interface.
- 6
- The type of interface. This example creates an Ethernet networking interface.
- 7
- The requested state for the interface after creation.
- 8
- Optional: If you do not use
dhcp, you can either set a static IP or leave the interface without an IP address. - 9
- Enables
ipv4in this example.
28.2.8.6. Example: Multiple interfaces in the same node network configuration policy Link kopierenLink in die Zwischenablage kopiert!
You can create multiple interfaces in the same node network configuration policy. These interfaces can reference each other, allowing you to build and deploy a network configuration by using a single policy manifest.
The following example YAML file creates a bond that is named
bond10
bond10.103
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bond-vlan
spec:
nodeSelector:
kubernetes.io/hostname: <node01>
desiredState:
interfaces:
- name: bond10
description: Bonding eth2 and eth3
type: bond
state: up
link-aggregation:
mode: balance-xor
options:
miimon: '140'
port:
- eth2
- eth3
- name: bond10.103
description: vlan using bond10
type: vlan
state: up
vlan:
base-iface: bond10
id: 103
ipv4:
dhcp: true
enabled: true
- 1
- Name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. - 3
- This example uses
hostnamenode selector. - 4 11
- Name of the interface.
- 5 12
- Optional: Human-readable description of the interface.
- 6 13
- The type of interface.
- 7 14
- The requested state for the interface after creation.
- 8
- The driver mode for the bond.
- 9
- Optional: This example uses miimon to inspect the bond link every 140ms.
- 10
- The subordinate node NICs in the bond.
- 15
- The node NIC to which the VLAN is attached.
- 16
- The VLAN tag.
- 17
- Optional: If you do not use dhcp, you can either set a static IP or leave the interface without an IP address.
- 18
- Enables ipv4 in this example.
28.2.8.7. Example: Network interface with a VRF instance node network configuration policy Link kopierenLink in die Zwischenablage kopiert!
Associate a Virtual Routing and Forwarding (VRF) instance with a network interface by applying a
NodeNetworkConfigurationPolicy
Associating a VRF instance with a network interface is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By associating a VRF instance with a network interface, you can support traffic isolation, independent routing decisions, and the logical separation of network resources.
When configuring Virtual Route Forwarding (VRF), you must change the VRF value to a table ID lower than
1000
1000
In a bare-metal environment, you can announce load balancer services through interfaces belonging to a VRF instance by using MetalLB. For more information, see the Additional resources section.
The following YAML file is an example of associating a VRF instance to a network interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: vrfpolicy
spec:
nodeSelector:
vrf: "true"
maxUnavailable: 3
desiredState:
interfaces:
- name: ens4vrf
type: vrf
state: up
vrf:
port:
- ens4
route-table-id: 2
28.2.9. Capturing the static IP of a NIC attached to a bridge Link kopierenLink in die Zwischenablage kopiert!
Capturing the static IP of a NIC is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
28.2.9.1. Example: Linux bridge interface node network configuration policy to inherit static IP address from the NIC attached to the bridge Link kopierenLink in die Zwischenablage kopiert!
Create a Linux bridge interface on nodes in the cluster and transfer the static IP configuration of the NIC to the bridge by applying a single
NodeNetworkConfigurationPolicy
The following YAML file is an example of a manifest for a Linux bridge interface. It includes sample values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-eth1-copy-ipv4-policy
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""
capture:
eth1-nic: interfaces.name=="eth1"
eth1-routes: routes.running.next-hop-interface=="eth1"
br1-routes: capture.eth1-routes | routes.running.next-hop-interface := "br1"
desiredState:
interfaces:
- name: br1
description: Linux bridge with eth1 as a port
type: linux-bridge
state: up
ipv4: "{{ capture.eth1-nic.interfaces.0.ipv4 }}"
bridge:
options:
stp:
enabled: false
port:
- name: eth1
routes:
config: "{{ capture.br1-routes.routes.running }}"
- 1
- The name of the policy.
- 2
- Optional: If you do not include the
nodeSelectorparameter, the policy applies to all nodes in the cluster. This example uses thenode-role.kubernetes.io/worker: ""node selector to select all worker nodes in the cluster. - 3
- The reference to the node NIC to which the bridge attaches.
- 4
- The type of interface. This example creates a bridge.
- 5
- The IP address of the bridge interface. This value matches the IP address of the NIC which is referenced by the
spec.capture.eth1-nicentry. - 6
- The node NIC to which the bridge attaches.
28.2.10. Examples: IP management Link kopierenLink in die Zwischenablage kopiert!
The following example configuration snippets show different methods of IP management.
These examples use the
ethernet
28.2.10.1. Static Link kopierenLink in die Zwischenablage kopiert!
The following snippet statically configures an IP address on the Ethernet interface:
# ...
interfaces:
- name: eth1
description: static IP on eth1
type: ethernet
state: up
ipv4:
dhcp: false
address:
- ip: 192.168.122.250
prefix-length: 24
enabled: true
# ...
- 1
- Replace this value with the static IP address for the interface.
28.2.10.2. No IP address Link kopierenLink in die Zwischenablage kopiert!
The following snippet ensures that the interface has no IP address:
# ...
interfaces:
- name: eth1
description: No IP on eth1
type: ethernet
state: up
ipv4:
enabled: false
# ...
Always set the
state
up
ipv4.enabled
ipv6.enabled
false
state: down
28.2.10.3. Dynamic host configuration Link kopierenLink in die Zwischenablage kopiert!
The following snippet configures an Ethernet interface that uses a dynamic IP address, gateway address, and DNS:
# ...
interfaces:
- name: eth1
description: DHCP on eth1
type: ethernet
state: up
ipv4:
dhcp: true
enabled: true
# ...
The following snippet configures an Ethernet interface that uses a dynamic IP address but does not use a dynamic gateway address or DNS:
# ...
interfaces:
- name: eth1
description: DHCP without gateway or DNS on eth1
type: ethernet
state: up
ipv4:
dhcp: true
auto-gateway: false
auto-dns: false
enabled: true
# ...
28.2.10.4. DNS Link kopierenLink in die Zwischenablage kopiert!
By default, the
nmstate
Setting a DNS configuration is comparable to modifying the
/etc/resolv.conf
To define a DNS configuration for a network interface, you must initially specify the
dns-resolver
oc apply -f <nncp_file_name>
The following example shows a default situation that stores DNS values globally:
Configure a static DNS without a network interface. Note that when updating the
file on a host node, you do not need to specify an interface, IPv4 or IPv6, in the/etc/resolv.conf(NNCP) manifest.NodeNetworkConfigurationPolicyExample of a DNS configuration for a network interface that globally stores DNS values
apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: worker-0-dns-testing spec: nodeSelector: kubernetes.io/hostname: <target_node> desiredState: dns-resolver: config: server: - 2001:db8:f::1 - 192.0.2.251 search: - example.com - example.org # ...ImportantYou can specify DNS options under the
section of your NNCP file as demonstrated in the following example:dns-resolver.config# ... desiredState: dns-resolver: config: options: - timeout:2 - attempts:3 # ...If you want to remove the DNS options from your network interface, apply the following configuration to your NNCP and then run the
command:oc apply -f <nncp_file_name># ... dns-resolver: config: {} interfaces: [] # ...
The following examples show situations that require configuring a network interface to store DNS values:
If you want to rank a static DNS name server over a dynamic DNS name server, define the interface that runs either the Dynamic Host Configuration Protocol (DHCP) or the IPv6 Autoconfiguration (
) mechanism in the network interface YAML configuration file.autoconfExample configuration that adds
192.0.2.1to DNS name servers retrieved from the DHCPv4 network protocol# ... dns-resolver: config: server: - 192.0.2.1 interfaces: - name: eth1 type: ethernet state: up ipv4: enabled: true dhcp: true auto-dns: true # ...If you need to configure a network interface to store DNS values instead of adopting the default method, which uses the
API to store DNS values globally, you can set static DNS values and static IP addresses in the network interface YAML file.nmstateImportantStoring DNS values at the network interface level might cause name resolution issues after you attach the interface to network components, such as an Open vSwitch (OVS) bridge, a Linux bridge, or a bond.
Example configuration that stores DNS values at the interface level
# ... dns-resolver: config: server: - 2001:db8:1::d1 - 2001:db8:1::d2 - 192.0.2.1 search: - example.com - example.org interfaces: - name: eth1 type: ethernet state: up ipv4: address: - ip: 192.0.2.251 prefix-length: 24 dhcp: false enabled: true ipv6: address: - ip: 2001:db8:1::1 prefix-length: 64 dhcp: false enabled: true autoconf: false # ...If you want to set static DNS search domains and static DNS name servers for your network interface, define the static interface that runs either the Dynamic Host Configuration Protocol (DHCP) or the IPv6 Autoconfiguration (
) mechanism in the network interface YAML configuration file.autoconfImportantSpecifying the following
configurations in the network interface YAML file might cause a race condition at reboot that prevents thedns-resolver(NNCP) from applying to pods that run in your cluster:NodeNetworkConfigurationPolicy- Setting static DNS search domains and dynamic DNS name servers for your network interface.
-
Specifying domain suffixes for the parameter and not setting IP addresses for the
searchparameter.server
Example configuration that sets
example.comandexample.orgstatic DNS search domains along with static DNS name server settings# ... dns-resolver: config: server: - 2001:db8:f::1 - 192.0.2.251 search: - example.com - example.org interfaces: - name: eth1 type: ethernet state: up ipv4: enabled: true dhcp: true auto-dns: true ipv6: enabled: true dhcp: true autoconf: true auto-dns: true # ...
28.2.10.5. Static routing Link kopierenLink in die Zwischenablage kopiert!
The following snippet configures a static route and a static IP on interface
eth1
dns-resolver:
config:
# ...
interfaces:
- name: eth1
description: Static routing on eth1
type: ethernet
state: up
ipv4:
dhcp: false
enabled: true
address:
- ip: 192.0.2.251
prefix-length: 24
routes:
config:
- destination: 198.51.100.0/24
metric: 150
next-hop-address: 192.0.2.1
next-hop-interface: eth1
table-id: 254
# ...
You cannot use the OVN-Kubernetes
br-ex
28.3. Troubleshooting node network configuration Link kopierenLink in die Zwischenablage kopiert!
If the node network configuration encounters an issue, the policy is automatically rolled back and the enactments report failure. This includes issues such as:
- The configuration fails to be applied on the host.
- The host loses connection to the default gateway.
- The host loses connection to the API server.
28.3.1. Troubleshooting an incorrect node network configuration policy configuration Link kopierenLink in die Zwischenablage kopiert!
You can apply changes to the node network configuration across your entire cluster by applying a node network configuration policy.
If you applied an incorrect configuration, you can use the following example to troubleshoot and correct the failed node network policy. The example attempts to apply a Linux bridge policy to a cluster that has three control plane nodes and three compute nodes. The policy is not applied because the policy references the wrong interface.
To find an error, you need to investigate the available NMState resources. You can then update the policy with the correct configuration.
Prerequisites
-
You ensured that an interface does not exist on your Linux system.
ens01
Procedure
Create a policy on your cluster. The following example creates a simple bridge,
that hasbr1as its member:ens01apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: ens01-bridge-testfail spec: desiredState: interfaces: - name: br1 description: Linux bridge with the wrong port type: linux-bridge state: up ipv4: dhcp: true enabled: true bridge: options: stp: enabled: false port: - name: ens01 # ...Apply the policy to your network interface:
$ oc apply -f ens01-bridge-testfail.yamlExample output
nodenetworkconfigurationpolicy.nmstate.io/ens01-bridge-testfail createdVerify the status of the policy by running the following command:
$ oc get nncpThe output shows that the policy failed:
Example output
NAME STATUS ens01-bridge-testfail FailedToConfigureThe policy status alone does not indicate if it failed on all nodes or a subset of nodes.
List the node network configuration enactments to see if the policy was successful on any of the nodes. If the policy failed for only a subset of nodes, the output suggests that the problem is with a specific node configuration. If the policy failed on all nodes, the output suggests that the problem is with the policy.
$ oc get nnceThe output shows that the policy failed on all nodes:
Example output
NAME STATUS control-plane-1.ens01-bridge-testfail FailedToConfigure control-plane-2.ens01-bridge-testfail FailedToConfigure control-plane-3.ens01-bridge-testfail FailedToConfigure compute-1.ens01-bridge-testfail FailedToConfigure compute-2.ens01-bridge-testfail FailedToConfigure compute-3.ens01-bridge-testfail FailedToConfigureView one of the failed enactments. The following command uses the output tool
to filter the output:jsonpath$ oc get nnce compute-1.ens01-bridge-testfail -o jsonpath='{.status.conditions[?(@.type=="Failing")].message}'Example output
[2024-10-10T08:40:46Z INFO nmstatectl] Nmstate version: 2.2.37 NmstateError: InvalidArgument: Controller interface br1 is holding unknown port ens01The previous example shows the output from an
error that indicates that theInvalidArgumentis an unknown port. For this example, you might need to change the port configuration in the policy configuration file.ens01To ensure that the policy is configured properly, view the network configuration for one or all of the nodes by requesting the
object. The following command returns the network configuration for theNodeNetworkStatenode:control-plane-1$ oc get nns control-plane-1 -o yamlThe output shows that the interface name on the nodes is
but the failed policy incorrectly usesens1:ens01Example output
- ipv4: # ... name: ens1 state: up type: ethernetCorrect the error by editing the existing policy:
$ oc edit nncp ens01-bridge-testfail# ... port: - name: ens1Save the policy to apply the correction.
Check the status of the policy to ensure it updated successfully:
$ oc get nncpExample output
NAME STATUS ens01-bridge-testfail SuccessfullyConfiguredThe updated policy is successfully configured on all nodes in the cluster.
28.3.2. Troubleshooting DNS connectivity issues in a disconnected environment Link kopierenLink in die Zwischenablage kopiert!
If you experience DNS connectivity issues when configuring
nmstate
root-servers.net
Ensure that the DNS server includes a name server (NS) entry for the
root-servers.net
28.3.2.1. Configuring the bind9 DNS named server Link kopierenLink in die Zwischenablage kopiert!
For a cluster configured to query a
bind9
root-servers.net
/var/named/named.localhost
Procedure
Add the
zone at the end of theroot-servers.netconfiguration file by running the following command:/etc/named.conf$ cat >> /etc/named.conf <<EOF zone "root-servers.net" IN { type master; file "named.localhost"; }; EOFRestart the
service by running the following command:named$ systemctl restart namedConfirm that the
zone is present by running the following command:root-servers.net$ journalctl -u named|grep root-servers.netExample output
Jul 03 15:16:26 rhel-8-10 bash[xxxx]: zone root-servers.net/IN: loaded serial 0 Jul 03 15:16:26 rhel-8-10 named[xxxx]: zone root-servers.net/IN: loaded serial 0Verify that the DNS server can resolve the NS record for the
domain by running the following command:root-servers.net$ host -t NS root-servers.net. 127.0.0.1Example output
Using domain server: Name: 127.0.0.1 Address: 127.0.0.53 Aliases: root-servers.net name server root-servers.net.
28.3.2.2. Configuring the dnsmasq DNS server Link kopierenLink in die Zwischenablage kopiert!
If you are using
dnsmasq
root-servers.net
root-servers.net
Create a configuration file that delegates the domain
to another DNS server by running the following command:root-servers.net$ echo 'server=/root-servers.net/<DNS_server_IP>'> /etc/dnsmasq.d/delegate-root-servers.net.confRestart the
service by running the following command:dnsmasq$ systemctl restart dnsmasqConfirm that the
domain is delegated to another DNS server by running the following command:root-servers.net$ journalctl -u dnsmasq|grep root-servers.netExample output
Jul 03 15:31:25 rhel-8-10 dnsmasq[1342]: using nameserver 192.168.1.1#53 for domain root-servers.netVerify that the DNS server can resolve the NS record for the
domain by running the following command:root-servers.net$ host -t NS root-servers.net. 127.0.0.1Example output
Using domain server: Name: 127.0.0.1 Address: 127.0.0.1#53 Aliases: root-servers.net name server root-servers.net.