Chapter 1. Kubernetes NMState Operator
The Kubernetes NMState Operator provides a Kubernetes API for performing state-driven network configuration across the OpenShift Container Platform cluster’s nodes with NMState. The Kubernetes NMState Operator provides users with functionality to configure various network interface types, DNS, and routing on cluster nodes. Additionally, the daemons on the cluster nodes periodically report on the state of each node’s network interfaces to the API server.
Red Hat supports the Kubernetes NMState Operator in production environments on bare-metal, IBM Power®, IBM Z®, IBM® LinuxONE, VMware vSphere, and Red Hat OpenStack Platform (RHOSP) installations.
Red Hat support exists for using the Kubernetes NMState Operator on Microsoft Azure but in a limited capacity. Support is limited to configuring DNS servers on your system as a postinstallation task.
Before you can use NMState with OpenShift Container Platform, you must install the Kubernetes NMState Operator. After you install the Kubernetes NMState Operator, you can complete the following tasks:
- Observing and updating the node network state and configuration
-
Creating a manifest object that includes a customized
br-ex
bridge
For more information on these tasks, see the Additional resources section.
The Kubernetes NMState Operator updates the network configuration of a secondary NIC. The Operator cannot update the network configuration of the primary NIC, or update the br-ex
bridge on most on-premise networks.
On a bare-metal platform, using the Kubernetes NMState Operator to update the br-ex
bridge network configuration is only supported if you set the br-ex
bridge as the interface in a machine config manifest file. To update the br-ex
bridge as a postinstallation task, you must set the br-ex
bridge as the interface in the NMState configuration of the NodeNetworkConfigurationPolicy
custom resource (CR) for your cluster. For more information, see Creating a manifest object that includes a customized br-ex bridge in Postinstallation configuration.
OpenShift Container Platform uses nmstate
to report on and configure the state of the node network. This makes it possible to modify the network policy configuration, such as by creating a Linux bridge on all nodes, by applying a single configuration manifest to the cluster.
Node networking is monitored and updated by the following objects:
NodeNetworkState
- Reports the state of the network on that node.
NodeNetworkConfigurationPolicy
-
Describes the requested network configuration on nodes. You update the node network configuration, including adding and removing interfaces, by applying a
NodeNetworkConfigurationPolicy
CR to the cluster. NodeNetworkConfigurationEnactment
- Reports the network policies enacted upon each node.
1.1. Installing the Kubernetes NMState Operator Copy linkLink copied to clipboard!
You can install the Kubernetes NMState Operator by using the web console or the CLI.
1.1.1. Installing the Kubernetes NMState Operator by using the web console Copy linkLink copied to clipboard!
You can install the Kubernetes NMState Operator by using the web console. After you install the Kubernetes NMState Operator, the Operator has deployed the NMState State Controller as a daemon set across all of the cluster nodes.
Prerequisites
-
You are logged in as a user with
cluster-admin
privileges.
Procedure
-
Select Operators
OperatorHub. -
In the search field below All Items, enter
nmstate
and click Enter to search for the Kubernetes NMState Operator. - Click on the Kubernetes NMState Operator search result.
- Click on Install to open the Install Operator window.
- Click Install to install the Operator.
- After the Operator finishes installing, click View Operator.
-
Under Provided APIs, click Create Instance to open the dialog box for creating an instance of
kubernetes-nmstate
. In the Name field of the dialog box, ensure the name of the instance is
nmstate.
NoteThe name restriction is a known issue. The instance is a singleton for the entire cluster.
- Accept the default settings and click Create to create the instance.
1.1.2. Installing the Kubernetes NMState Operator by using the CLI Copy linkLink copied to clipboard!
You can install the Kubernetes NMState Operator by using the OpenShift CLI (oc)
. After it is installed, the Operator can deploy the NMState State Controller as a daemon set across all of the cluster nodes.
Prerequisites
-
You have installed the OpenShift CLI (
oc
). -
You are logged in as a user with
cluster-admin
privileges.
Procedure
Create the
nmstate
Operator namespace:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
OperatorGroup
:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Subscribe to the
nmstate
Operator:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm the
ClusterServiceVersion
(CSV) status for thenmstate
Operator deployment equalsSucceeded
:oc get clusterserviceversion -n openshift-nmstate \ -o custom-columns=Name:.metadata.name,Phase:.status.phase
$ oc get clusterserviceversion -n openshift-nmstate \ -o custom-columns=Name:.metadata.name,Phase:.status.phase
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create an instance of the
nmstate
Operator:Copy to Clipboard Copied! Toggle word wrap Toggle overflow If your cluster has problems with the DNS health check probe because of DNS connectivity issues, you can add the following DNS host name configuration to the
NMState
CRD to build in health checks that can resolve these issues:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the DNS host name configuration to your cluster network by running the following command. Ensure that you replace
<filename>
with the name of your CRD file.$ oc apply -f <filename>.yaml
$ oc apply -f <filename>.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Verify that all pods for the NMState Operator have the
Running
status by entering the following command:oc get pod -n openshift-nmstate
$ oc get pod -n openshift-nmstate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.1.3. Viewing metrics collected by the Kubernetes NMState Operator Copy linkLink copied to clipboard!
The Kubernetes NMState Operator, kubernetes-nmstate-operator
, can collect metrics from the kubernetes_nmstate_features_applied
component and expose them as ready-to-use metrics. As a use case for viewing metrics, consider a situation where you created a NodeNetworkConfigurationPolicy
custom resource and you want to confirm that the policy is active.
The kubernetes_nmstate_features_applied
metrics are not an API and might change between OpenShift Container Platform versions.
In the web console, the Metrics UI includes some predefined CPU, memory, bandwidth, and network packet queries for the selected project. You can run custom Prometheus Query Language (PromQL) queries for CPU, memory, bandwidth, network packet and application metrics for the project.
The following example demonstrates a NodeNetworkConfigurationPolicy
manifest example that is applied to an OpenShift Container Platform cluster:
The NodeNetworkConfigurationPolicy
manifest exposes metrics and makes them available to the Cluster Monitoring Operator (CMO). The following example shows some exposed metrics:
Prerequisites
-
You have installed the OpenShift CLI (
oc
). - You have logged in to the web console as the administrator and installed the Kubernetes NMState Operator.
- You have access to the cluster as a developer or as a user with view permissions for the project that you are viewing metrics for.
- You have enabled monitoring for user-defined projects.
- You have deployed a service in a user-defined project.
-
You have created a
NodeNetworkConfigurationPolicy
manifest and applied it to your cluster.
Starting with OpenShift Container Platform 4.19, the perspectives in the web console have unified. The Developer perspective is no longer enabled by default.
All users can interact with all OpenShift Container Platform web console features. However, if you are not the cluster owner, you might need to request permission to access certain features from the cluster owner.
You can still enable the Developer perspective. On the Getting Started pane in the web console, you can take a tour of the console, find information on setting up your cluster, view a quick start for enabling the Developer perspective, and follow links to explore new features and capabilities.
Procedure
If you want to view the metrics from the Developer perspective in the OpenShift Container Platform web console, complete the following tasks:
- Click Observe.
-
To view the metrics of a specific project, select the project in the Project: list. For example,
openshift-nmstate
. - Click the Metrics tab.
To visualize the metrics on the plot, select a query from the Select query list or create a custom PromQL query based on the selected query by selecting Show PromQL.
NoteYou can only run one query at a time with the developer role.
If you want to view the metrics in the OpenShift Container Platform web console as an administrator, complete the following tasks:
-
Click Observe
Metrics. -
Enter
kubernetes_nmstate_features_applied
in the Expression field. - Click Add query and then Run queries.
-
Click Observe
To explore the visualized metrics, do any of the following tasks:
To zoom into the plot and change the time range, do any of the following tasks:
- To visually select the time range, click and drag on the plot horizontally.
- To select the time range, use the menu which is in the upper left of the console.
- To reset the time range, select Reset zoom.
- To display the output for all the queries at a specific point in time, hold the mouse cursor on the plot at that point. The query output displays in a pop-up box.
1.2. Uninstalling the Kubernetes NMState Operator Copy linkLink copied to clipboard!
You can use the Operator Lifecycle Manager (OLM) to uninstall the Kubernetes NMState Operator, but by design OLM does not delete any associated custom resource definitions (CRDs), custom resources (CRs), or API Services.
Before you uninstall the Kubernetes NMState Operator from the Subcription
resource used by OLM, identify what Kubernetes NMState Operator resources to delete. This identification ensures that you can delete resources without impacting your running cluster.
If you need to reinstall the Kubernetes NMState Operator, see "Installing the Kubernetes NMState Operator by using the CLI" or "Installing the Kubernetes NMState Operator by using the web console".
Prerequisites
-
You have installed the OpenShift CLI (
oc
). -
You have installed the
jq
CLI tool. -
You are logged in as a user with
cluster-admin
privileges.
Procedure
Unsubscribe the Kubernetes NMState Operator from the
Subcription
resource by running the following command:oc delete --namespace openshift-nmstate subscription kubernetes-nmstate-operator
$ oc delete --namespace openshift-nmstate subscription kubernetes-nmstate-operator
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Find the
ClusterServiceVersion
(CSV) resource that associates with the Kubernetes NMState Operator:oc get --namespace openshift-nmstate clusterserviceversion
$ oc get --namespace openshift-nmstate clusterserviceversion
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output that lists a CSV resource
NAME DISPLAY VERSION REPLACES PHASE kubernetes-nmstate-operator.v4.19.0 Kubernetes NMState Operator 4.19.0 Succeeded
NAME DISPLAY VERSION REPLACES PHASE kubernetes-nmstate-operator.v4.19.0 Kubernetes NMState Operator 4.19.0 Succeeded
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the CSV resource. After you delete the file, OLM deletes certain resources, such as
RBAC
, that it created for the Operator.oc delete --namespace openshift-nmstate clusterserviceversion kubernetes-nmstate-operator.v4.19.0
$ oc delete --namespace openshift-nmstate clusterserviceversion kubernetes-nmstate-operator.v4.19.0
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
nmstate
CR and any associatedDeployment
resources by running the following commands:oc -n openshift-nmstate delete nmstate nmstate
$ oc -n openshift-nmstate delete nmstate nmstate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete --all deployments --namespace=openshift-nmstate
$ oc delete --all deployments --namespace=openshift-nmstate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After you deleted the
nmstate
CR, remove thenmstate-console-plugin
console plugin name from theconsole.operator.openshift.io/cluster
CR.Store the position of the
nmstate-console-plugin
entry that exists among the list of enable plugins by running the following command. The following command uses thejq
CLI tool to store the index of the entry in an environment variable namedINDEX
:INDEX=$(oc get console.operator.openshift.io cluster -o json | jq -r '.spec.plugins | to_entries[] | select(.value == "nmstate-console-plugin") | .key')
INDEX=$(oc get console.operator.openshift.io cluster -o json | jq -r '.spec.plugins | to_entries[] | select(.value == "nmstate-console-plugin") | .key')
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the
nmstate-console-plugin
entry from theconsole.operator.openshift.io/cluster
CR by running the following patch command:oc patch console.operator.openshift.io cluster --type=json -p "[{\"op\": \"remove\", \"path\": \"/spec/plugins/$INDEX\"}]"
$ oc patch console.operator.openshift.io cluster --type=json -p "[{\"op\": \"remove\", \"path\": \"/spec/plugins/$INDEX\"}]"
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
INDEX
is an auxiliary variable. You can specify a different name for this variable.
Delete all the custom resource definitions (CRDs), such as
nmstates.nmstate.io
, by running the following commands:oc delete crd nmstates.nmstate.io
$ oc delete crd nmstates.nmstate.io
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete crd nodenetworkconfigurationenactments.nmstate.io
$ oc delete crd nodenetworkconfigurationenactments.nmstate.io
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete crd nodenetworkstates.nmstate.io
$ oc delete crd nodenetworkstates.nmstate.io
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete crd nodenetworkconfigurationpolicies.nmstate.io
$ oc delete crd nodenetworkconfigurationpolicies.nmstate.io
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the namespace:
oc delete namespace kubernetes-nmstate
$ oc delete namespace kubernetes-nmstate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow