Red Hat Summit Red Hat SummitApoyoConsolaDesarrolladoresIniciar una pruebaContacto

Este contenido no está disponible en el idioma seleccionado.

Chapter 11. Node maintenance

11.1. About node maintenance
Copiar enlace

11.1.1. Understanding node maintenance mode
Copiar enlace

Nodes can be placed into maintenance mode using the 

oc adm
utility, or using 
NodeMaintenance
custom resources (CRs).

Placing a node into maintenance marks the node as unschedulable and drains all the virtual machines and pods from it. Virtual machine instances that have a 

LiveMigrate
eviction strategy are live migrated to another node without loss of service. This eviction strategy is configured by default in virtual machine created from common templates but must be configured manually for custom virtual machines.

Virtual machine instances without an eviction strategy are shut down. Virtual machines with a 

RunStrategy
of 
Running
or 
RerunOnFailure
are recreated on another node. Virtual machines with a 
RunStrategy
of 
Manual
are not automatically restarted.

Important

Virtual machines must have a persistent volume claim (PVC) with a shared 

ReadWriteMany
(RWX) access mode to be live migrated.

When installed as part of OpenShift Virtualization, Node Maintenance Operator watches for new or deleted 

NodeMaintenance
CRs. When a new 
NodeMaintenance
CR is detected, no new workloads are scheduled and the node is cordoned off from the rest of the cluster. All pods that can be evicted are evicted from the node. When a 
NodeMaintenance
CR is deleted, the node that is referenced in the CR is made available for new workloads.

Note

Using a 

NodeMaintenance
CR for node maintenance tasks achieves the same results as the 
oc adm cordon
and 
oc adm drain
commands using standard OpenShift Container Platform custom resource processing.

11.1.2. Maintaining bare metal nodes
Copiar enlace

When you deploy OpenShift Container Platform on bare metal infrastructure, there are additional considerations that must be taken into account compared to deploying on cloud infrastructure. Unlike in cloud environments where the cluster nodes are considered ephemeral, re-provisioning a bare metal node requires significantly more time and effort for maintenance tasks.

When a bare metal node fails, for example, if a fatal kernel error happens or a NIC card hardware failure occurs, workloads on the failed node need to be restarted elsewhere else on the cluster while the problem node is repaired or replaced. Node maintenance mode allows cluster administrators to gracefully power down nodes, moving workloads to other parts of the cluster and ensuring workloads do not get interrupted. Detailed progress and node status details are provided during maintenance.

11.2. Setting a node to maintenance mode
Copiar enlace

Place a node into maintenance from the web console, CLI, or using a 

NodeMaintenance
custom resource.

11.2.1. Setting a node to maintenance mode in the web console
Copiar enlace

Set a node to maintenance mode using the Options menu kebab found on each node in the Compute Nodes list, or using the Actions control of the Node Details screen.

Procedure

  1. In the OpenShift Virtualization console, click Compute Nodes.

  2. You can set the node to maintenance from this screen, which makes it easier to perform actions on multiple nodes in the one screen or from the Node Details screen where you can view comprehensive details of the selected node:

    • Click the Options menu kebab at the end of the node and select Start Maintenance.
    • Click the node name to open the Node Details screen and click Actions Start Maintenance.
  3. Click Start Maintenance in the confirmation window.

The node will live migrate virtual machine instances that have the 

LiveMigration
eviction strategy, and the node is no longer schedulable. All other pods and virtual machines on the node are deleted and recreated on another node.

11.2.2. Setting a node to maintenance mode in the CLI
Copiar enlace

Set a node to maintenance mode by marking it as unschedulable and using the 

oc adm drain
command to evict or delete pods from the node.

Procedure

  1. Mark the node as unschedulable. The node status changes to 

    NotReady,SchedulingDisabled
    . 

    $ oc adm cordon <node1>

  2. Drain the node in preparation for maintenance. The node live migrates virtual machine instances that have the 

    LiveMigratable
    condition set to 
    True
    and the 
    spec:evictionStrategy
    field set to 
    LiveMigrate
    . All other pods and virtual machines on the node are deleted and recreated on another node. 

    $ oc adm drain <node1> --delete-emptydir-data --ignore-daemonsets=true --force
    • The 
      --delete-emptydir-data
      flag removes any virtual machine instances on the node that use 
      emptyDir
      volumes. Data in these volumes is ephemeral and is safe to be deleted after termination.
    • The 
      --ignore-daemonsets=true
      flag ensures that daemon sets are ignored and pod eviction can continue successfully.
    • The 
      --force
      flag is required to delete pods that are not managed by a replica set or daemon set controller.

You can put a node into maintenance mode with a 

NodeMaintenance
custom resource (CR). When you apply a 
NodeMaintenance
CR, all allowed pods are evicted and the node is shut down. Evicted pods are queued to be moved to another node in the cluster.

Prerequisites

  • Install the OpenShift Container Platform CLI 
    oc
    .
  • Log in to the cluster as a user with 
    cluster-admin
    privileges.

Procedure

  1. Create the following node maintenance CR, and save the file as 

    nodemaintenance-cr.yaml
    : 

    apiVersion: nodemaintenance.kubevirt.io/v1beta1
kind: NodeMaintenance
metadata:
  name: maintenance-example
    1 
    
spec:
  nodeName: node-1.example.com
    2 
    
  reason: "Node maintenance"
    3
    1
    Node maintenance CR name
    2
    The name of the node to be put into maintenance mode
    3
    Plain text description of the reason for maintenance

  2. Apply the node maintenance schedule by running the following command: 

    $ oc apply -f nodemaintenance-cr.yaml

  3. Check the progress of the maintenance task by running the following command, replacing 

    <node-name>
    with the name of your node: 

    $ oc describe node <node-name>

    Example output

    Events:
  Type     Reason                     Age                   From     Message
  ----     ------                     ----                  ----     -------
  Normal   NodeNotSchedulable         61m                   kubelet  Node node-1.example.com status is now: NodeNotSchedulable

11.2.3.1. Checking status of current NodeMaintenance CR tasks
Copiar enlace

You can check the status of current 

NodeMaintenance
CR tasks.

Prerequisites

  • Install the OpenShift Container Platform CLI 
    oc
    .
  • Log in as a user with 
    cluster-admin
    privileges.

Procedure

  • Check the status of current node maintenance tasks by running the following command: 

    $ oc get NodeMaintenance -o yaml

    Example output

    apiVersion: v1
items:
- apiVersion: nodemaintenance.kubevirt.io/v1beta1
  kind: NodeMaintenance
  metadata:
...
  spec:
    nodeName: node-1.example.com
    reason: Node maintenance
  status:
    evictionPods: 3
    1 
    
    pendingPods:
    - pod-example-workload-0
    - httpd
    - httpd-manual
    phase: Running
    lastError: "Last failure message"
    2 
    
    totalpods: 5
...

    1
    evictionPods is the number of pods scheduled for eviction.
    2
    lastError records the latest eviction error, if any.

11.3. Resuming a node from maintenance mode
Copiar enlace

Resuming a node brings it out of maintenance mode and makes it schedulable again.

Resume a node from maintenance mode from the web console, CLI, or by deleting the 

NodeMaintenance
custom resource.

11.3.1. Resuming a node from maintenance mode in the web console
Copiar enlace

Resume a node from maintenance mode using the Options menu kebab found on each node in the Compute Nodes list, or using the Actions control of the Node Details screen.

Procedure

  1. In the OpenShift Virtualization console, click Compute Nodes.

  2. You can resume the node from this screen, which makes it easier to perform actions on multiple nodes in the one screen, or from the Node Details screen where you can view comprehensive details of the selected node:

    • Click the Options menu kebab at the end of the node and select Stop Maintenance.
    • Click the node name to open the Node Details screen and click Actions Stop Maintenance.
  3. Click Stop Maintenance in the confirmation window.

The node becomes schedulable, but virtual machine instances that were running on the node prior to maintenance will not automatically migrate back to this node.

11.3.2. Resuming a node from maintenance mode in the CLI
Copiar enlace

Resume a node from maintenance mode by making it schedulable again.

Procedure

  • Mark the node as schedulable. You can then resume scheduling new workloads on the node. 

    $ oc adm uncordon <node1>

You can resume a node by deleting the 

NodeMaintenance
CR.

Prerequisites

  • Install the OpenShift Container Platform CLI 
    oc
    .
  • Log in to the cluster as a user with 
    cluster-admin
    privileges.

Procedure

  • When your node maintenance task is complete, delete the active 

    NodeMaintenance
    CR: 

    $ oc delete -f nodemaintenance-cr.yaml

    Example output

    nodemaintenance.nodemaintenance.kubevirt.io "maintenance-example" deleted

11.4. Automatic renewal of TLS certificates
Copiar enlace

All TLS certificates for OpenShift Virtualization components are renewed and rotated automatically. You are not required to refresh them manually.

11.4.1. TLS certificates automatic renewal schedules
Copiar enlace

TLS certificates are automatically deleted and replaced according to the following schedule:

  • KubeVirt certificates are renewed daily.
  • Containerized Data Importer controller (CDI) certificates are renewed every 15 days.
  • MAC pool certificates are renewed every year.

Automatic TLS certificate rotation does not disrupt any operations. For example, the following operations continue to function without any disruption:

  • Migrations
  • Image uploads
  • VNC and console connections

11.5. Managing node labeling for obsolete CPU models
Copiar enlace

You can schedule a virtual machine (VM) on a node as long as the VM CPU model and policy are supported by the node.

11.5.1. About node labeling for obsolete CPU models
Copiar enlace

The OpenShift Virtualization Operator uses a predefined list of obsolete CPU models to ensure that a node supports only valid CPU models for scheduled VMs.

By default, the following CPU models are eliminated from the list of labels generated for the node:

Example 11.1. Obsolete CPU models

"486"
Conroe
athlon
core2duo
coreduo
kvm32
kvm64
n270
pentium
pentium2
pentium3
pentiumpro
phenom
qemu32
qemu64

This predefined list is not visible in the 

HyperConverged
CR. You cannot remove CPU models from this list, but you can add to the list by editing the 
spec.obsoleteCPUs.cpuModels
field of the 
HyperConverged
CR.

11.5.2. About node labeling for CPU features
Copiar enlace

Through the process of iteration, the base CPU features in the minimum CPU model are eliminated from the list of labels generated for the node.

For example:

  • An environment might have two supported CPU models: 
    Penryn
    and 
    Haswell
    .

  • If 

    Penryn
    is specified as the CPU model for 
    minCPU
    , each base CPU feature for 
    Penryn
    is compared to the list of CPU features supported by 
    Haswell
    .

    Example 11.2. CPU features supported by Penryn

    apic
clflush
cmov
cx16
cx8
de
fpu
fxsr
lahf_lm
lm
mca
mce
mmx
msr
mtrr
nx
pae
pat
pge
pni
pse
pse36
sep
sse
sse2
sse4.1
ssse3
syscall
tsc

    Example 11.3. CPU features supported by Haswell

    aes
apic
avx
avx2
bmi1
bmi2
clflush
cmov
cx16
cx8
de
erms
fma
fpu
fsgsbase
fxsr
hle
invpcid
lahf_lm
lm
mca
mce
mmx
movbe
msr
mtrr
nx
pae
pat
pcid
pclmuldq
pge
pni
popcnt
pse
pse36
rdtscp
rtm
sep
smep
sse
sse2
sse4.1
sse4.2
ssse3
syscall
tsc
tsc-deadline
x2apic
xsave

  • If both 

    Penryn
    and 
    Haswell
    support a specific CPU feature, a label is not created for that feature. Labels are generated for CPU features that are supported only by 
    Haswell
    and not by 
    Penryn
    .

    Example 11.4. Node labels created for CPU features after iteration

    aes
avx
avx2
bmi1
bmi2
erms
fma
fsgsbase
hle
invpcid
movbe
pcid
pclmuldq
popcnt
rdtscp
rtm
sse4.2
tsc-deadline
x2apic
xsave

11.5.3. Configuring obsolete CPU models
Copiar enlace

You can configure a list of obsolete CPU models by editing the 

HyperConverged
custom resource (CR).

Procedure

  • Edit the 

    HyperConverged
    custom resource, specifying the obsolete CPU models in the 
    obsoleteCPUs
    array. For example: 

    apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  obsoleteCPUs:
    cpuModels:
    1 
    
      - "<obsolete_cpu_1>"
      - "<obsolete_cpu_2>"
    minCPUModel: "<minimum_cpu_model>"
    2
    1
    Replace the example values in the cpuModels array with obsolete CPU models. Any value that you specify is added to a predefined list of obsolete CPU models. The predefined list is not visible in the CR.
    2
    Replace this value with the minimum CPU model that you want to use for basic CPU features. If you do not specify a value, Penryn is used by default.

11.6. Preventing node reconciliation
Copiar enlace

Use 

skip-node
annotation to prevent the 
node-labeller
from reconciling a node.

11.6.1. Using skip-node annotation
Copiar enlace

If you want the 

node-labeller
to skip a node, annotate that node by using the 
oc
CLI.

Prerequisites

  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  • Annotate the node that you want to skip by running the following command: 

    $ oc annotate node <node_name> node-labeller.kubevirt.io/skip-node=true
    1
    1
    Replace <node_name> with the name of the relevant node to skip.

    Reconciliation resumes on the next cycle after the node annotation is removed or set to false.

Red Hat logoGithubredditYoutubeTwitter

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Ayudamos a los usuarios de Red Hat a innovar y alcanzar sus objetivos con nuestros productos y servicios con contenido en el que pueden confiar. Explore nuestras recientes actualizaciones.

Hacer que el código abierto sea más inclusivo

Red Hat se compromete a reemplazar el lenguaje problemático en nuestro código, documentación y propiedades web. Para más detalles, consulte el Blog de Red Hat.

Acerca de Red Hat

Ofrecemos soluciones reforzadas que facilitan a las empresas trabajar en plataformas y entornos, desde el centro de datos central hasta el perímetro de la red.

Theme

© 2026 Red HatVolver arriba