Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 6. Boot image management

For Google Cloud and Amazon Web Services (AWS) clusters, by default the Machine Config Operator (MCO) manages and updates the boot image that is used to scale up your nodes. This means that by default, the MCO updates the boot images whenever you upgrade your cluster.

For VMware vSphere, you can enable boot image management as a Technology Preview feature.

For all other platforms, the MCO does not update the boot image with each cluster update.

Important

Boot image management on vSphere is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

6.1. About boot image management
Copy link

By default, for Google Cloud and Amazon Web Services (AWS) clusters, the Machine Config Operator (MCO) updates the boot image in the machine sets in your cluster whenever you update your cluster.

For VMware vSphere, you can enable boot image management as a Technology Preview feature. For information on how to enable this feature, see "Enabling boot image management".

Important

Boot image management on vSphere is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can disable the boot image management feature, if needed. When the feature is disabled, the boot image no longer updates with the cluster. For example, with the feature disabled, if your cluster was originally created with OpenShift Container Platform 4.16, the boot image that the MCO would use to create nodes is the same 4.16 version, even if your cluster is at a later version.

However, using an older boot image could cause the following issues:

  • Extra time to start nodes
  • Certificate expiration issues
  • Version skew issues

For information on how to disable this feature, see "Disabling boot image management". If you disable this feature, you can re-enable the feature at any time. For information, see "Enabling boot image management".

Note

The ability to configure boot image management is available for only Google Cloud and AWS clusters. It is not supported for clusters managed by the Cluster CAPI Operator.

How the cluster behaves after disabling or re-enabling the feature, depends upon when you made the change, including the following scenarios:

  • If you disable the feature before updating to a new OpenShift Container Platform version:

    • The boot image version used by the machine sets remains the same OpenShift Container Platform version as when the feature was disabled.
    • When you scale up nodes, the new nodes use that same OpenShift Container Platform version.

  • If you disable the feature after updating to a new OpenShift Container Platform version:

    • The boot image version used by the machine sets is updated to match the updated OpenShift Container Platform version.
    • When you scale up nodes, the new nodes use the updated OpenShift Container Platform version.
    • If you update to a later OpenShift Container Platform version, the boot image version in the machine sets remains at the current version and is not updated with the cluster.

  • If you enable the feature after disabling:

    • The boot image version used by the machine sets is updated to the current OpenShift Container Platform version, if different.
    • When you scale up nodes, the new nodes use the current OpenShift Container Platform version in the cluster.
Note

Because a boot image is used only when a node is scaled up, this feature has no effect on existing nodes.

To view the current boot image used in your cluster, use one of the following methods, based on your platform:

  • For Google Cloud and AWS, you can examine a machine set.

    Note

    The location and format of the boot image within the machine set differs, based on the platform. However, the boot image is always listed in the spec.template.spec.providerSpec. parameter.

    Example Google Cloud machine set with the boot image reference

    apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: ci-ln-hmy310k-72292-5f87z-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
# ...
        value:
          disks:
          - autoDelete: true
            boot: true
            image: projects/rhcos-cloud/global/images/rhcos-412-85-202203181601-0-gcp-x86-64
    1 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    This boot image is the same as the originally-installed OpenShift Container Platform version, in this example OpenShift Container Platform 4.12, regardless of the current version of the cluster. The way that the boot image is represented in the machine set depends on the platform, as the structure of the providerSpec field differs from platform to platform.

    Example AWS machine set with the boot image reference

    apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: ci-ln-hmy310k-72292-5f87z-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
         value:
          ami:
            id: ami-0e8fd9094e487d1ff
# ...
    Copy to Clipboard Toggle word wrap

  • For VMware vSphere, examine an affected node by opening an oc debug session to the node and using the rpm-ostree status command. 

    sh-5.1# rpm-ostree status
    Copy to Clipboard Toggle word wrap

    Example AWS node with the boot image reference

    State: idle
Deployments:
* ostree-unverified-registry:quay.io/my-registry/...
                   Digest: sha256:...
    Copy to Clipboard Toggle word wrap

Important

If any of the machine sets for which you want to enable boot image management use a *-user-data secret that is based on Ignition version 2.2.0, the Machine Config Operator converts the Ignition version to 3.4.0 when you enable the feature. OpenShift Container Platform versions 4.5 and lower use Ignition version 2.2.0. If this conversion fails, the MCO or your cluster could degrade. An error message that includes err: converting ignition stub failed: failed to parse Ignition config is added to the output of the oc get ClusterOperator machine-config command. You can use the following general steps to correct the problem:

  1. Disable the boot image management feature. For information, see "Disabling boot image management".
  2. Manually update the *-user-data secret to use Ignition version to 3.2.0.
  3. Enable the boot image management feature. For information, see "Enabling boot image management".

6.2. Disabling boot image management
Copy link

By default, for Google Cloud and Amazon Web Services (AWS) clusters, the Machine Config Operator (MCO) manages and updates the boot image in the machine sets in your cluster whenever you update your cluster. For VMware vSphere, you can enable boot image management as a Technology Preview feature.

You can disable the boot image management feature for your cluster by editing the MachineConfiguration object. When disabled, the Machine Config Operator (MCO) no longer manages the boot image in your cluster and no longer updates the boot image with each cluster update.

Disabling this feature does not rollback the nodes or machine sets to the originally-installed boot image. The machine sets retain the boot image version that was present when the feature was disabled and is not updated if the cluster is upgraded to a new OpenShift Container Platform version in the future. This feature has no effect on existing nodes.

After disabling the feature, you can re-enable the feature at any time. For more information, see "Enabling updated boot images".

Procedure

  1. Edit the MachineConfiguration object to disable the boot image management feature for some or all of your machine sets: 

    $ oc edit MachineConfiguration cluster
    Copy to Clipboard Toggle word wrap

    • Optional: Disable the feature for all machine sets: 

      apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
  namespace: openshift-machine-config-operator
spec:
# ...
  managedBootImages:
      1 
      
    machineManagers:
    - apiGroup: machine.openshift.io
      2 
      
      resource: machinesets
      3 
      
      selection:
        mode: None
      4
      Copy to Clipboard Toggle word wrap
      1
      Configures the boot image management feature.
      2
      Specifies an API group. This must be machine.openshift.io.
      3
      Specifies the resource within the specified API group to apply the change. This must be machinesets.
      4
      Specifies that the feature is disabled for all machine sets in the cluster.

Verification

  • When the affected nodes return to the READY state, view the current state of the boot image management feature by viewing the machine configuration object: 

    $ oc get machineconfiguration cluster -o yaml
    Copy to Clipboard Toggle word wrap

    Example machine set with the boot image reference

    kind: MachineConfiguration
metadata:
  name: cluster
# ...
status:
  conditions:
  - lastTransitionTime: "2025-05-01T20:11:49Z"
    message: Reconciled 2 of 4 MAPI MachineSets | Reconciled 0 of 0 CAPI MachineSets
      | Reconciled 0 of 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "True"
    type: BootImageUpdateProgressing
  - lastTransitionTime: "2025-05-01T19:30:13Z"
    message: 0 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "False"
    type: BootImageUpdateDegraded
  managedBootImagesStatus:
    machineManagers:
    - apiGroup: machine.openshift.io
      resource: machinesets
      selection:
        mode: All
    Copy to Clipboard Toggle word wrap

  • When the affected nodes return to the READY state, check the current boot image by using one of the following methods:

    • For Google Cloud and AWS, get the boot image version by running the following command. The location and format of the boot image within the machine set differs, based on the platform. However, the boot image is always listed in the spec.template.spec.providerSpec. parameter. 

      $ oc get machinesets <machineset_name> -n openshift-machine-api -o yaml
      Copy to Clipboard Toggle word wrap

      Example machine set with the boot image reference

      apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: ci-ln-77hmkpt-72292-d4pxp
    update-boot-image: "true"
  name: ci-ln-77hmkpt-72292-d4pxp-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
# ...
        value:
          disks:
          - autoDelete: true
            boot: true
            image: projects/rhcos-cloud/global/images/<boot_image>
      1 
      
# ...
      Copy to Clipboard Toggle word wrap

      1
      This boot image is the same as the current OpenShift Container Platform version.

    • For VMware vSphere, get the boot image version from an affected node:

      1. Open an oc debug session to the node by running a command similar to the following: 

        $ oc debug node/<node_name>
        Copy to Clipboard Toggle word wrap

      2. Set /host as the root directory within the debug shell by running the following command: 

        sh-5.1# chroot /host
        Copy to Clipboard Toggle word wrap

      3. Run the rpm-ostree status command to view that the custom layered image is in use: 

        sh-5.1# rpm-ostree status
        Copy to Clipboard Toggle word wrap

        Example output

        State: idle
Deployments:
* ostree-unverified-registry:quay.io/my-registry/...
                   Digest: sha256:...
        Copy to Clipboard Toggle word wrap

6.3. Enabling boot image management
Copy link

By default, for Google Cloud and Amazon Web Services (AWS) clusters, the Machine Config Operator (MCO) updates the boot image in the machine sets in your cluster whenever you update your cluster.

If you disabled the boot image management feature, so that the boot images are not updated, you can re-enable the feature by editing the MachineConfiguration object.

For VMware vSphere, you can enable boot image management as a Technology Preview feature.

Important

Boot image management on vSphere is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Enabling the feature updates the boot image to the current OpenShift Container Platform version. If the cluster is again updated to a new OpenShift Container Platform version in the future, the boot image is updated again. New nodes created after enabling the feature use the updated boot image. This feature has no effect on existing nodes.

Prerequisites

  • For vSphere, enable the TechPreviewNoUpgrade feature set on the cluster. For more information, see "Enabling features using feature gates".

    Note

    Enabling the TechPreviewNoUpgrade feature set cannot be undone and prevents minor version updates. These feature sets are not recommended on production clusters.

    Wait until the managedBootImagesStatus stanza displays in the MachineConfiguration object. 

    apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
# ...
status:
# ...
  managedBootImagesStatus:
    machineManagers:
    - apiGroup: machine.openshift.io
      resource: machinesets
      selection:
        mode: None
    Copy to Clipboard Toggle word wrap

Procedure

  1. Edit the MachineConfiguration object, named cluster, to enable the boot image management feature for some or all of your machine sets: 

    $ oc edit MachineConfiguration cluster
    Copy to Clipboard Toggle word wrap

    • Optional: Enable the boot image management feature for all machine sets: 

      apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
  namespace: openshift-machine-config-operator
spec:
# ...
  managedBootImages:
      1 
      
    machineManagers:
    - apiGroup: machine.openshift.io
      2 
      
      resource: machinesets
      3 
      
      selection:
        mode: All
      4
      Copy to Clipboard Toggle word wrap
      1
      Configures the boot image management feature.
      2
      Specifies the API group. This must be machine.openshift.io.
      3
      Specifies the resource within the specified API group to apply the change. This must be machinesets.
      4
      Specifies that the feature is enabled for all machine sets in the cluster.

    • Optional: Enable the boot image management feature for specific machine sets: 

      apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
  namespace: openshift-machine-config-operator
spec:
# ...
  managedBootImages:
      1 
      
    machineManagers:
    - apiGroup: machine.openshift.io
      2 
      
      resource: machinesets
      3 
      
      selection:
        mode: Partial
      4 
      
        partial:
          machineResourceSelector:
            matchLabels:
              region: "east"
      Copy to Clipboard Toggle word wrap
      1
      Configures the boot image update feature.
      2
      Specifies the API group. This must be machine.openshift.io.
      3
      Specifies the resource within the specified API group to apply the change. This must be machinesets.
      4
      Specifies that the feature is enabled for machine sets with the specified label.
      Tip

      If an appropriate label is not present on the machine set, add a key-value pair by running a command similar to following: 

      $ oc label machineset.machine ci-ln-hmy310k-72292-5f87z-worker-a region="east" -n openshift-machine-api
      Copy to Clipboard Toggle word wrap

Verification

  • When the affected nodes return to the READY state, view the current state of the boot image management feature by viewing the machine configuration object: 

    $ oc get machineconfiguration cluster -o yaml
    Copy to Clipboard Toggle word wrap

    Example machine set with the boot image reference

    kind: MachineConfiguration
metadata:
  name: cluster
# ...
status:
  conditions:
  - lastTransitionTime: "2025-05-01T20:11:49Z"
    message: Reconciled 2 of 4 MAPI MachineSets | Reconciled 0 of 0 CAPI MachineSets
      | Reconciled 0 of 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "True"
    type: BootImageUpdateProgressing
  - lastTransitionTime: "2025-05-01T19:30:13Z"
    message: 0 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments
    reason: BootImageUpdateConfigurationUpdated
    status: "False"
    type: BootImageUpdateDegraded
  managedBootImagesStatus:
    machineManagers:
    - apiGroup: machine.openshift.io
      resource: machinesets
      selection:
        mode: All
    Copy to Clipboard Toggle word wrap

  • When the affected nodes return to the READY state, check the current boot image by using one of the following methods:

    • For Google Cloud and AWS, get the boot image version by running the following command. The location and format of the boot image within the machine set differs, based on the platform. However, the boot image is always listed in the spec.template.spec.providerSpec. parameter. 

      $ oc get machinesets <machineset_name> -n openshift-machine-api -o yaml
      Copy to Clipboard Toggle word wrap

      Example machine set with the boot image reference

      apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: ci-ln-77hmkpt-72292-d4pxp
    update-boot-image: "true"
  name: ci-ln-77hmkpt-72292-d4pxp-worker-a
  namespace: openshift-machine-api
spec:
# ...
  template:
# ...
    spec:
# ...
      providerSpec:
# ...
        value:
          disks:
          - autoDelete: true
            boot: true
            image: projects/rhcos-cloud/global/images/<boot_image>
      1 
      
# ...
      Copy to Clipboard Toggle word wrap

      1
      This boot image is the same as the current OpenShift Container Platform version.

    • For VMware vSphere, get the boot image version from an affected node:

      1. Open an oc debug session to the node by running a command similar to the following: 

        $ oc debug node/<node_name>
        Copy to Clipboard Toggle word wrap

      2. Set /host as the root directory within the debug shell by running the following command: 

        sh-5.1# chroot /host
        Copy to Clipboard Toggle word wrap

      3. Run the rpm-ostree status command to view that the custom layered image is in use: 

        sh-5.1# rpm-ostree status
        Copy to Clipboard Toggle word wrap

        Example output

        State: idle
Deployments:
* ostree-unverified-registry:quay.io/my-registry/...
                   Digest: sha256:...
        Copy to Clipboard Toggle word wrap

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat