Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 11. Managing RHEL bootc images

After installing and deploying RHEL bootc images, you can perform management operations on your container images, such as changing or updating the systems. The system supports in-place transactional updates with rollback after deployment.

This kind of management, also known as Day 2 management baseline, consists of transactionally fetching new operating system updates from a container registry and booting the system into them, while supporting manual, or automated rollbacks in case of failures.

You can also rely on automatic updates, that are turned on by default. The systemd service unit and the systemd timer unit files check the container registry for updates and apply them to the system. You can trigger an update process with different events, such as updating an application. There are automation tools watching these updates and then triggering the CI/CD pipelines. A reboot is required, because the updates are transactional. For environments that require more sophisticated or scheduled rollouts, you must disable auto updates and use the bootc utility to update your operating system.

See Day 2 operations support for more details.

Note

The rhel-bootc images are rebuilt whenever their underlying inputs, such as RPM packages, are updated. These rebuilds occur at least monthly, or more frequently if critical updates are released. As a user, you maintain full control over when to push the update images. A newly published base image does not trigger automatic rebuilds or redeployments of your custom images. You configure the update cadence and only push changes as required.

Figure 11.1. Manually updating an installed operating system, changing the container image reference or rolling back changes if needed

Bootc update diagram

11.1. Switching the container image reference

You can change the container image reference used for upgrades by using the bootc switch command. For example, you can switch from the stage to the production tag. The bootc switch command performs the same operations as the bootc upgrade command and additionally changes the container image reference.

To manually switch an existing ostree-based container image reference, use the bootc switch command.

Warning

The use of rpm-ostree to make changes, or install content, is not supported.

Prerequisites

  • A booted system using bootc.

Procedure

  • Run the following command:

    Copy to Clipboard Toggle word wrap 
    $ sudo bootc switch [--apply] quay.io/<namespace>/<image>:<tag>

    Optionally, you can use the --apply option when you want to automatically take actions, such as rebooting if the system has changed.

Note

The bootc switch command has the same effect as bootc upgrade. The only difference is the container image reference is changed. This allows preserving the existing states in /etc and /var, for example, host SSH keys and home directories.

Additional resources

11.2. Adding modules to the bootc image initramfs

The rhel9/rhel-bootc image uses the dracut infrastructure to build an initial RAM disk, the initrd during the image build time. The initrd is built and included in the /usr/lib/modules/$kver/initramfs.img location inside the container.

You can use a drop-in configuration file to override the dracut configuration, and place it in /usr/lib/dracut/dracut.conf.d/<50-custom-added-modules.conf> And thus re-create initrd with the modules you want to add.

Prerequisites

  • A booted system using bootc.

Procedure

  • Re-create the initrd as part of a container build:

    Copy to Clipboard Toggle word wrap 
    FROM <baseimage>
COPY <50-custom-added-modules>.conf /usr/lib/dracut/dracut.conf.d
RUN set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kver
    Note

    By default the command attempts to pull the running kernel version, which causes an error. Explicitly pass to dracut the kernel version of the target to avoid errors.

11.3. Modifying and regenerating initrd

The default container image includes a pre-generated initial RAM disk (initrd) in /usr/lib/modules/$kver/initramfs.img. To regenerate the initrd, for example, to add a dracut module, follow the steps:

Procedure

  1. Write your drop-in configuration file. For example:

    Copy to Clipboard Toggle word wrap 
    dracutmodules = "module"

  2. Place your drop-in configuration file in the location that dracut normally uses: /usr. For example:

    Copy to Clipboard Toggle word wrap 
    /usr/lib/dracut/dracut.conf.d/50-custom-added-modules.conf

  3. Regenerate the initrd as part of the container build. You must explicitly pass the kernel version to target to dracut, because it tries to pull the running kernel version, which can cause an error. The following is an example:

    Copy to Clipboard Toggle word wrap 
    FROM <baseimage>
COPY 50-custom-added-modules.conf /usr/lib/dracut/dracut.conf.d
RUN set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kver

11.4. Performing manual updates from an installed operating system

Installing image mode for RHEL is a one time task. You can perform any other management task, such as changing or updating the system, by pushing the changes to the container registry.

When using image mode for RHEL, you can choose to perform manual updates for your systems. Manual updates are also useful if you have an automated way to perform updates, for example, by using Ansible. Because the automatic updates are enabled by default, to perform manual updates you must turn the automatic updates off. You can do this by choosing one of the following options:

  • Running the bootc upgrade command
  • Modifying the systemd timer file

11.5. Turning off automatic updates

To perform manual updates you must turn off automatic updates. You can do this by choosing one of the following options in the procedure below.

Procedure

  • Disable the timer of a container build.

    • By running the systemctl mask command:

      Copy to Clipboard Toggle word wrap 
      $ systemctl mask bootc-fetch-apply-updates.timer

    • By modifying the systemd timer file. Use systemd "drop-ins" to override the timer. In the following example, updates are scheduled for once a week.

      1. Create an updates.conf file with the following content:

        Copy to Clipboard Toggle word wrap 
        [Timer]
# Clear previous timers
OnBootSec= OnBootSec=1w OnUnitInactiveSec=1w

      2. Add you file to the directory you created:

        Copy to Clipboard Toggle word wrap 
        $ mkdir -p /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d
$ cp updates.conf /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d

11.6. Manually updating an installed operating system

To manually fetch updates from a registry and boot the system into the new updates, use bootc upgrade. This command fetches the transactional in-place updates from the installed operating system to the container image registry. The command queries the registry and queues an updated container image for the next boot. It stages the changes to the base image, while not changing the running system by default.

Procedure

  • Run the following command:

    Copy to Clipboard Toggle word wrap 
    $ bootc upgrade [--apply]

    The apply argument is optional and you can use it when you want to automatically take actions, such as rebooting if the system has changed.

Note

The bootc upgrade and bootc update commands are aliases.

Additional resources

11.7. Performing rollbacks from a updated operating system

You can roll back to a previous boot entry to revert changes by using the bootc rollback command. This command changes the boot loader entry ordering by making the deployment under rollback queued for the next boot. The current deployment then becomes the rollback. Any staged changes, such as a queued upgrade that was not applied, are discarded.

After a rollback completes, the system reboots and the update timer runs within 1 to 3 hours which automatically updates and reboots your system to the image you just rolled back from.

Warning

If you perform a rollback, the system will automatically update again unless you turn off auto-updates. See Turning off automatic updates.

Note

When performing a rollback, for example, by using the bootc rollback command, changes made to files in the /etc directory do not carry over to the rolled-back deployment. Instead, the files in the /etc directory revert to the state they were in during the previous deployment.

The bootc rollback command reorders existing deployments but does not create new ones. The /etc directory is merged when new deployments are created.

To preserve a modified /etc file for use after a rollback, copy it to a directory under /var, such as /var/home/<user>, for a specific <user>, or under /var/root/, for the root user. These directories are unaffected by rollbacks, as they store user content.

When returning to the original state, either through a temporary rollback or another bootc rollback, the /etc directory reverts to its state from the original deployment.

Alternatively, if the issue you are rolling back does not involve configuration files in the /etc directory and you want to revert to an older deployment, use the bootc switch command. This command performs the necessary /etc merge and deploy the previous version of the software.

Prerequisites

  • You performed an update to the system.

Procedure

  • Run the following command:

    Copy to Clipboard Toggle word wrap 
    $ bootc rollback [-h|--help] [-V|--version]

Verification

  • Use systemd journal to check the logged message for the detected rollback invocation.

    Copy to Clipboard Toggle word wrap 
    $ journalctl -b

    You can see a log similar to:

    Copy to Clipboard Toggle word wrap 
    MESSAGE_ID=26f3b1eb24464d12aa5e7b544a6b5468

Additional resources

11.8. Deploying updates to system groups

You can change the configuration of your operating system by modifying the Containerfile. Then you can build and push your container image to the registry. When you next boot your operating system, an update will be applied.

You can also change the container image source by using the bootc switch command. The container registry is the source of truth. See Switching the container image reference.

Usually, when deploying updates to system groups, you can use a central management service to provide a client to be installed on each system which connects to the central service. Often, the management service requires the client to perform a one time registration. The following is an example on how to deploy updates to system groups. You can modify it to create a persistent systemd service, if required.

Note

For clarity reasons, the Containerfile in the example is not optimized. For example, a better optimization to avoid creating multiple layers in the image is by invoking RUN a single time.

You can install a client into an image mode for RHEL image and run it at startup to register the system.

Prerequisites

  • The management-client handles future connections to the server, by using a cron job or a separate systemd service.

Procedure

  • Create a management service with the following characteristics. It determines when to upgrade the system.

    1. Disable bootc-fetch-apply-updates.timer if it is included in the base image.
    2. Install the client by using dnf, or some other method that applies for your client.
    3. Inject the credentials for the management service into the image.

11.9. Checking inventory health

Health checks are one of the Day 2 Operations. You can manually check the system health of the container images and events that are running inside the container.

You can set health checks by creating the container on the command line. You can display the health check status of a container by using the podman inspect or podman ps commands.

You can monitor and print events that occur in Podman by using the podman events command. Each event includes a timestamp, a type, a status, a name, if applicable, and an image, if applicable.

For more information about health checks and events, see chapter Monitoring containers.

11.10. Automation and GitOps

You can automate the building process by using CI/CD (Continuous Integration and Continuous Delivery) pipelines so that an update process can be triggered by events, such as updating an application. You can use automation tools that track these updates and trigger the CI/CD pipelines. The pipeline keeps the systems up to date by using the transactional background operating system updates.

For more details on resources to create image mode for RHEL instances, check the specific implementations available to create image mode for RHEL instances:RHEL Image Mode CI/CD.

11.11. Using Toolbx to inspect bootc containers

Installing software on a system presents certain risks: it can change a system’s behavior, and can leave unwanted files and directories behind after they are no longer needed. You can prevent these risks by installing your favorite development and debugging tools, editors, and software development kits (SDKs) into the Toolbx utility included in the RHEL bootc, an image fully mutable container without affecting the base operating system. You can perform changes on the host system with commands such as less, lsof, rsync, ssh, sudo, and unzip.

The Toolbx utility performs the following actions:

  1. Pulling the registry.access.redhat.com/ubi9/toolbox:latest image to your local system
  2. Starting up a container from the image
  3. Running a shell inside the container from which you can access the host system
Note

Toolbx can run a root container or a rootless container, depending on the rights of the user who creates the Toolbx container. Utilities that would require root rights on the host system also should be run in root containers.

The default container name is rhel-toolbox. To inspect bootc containers, follow the steps:

Procedure

  1. Start a Toolbx container by using the toolbox create command and enter the container with the toolbox enter command.

    • As a rootless user:

      Copy to Clipboard Toggle word wrap 
      $ toolbox create <mytoolbox>

    • As a root user:

      Copy to Clipboard Toggle word wrap 
      $ sudo toolbox create <mytoolbox>
Created container: <mytoolbox>
Enter with: toolbox enter

    • Verify that you pulled the correct image:

      Copy to Clipboard Toggle word wrap 
      [user@toolbox ~]$ toolbox list
IMAGE ID      IMAGE NAME    CREATED
fe0ae375f149   registry.access.redhat.com/ubi{ProductVersion}/toolbox 5 weeks ago

CONTAINER ID  CONTAINER NAME  CREATED         STATUS   IMAGE NAME
5245b924c2cb  <mytoolbox>       7 minutes ago   created  registry.access.redhat.com/ubi{ProductVersion}/toolbox:8.9-6

      1. Enter the Toolbx container:

        Copy to Clipboard Toggle word wrap 
        [user@toolbox ~]$ toolbox enter <mytoolbox>
      2. Optional: Check if you pulled the correct image

    • Enter a command inside the <mytoolbox> container and display the name of the container and the image:

      Copy to Clipboard Toggle word wrap 
      ⬢ [user@toolbox ~]$ cat /run/.containerenv
engine="podman-4.8.2"
name="<mytoolbox>"
id="5245b924c2cb..."
image="registry.access.redhat.com/ubi{ProductVersion}/toolbox"
imageid="fe0ae375f14919cbc0596142e3aff22a70973a36e5a165c75a86ea7ec5d8d65c"

  2. Use the Toolbx to install the development tools:

    1. Install the tools of your choice, for example, the Emacs text editor, GCC compiler and GNU Debugger (GDB):

      Copy to Clipboard Toggle word wrap 
      ⬢[user@toolbox ~]$ sudo dnf install emacs gcc gdb

    2. Optional: Verify that the tools are installed:

      Copy to Clipboard Toggle word wrap 
      ⬢[user@toolbox ~]$  dnf repoquery --info --installed <package_name>

      After installation, you can continue using those tools as a rootless user.

  3. Use Toolbx to troubleshoot the host system without installing them on the host system.

    1. Install the systemd suite to be able to run the journalctl command:

      Copy to Clipboard Toggle word wrap 
      ⬢[root@toolbox ~]# dnf install systemd

    2. Display log messages for all processes running on the host:

      Copy to Clipboard Toggle word wrap 
      ⬢[root@toolbox ~]# j journalctl --boot -0
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: microcode: updated ear>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Linux version 6.6.8-10>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Command line: BOOT_IMA>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: x86/split lock detecti>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-provided physical>

    3. Display log messages for the kernel:

      Copy to Clipboard Toggle word wrap 
      ⬢[root@toolbox ~]# journalctl --boot -0 --dmesg
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: microcode: updated ear>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Linux version 6.6.8-10>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Command line: BOOT_IMA>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: x86/split lock detecti>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-provided physical>
Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-e820: [mem 0x0000>

    4. Install the nmap network scanning tool:

      Copy to Clipboard Toggle word wrap 
      ⬢[root@toolbox ~]# dnf install nmap

    5. Scan IP addresses and ports in a network:

      Copy to Clipboard Toggle word wrap 
      ⬢[root@toolbox ~]# nmap -sS scanme.nmap.org
Starting Nmap 7.93 ( https://nmap.org ) at 2024-01-02 10:39 CET
Stats: 0:01:01 elapsed; 0 hosts completed (0 up), 256 undergoing Ping Scan
Ping Scan Timing: About 29.79% done; ETC: 10:43 (0:02:24 remaining)
Nmap done: 256 IP addresses (0 hosts up) scanned in 206.45 seconds
      • The -sS option performs a TCP SYN scan. Most of Nmap’s scan types are only available to privileged users, because they send and receive raw packets, which requires root access on UNIX systems.

  4. Stop the Toolbx bootc container.

    1. Leave the container and return to the host:

      Copy to Clipboard Toggle word wrap 
      ⬢ [user@toolbox ~]$ exit

    2. Stop the toolbox container:

      Copy to Clipboard Toggle word wrap 
      ⬢ [user@toolbox ~]$ podman stop <mytoolbox>

    3. Optional: Remove the toolbox container:

      Copy to Clipboard Toggle word wrap 
      ⬢ [user@toolbox ~]$ toolbox rm <mytoolbox>

      Alternatively, you can also use the podman rm command to remove the bootc container.

Additional resources

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat, Inc.