Chapter 9. Managing RHEL bootc images
After installing and deploying RHEL bootc images, you can perform management operations on your container images, such as changing or updating the systems. The system supports in-place transactional updates with rollback after deployment.
This kind of management, also known as Day 2 management baseline, consists of transactionally fetching new operating system updates from a container registry and booting the system into them, while supporting manual, or automated rollbacks in case of failures.
You can also rely on automatic updates, that are turned on by default. The systemd service unit
and the systemd timer unit
files check the container registry for updates and apply them to the system. You can trigger an update process with different events, such as updating an application. There are automation tools watching these updates and then triggering the CI/CD pipelines. A reboot is required, because the updates are transactional. For environments that require more sophisticated or scheduled rollouts, you must disable auto updates and use the bootc
utility to update your operating system.
See Day 2 operations support for more details.
Figure 9.1. Manually updating an installed operating system, changing the container image reference or rolling back changes if needed

9.1. Switching the container image reference
You can change the container image reference used for upgrades by using the bootc switch
command. For example, you can switch from the stage to the production tag. The bootc switch
command performs the same operations as the bootc upgrade
command and additionally changes the container image reference.
To manually switch an existing ostree-based
container image reference, use the bootc switch
command.
The use of rpm-ostree
to make changes, or install content, is not supported.
Prerequisites
-
A booted system using
bootc
.
Procedure
Run the following command:
$ sudo bootc switch [--apply] quay.io/<namespace>/<image>:<tag>
Optionally, you can use the
--apply
option when you want to automatically take actions, such as rebooting if the system has changed.
The bootc switch
command has the same effect as bootc upgrade
. The only difference is the container image reference is changed. This allows preserving the existing states in /etc
and /var
, for example, host SSH keys and home directories.
Additional resources
-
The
bootc-switch
man page
9.2. Adding modules to the bootc image initramfs
The rhel9/rhel-bootc
image uses the dracut
infrastructure to build an initial RAM disk, the initrd
during the image build time. The initrd
is built and included in the /usr/lib/modules/$kver/initramfs.img
location inside the container.
You can use a drop-in configuration file to override the dracut
configuration, and place it in /usr/lib/dracut/dracut.conf.d/<50-custom-added-modules.conf>
And thus re-create initrd
with the modules you want to add.
Prerequisites
- A booted system using bootc.
Procedure
Re-create the
initrd
as part of a container build:FROM <baseimage> COPY <50-custom-added-modules>.conf /usr/lib/dracut/dracut.conf.d RUN set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kver
NoteBy default the command attempts to pull the running kernel version, which causes an error. Explicitly pass to
dracut
the kernel version of the target to avoid errors.
9.3. Modifying and regenerating initrd
The default container image includes a pre-generated initial RAM disk (initrd) in /usr/lib/modules/$kver/initramfs.img
. To regenerate the initrd
, for example, to add a dracut module, follow the steps:
Procedure
Write your drop-in configuration file. For example:
dracutmodules = "module"
Place your drop-in configuration file in the location that
dracut
normally uses:/usr
. For example:/usr/lib/dracut/dracut.conf.d/50-custom-added-modules.conf
Regenerate the
initrd
as part of the container build. You must explicitly pass the kernel version to target todracut
, because it tries to pull the running kernel version, which can cause an error. The following is an example:FROM <baseimage> COPY 50-custom-added-modules.conf /usr/lib/dracut/dracut.conf.d RUN set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kver
9.4. Performing manual updates from an installed operating system
Installing image mode for RHEL is a one time task. You can perform any other management task, such as changing or updating the system, by pushing the changes to the container registry.
When using image mode for RHEL, you can choose to perform manual updates for your systems. Manual updates are also useful if you have an automated way to perform updates, for example, by using Ansible. Because the automatic updates are enabled by default, to perform manual updates you must turn the automatic updates off. You can do this by choosing one of the following options:
-
Running the
bootc upgrade
command -
Modifying the
systemd
timer file
9.5. Turning off automatic updates
To perform manual updates you must turn off automatic updates. You can do this by choosing one of the following options in the procedure below.
Procedure
Disable the timer of a container build.
By running the
systemctl mask
command:$ systemctl mask bootc-fetch-apply-updates.timer
By modifying the
systemd
timer file. Usesystemd
"drop-ins" to override the timer. In the following example, updates are scheduled for once a week.Create an
updates.conf
file with the following content:[Timer] # Clear previous timers OnBootSec= OnBootSec=1w OnUnitInactiveSec=1w
Add you file to the directory you created:
$ mkdir -p /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d $ cp updates.conf /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d
9.6. Manually updating an installed operating system
To manually fetch updates from a registry and boot the system into the new updates, use bootc upgrade
. This command fetches the transactional in-place updates from the installed operating system to the container image registry. The command queries the registry and queues an updated container image for the next boot. It stages the changes to the base image, while not changing the running system by default.
Procedure
Run the following command:
$ bootc upgrade [--apply]
The
apply
argument is optional and you can use it when you want to automatically take actions, such as rebooting if the system has changed.
The bootc upgrade
and bootc update
commands are aliases.
Additional resources
-
The
bootc-upgrade
man page
9.7. Performing rollbacks from a updated operating system
You can roll back to a previous boot entry to revert changes by using the bootc rollback
command. This command changes the boot loader entry ordering by making the deployment under rollback
queued for the next boot. The current deployment then becomes the rollback. Any staged changes, such as a queued upgrade that was not applied, are discarded.
After a rollback completes, the system reboots and the update timer runs within 1 to 3 hours which automatically updates and reboots your system to the image you just rolled back from.
If you perform a rollback, the system will automatically update again unless you turn off auto-updates. See Turning off automatic updates.
Prerequisites
- You performed an update to the system.
Procedure
Run the following command:
$ bootc rollback [-h|--help] [-V|--version]
The bootc rollback
command has the same effect as bootc upgrade
. The only difference is the container image being tracked. This enables preserving the existing states in /etc
and /var
, for example, host SSH keys and home directories.
Verification
Use
systemd journal
to check the logged message for the detected rollback invocation.$ journalctl -b
You can see a log similar to:
MESSAGE_ID=26f3b1eb24464d12aa5e7b544a6b5468
Additional resources
-
The
bootc-rollback
man page
9.8. Deploying updates to system groups
You can change the configuration of your operating system by modifying the Containerfile. Then you can build and push your container image to the registry. When you next boot your operating system, an update will be applied.
You can also change the container image source by using the bootc switch
command. The container registry is the source of truth. See Switching the container image reference.
Usually, when deploying updates to system groups, you can use a central management service to provide a client to be installed on each system which connects to the central service. Often, the management service requires the client to perform a one time registration. The following is an example on how to deploy updates to system groups. You can modify it to create a persistent systemd
service, if required.
For clarity reasons, the Containerfile in the example is not optimized. For example, a better optimization to avoid creating multiple layers in the image is by invoking RUN a single time.
You can install a client into an image mode for RHEL image and run it at startup to register the system.
Prerequisites
-
The management-client handles future connections to the server, by using a
cron
job or a separatesystemd
service.
Procedure
Create a management service with the following characteristics. It determines when to upgrade the system.
-
Disable
bootc-fetch-apply-updates.timer
if it is included in the base image. -
Install the client by using
dnf
, or some other method that applies for your client. - Inject the credentials for the management service into the image.
-
Disable
9.9. Checking inventory health
Health checks are one of the Day 2 Operations. You can manually check the system health of the container images and events that are running inside the container.
You can set health checks by creating the container on the command line. You can display the health check status of a container by using the podman inspect
or podman ps
commands.
You can monitor and print events that occur in Podman by using the podman events
command. Each event includes a timestamp, a type, a status, a name, if applicable, and an image, if applicable.
For more information about health checks and events, see chapter Monitoring containers.
9.10. Automation and GitOps
You can automate the building process by using CI/CD pipelines so that an update process can be triggered by events, such as updating an application. You can use automation tools that track these updates and trigger the CI/CD pipelines. The pipeline keeps the systems up to date by using the transactional background operating system updates.
9.11. Using Toolbx to inspect bootc containers
Installing software on a system presents certain risks: it can change a system’s behavior, and can leave unwanted files and directories behind after they are no longer needed. You can prevent these risks by installing your favorite development and debugging tools, editors, and software development kits (SDKs) into the Toolbx utility included in the RHEL bootc, an image fully mutable container without affecting the base operating system. You can perform changes on the host system with commands such as less
, lsof
, rsync
, ssh
, sudo
, and unzip
.
The Toolbx utility performs the following actions:
-
Pulling the
registry.access.redhat.com/ubi9/toolbox:latest
image to your local system - Starting up a container from the image
- Running a shell inside the container from which you can access the host system
Toolbx can run a root container or a rootless container, depending on the rights of the user who creates the Toolbx container. Utilities that would require root rights on the host system also should be run in root containers.
The default container name is rhel-toolbox
. To inspect bootc containers, follow the steps:
Procedure
Start a Toolbx container by using the
toolbox create
command and enter the container with thetoolbox enter
command.As a rootless user:
$ toolbox create <mytoolbox>
As a root user:
$ sudo toolbox create <mytoolbox> Created container: <mytoolbox> Enter with: toolbox enter
Verify that you pulled the correct image:
[user@toolbox ~]$ toolbox list IMAGE ID IMAGE NAME CREATED fe0ae375f149 registry.access.redhat.com/ubi{ProductVersion}/toolbox 5 weeks ago CONTAINER ID CONTAINER NAME CREATED STATUS IMAGE NAME 5245b924c2cb <mytoolbox> 7 minutes ago created registry.access.redhat.com/ubi{ProductVersion}/toolbox:8.9-6
Enter the Toolbx container:
[user@toolbox ~]$ toolbox enter <mytoolbox>
- Optional: Check if you pulled the correct image
Enter a command inside the
<mytoolbox>
container and display the name of the container and the image:⬢ [user@toolbox ~]$ cat /run/.containerenv engine="podman-4.8.2" name="<mytoolbox>" id="5245b924c2cb..." image="registry.access.redhat.com/ubi{ProductVersion}/toolbox" imageid="fe0ae375f14919cbc0596142e3aff22a70973a36e5a165c75a86ea7ec5d8d65c"
Use the Toolbx to install the development tools:
Install the tools of your choice, for example, the Emacs text editor, GCC compiler and GNU Debugger (GDB):
⬢[user@toolbox ~]$ sudo dnf install emacs gcc gdb
Optional: Verify that the tools are installed:
⬢[user@toolbox ~]$ dnf repoquery --info --installed <package_name>
After installation, you can continue using those tools as a rootless user.
Use Toolbx to troubleshoot the host system without installing them on the host system.
Install the
systemd
suite to be able to run thejournalctl
command:⬢[root@toolbox ~]# dnf install systemd
Display log messages for all processes running on the host:
⬢[root@toolbox ~]# j journalctl --boot -0 Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: microcode: updated ear> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Linux version 6.6.8-10> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Command line: BOOT_IMA> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: x86/split lock detecti> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-provided physical>
Display log messages for the kernel:
⬢[root@toolbox ~]# journalctl --boot -0 --dmesg Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: microcode: updated ear> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Linux version 6.6.8-10> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Command line: BOOT_IMA> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: x86/split lock detecti> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-provided physical> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-e820: [mem 0x0000>
Install the
nmap
network scanning tool:⬢[root@toolbox ~]# dnf install nmap
Scan IP addresses and ports in a network:
⬢[root@toolbox ~]# nmap -sS scanme.nmap.org Starting Nmap 7.93 ( https://nmap.org ) at 2024-01-02 10:39 CET Stats: 0:01:01 elapsed; 0 hosts completed (0 up), 256 undergoing Ping Scan Ping Scan Timing: About 29.79% done; ETC: 10:43 (0:02:24 remaining) Nmap done: 256 IP addresses (0 hosts up) scanned in 206.45 seconds
-
The
-sS
option performs a TCP SYN scan. Most of Nmap’s scan types are only available to privileged users, because they send and receive raw packets, which requires root access on UNIX systems.
-
The
Stop the Toolbx bootc container.
Leave the container and return to the host:
⬢ [user@toolbox ~]$ exit
Stop the toolbox container:
⬢ [user@toolbox ~]$ podman stop <mytoolbox>
Optional: Remove the toolbox container:
⬢ [user@toolbox ~]$ toolbox rm <mytoolbox>
Alternatively, you can also use the
podman rm
command to remove the bootc container.
Additional resources
- Debugging image mode hosts article