Chapter 11. Managing RHEL bootc images
After installing and deploying RHEL bootc images, you can perform management operations on your container images, such as changing or updating the systems. The system supports in-place transactional updates with rollback after deployment. This kind of management, also known as Day 2 management baseline, consists of transactionally fetching new operating system updates from a container registry and booting the system into them, while supporting manual, or automated rollbacks in case of failures.
See Day 2 operations support for more details.
The rhel-bootc images are rebuilt whenever their underlying inputs, such as RPM packages, are updated. These rebuilds occur at least monthly, or more frequently if critical updates are released. As a user, you maintain full control over when to push the update images. A newly published base image does not trigger automatic rebuilds or redeployments of your custom images. You configure the update cadence and only push changes as required.
11.1. Switching the container image reference Copy linkLink copied to clipboard!
You can change the container image reference used for upgrades by using the bootc switch command. For example, you can switch from the stage to the production tag. To manually switch an existing ostree-based container image reference, use the bootc switch command.
Prerequisites
-
A booted system using
bootc.
Procedure
Run the following command:
$ sudo bootc switch [--apply] quay.io/<namespace>/<image>:<tag>Optionally, you can use the
--applyoption when you want to automatically take actions, such as rebooting if the system has changed.NoteThe
bootc switchcommand has the same effect asbootc upgrade. The only difference is the container image reference is changed. This enables preserving the existing states in/etcand/var, for example, host SSH keys and home directories.
11.2. Adding or changing content in the bootc image initramfs Copy linkLink copied to clipboard!
The rhel10/rhel-bootc image uses the dracut infrastructure to build an initial RAM disk (initrd) during the image build time. A default initrd is built and included in the /usr/lib/modules/<kernel_version>/initramfs.img location inside the container image. You can use a drop-in configuration file to extend the dracut configuration, and place it in a file in the /usr/lib/dracut/dracut.conf.d/ directory. This re-creates `initrd with the modules you want to add.
Prerequisites
- A booted system using bootc.
Procedure
Re-create the
initrdas part of a container build:FROM <baseimage> COPY <custom_modules_list>.conf /usr/lib/dracut/dracut.conf.d RUN set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kverNoteBy default the
dracutcommand attempts to pull the running kernel version, which causes an error. Explicitly pass to dracut the kernel version of the target to avoid errors.
11.3. Performing manual updates from an installed operating system Copy linkLink copied to clipboard!
With image mode for RHEL, you can perform any other management task, such as changing or updating the system, by pushing the changes to the container registry.
When using image mode for RHEL, you can choose to perform manual updates for your systems. If you have automatic updates enabled, you must turn them off to perform a manual update. To do so, use one of the following options:
-
Running the
bootc upgradecommand -
Modifying the
systemdtimer file
11.3.1. Turning off automatic updates Copy linkLink copied to clipboard!
To perform manual updates you must turn off automatic updates. You can do this by disabling the timer of the container build, by using one of the following options described in the procedure.
Prerequisites
- A booted system using bootc.
Procedure
Disable the timer of a container build.
By running the
systemctl maskcommand:$ systemctl mask bootc-fetch-apply-updates.timerBy modifying the
systemdtimer file. Usesystemd"drop-ins" to override the timer. In the following example, updates are scheduled for once a week.Create an
updates.conffile with the following content:[Timer] # Clear previous timers OnBootSec= OnBootSec=1w OnUnitInactiveSec=1wAdd you file to the directory that you created:
$ mkdir -p /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d $ cp updates.conf /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d
11.3.2. Manually updating an installed operating system Copy linkLink copied to clipboard!
To manually fetch updates from a registry and boot the system into the new updates, use bootc upgrade. This command fetches the transactional in-place updates from the installed operating system to the container image registry. The command queries the registry and queues an updated container image for the next boot. It stages the changes to the base image, but does not change the running system by default.
Prerequisites
- A booted system using bootc.
Procedure
Run the following command:
$ bootc upgrade [--apply]The
applyargument is optional and you can use it when you want to automatically take actions, such as rebooting if the system has changed.NoteThe
bootc upgradecommand is an alias forbootc update. Both commands have the same effect.For more information, see the
bootc-upgrademan page on your system.
11.3.3. Upgrading images by using download-only mode Copy linkLink copied to clipboard!
You can use the download-only mode to validate updates before application, manage fleet-wide rollouts that include network-intensive downloads, and handle scenarios where system-altering reboots occur at different times. By using the bootc upgrade command with the --download-only flag you can download and stage an update without the system automatically applying it on reboot.
By default, the bootc upgrade command downloads a new container image and stages it for automatic application on the next reboot. You can separate the download phase from the apply phase by using the download-mode only. This mode prevents unintended updates during routine reboots in the production environments with strict maintenance windows.
Prerequisites
- The system is booted system by using bootc.
- Automatic updates are turned off. See Turning off automatic updates.
Procedure
Download the update in download-only mode:
$ bootc upgrade --download-onlyVerify the mode of the staged deployment:
$ bootc status --verboseStaged: Image: quay.io/example/rhel-guest:latest Version: 10.2.20260126 download-only: yesYou can not automatically switch to this version on the next boot, as confirmed by
download-only: yesline in the output.In the maintenance window, you can apply staged updates in one of the following ways in the download-only mode:
Apply staged update without fetching from image source:
$ bootc upgrade --from-downloadedApply the staged update and reboot immediately:
$ bootc upgrade --from-downloaded --applyCheck for newer updates and apply:
$ bootc upgradeRunning the
bootc upgradecommand without flags pulls from the container image source to check for updates. If the staged deployment matches the latest available update, it unlocks the staged deployment. If a newer update is available, the newer version replaces the staged deployment.
Verification
Verify that the staged deployment is in the download-only mode by checking that the output shows the
download-only: yesline:$ bootc status --verboseStaged: Image: quay.io/example/rhel-guest:latest Version: 10.2.20260126 download-only: yes
11.4. Performing rollbacks from an updated operating system Copy linkLink copied to clipboard!
You can roll back to a previous boot entry to revert changes in the system by using the bootc rollback command. This command changes the boot loader entry ordering by making the deployment under rollback queued for the next boot. The current deployment then becomes the rollback. Any staged changes, such as a queued upgrade that was not applied, are discarded.
After a rollback completes, the system reboots and the update timer run within 1 to 3 hours which automatically update and reboot your system to the image you just rolled back from.
If you perform a rollback, the system will automatically update again unless you turn off auto-updates. See Turning off automatic updates.
When performing a rollback, for example, by using the bootc rollback command, changes made to files in the /etc directory do not carry over to the rolled-back deployment. Instead, the files in the /etc directory revert to the state they were in during the previous deployment.
The bootc rollback command reorders existing deployments but does not create new ones. The /etc directory is merged when new deployments are created.
To preserve a modified /etc file for use after a rollback, copy it to a directory under /var, such as /var/home/<user> for a specific <user>, or under /var/root/, for the root user. These directories are unaffected by rollbacks, as they store user content.
When returning to the original state, either through a temporary rollback or another bootc rollback, the /etc directory reverts to its state from the original deployment.
Alternatively, if the issue you are rolling back does not involve configuration files in the /etc directory and you want to revert to an older deployment, use the bootc switch command. This command performs the necessary /etc merge and deploy the previous version of the software.
Prerequisites
- You performed an update to the system.
Procedure
Run the following command:
$ bootc rollback [-h|--help] [-V|--version]NoteThe
bootc rollbackcommand has the same effect asbootc upgrade. The only difference is the container image being tracked. This enables preserving the existing states in /etc and /var, for example, host SSH keys and home directories.
Verification
Use
systemd journalto check the logged message for the detected rollback invocation.$ journalctl -bYou can see a log similar to:
MESSAGE_ID=26f3b1eb24464d12aa5e7b544a6b5468For more information, see the
bootc-rollback(8)man page on your system.
11.5. Deploying updates to system groups Copy linkLink copied to clipboard!
You can change the configuration of your operating system by modifying the Containerfile. The update will be applied after you build and push your container image to the registry, and reboot the operating system.
You can also change the container image source by using the bootc switch command. The content in the container registry determines the specific configuration of the RHEL Image Mode operating system. See Switching the container image reference.
Usually, when deploying updates to system groups, you can use a central management service to provide a client to be installed on each system which connects to the central service. Often, the management service requires the client to perform a one time registration. The following is an example on how to deploy updates to system groups. You can modify, by injecting the credentials for the management service into the image, to create a persistent systemd service, if required.
For clarity reasons, the Containerfile in the example is not optimized. For example, a better optimization to avoid creating multiple layers in the image is by invoking RUN a single time.
You can install a client into a image mode for RHEL image and run it at startup to register the system.
Prerequisites
-
The management-client handles future connections to the server, by using a
cronjob or a separatesystemdservice.
Procedure
Create a management service with the following characteristics. It determines when to upgrade the system.
FROM registry.redhat.io/rhel10/rhel-bootc:latest # Management services determine when to upgrade the system. # Disable bootc-fetch-apply-updates.timer if it is included in the base image. RUN systemctl disable bootc-fetch-apply-updates.timer # Install the client from dnf, or some other method that applies for your client RUN dnf install management-client -y && dnf clean all # Inject the credentials for the management service into the image ARG activation_key= # The existence of .run_next_boot acts as a flag to determine if the # registration is required to run when booting RUN touch /etc/management-client/.run_next_boot COPY <<"EOT" /usr/lib/systemd/system/management-client.service [Unit] Description=Run management client at boot After=network-online.target ConditionPathExists=/etc/management-client/.run_client_next_boot [Service] Type=oneshot EnvironmentFile=/etc/management-client/.credentials ExecStart=/usr/bin/management-client register --activation-key ${CLIENT_ACTIVATION_KEY} ExecStartPre=/bin/rm -f /etc/management-client/.run_next_boot ExecStop=/bin/rm -f /etc/management-client/.credentials [Install] WantedBy=multi-user.target EOT # Link the service to run at startup RUN ln -s /usr/lib/systemd/system/management-client.service /usr/lib/systemd/system/multi-user.target.wants/management-client.service # Store the credentials in a file to be used by the systemd service RUN echo -e "CLIENT_ACTIVATION_KEY=${activation_key}" > /etc/management-client/.credentials # Set the flag to enable the service to run one time # The systemd service will remove this file after the registration completes the first time RUN touch /etc/management-client/.run_next_boot-
Disable
bootc-fetch-apply-updates.timerif it is included in the base image. -
Install the client by using
dnf, or some other method that applies for your client. - Inject the credentials for the management service into the image.
-
Disable
11.6. Checking inventory health Copy linkLink copied to clipboard!
Assess the status of your Red Hat Enterprise Linux managed nodes to verify they are online and reachable. Checking inventory health ensures that your systems are ready to receive updates or configuration changes without interruption. You can manually check the system health of the container images and events that are running inside the container.
Prerequisites
- You pushed the container image to an accessible repository.
-
The
container-toolsmeta-package is installed.
Procedure
Display the health check status of a container.
By using the
podman inspect: orpodman pscommands.$ podman inspect --format='{{json .State.Health.Status}}' <container> healthyBy using the
podman pscommands.$ podman healthcheck run <container> healthy
Monitor and print events that occur in Podman by using the
podman eventscommand. Each event includes a timestamp, a type, a status, a name, if applicable, and an image, if applicable.$ now=$(date --iso-8601=seconds) $ podman events --since=now --stream=false healthy
11.7. Automation and GitOps Copy linkLink copied to clipboard!
You can use CI/CD (Continuous Integration and Continuous Delivery) pipelines to automate the building process of your RHEL bootc images. By using a CI/CD , an event can trigger an update, such as updating an application.
You can use automation tools that track these updates and trigger the CI/CD pipelines, such as GitHub Actions, and GitLab CI. The pipeline keeps the systems up to date by using the transactional background operating system updates.
For more details on resources to create image mode for RHEL instances, check the specific implementations available to create image mode for RHEL instances:RHEL Image Mode CI/CD.