Using image mode for RHEL to build, deploy, and manage operating systems

Use the podman build command to build an image using instructions from a Containerfile.

Use the podman run command to run and test your container.

Use the podman push command to push an image to your own, or a third party, registry and share it with others. The following procedure uses the Red Hat Quay registry.

With the bootc-image-builder tool, you can convert bootc images into disk images for a variety of different platforms and formats. Converting bootc images into disk images is equivalent to installing a bootc. After you deploy these disk images to the target environment, you can update them directly from the container registry.

The bootc-image-builder tool supports generating the following image types:

Deploying from a container image is beneficial when you run VMs or servers because you can achieve the same installation result. That consistency extends across multiple different image types and platforms when you build them from the same container image. Consequently, you can minimize the effort in maintaining operating system images across platforms. You can also update systems that you deploy from these disk images by using the bootc tool, instead of re-creating and uploading new disk images with bootc-image-builder.

Although you can deploy a rhel-9-bootc image directly, you can also create your own customized images that are derived from this bootc image. The bootc-image-builder tool takes the rhel-9-bootc OCI container image as an input.

The bootc-image-builder is intended to be used as a container and it is not available as an RPM package in RHEL. To access it, follow the procedure.

Build a RHEL bootc image into a QEMU Disk Images (QCOW2) image for the architecture that you are running the commands on.

The RHEL base image does not include a default user. Optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

Create an Amazon Machine Image (AMI) from a bootc image and use it to launch an Amazon Web Service EC2 (Amazon Elastic Compute Cloud) instance.

You can convert a bootc image to a Raw image with an MBR or GPT partition table by using bootc-image-builder. The RHEL base image does not include a default user, so optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

You can use bootc-image-builder to create an ISO from which you can perform an offline deployment of a bootable container.

After creating a QEMU disk image from a RHEL bootc image by using the bootc-image-builder tool, you can use a virtualization software to boot it.

After using the bootc-image-builder tool to create an AMI from a bootc image, and uploading it to a AWS s3 bucket, you can deploy a container image to AWS with the AMI disk image.

After you convert your bootc image to an ISO image by using bootc-image-builder, you can deploy the ISO image by using Anaconda and Kickstart to install your container image. The installable boot ISO already contains the ostreecontainer Kickstart file configured that you can use to provision your custom container image.

Convert a bootc image to an ISO image by using bootc-image-builder. This creates a system similar to the RHEL ISOs available for download, except that your container image content is embedded in the ISO disk image. You do not need to have access to the network during installation. Then, you install the ISO disk image that you created from bootc-image-builder to a bare metal system.

You can use a network installation to deploy the RHEL ISO image over PXE boot to run your ISO bootc image.

You can inject configuration into a custom image by using a Containerfile.

With bootc, you have a container that is the source of truth. It contains a basic build installer and it is available as bootc install to-disk or bootc install to-filesystem. By using the bootc install methods you do not need to perform any additional steps to deploy the container image, because the container images include a basic installer.

With image mode for RHEL, you can install unconfigured images, for example, images that do not have a default password or SSH key.

Perform a bare-metal installation to a device by using a RHEL ISO image.

The bootc install contains two subcommands: bootc install to-disk and bootc install to-filesystem.

You can change the container image reference used for upgrades by using the bootc switch command. For example, you can switch from the stage to the production tag. The bootc switch command performs the same operations as the bootc upgrade command and additionally changes the container image reference.

To manually switch an existing ostree-based container image reference, use the bootc switch command.

The rhel9/rhel-bootc image uses the dracut infrastructure to build an initial RAM disk, the initrd during the image build time. The initrd is built and included in the /usr/lib/modules/$kver/initramfs.img location inside the container.

You can use a drop-in configuration file to override the dracut configuration, and place it in /usr/lib/dracut/dracut.conf.d/<50-custom-added-modules.conf> And thus re-create initrd with the modules you want to add.

The default container image includes a pre-generated initial RAM disk (initrd) in /usr/lib/modules/$kver/initramfs.img. To regenerate the initrd, for example, to add a dracut module, follow the steps:

Installing image mode for RHEL is a one time task. You can perform any other management task, such as changing or updating the system, by pushing the changes to the container registry.

When using image mode for RHEL, you can choose to perform manual updates for your systems. Manual updates are also useful if you have an automated way to perform updates, for example, by using Ansible. Because the automatic updates are enabled by default, to perform manual updates you must turn the automatic updates off. You can do this by choosing one of the following options:

To perform manual updates you must turn off automatic updates. You can do this by choosing one of the following options in the procedure below.

To manually fetch updates from a registry and boot the system into the new updates, use bootc upgrade. This command fetches the transactional in-place updates from the installed operating system to the container image registry. The command queries the registry and queues an updated container image for the next boot. It stages the changes to the base image, while not changing the running system by default.

You can roll back to a previous boot entry to revert changes by using the bootc rollback command. This command changes the boot loader entry ordering by making the deployment under rollback queued for the next boot. The current deployment then becomes the rollback. Any staged changes, such as a queued upgrade that was not applied, are discarded.

After a rollback completes, the system reboots and the update timer run within 1 to 3 hours which automatically update and reboot your system to the image you just rolled back from.

You can change the configuration of your operating system by modifying the Containerfile. Then you can build and push your container image to the registry. When you next boot your operating system, an update will be applied.

You can also change the container image source by using the bootc switch command. The container registry is the source of truth. See Switching the container image reference.

Usually, when deploying updates to system groups, you can use a central management service to provide a client to be installed on each system which connects to the central service. Often, the management service requires the client to perform a one time registration. The following is an example on how to deploy updates to system groups. You can modify it to create a persistent systemd service, if required.

You can install a client into a image mode for RHEL image and run it at startup to register the system.

Health checks are one of the Day 2 Operations. You can manually check the system health of the container images and events that are running inside the container.

You can set health checks by creating the container on the command line. You can display the health check status of a container by using the podman inspect or podman ps commands.

You can monitor and print events that occur in Podman by using the podman events command. Each event includes a timestamp, a type, a status, a name, if applicable, and an image, if applicable.

For more information about health checks and events, see chapter Monitoring containers.

You can automate the building process by using CI/CD pipelines so that an update process can be triggered by events, such as updating an application. You can use automation tools that track these updates and trigger the CI/CD pipelines. The pipeline keeps the systems up to date by using the transactional background operating system updates.

The bootc tool uses generic operating system kernels. You can add support to inject kernel arguments by adding a custom configuration, in the TOML format, in /usr/lib/bootc/kargs.d. For example:

# /usr/lib/bootc/kargs.d/10-example.toml
kargs = ["mitigations=auto,nosmt"]

You can also make these kernel arguments architecture-specific by using the match-architectures key. For example:

# /usr/lib/bootc/kargs.d/00-console.toml
kargs = ["console=ttyS0,114800n8"]
match-architectures = ["x86_64"]

You can use bootc install to add kernel arguments during the install time in the following ways:

You can use the kernel arguments on Day 2 operations, by adding the arguments and applying them on a switch, upgrade, or edit. Adding kernel arguments and using it for Day 2 operations involves the following high-level steps:

To add kernel arguments into a container image, use a Containerfile. The following is an example:

FROM registry.redhat.io/rhel9/rhel-bootc:latest

RUN mkdir -p /usr/lib/bootc/kargs.d
RUN cat <<EOF >> /usr/lib/bootc/kargs.d/console.toml
kargs = ["console=ttyS0,114800n8"]
match-architectures = ["x86_64"]
EOF

RUN cat <<EOF >> /usr/lib/bootc/kargs.d/01-mitigations.toml
kargs = ["mitigations=on", "systemd.unified_cgroup_hierarchy=0"]
match-architectures = ["x86_64", "aarch64"]
EOF

You can use boot install with the --karg to inject kernel arguments during installation time. As a result, the kernel arguments become machine-local state.

For example, to inject kernel arguments, use the following command:

# bootc install to-filesystem --karg

The bootc-image-builder tool supports the customizations.kernel.append during install-time.

To add the kernel arguments with bootc-image-builder, use the following customization:

{
  "customizations": {
    "kernel": {
      "append": "mitigations=auto,nosmt"
    }
  }
}

The changes that you make to kargs.d files and include in a container build are applied after the installation, and the difference between the set of kernel arguments is applied to the current boot loader configuration. This preserves any machine-local kernel arguments. You can use any tool to edit the /boot/loader/entries files, which are in a standardized format. The /boot file has read-only access to limit the set of tools that can write to this filesystem.

To perform machine local changes, you also can edit kernel arguments on a bootc system or an`rpm-ostree` system, by using the rpm-ostree kargs command. The changes are made through the user/lib/bootc/kargs.d path, which also handles "Day 2" changes, besides the first boot changes.

The following are the options that you can use to add, modify or remove kernel arguments.

rpm-ostree kargs [option]

For more information, check the help:

# rpm-ostree kargs --help

The following is an example:

# rpm-ostree kargs --append debug
Staging deployment... done
Freed: 40.1 MB (pkgcache branches: 0)
Changes queued for next boot. Run "systemctl reboot" to start a reboot

When a system is fully booted, it is similar to chroot, that is, the operating system changes the apparent root directory for the current running process and its children. The physical host root filesystem is mounted at /sysroot. The chroot filesystem is called a deployment root.

The remaining filesystem paths are part of a deployment root which is used as a final target for the system boot. The system uses the ostree=kernel argument to find the deployment root.

The ostree-finalize-staged.service executes these tasks during shutdown time, before creating the new boot loader entry.

This happens because many components of a Linux system ship default configuration files in the /etc directory. Even if the default package does not ship it, by default the software only checks for config files in /etc. Non bootc image based update systems with no distinct versions of /etc are populated only during the installation time, and will not be changed at any point after installation. This causes the /etc system state to be influenced by the initial image version and can lead to problems to apply a change, for example, to /etc/sudoers.conf, and requires external intervention. For more details about file configuration, see Building and testing RHEL bootc images.

There is just one /var directory. If it is not a distinct partition, then physically the /var directory is a bind mount into /ostree/deploy/$stateroot/var and is shared across the available boot loader entries deployments.

By default, the content in /var acts as a volume, that is, the content from the container image is copied during the initial installation time, and is not updated thereafter.

The /var and the /etc directories are different. You can use /etc for relatively small configuration files, and the expected configuration files are often bound to the operating system binaries in /usr. The /var directory has arbitrarily large data, for example, system logs, databases, and by default, will not be rolled back if the operating system state is rolled back.

For example, making an update such as dnf downgrade postgresql should not affect the physical database in /var/lib/postgres. Similarly, making a bootc update or bootc rollback do not affect this application data.

Having /var separate also makes it work cleanly to stage new operating system updates before applying them, that is, updates are downloaded and ready, but only take effect on reboot. The same applies for Docker volume, as it decouples the application code from its data.

You can use this case if you want applications to have a pre-created directory structure, for example, /var/lib/postgresql. Use systemd tmpfiles.d for this. You can also use StateDirectory=<directory> in units.

When a software needs to write to its own directory in /opt/exampleapp, a common pattern is to use a symbolic link to redirect to, for example, /var for operations such as log files:

RUN rmdir /opt/exampleapp/logs && ln -sr /var/log/exampleapp /opt/exampleapp/logs

Optionally, you can configure the systemd unit to launch the service to do these mounts dynamically. For example:

BindPaths=/var/log/exampleapp:/opt/exampleapp/logs

[root]
transient = true

This enables a software to transiently writes to /opt, with symlinks to /var for content that must persist.

Image mode for RHEL uses GRUB by default, with exception to s390x architectures. Each version of image mode for RHEL currently available on a system has a menu entry.

The menu entry references an OSTree deployment which consists of a Linux kernel, an initramfs and a hash linking to an OSTree commit, that you can pass by using the ostree=kernel argument.

During bootup, OSTree reads the kernel argument to determine which deployment to use as the root filesystem. Each update or change to the system, such as package installation, addition of kernel arguments, creates a new deployment.

This enables rolling back to a previous deployment if the update causes problems.

Image mode for RHEL is a generic operating system update and configuration mechanism. You cannot use it to configure users or groups. The only exception is the bootc install command that has the --root-ssh-authorized-keys option.

Image mode for RHEL does not have an opinionated mechanism for secrets. You can inject container pull secrets in your system for some cases, for example: