Using image mode for RHEL to build, deploy, and manage operating systems

Use the podman build command to build an image using instructions from a Containerfile.

The deployment image should include only the application and its required runtime, without adding any build tools or unnecessary libraries. To achieve this, use a two-stage Containerfile: one stage for building the artifacts and another for hosting the application.

With multi-stage builds, you use multiple FROM instructions in your Containerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build. You can selectively copy artifacts from one stage to another, and exclude everything you do not need in the final image.

Multi-stage builds offer several advantages:

The following Containerfile consists of two stages. The first stage is typically named builder and it compiles a golang binary. The second stage copies the binary from the first stage. The default working directory for the go-toolset builder is opt/ap-root/src.

FROM registry.access.redhat.com/ubi9/go-toolset:latest as builder
RUN echo 'package main; import "fmt"; func main() { fmt.Println("hello world") }' > helloworld.go
RUN go build helloworld.go

FROM registry.redhat.io/rhel9/rhel-bootc:latest
COPY --from=builder /opt/app-root/src/helloworld /
CMD ["/helloworld"]

As a result, the final container image includes the helloworld binary but no data from the previous stage.

You can also use multi-stage builds to perform the following scenarios:

$ podman build --target build -t hello .

For example, you can use this approach to debugging a specific build stage.

COPY --from=<image> <source_path> <destination_path>_

FROM ubi9 AS stage1
[...]

FROM stage1 AS stage2
[...]

FROM ubi9 AS final-stage
[...]

Use the podman run command to run and test your container.

Use the podman push command to push an image to your own, or a third party, registry and share it with others. The following procedure uses the Red Hat Quay registry.

Logically bound images enable an association of the container application images to a base bootc system image. The term logically bound is used to contrast with physically bound images. The logically bound images offer the following benefits:

The following are examples for lifecycle bound workloads, whose activities are usually not updated outside of the host:

Another important property of the logically bound images is that they must be present and available on the host, possibly from a very early stage in the boot process.

Differently from the default usage of tools like Podman or Docker, images might be pulled dynamically after the boot starts, which requires a functioning network. For example, if the remote registry is temporarily unavailable, the host system might run longer without log forwarding or monitoring, which is not desirable. Logically bound images enable you to reference container images similarly to you can with ExecStart= in a systemd unit.

When using logically bound images, you must manage multiple container images for the system to install the logically bound images. This is an advantage and also a disadvantage. For example, for a disconnected or offline installation, you must mirror all the containers, not just one. The application images are only referenced from the base image as .image files or an equivalent.

Each logically bound image is defined in a Podman Quadlet .image or .container file. To use logically bound images, follow the steps:

The bootc image store is owned by bootc. The logically bound images will be garbage collected when they are no longer referenced by a file in the /usr/lib/bootc/bound-images.d directory.

With the bootc-image-builder tool, you can convert bootc images into disk images for a variety of different platforms and formats. Converting bootc images into disk images is equivalent to installing a bootc. After you deploy these disk images to the target environment, you can update them directly from the container registry.

The bootc-image-builder tool supports generating the following image types:

Deploying from a container image is beneficial when you run VMs or servers because you can achieve the same installation result. That consistency extends across multiple different image types and platforms when you build them from the same container image. Consequently, you can minimize the effort in maintaining operating system images across platforms. You can also update systems that you deploy from these disk images by using the bootc tool, instead of re-creating and uploading new disk images with bootc-image-builder.

Although you can deploy a rhel-9-bootc image directly, you can also create your own customized images that are derived from this bootc image. The bootc-image-builder tool takes the rhel-9-bootc OCI container image as an input.

The bootc-image-builder is intended to be used as a container and it is not available as an RPM package in RHEL. To access it, follow the procedure.

Build a RHEL bootc image into a QEMU Disk Images (QCOW2) image for the architecture that you are running the commands on.

The RHEL base image does not include a default user. Optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

Create a Virtual Machine Disk (VMDK) from a bootc image and use it within VMware’s virtualization platforms, such as vSphere, or use the Virtual Machine Disk (VMDK) in VirtualBox.

A VMDK disk file for the bootc image is stored in the output/vmdk directory.

Build a RHEL bootc image into a gce image for the architecture that you are running the commands on. The RHEL base image does not include a default user. Optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

Create an Amazon Machine Image (AMI) from a bootc image and use it to launch an Amazon Web Service EC2 (Amazon Elastic Compute Cloud) instance.

You can convert a bootc image to a Raw image with an MBR or GPT partition table by using bootc-image-builder. The RHEL base image does not include a default user, so optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

You can use bootc-image-builder to create an ISO from which you can perform an offline deployment of a bootable container.

You can use a Kickstart file to configure various parts of the installation process, such as setting up users, customizing partitioning, and adding an SSH key. You can include the Kickstart file in an ISO build to configure any part of the installation process, except the deployment of the base image. For ISOs with bootc container base images, you can use a Kickstart file to configure anything except the ostreecontainer command.

For example, you can use a Kickstart to perform either a partial installation, a full installation, or even omit the user creation. Use bootc-image-builder to build an ISO image that contains the custom Kickstart to configure your installation process.

The following are the general principles about partitions:

The disk customizations option provides a more powerful interface to control the whole partitioning layout of the image.

When using the Disk customizations, you can describe the partition table almost entirely by using a blueprint. The customizations have the following structure:

The order of each element in a list is respected when creating the partition table. The partitions are created in the order they are defined, regardless of their type.

Incomplete partitioning descriptions are valid. Partitions, LVM logical volumes, are added automatically to create a valid partition table. The following rules are applied:

You can define multiple partitions. The following combination of partition types are valid:

The following blueprint defines two partitions. The first is a 50 GiB partition with an ext4 filesystem that will be mounted at /data. The second is an LVM volume group with three logical volumes, one for root /, one for home directories /home, and a swap space in that order. The LVM volume group will have 15 GiB of non-allocated space.

[[customizations.disk.partitions]]
type = "plain"
label = "data"
mountpoint = "/data"
fs_type = "ext4"
minsize = "50 GiB"

[[customizations.disk.partitions]]
type = "lvm"
name = "mainvg"
minsize = "20 GiB"

[[customizations.disk.partitions.logical_volumes]]
name = "rootlv"
mountpoint = "/"
label = "root"
fs_type = "ext4"
minsize = "2 GiB"

[[customizations.disk.partitions.logical_volumes]]
name = "homelv"
mountpoint = "/home"
label = "home"
fs_type = "ext4"
minsize = "2 GiB"

[[customizations.disk.partitions.logical_volumes]]
name = "swaplv"
fs_type = "swap"
minsize = "1 GiB"

The filesystem customization option provides the final partition table of an image that you built with image builder, and it is determined by a combination of the following factors:

The following describes how these factors affect the final layout of the partition table.

You can modify the partition table by taking the following aspects in consideration:

The partitioning mode controls how the partition table is modified from the image type’s default layout.

You can define new filesystems and minimum partition sizes by using the filesystems customization in the blueprint. By default, if new mountpoints are created, a partition table is automatically converted to LVM. See the Partitioning modes​ entry for more details.

To create a disk image of a very specific size, you must ensure that the following requirements are met:

The following are overall steps to create a disk image of a very specific size in the TOML file:

With this, you create a disk with a partition table of the desired size with each partition sized to fit the desired mountpoints. The root partition, root LVM Logical Volume, will be at least 3 GiB, or 1 GiB if /usr is specified. See Understanding partitions for more details.

Create image mode disk images with advanced partitioning by using bootc-image-builder. The image mode disk images that you create of image mode RHEL with custom mount points, include custom mount options, LVM-based partitions and LVM-based SWAP. With that you can, for example, change the size of the / and the /boot directories by using a config.toml file. When installing the RHEL image mode on bare-metal machines, you can benefit from all partitioning features available on Anaconda.

Use bound mounts to override the container’s store with the host’s.

Create a new container image with your custom certificates by building it using a Containerfile.

After creating a QEMU disk image from a RHEL bootc image by using the bootc-image-builder tool, you can use a virtualization software to boot it.

After using the bootc-image-builder tool to create an AMI from a bootc image, and uploading it to a AWS s3 bucket, you can deploy a container image to AWS with the AMI disk image.

You can deploy the RHEL ISO image that you downloaded from Red Hat by using Anaconda and Kickstart to install your container image.

After you build an ISO image by using bootc-image-builder, the resulting image is a system similar to the RHEL ISOs available for download, except that your container image content is embedded in the ISO disk image. You do not need to have access to the network during installation. You can install the resulting ISO disk image to a bare metal system. See Creating ISO images by using bootc-image-builder.

You can use a network installation to deploy the RHEL ISO image over PXE boot to run your ISO bootc image.

You can inject configuration into a custom image by using a Containerfile.

With bootc, you have a container that is the source of truth. It contains a basic build installer and it is available as bootc install to-disk or bootc install to-filesystem. By using the bootc install methods you do not need to perform any additional steps to deploy the container image, because the container images include a basic installer.

With image mode for RHEL, you can install unconfigured images, for example, images that do not have a default password or SSH key.

Perform a bare-metal installation to a device by using a RHEL ISO image.

The bootc install contains two subcommands: bootc install to-disk and bootc install to-filesystem.

Create a disk image by using bootc-image-builder or bootc install to-disk, and enable the FIPS mode by passing the custom Containerfile as an argument when building the image.

To create a disk image and enable the FIPS mode when performing an Anaconda installation, follow the steps:

You can change the container image reference used for upgrades by using the bootc switch command. For example, you can switch from the stage to the production tag. The bootc switch command performs the same operations as the bootc upgrade command and additionally changes the container image reference.

To manually switch an existing ostree-based container image reference, use the bootc switch command.

The rhel9/rhel-bootc image uses the dracut infrastructure to build an initial RAM disk, the initrd during the image build time. The initrd is built and included in the /usr/lib/modules/$kver/initramfs.img location inside the container.

You can use a drop-in configuration file to override the dracut configuration, and place it in /usr/lib/dracut/dracut.conf.d/<50-custom-added-modules.conf> And thus re-create initrd with the modules you want to add.

The default container image includes a pre-generated initial RAM disk (initrd) in /usr/lib/modules/$kver/initramfs.img. To regenerate the initrd, for example, to add a dracut module, follow the steps:

Installing image mode for RHEL is a one time task. You can perform any other management task, such as changing or updating the system, by pushing the changes to the container registry.

When using image mode for RHEL, you can choose to perform manual updates for your systems. Manual updates are also useful if you have an automated way to perform updates, for example, by using Ansible. Because the automatic updates are enabled by default, to perform manual updates you must turn the automatic updates off. You can do this by choosing one of the following options:

To perform manual updates you must turn off automatic updates. You can do this by choosing one of the following options in the procedure below.

To manually fetch updates from a registry and boot the system into the new updates, use bootc upgrade. This command fetches the transactional in-place updates from the installed operating system to the container image registry. The command queries the registry and queues an updated container image for the next boot. It stages the changes to the base image, while not changing the running system by default.

You can roll back to a previous boot entry to revert changes by using the bootc rollback command. This command changes the boot loader entry ordering by making the deployment under rollback queued for the next boot. The current deployment then becomes the rollback. Any staged changes, such as a queued upgrade that was not applied, are discarded.

After a rollback completes, the system reboots and the update timer runs within 1 to 3 hours which automatically updates and reboots your system to the image you just rolled back from.

You can change the configuration of your operating system by modifying the Containerfile. Then you can build and push your container image to the registry. When you next boot your operating system, an update will be applied.

You can also change the container image source by using the bootc switch command. The container registry is the source of truth. See Switching the container image reference.

Usually, when deploying updates to system groups, you can use a central management service to provide a client to be installed on each system which connects to the central service. Often, the management service requires the client to perform a one time registration. The following is an example on how to deploy updates to system groups. You can modify it to create a persistent systemd service, if required.

You can install a client into an image mode for RHEL image and run it at startup to register the system.

Health checks are one of the Day 2 Operations. You can manually check the system health of the container images and events that are running inside the container.

You can set health checks by creating the container on the command line. You can display the health check status of a container by using the podman inspect or podman ps commands.

You can monitor and print events that occur in Podman by using the podman events command. Each event includes a timestamp, a type, a status, a name, if applicable, and an image, if applicable.

For more information about health checks and events, see chapter Monitoring containers.

You can automate the building process by using CI/CD pipelines so that an update process can be triggered by events, such as updating an application. You can use automation tools that track these updates and trigger the CI/CD pipelines. The pipeline keeps the systems up to date by using the transactional background operating system updates.

Installing software on a system presents certain risks: it can change a system’s behavior, and can leave unwanted files and directories behind after they are no longer needed. You can prevent these risks by installing your favorite development and debugging tools, editors, and software development kits (SDKs) into the Toolbx utility included in the RHEL bootc, an image fully mutable container without affecting the base operating system. You can perform changes on the host system with commands such as less, lsof, rsync, ssh, sudo, and unzip.

The Toolbx utility performs the following actions:

The default container name is rhel-toolbox. To inspect bootc containers, follow the steps:

The bootc tool uses generic operating system kernels. You can add support to inject kernel arguments by adding a custom configuration, in the TOML format, in /usr/lib/bootc/kargs.d. For example:

# /usr/lib/bootc/kargs.d/10-example.toml
kargs = ["mitigations=auto,nosmt"]

You can also make these kernel arguments architecture-specific by using the match-architectures key. For example:

# /usr/lib/bootc/kargs.d/00-console.toml
kargs = ["console=ttyS0,114800n8"]
match-architectures = ["x86_64"]

You can use bootc install to add kernel arguments during the install time in the following ways:

You can use the kernel arguments on Day 2 operations, by adding the arguments and applying them on a switch, upgrade, or edit. Adding kernel arguments and using it for Day 2 operations involves the following high-level steps:

To add kernel arguments into a container image, use a Containerfile. The following is an example:

FROM registry.redhat.io/rhel9/rhel-bootc:latest

RUN mkdir -p /usr/lib/bootc/kargs.d
RUN cat <<EOF >> /usr/lib/bootc/kargs.d/console.toml
kargs = ["console=ttyS0,114800n8"]
match-architectures = ["x86_64"]
EOF

RUN cat <<EOF >> /usr/lib/bootc/kargs.d/01-mitigations.toml
kargs = ["mitigations=on", "systemd.unified_cgroup_hierarchy=0"]
match-architectures = ["x86_64", "aarch64"]
EOF

You can use boot install with the --karg to inject kernel arguments during installation time. As a result, the kernel arguments become machine-local state.

For example, to inject kernel arguments, use the following command:

# bootc install to-filesystem --karg

The bootc-image-builder tool supports the customizations.kernel.append during install-time.

To add the kernel arguments with bootc-image-builder, use the following customization:

{
  "customizations": {
    "kernel": {
      "append": "mitigations=auto,nosmt"
    }
  }
}

The changes that you make to kargs.d files and include in a container build are applied after the installation, and the difference between the set of kernel arguments is applied to the current boot loader configuration. This preserves any machine-local kernel arguments. You can use any tool to edit the /boot/loader/entries files, which are in a standardized format. The /boot file has read-only access to limit the set of tools that can write to this filesystem.

To perform machine local changes, you also can edit kernel arguments on a bootc system or an`rpm-ostree` system, by using the rpm-ostree kargs command. The changes are made through the user/lib/bootc/kargs.d path, which also handles "Day 2" changes, besides the first boot changes.

The following are the options that you can use to add, modify or remove kernel arguments.

rpm-ostree kargs [option]

For more information, check the help:

# rpm-ostree kargs --help

The following is an example:

# rpm-ostree kargs --append debug
Staging deployment... done
Freed: 40.1 MB (pkgcache branches: 0)
Changes queued for next boot. Run "systemctl reboot" to start a reboot

When a system is fully booted, it is similar to chroot, that is, the operating system changes the apparent root directory for the current running process and its children. The physical host root filesystem is mounted at /sysroot. The chroot filesystem is called a deployment root.

The remaining filesystem paths are part of a deployment root which is used as a final target for the system boot. The system uses the ostree=kernel argument to find the deployment root.

The ostree-finalize-staged.service executes these tasks during shutdown time, before creating the new boot loader entry.

This happens because many components of a Linux system ship default configuration files in the /etc directory. Even if the default package does not ship it, by default the software only checks for config files in /etc. Non bootc image based update systems with no distinct versions of /etc are populated only during the installation time, and will not be changed at any point after installation. This causes the /etc system state to be influenced by the initial image version and can lead to problems to apply a change, for example, to /etc/sudoers.conf, and requires external intervention. For more details about file configuration, see Building and testing RHEL bootc images.

There is just one /var directory. If it is not a distinct partition, then physically the /var directory is a bind mount into /ostree/deploy/$stateroot/var and is shared across the available boot loader entries deployments.

By default, the content in /var acts as a volume, that is, the content from the container image is copied during the initial installation time, and is not updated thereafter.

The /var and the /etc directories are different. You can use /etc for relatively small configuration files, and the expected configuration files are often bound to the operating system binaries in /usr. The /var directory has arbitrarily large data, for example, system logs, databases, and by default, will not be rolled back if the operating system state is rolled back.

For example, making an update such as dnf downgrade postgresql should not affect the physical database in /var/lib/postgres. Similarly, making a bootc update or bootc rollback do not affect this application data.

Having /var separate also makes it work cleanly to stage new operating system updates before applying them, that is, updates are downloaded and ready, but only take effect on reboot. The same applies for Docker volume, as it decouples the application code from its data.

You can use this case if you want applications to have a pre-created directory structure, for example, /var/lib/postgresql. Use systemd tmpfiles.d for this. You can also use StateDirectory=<directory> in units.

When a software needs to write to its own directory in /opt/exampleapp, a common pattern is to use a symbolic link to redirect to, for example, /var for operations such as log files:

RUN rmdir /opt/exampleapp/logs && ln -sr /var/log/exampleapp /opt/exampleapp/logs

Optionally, you can configure the systemd unit to launch the service to do these mounts dynamically. For example:

BindPaths=/var/log/exampleapp:/opt/exampleapp/logs

[root]
transient = true

This enables a software to transiently write to /opt, with symlinks to /var for content that must persist.

Image mode for RHEL uses GRUB by default, with exception to s390x architectures. Each version of image mode for RHEL currently available on a system has a menu entry.

The menu entry references an OSTree deployment which consists of a Linux kernel, an initramfs and a hash linking to an OSTree commit, that you can pass by using the ostree=kernel argument.

During bootup, OSTree reads the kernel argument to determine which deployment to use as the root filesystem. Each update or change to the system, such as package installation, addition of kernel arguments, creates a new deployment.

This enables rolling back to a previous deployment if the update causes problems.

Image mode for RHEL is a generic operating system update and configuration mechanism. You cannot use it to configure users or groups. The only exception is the bootc install command that has the --root-ssh-authorized-keys option.

Image mode for RHEL does not have an opinionated mechanism for secrets. You can inject container pull secrets in your system for some cases, for example:

To be able to fetch container images, you must configure a host system with a "pull secret", which includes the host updates itself. See the appendix about Injecting secrets in image mode for RHEL documentation for more details.

You can configure the container pull secrets to an image already built. If you use an external installer such as Anaconda for bare metal, or bootc-image-builder, you must configure the systems with any applicable pull secrets.

The host bootc updates write the configuration to the /etc/ostree/auth.json file, which is shared with rpm-ostree.

Podman does not have system wide credentials. Podman accepts the containers-auth locations that are underneath the following directories:

To unify bootc and Podman credentials, use a single default global pull secret for both bootc and Podman. The following container build is an example to unify the bootc and the Podman credentials. The example expects a secret named creds to contain the registry pull secret to build.

You can configure container images, pull secrets, and disable TLS for a registry within a system. These actions enable containerized environments to pull images from private or insecure registries.

You can include container pull secrets and other configuration to access a registry inside the base image. However, when installing by using Anaconda, the installation environment might need a duplicate copy of "bootstrap" configuration to access the targeted registry when fetching over the network.

To perform arbitrary changes to the installation environment before the target bootc container image is fetched, you can use the Anaconda %pre command.

See the containers-auth.json(5) for more detailed information about format and configurations of the auth.json file.

You can also use %pre to:

You can configure insecure registries similarly by modifying the /etc/containers directory.

You can perform a dynamic reconfiguration in the base image configuration. For example, you can run the firewall-cmd --permanent command to achieve persistent changes across a reboot.

In the default configuration, first make the changes in the base image, then queue the changes without restarting running systems, and then simultaneously write to apply the changes to existing systems only in memory.

You can configure the /etc directory to be transient by using bind mounts. In this case, the etc directory is a part of the machine’s local root filesystem. For example, if you inject static IP addresses by using Anaconda kickstarts, they persist across upgrades.

A 3-way merge is applied across upgrades and each "deployment" has its own copy of /etc.

In the Push model, some workloads are implemented by tooling such as Ansible.

The rhel9/rhel-bootc container image includes dnf. There are several use cases:

You can use the bootc-usr-overlay command to create a writable overlay filesystem for /usr directory. The dnf install writes to this overlay. You can use this feature for installing debugging tools. Note that changes will be lost on reboot.

You can add other storage packages to the host system.

Storage with bootc install You can use the bootc install to-disk command for flat storage configurations and bootc install to-filesytem command for more advanced installations. For more information see Advanced installation with to-filesystem.

To set a custom hostname for your system, modify the /etc/hostname file. You can set the hostname by using Anaconda, or with a privileged container.

Once you boot a system, you can verify the hostname by using the hostnamectl command.

If you are deploying to an environment requiring internet access by using a proxy, you need to configure services so that they can access resources as intended.

This is done by defining a single file with required environment variables in your configuration, and to reference this by using systemd drop-in unit files for all such services.

# /etc/example-proxy.env
https_proxy="http://example.com:8080"
all_proxy="http://example.com:8080"
http_proxy="http://example.com:8080"
HTTP_PROXY="http://example.com:8080"
HTTPS_PROXY="http://example.com:8080"
no_proxy="*.example.com,127.0.0.1,0.0.0.0,localhost"

# /usr/lib/systemd/system/bootc-fetch-apply-updates.service.d/99-proxy.conf
[Service]
EnvironmentFile=/etc/example-proxy.env