Embedding in a RHEL for Edge image
Embedding in a RHEL for Edge image
Abstract
Chapter 1. Embedding in a RHEL for Edge image using image builder
You can embed MicroShift into a Red Hat Enterprise Linux for Edge (RHEL for Edge) image. Use this guide to build a RHEL image containing MicroShift.
1.1. Preparing for image building
Read Composing, installing, and managing RHEL for Edge images.
To build an Red Hat Enterprise Linux for Edge (RHEL for Edge) 9.4 image for a given CPU architecture, you need a RHEL 9.4 build host of the same CPU architecture that meets the Image builder system requirements.
Follow the instructions in Installing image builder to install image builder and the composer-cli
tool.
1.2. Adding MicroShift repositories to image builder
Use the following procedure to add the MicroShift repositories to image builder on your build host.
Prerequisites
- Your build host meets the image builder system requirements.
-
You have installed and set up image builder and the
composer-cli
tool. - You have root-user access to your build host.
Procedure
Create an image builder configuration file for adding the
rhocp-4.17
RPM repository source required to pull MicroShift RPMs by running the following command:cat > rhocp-4.17.toml <<EOF id = "rhocp-4.17" name = "Red Hat OpenShift Container Platform 4.17 for RHEL 9" type = "yum-baseurl" url = "https://cdn.redhat.com/content/dist/layered/rhel9/$(uname -m)/rhocp/4.17/os" check_gpg = true check_ssl = true system = false rhsm = true EOF
Create an image builder configuration file for adding the
fast-datapath
RPM repository by running the following command:cat > fast-datapath.toml <<EOF id = "fast-datapath" name = "Fast Datapath for RHEL 9" type = "yum-baseurl" url = "https://cdn.redhat.com/content/dist/layered/rhel9/$(uname -m)/fast-datapath/os" check_gpg = true check_ssl = true system = false rhsm = true EOF
Add the sources to the image builder by running the following commands:
$ sudo composer-cli sources add rhocp-4.17.toml
$ sudo composer-cli sources add fast-datapath.toml
Verification
Confirm that the sources were added properly by running the following command:
$ sudo composer-cli sources list
Example output
appstream baseos fast-datapath rhocp-4.17
Additional resources
1.3. Adding the MicroShift service to a blueprint
Adding the MicroShift RPM package to an image builder blueprint enables the build of a RHEL for Edge image with MicroShift embedded.
- Start with step 1 to create your own minimal blueprint file which results in a faster MicroShift installation.
Start with step 2 to use the generated blueprint for installation which includes all the RPM packages and container images. This is a longer installation process, but a faster start up because container references are accessed locally.
Important-
Replace
<microshift_blueprint.toml>
in the following procedures with the name of the TOML file you are using. -
Replace
<microshift_blueprint>
in the following procedures with the name you want to use for your blueprint.
-
Replace
Procedure
Use the following example to create your own blueprint file:
Custom image builder blueprint example
cat > <microshift_blueprint.toml> <<EOF 1 name = "<microshift_blueprint>" 2 description = "" version = "0.0.1" modules = [] groups = [] [[packages]] name = "microshift" version = "*" [customizations.services] enabled = ["microshift"] EOF
NoteThe wildcard
*
in the commands uses the latest MicroShift RPMs. If you need a specific version, substitute the wildcard for the version you want. For example, insert4.17.1
to download the MicroShift 4.17.1 RPMs.Optional. Use the blueprint installed in the
/usr/share/microshift/blueprint
directory that is specific to your platform architecture. See the following example snippet for an explanation of the blueprint sections:Generated image builder blueprint example snippet
name = "microshift_blueprint" description = "MicroShift 4.17.1 on x86_64 platform" version = "0.0.1" modules = [] groups = [] [[packages]] 1 name = "microshift" version = "4.17.1" ... ... [customizations.services] 2 enabled = ["microshift"] [customizations.firewall] ports = ["22:tcp", "80:tcp", "443:tcp", "5353:udp", "6443:tcp", "30000-32767:tcp", "30000-32767:udp"] ... ... [[containers]] 3 source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41e79c17e8b41f1b0a5a32c3e2dd7cd15b8274554d3f1ba12b2598a347475f4" [[containers]] source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbc65f1fba7d92b36cf7514cd130fe83a9bd211005ddb23a8dc479e0eea645fd" ... … EOF
- 1
- References for all non-optional MicroShift RPM packages using the same version compatible with the
microshift-release-info
RPM. - 2
- References for automatically enabling MicroShift on system startup and applying default networking settings.
- 3
- References for all non-optional MicroShift container images necessary for an offline deployment.
Add the blueprint to the image builder by running the following command:
$ sudo composer-cli blueprints push <microshift_blueprint.toml> 1
- 1
- Replace
<microshift_blueprint.toml>
with the name of your TOML file.
Verification
Verify the image builder configuration listing only MicroShift packages by running the following command:
$ sudo composer-cli blueprints depsolve <microshift_blueprint> | grep microshift 1
- 1
- Replace
<microshift_blueprint>
with the name of your blueprint.
Example output
blueprint: microshift_blueprint v0.0.1 microshift-greenboot-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch microshift-networking-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.x86_64 microshift-release-info-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch microshift-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.x86_64 microshift-selinux-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch
Optional: Verify the image builder configuration listing all components to be installed by running the following command:
$ sudo composer-cli blueprints depsolve <microshift_blueprint> 1
- 1
- Replace
<microshift_blueprint>
with the name of your blueprint.
1.4. Adding other packages to a blueprint
Add the references for optional RPM packages to your ostree
blueprint to enable them.
Prerequisites
- You created an image builder blueprint file.
Procedure
Edit your
ostree
blueprint by running the following command:$ vi <microshift_blueprint.toml> 1
- 1
- Replace
<microshift_blueprint.toml>
with the name of the blueprint file used for the MicroShift service.
Add the following example text to your blueprint:
[[packages]] 1 name = "<microshift-additional-package-name>" 2 version = "*"
Next steps
- Add custom certificate authorities to the blueprint as needed.
After you are done adding to your blueprint, you can apply the manifests to an active cluster by building a new
ostree
system and deploying it on the client:- Create the ISO.
- Add the blueprint and build the ISO.
- Download the ISO and prepare it for use.
- Do any provisioning that is needed.
1.5. Adding a certificate authority bundle
MicroShift uses the host trust bundle when clients evaluate server certificates. You can also use a customized security certificate chain to improve the compatibility of your endpoint certificates with clients specific to your deployments. To do this, you can add a certificate authority (CA) bundle with root and intermediate certificates to the Red Hat Enterprise Linux for Edge (RHEL for Edge) system-wide trust store.
1.5.1. Adding a certificate authority bundle to an rpm-ostree image
You can include additional trusted certificate authorities (CAs) to the Red Hat Enterprise Linux for Edge (RHEL for Edge) rpm-ostree
image by adding them to the blueprint that you use to create the image. Using the following procedure sets up additional CAs to be trusted by the operating system when pulling images from an image registry.
This procedure requires you to configure the CA bundle customizations in the blueprint, and then add steps to your kickstart file to enable the bundle. In the following steps, data
is the key, and <value>
represents the PEM-encoded certificate.
Prerequisites
- You have root user access to your build host.
- Your build host meets the image builder system requirements.
-
You have installed and set up image builder and the
composer-cli
tool.
Procedure
Add the following custom values to your blueprint to add a directory.
Add instructions to your blueprint on the host where the image is built to create the directory, for example,
/etc/pki/ca-trust/source/anchors/
for your certificate bundles.[[customizations.directories]] path = "/etc/pki/ca-trust/source/anchors"
After the image has booted, create the certificate bundles, for example,
/etc/pki/ca-trust/source/anchors/cert1.pem
:[[customizations.files]] path = "/etc/pki/ca-trust/source/anchors/cert1.pem" data = "<value>"
To enable the certificate bundle in the system-wide trust store configuration, use the
update-ca-trust
command on the host where the image you are using has booted, for example:$ sudo update-ca-trust
The update-ca-trust
command might be included in the %post
section of a kickstart file used for MicroShift host installation so that all the necessary certificate trust is enabled on the first boot. You must configure the CA bundle customizations in the blueprint before adding steps to your kickstart file to enable the bundle.
%post # Update certificate trust storage in case new certificates were # installed at /etc/pki/ca-trust/source/anchors directory update-ca-trust %end
1.6. Creating the RHEL for Edge image
Use the following procedure to create the ISO. The RHEL for Edge Installer image pulls the commit from the running container and creates an installable boot ISO with a Kickstart file configured to use the embedded rpm-ostree
commit.
Prerequisites
- Your build host meets the Image Builder system requirements.
-
You have installed and set up Image Builder and the
composer-cli
tool. - You have root-user access to your build host.
-
You have installed the
podman
tool.
Procedure
Start an
ostree
container image build by running the following command:$ BUILDID=$(sudo composer-cli compose start-ostree --ref "rhel/{op-system-version-major}/$(uname -m)/edge" <microshift_blueprint> edge-container | awk '/^Compose/ {print $2}') 1
- 1
- Replace
<microshift_blueprint>
with the name of your blueprint.
This command also returns the identification (ID) of the build for monitoring.
You can check the status of the build periodically by running the following command:
$ sudo composer-cli compose status
Example output of a running build
ID Status Time Blueprint Version Type Size cc3377ec-4643-4483-b0e7-6b0ad0ae6332 RUNNING Wed Jun 7 12:26:23 2023 microshift_blueprint 0.0.1 edge-container
Example output of a completed build
ID Status Time Blueprint Version Type Size cc3377ec-4643-4483-b0e7-6b0ad0ae6332 FINISHED Wed Jun 7 12:32:37 2023 microshift_blueprint 0.0.1 edge-container
NoteYou can use the
watch
command to monitor your build if you are familiar with how to start and stop it.Download the container image using the ID and get the image ready for use by running the following command:
$ sudo composer-cli compose image ${BUILDID}
Change the ownership of the downloaded container image to the current user by running the following command:
$ sudo chown $(whoami). ${BUILDID}-container.tar
Add read permissions for the current user to the image by running the following command:
$ sudo chmod a+r ${BUILDID}-container.tar
Bootstrap a server on port 8085 for the
ostree
container image to be consumed by the ISO build by completing the following steps:Get the
IMAGEID
variable result by running the following command:$ IMAGEID=$(cat < "./${BUILDID}-container.tar" | sudo podman load | grep -o -P '(?<=sha256[@:])[a-z0-9]*')
Use the
IMAGEID
variable result to execute the podman command step by running the following command:$ sudo podman run -d --name=minimal-microshift-server -p 8085:8080 ${IMAGEID}
This command also returns the ID of the container saved in the
IMAGEID
variable for monitoring.
Generate the installer blueprint file by running the following command:
cat > microshift-installer.toml <<EOF name = "microshift-installer" description = "" version = "0.0.0" modules = [] groups = [] packages = [] EOF
1.7. Add the blueprint to image builder and build the ISO
Add the blueprint to the image builder by running the following command:
$ sudo composer-cli blueprints push microshift-installer.toml
Start the
ostree
ISO build by running the following command:$ BUILDID=$(sudo composer-cli compose start-ostree --url http://localhost:8085/repo/ --ref "rhel/9/$(uname -m)/edge" microshift-installer edge-installer | awk '{print $2}')
This command also returns the identification (ID) of the build for monitoring.
You can check the status of the build periodically by running the following command:
$ sudo composer-cli compose status
Example output for a running build
ID Status Time Blueprint Version Type Size c793c24f-ca2c-4c79-b5b7-ba36f5078e8d RUNNING Wed Jun 7 13:22:20 2023 microshift-installer 0.0.0 edge-installer
Example output for a completed build
ID Status Time Blueprint Version Type Size c793c24f-ca2c-4c79-b5b7-ba36f5078e8d FINISHED Wed Jun 7 13:34:49 2023 microshift-installer 0.0.0 edge-installer
1.8. Download the ISO and prepare it for use
Download the ISO using the ID by running the following command:
$ sudo composer-cli compose image ${BUILDID}
Change the ownership of the downloaded container image to the current user by running the following command:
$ sudo chown $(whoami). ${BUILDID}-installer.iso
Add read permissions for the current user to the image by running the following command:
$ sudo chmod a+r ${BUILDID}-installer.iso
1.9. Provisioning a machine for MicroShift
Provision a machine with your RHEL for Edge image by using the procedures from the RHEL for Edge documentation.
To use MicroShift, you must provision the system so that it meets the following requirements:
- The machine you are provisioning must meet the system requirements for installing MicroShift.
- The file system must have a logical volume manager (LVM) volume group (VG) with sufficient capacity for the persistent volumes (PVs) of your workload.
-
A pull secret from the Red Hat Hybrid Cloud Console must be present as
/etc/crio/openshift-pull-secret
and have root user-only read/write permissions. - The firewall must be configured with the required settings.
If you are using a Kickstart such as the RHEL for Edge Installer (ISO) image, you can update your Kickstart file to meet the provisioning requirements.
Prerequisites
You have created a RHEL for Edge Installer (ISO) image containing your RHEL for Edge commit with MicroShift.
- This requirement includes the steps of composing an RFE Container image, creating the RFE Installer blueprint, starting the RFE container, and composing the RFE Installer image.
Create a Kickstart file or use an existing one. In the Kickstart file, you must include:
- Detailed instructions about how to create a user.
- How to fetch and deploy the RHEL for Edge image.
For more information, read "Additional resources."
Procedure
In the main section of the Kickstart file, update the setup of the filesystem such that it contains an LVM volume group called
rhel
with at least 10GB system root. Leave free space for the LVMS CSI driver to use for storing the data for your workloads.Example kickstart snippet for configuring the filesystem
# Partition disk such that it contains an LVM volume group called `rhel` with a # 10GB+ system root but leaving free space for the LVMS CSI driver for storing data. # # For example, a 20GB disk would be partitioned in the following way: # # NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT # sda 8:0 0 20G 0 disk # ├─sda1 8:1 0 200M 0 part /boot/efi # ├─sda1 8:1 0 800M 0 part /boot # └─sda2 8:2 0 19G 0 part # └─rhel-root 253:0 0 10G 0 lvm /sysroot # ostreesetup --nogpg --osname=rhel --remote=edge \ --url=file:///run/install/repo/ostree/repo --ref=rhel/<RHEL VERSION NUMBER>/x86_64/edge zerombr clearpart --all --initlabel part /boot/efi --fstype=efi --size=200 part /boot --fstype=xfs --asprimary --size=800 # Uncomment this line to add a SWAP partition of the recommended size #part swap --fstype=swap --recommended part pv.01 --grow volgroup rhel pv.01 logvol / --vgname=rhel --fstype=xfs --size=10000 --name=root # To add users, use a line such as the following user --name=<YOUR_USER_NAME> \ --password=<YOUR_HASHED_PASSWORD> \ --iscrypted --groups=<YOUR_USER_GROUPS>
In the
%post
section of the Kickstart file, add your pull secret and the mandatory firewall rules.Example Kickstart snippet for adding the pull secret and firewall rules
%post --log=/var/log/anaconda/post-install.log --erroronfail # Add the pull secret to CRI-O and set root user-only read/write permissions cat > /etc/crio/openshift-pull-secret << EOF YOUR_OPENSHIFT_PULL_SECRET_HERE EOF chmod 600 /etc/crio/openshift-pull-secret # Configure the firewall with the mandatory rules for MicroShift firewall-offline-cmd --zone=trusted --add-source=10.42.0.0/16 firewall-offline-cmd --zone=trusted --add-source=169.254.169.1 %end
Install the
mkksiso
tool by running the following command:$ sudo yum install -y lorax
Update the Kickstart file in the ISO with your new Kickstart file by running the following command:
$ sudo mkksiso <your_kickstart>.ks <your_installer>.iso <updated_installer>.iso
1.10. How to access the MicroShift cluster
Use the procedures in this section to access the MicroShift cluster by using the OpenShift CLI (oc
).
- You can access the cluster from either the same machine running the MicroShift service or from a remote location.
- You can use this access to observe and administrate workloads.
-
When using the following steps, choose the
kubeconfig
file that contains the host name or IP address you want to connect to and place it in the relevant directory.
1.10.1. Accessing the MicroShift cluster locally
Use the following procedure to access the MicroShift cluster locally by using a kubeconfig
file.
Prerequisites
-
You have installed the
oc
binary.
Procedure
Optional: to create a
~/.kube/
folder if your Red Hat Enterprise Linux (RHEL) machine does not have one, run the following command:$ mkdir -p ~/.kube/
Copy the generated local access
kubeconfig
file to the~/.kube/
directory by running the following command:$ sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config
Update the permissions on your
~/.kube/config
file by running the following command:$ chmod go-r ~/.kube/config
Verification
Verify that MicroShift is running by entering the following command:
$ oc get all -A
1.10.2. Opening the firewall for remote access to the MicroShift cluster
Use the following procedure to open the firewall so that a remote user can access the MicroShift cluster. This procedure must be completed before a workstation user can access the cluster remotely.
For this procedure, user@microshift
is the user on the MicroShift host machine and is responsible for setting up that machine so that it can be accessed by a remote user on a separate workstation.
Prerequisites
-
You have installed the
oc
binary. - Your account has cluster administration privileges.
Procedure
As
user@microshift
on the MicroShift host, open the firewall port for the Kubernetes API server (6443/tcp
) by running the following command:[user@microshift]$ sudo firewall-cmd --permanent --zone=public --add-port=6443/tcp && sudo firewall-cmd --reload
Verification
As
user@microshift
, verify that MicroShift is running by entering the following command:[user@microshift]$ oc get all -A
1.10.3. Accessing the MicroShift cluster remotely
Use the following procedure to access the MicroShift cluster from a remote location by using a kubeconfig
file.
The user@workstation
login is used to access the host machine remotely. The <user>
value in the procedure is the name of the user that user@workstation
logs in with to the MicroShift host.
Prerequisites
-
You have installed the
oc
binary. -
The
user@microshift
has opened the firewall from the local host.
Procedure
As
user@workstation
, create a~/.kube/
folder if your Red Hat Enterprise Linux (RHEL) machine does not have one by running the following command:[user@workstation]$ mkdir -p ~/.kube/
As
user@workstation
, set a variable for the hostname of your MicroShift host by running the following command:[user@workstation]$ MICROSHIFT_MACHINE=<name or IP address of MicroShift machine>
As
user@workstation
, copy the generatedkubeconfig
file that contains the host name or IP address you want to connect with from the RHEL machine running MicroShift to your local machine by running the following command:[user@workstation]$ ssh <user>@$MICROSHIFT_MACHINE "sudo cat /var/lib/microshift/resources/kubeadmin/$MICROSHIFT_MACHINE/kubeconfig" > ~/.kube/config
NoteTo generate the
kubeconfig
files for this step, see Generating additional kubeconfig files for remote access.As
user@workstation
, update the permissions on your~/.kube/config
file by running the following command:$ chmod go-r ~/.kube/config
Verification
As
user@workstation
, verify that MicroShift is running by entering the following command:[user@workstation]$ oc get all -A
Additional resources
Chapter 2. Mirroring container images for disconnected installations
You can use a custom container registry when you deploy MicroShift in a disconnected network. Running your cluster in a restricted network without direct internet connectivity is possible by installing the cluster from a mirrored set of container images in a private registry.
2.1. Mirror container images into an existing registry
Using a custom air-gapped container registry, or mirror, is necessary with certain user environments and workload requirements. Mirroring allows for the transfer of container images and updates to air-gapped environments where they can be installed on a MicroShift instance.
To create an air-gapped mirror registry for MicroShift containers, you must complete the following steps:
- Get the container image list to be mirrored.
- Configure the mirroring prerequisites.
- Download images on a host with internet access.
- Copy the downloaded image directory to an air-gapped site.
- Upload images to a mirror registry in an air-gapped site.
- Configure your MicroShift hosts to use the mirror registry.
Additional resources
2.2. Getting the mirror registry container image list
To use a mirror registry, you must know which container image references are used by a specific version of MicroShift. These references are provided in the release-<arch>.json
files that are part of the microshift-release-info
RPM package.
To mirror the Operator Lifecycle Manager (OLM) in disconnected environments, add the references provided in the release-olm-$ARCH.json
that is included in the microshift-olm
RPM and follow the same procedure. Use oc-mirror
for mirroring Operator catalogs and Operators.
Prerequisites
- You have installed jq.
Procedure
Access the list of container image references by using one of the following methods:
If the package is installed on the MicroShift host, get the location of the files by running the following command:
$ rpm -ql microshift-release-info
Example output
/usr/share/microshift/release/release-x86_64.json
If the package is not installed on a MicroShift host, download and unpack the RPM package without installing it by running the following command:
$ rpm2cpio microshift-release-info*.noarch.rpm | cpio -idmv
Example output
/usr/share/microshift/release/release-x86_64.json
Extract the list of container images into the
microshift-container-refs.txt
file by running the following commands:$ RELEASE_FILE=/usr/share/microshift/release/release-$(uname -m).json
$ jq -r '.images | .[]' ${RELEASE_FILE} > microshift-container-refs.txt
After the microshift-container-refs.txt
file is created with the MicroShift container image list, you can append the file with other user-specific image references before running the mirroring procedure.
2.3. Configuring mirroring prerequisites
You must create a container image registry credentials file that allows the mirroring of images from your internet-connected mirror host to your air-gapped mirror. Follow the instructions in the "Configuring credentials that allow images to be mirrored" link provided in the "Additional resources" section. These instructions guide you to create a ~/.pull-secret-mirror.json
file on the mirror registry host that includes the user credentials for accessing the mirror.
2.3.1. Example mirror registry pull secret entry
For example, the following section is added to the pull secret file for the microshift_quay:8443
mirror registry using microshift:microshift
as username and password.
Example mirror registry section for pull secret file
"<microshift_quay:8443>": { 1 "auth": "<microshift_auth>", 2 "email": "<microshift_quay@example.com>" 3 },
Additional resources
2.4. Downloading container images
After you have located the container list and completed the mirroring prerequisites, download the container images to a host with internet access.
Prerequisites
- You are logged into a host with access to the internet.
-
You have ensured that the
.pull-secret-mirror.json
file andmicroshift-containers
directory contents are available locally.
Procedure
Install the
skopeo
tool used for copying the container images by running the following command:$ sudo dnf install -y skopeo
Set the environment variable that points to the pull secret file:
$ PULL_SECRET_FILE=~/.pull-secret-mirror.json
Set the environment variable that points to the list of container images:
$ IMAGE_LIST_FILE=~/microshift-container-refs.txt
Set the environment variable that points to the destination directory for storing the downloaded data:
$ IMAGE_LOCAL_DIR=~/microshift-containers
Run the following script to download the container images to the
${IMAGE_LOCAL_DIR}
directory:while read -r src_img ; do # Remove the source registry prefix dst_img=$(echo "${src_img}" | cut -d '/' -f 2-) # Run the image download command echo "Downloading '${src_img}' to '${IMAGE_LOCAL_DIR}'" mkdir -p "${IMAGE_LOCAL_DIR}/${dst_img}" skopeo copy --all --quiet \ --preserve-digests \ --authfile "${PULL_SECRET_FILE}" \ docker://"${src_img}" dir://"${IMAGE_LOCAL_DIR}/${dst_img}" done < "${IMAGE_LIST_FILE}"
- Transfer the image set to the target environment, such as air-gapped site. Then you can upload the image set into the mirror registry.
2.5. Uploading container images to a mirror registry
To use your container images at an air-gapped site, upload them to the mirror registry using the following procedure.
Prerequisites
-
You are logged into a host with access to
microshift-quay
. -
The
.pull-secret-mirror.json
file is available locally. -
The
microshift-containers
directory contents are available locally.
Procedure
Install the
skopeo
tool used for copying the container images by running the following command:$ sudo dnf install -y skopeo
Set the environment variables pointing to the pull secret file:
$ IMAGE_PULL_FILE=~/.pull-secret-mirror.json
Set the environment variables pointing to the local container image directory:
$ IMAGE_LOCAL_DIR=~/microshift-containers
Set the environment variables pointing to the mirror registry URL for uploading the container images:
$ TARGET_REGISTRY=<registry_host>:<port> 1
- 1
- Replace
<registry_host>:<port>
with the host name and port of your mirror registry server.
Run the following script to upload the container images to the
${TARGET_REGISTRY}
mirror registry:image_tag=mirror-$(date +%y%m%d%H%M%S) image_cnt=1 # Uses timestamp and counter as a tag on the target images to avoid # their overwrite by the 'latest' automatic tagging pushd "${IMAGE_LOCAL_DIR}" >/dev/null while read -r src_manifest ; do # Remove the manifest.json file name src_img=$(dirname "${src_manifest}") # Add the target registry prefix and remove SHA dst_img="${TARGET_REGISTRY}/${src_img}" dst_img=$(echo "${dst_img}" | awk -F'@' '{print $1}') # Run the image upload command echo "Uploading '${src_img}' to '${dst_img}'" skopeo copy --all --quiet \ --preserve-digests \ --authfile "${IMAGE_PULL_FILE}" \ dir://"${IMAGE_LOCAL_DIR}/${src_img}" docker://"${dst_img}:${image_tag}-${image_cnt}" # Increment the counter (( image_cnt += 1 )) done < <(find . -type f -name manifest.json -printf '%P\n') popd >/dev/null
2.6. Configuring hosts for mirror registry access
To configure a MicroShift host to use a mirror registry, you must give the MicroShift host access to the registry by creating a configuration file that maps the Red Hat registry host names to the mirror.
Prerequisites
- Your mirror host has access to the internet.
- The mirror host can access the mirror registry.
- You configured the mirror registry for use in your restricted network.
- You downloaded the pull secret and modified it to include authentication to your mirror repository.
Procedure
- Log into your MicroShift host.
Enable the SSL certificate trust on any host accessing the mirror registry by completing the following steps:
-
Copy the
rootCA.pem
file from the mirror registry, for example,<registry_path>/quay-rootCA
, to the MicroShift host at the/etc/pki/ca-trust/source/anchors
directory. Enable the certificate in the system-wide trust store configuration by running the following command:
$ sudo update-ca-trust
-
Copy the
Create the
/etc/containers/registries.conf.d/999-microshift-mirror.conf
configuration file that maps the Red Hat registry host names to the mirror registry:Example mirror configuration file
[[registry]] prefix = "" location = "<registry_host>:<port>" 1 mirror-by-digest-only = true insecure = false [[registry]] prefix = "" location = "quay.io" mirror-by-digest-only = true [[registry.mirror]] location = "<registry_host>:<port>" insecure = false [[registry]] prefix = "" location = "registry.redhat.io" mirror-by-digest-only = true [[registry.mirror]] location = "<registry_host>:<port>" insecure = false [[registry]] prefix = "" location = "registry.access.redhat.com" mirror-by-digest-only = true [[registry.mirror]] location = "<registry_host>:<port>" insecure = false
- 1
- Replace
<registry_host>:<port>
with the host name and port of your mirror registry server, for example,<microshift-quay:8443>
.
Enable the MicroShift service by running the following command:
$ sudo systemctl enable microshift
Reboot the host by running the following command:
$ sudo reboot
Chapter 3. Embedding in a RHEL for Edge image for offline use
Embedding MicroShift containers in an rpm-ostree
commit means that you can run a cluster in air-gapped, disconnected, or offline environments. You can embed MicroShift containers in a Red Hat Enterprise Linux for Edge (RHEL for Edge) image so that container engines do not need to pull images over a network from a container registry. Workloads can start up immediately without network connectivity.
3.1. Embedding MicroShift containers for offline deployments
You can use image builder to create rpm-ostree
system images with embedded MicroShift container images. To embed container images, you must add the image references to your image builder blueprint.
Prerequisites
- You have root-user access to your build host.
- Your build host meets the image builder system requirements.
-
You have installed and set up image builder and the
composer-cli
tool. - You have created a RHEL for Edge image blueprint.
- You have installed jq.
Procedure
-
Get the exact list of container image references used by the MicroShift version you are deploying. You can either install the
microshift-release-info
RPM package by following step 2 or download and unpack the RPM by following step 3. To install the
microshift-release-info
RPM package:Install the
microshift-release-info
RPM package by running the following command:$ sudo dnf install -y microshift-release-info-<release_version>
Replace
<release_version>
with the numerical value of the release you are deploying, using the entire version number, such as4.17.1
.List the contents of the
/usr/share/microshift/release
directory to verify the presence of the release information files by running the following command:$ ls /usr/share/microshift/release
Example output
release-x86_64.json release-aarch64.json
If you installed the
microshift-release-info
RPM, you can proceed to step 4.
If you did not complete step 2, download and unpack the
microshift-release-info
RPM without installing it:Download the RPM package by running the following command:
$ sudo dnf download microshift-release-info-<release_version>
Replace
<release_version>
with the numerical value of the release you are deploying, using the entire version number, such as4.17.1
.Example rpm
microshift-release-info-4.17.1.*.el9.noarch.rpm 1
- 1
- The
*
represents the date and commit ID. Your output should contain both, for example-202311101230.p0.g7dc6a00.assembly.4.17.1
.
Unpack the RPM package without installing it by running the following command:
$ rpm2cpio <my_microshift_release_info> | cpio -idmv 1 ./usr/share/microshift/release/release-aarch64.json ./usr/share/microshift/release/release-x86_64.json
- 1
- Replace
<my_microshift_release_info>
with the name of the RPM package from the previous step.
Define the location of your JSON file, which contains the container reference information, by running the following command:
$ RELEASE_FILE=</path/to/your/release-$(uname -m).json>
Replace
</path/to/your/release-$(uname -m).json>
with the full path to your JSON file. Be sure to use the file needed for your architecture.Define the location of your TOML file, which contains instructions for building the image, by running the following command:
$ BLUEPRINT_FILE=</path/to/your/blueprint.toml>
Replace
</path/to/your/blueprint.toml>
with the full path to your JSON file.Generate and then embed the container image references in your blueprint TOML file by running the following command:
$ jq -r '.images | .[] | ("[[containers]]\nsource = \"" + . + "\"\n")' "${RELEASE_FILE}" >> "${BLUEPRINT_FILE}"
Example resulting
<my_blueprint.toml>
fragment showing container references[[containers]] source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82cfef91557f9a70cff5a90accba45841a37524e9b93f98a97b20f6b2b69e5db" [[containers]] source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82cfef91557f9a70cff5a90accba45841a37524e9b93f98a97b20f6b2b69e5db"
You can manually embed any container image by adding it to the image builder blueprint using the following example:
Example section for manually embedding container image to image builder
[[containers]] source = "<my_image_pullspec_with_tag_or_digest>"
Replace
<my_image_pullspec_with_tag_or_digest>
with the exact reference to a container image used by the MicroShift version you are deploying.
3.2. Updating osbuilder worker configuration to prepare for image building
After you have updated the blueprint, you must update the osbuilder worker configuration to prepare for building the image with embedded MicroShift containers.
Prerequisites
- You have root-user access to your build host.
- Your build host meets the Image Builder system requirements.
-
You have installed and set up Image Builder and the
composer-cli
tool.
You can create an /etc/osbuild-worker/osbuild-worker.toml
directory and configuration file if they do not exist.
Procedure
Add a pull secret for authenticating to the registry by setting the
auth_file_path
in the[containers]
section of the/etc/osbuild-worker/osbuild-worker.toml
osbuilder worker configuration file:[containers] auth_file_path = "/etc/osbuild-worker/pull-secret.json"
-
Restart the
osbuild-worker
to apply configuration changes by restarting the host. Restarting the host ensures that allosbuild-worker
services currently running are restarted.
3.3. Build and use the rpm-ostree image for offline deployments
You can use image builder to create rpm-ostree
system images with embedded MicroShift container images. To embed container images, you must add the image references to your image builder blueprint. You can create the commit and ISO as needed for your use case.
Add the prerequisites listed here to the ones that are included in the procedures that follow.
3.3.1. Additional prerequisites for offline deployments
- You have created and updated a RHEL for Edge image blueprint for offline use. The following procedures use the example of a blueprint created with container images. You must use the updated blueprint you created in the "Embedding MicroShift containers for offline deployments" procedure.
-
You have updated the
/etc/osbuild-worker/osbuild-worker.toml
configuration file for offline use.
Replace minimal-microshift.toml
in the following procedures with the name of the TOML you updated for offline use, <my_blueprint_name>.
3.3.2. Adding the MicroShift service to a blueprint
Adding the MicroShift RPM package to an image builder blueprint enables the build of a RHEL for Edge image with MicroShift embedded.
- Start with step 1 to create your own minimal blueprint file which results in a faster MicroShift installation.
Start with step 2 to use the generated blueprint for installation which includes all the RPM packages and container images. This is a longer installation process, but a faster start up because container references are accessed locally.
Important-
Replace
<microshift_blueprint.toml>
in the following procedures with the name of the TOML file you are using. -
Replace
<microshift_blueprint>
in the following procedures with the name you want to use for your blueprint.
-
Replace
Procedure
Use the following example to create your own blueprint file:
Custom image builder blueprint example
cat > <microshift_blueprint.toml> <<EOF 1 name = "<microshift_blueprint>" 2 description = "" version = "0.0.1" modules = [] groups = [] [[packages]] name = "microshift" version = "*" [customizations.services] enabled = ["microshift"] EOF
NoteThe wildcard
*
in the commands uses the latest MicroShift RPMs. If you need a specific version, substitute the wildcard for the version you want. For example, insert4.17.1
to download the MicroShift 4.17.1 RPMs.Optional. Use the blueprint installed in the
/usr/share/microshift/blueprint
directory that is specific to your platform architecture. See the following example snippet for an explanation of the blueprint sections:Generated image builder blueprint example snippet
name = "microshift_blueprint" description = "MicroShift 4.17.1 on x86_64 platform" version = "0.0.1" modules = [] groups = [] [[packages]] 1 name = "microshift" version = "4.17.1" ... ... [customizations.services] 2 enabled = ["microshift"] [customizations.firewall] ports = ["22:tcp", "80:tcp", "443:tcp", "5353:udp", "6443:tcp", "30000-32767:tcp", "30000-32767:udp"] ... ... [[containers]] 3 source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41e79c17e8b41f1b0a5a32c3e2dd7cd15b8274554d3f1ba12b2598a347475f4" [[containers]] source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbc65f1fba7d92b36cf7514cd130fe83a9bd211005ddb23a8dc479e0eea645fd" ... … EOF
- 1
- References for all non-optional MicroShift RPM packages using the same version compatible with the
microshift-release-info
RPM. - 2
- References for automatically enabling MicroShift on system startup and applying default networking settings.
- 3
- References for all non-optional MicroShift container images necessary for an offline deployment.
Add the blueprint to the image builder by running the following command:
$ sudo composer-cli blueprints push <microshift_blueprint.toml> 1
- 1
- Replace
<microshift_blueprint.toml>
with the name of your TOML file.
Verification
Verify the image builder configuration listing only MicroShift packages by running the following command:
$ sudo composer-cli blueprints depsolve <microshift_blueprint> | grep microshift 1
- 1
- Replace
<microshift_blueprint>
with the name of your blueprint.
Example output
blueprint: microshift_blueprint v0.0.1 microshift-greenboot-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch microshift-networking-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.x86_64 microshift-release-info-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch microshift-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.x86_64 microshift-selinux-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch
Optional: Verify the image builder configuration listing all components to be installed by running the following command:
$ sudo composer-cli blueprints depsolve <microshift_blueprint> 1
- 1
- Replace
<microshift_blueprint>
with the name of your blueprint.
3.3.3. Creating the RHEL for Edge image
Use the following procedure to create the ISO. The RHEL for Edge Installer image pulls the commit from the running container and creates an installable boot ISO with a Kickstart file configured to use the embedded rpm-ostree
commit.
Prerequisites
- Your build host meets the Image Builder system requirements.
-
You have installed and set up Image Builder and the
composer-cli
tool. - You have root-user access to your build host.
-
You have installed the
podman
tool.
Procedure
Start an
ostree
container image build by running the following command:$ BUILDID=$(sudo composer-cli compose start-ostree --ref "rhel/{op-system-version-major}/$(uname -m)/edge" <microshift_blueprint> edge-container | awk '/^Compose/ {print $2}') 1
- 1
- Replace
<microshift_blueprint>
with the name of your blueprint.
This command also returns the identification (ID) of the build for monitoring.
You can check the status of the build periodically by running the following command:
$ sudo composer-cli compose status
Example output of a running build
ID Status Time Blueprint Version Type Size cc3377ec-4643-4483-b0e7-6b0ad0ae6332 RUNNING Wed Jun 7 12:26:23 2023 microshift_blueprint 0.0.1 edge-container
Example output of a completed build
ID Status Time Blueprint Version Type Size cc3377ec-4643-4483-b0e7-6b0ad0ae6332 FINISHED Wed Jun 7 12:32:37 2023 microshift_blueprint 0.0.1 edge-container
NoteYou can use the
watch
command to monitor your build if you are familiar with how to start and stop it.Download the container image using the ID and get the image ready for use by running the following command:
$ sudo composer-cli compose image ${BUILDID}
Change the ownership of the downloaded container image to the current user by running the following command:
$ sudo chown $(whoami). ${BUILDID}-container.tar
Add read permissions for the current user to the image by running the following command:
$ sudo chmod a+r ${BUILDID}-container.tar
Bootstrap a server on port 8085 for the
ostree
container image to be consumed by the ISO build by completing the following steps:Get the
IMAGEID
variable result by running the following command:$ IMAGEID=$(cat < "./${BUILDID}-container.tar" | sudo podman load | grep -o -P '(?<=sha256[@:])[a-z0-9]*')
Use the
IMAGEID
variable result to execute the podman command step by running the following command:$ sudo podman run -d --name=minimal-microshift-server -p 8085:8080 ${IMAGEID}
This command also returns the ID of the container saved in the
IMAGEID
variable for monitoring.
Generate the installer blueprint file by running the following command:
cat > microshift-installer.toml <<EOF name = "microshift-installer" description = "" version = "0.0.0" modules = [] groups = [] packages = [] EOF