Embedding in a RHEL for Edge image

Procedure

1. Create an image builder configuration file for adding the rhocp-4.17 RPM repository source required to pull MicroShift RPMs by running the following command:

   cat > rhocp-4.17.toml <<EOF
   id = "rhocp-4.17"
   name = "Red Hat OpenShift Container Platform 4.17 for RHEL 9"
   type = "yum-baseurl"
   url = "https://cdn.redhat.com/content/dist/layered/rhel9/$(uname -m)/rhocp/4.17/os"
   check_gpg = true
   check_ssl = true
   system = false
   rhsm = true
   EOF

2. Create an image builder configuration file for adding the fast-datapath RPM repository by running the following command:

   cat > fast-datapath.toml <<EOF
   id = "fast-datapath"
   name = "Fast Datapath for RHEL 9"
   type = "yum-baseurl"
   url = "https://cdn.redhat.com/content/dist/layered/rhel9/$(uname -m)/fast-datapath/os"
   check_gpg = true
   check_ssl = true
   system = false
   rhsm = true
   EOF

3. Add the sources to the image builder by running the following commands:

   $ sudo composer-cli sources add rhocp-4.17.toml

   $ sudo composer-cli sources add fast-datapath.toml

Procedure

1. Use the following example to create your own blueprint file:

   Custom image builder blueprint example

   cat > <microshift_blueprint.toml> <<EOF 1
   name = "<microshift_blueprint>" 2
   
   description = ""
   version = "0.0.1"
   modules = []
   groups = []
   
   [[packages]]
   name = "microshift"
   version = "*"
   
   [customizations.services]
   enabled = ["microshift"]
   EOF

   1

   The name of the TOML file.

   2

   The name of the blueprint.

   Note

   The wildcard * in the commands uses the latest MicroShift RPMs. If you need a specific version, substitute the wildcard for the version you want. For example, insert 4.17.1 to download the MicroShift 4.17.1 RPMs.

2. Optional. Use the blueprint installed in the /usr/share/microshift/blueprint directory that is specific to your platform architecture. See the following example snippet for an explanation of the blueprint sections:

   Generated image builder blueprint example snippet

   name = "microshift_blueprint"
   description = "MicroShift 4.17.1 on x86_64 platform"
   version = "0.0.1"
   modules = []
   groups = []
   
   [[packages]] 1
   name = "microshift"
   version = "4.17.1"
   ...
   ...
   
   [customizations.services] 2
   enabled = ["microshift"]
   
   [customizations.firewall]
   ports = ["22:tcp", "80:tcp", "443:tcp", "5353:udp", "6443:tcp", "30000-32767:tcp", "30000-32767:udp"]
   ...
   ...
   
   [[containers]] 3
   source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41e79c17e8b41f1b0a5a32c3e2dd7cd15b8274554d3f1ba12b2598a347475f4"
   
   [[containers]]
   source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbc65f1fba7d92b36cf7514cd130fe83a9bd211005ddb23a8dc479e0eea645fd"
   ...
   …
   EOF

   1

   References for all non-optional MicroShift RPM packages using the same version compatible with the microshift-release-info RPM.

   2

   References for automatically enabling MicroShift on system startup and applying default networking settings.

   3

   References for all non-optional MicroShift container images necessary for an offline deployment.

3. Add the blueprint to the image builder by running the following command:

   $ sudo composer-cli blueprints push <microshift_blueprint.toml> 1

   1

   Replace <microshift_blueprint.toml> with the name of your TOML file.

Verification

1. Verify the image builder configuration listing only MicroShift packages by running the following command:

   $ sudo composer-cli blueprints depsolve <microshift_blueprint> | grep microshift 1

   1

   Replace <microshift_blueprint> with the name of your blueprint.

   Example output

   blueprint: microshift_blueprint v0.0.1
       microshift-greenboot-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch
       microshift-networking-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.x86_64
       microshift-release-info-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch
       microshift-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.x86_64
       microshift-selinux-4.17.1-202305250827.p0.g4105d3b.assembly.4.17.1.el9.noarch

2. Optional: Verify the image builder configuration listing all components to be installed by running the following command:

   $ sudo composer-cli blueprints depsolve <microshift_blueprint> 1

   1

   Replace <microshift_blueprint> with the name of your blueprint.

Procedure

1. Edit your ostree blueprint by running the following command:

   $ vi <microshift_blueprint.toml> 1

   1

   Replace <microshift_blueprint.toml> with the name of the blueprint file used for the MicroShift service.

2. Add the following example text to your blueprint:

   [[packages]] 1
   name = "<microshift-additional-package-name>" 2
   version = "*"

   1

   Include one stanza for each additional service that you want to add.

   2

   Replace <microshift-additional-package-name> with the name the RPM for the service you want to include. For example, microshift-olm.

Next steps

1. Add custom certificate authorities to the blueprint as needed.

2. After you are done adding to your blueprint, you can apply the manifests to an active cluster by building a new ostree system and deploying it on the client:

   • Create the ISO.
   • Add the blueprint and build the ISO.
   • Download the ISO and prepare it for use.
   • Do any provisioning that is needed.

Procedure

1. Start an ostree container image build by running the following command:

   $ BUILDID=$(sudo composer-cli compose start-ostree --ref "rhel/{op-system-version-major}/$(uname -m)/edge" <microshift_blueprint> edge-container | awk '{print $2}') 1

   1

   Replace <microshift_blueprint> with the name of your blueprint.

   This command also returns the identification (ID) of the build for monitoring.

2. You can check the status of the build periodically by running the following command:

   $ sudo composer-cli compose status

   Example output of a running build

   ID                                     Status     Time                     Blueprint                 Version   Type               Size
   cc3377ec-4643-4483-b0e7-6b0ad0ae6332   RUNNING    Wed Jun 7 12:26:23 2023  microshift_blueprint      0.0.1     edge-container

   Example output of a completed build

   ID                                     Status     Time                      Blueprint              Version   Type               Size
   cc3377ec-4643-4483-b0e7-6b0ad0ae6332   FINISHED   Wed Jun 7 12:32:37 2023   microshift_blueprint   0.0.1     edge-container

   Note

   You can use the watch command to monitor your build if you are familiar with how to start and stop it.

3. Download the container image using the ID and get the image ready for use by running the following command:

   $ sudo composer-cli compose image ${BUILDID}

4. Change the ownership of the downloaded container image to the current user by running the following command:

   $ sudo chown $(whoami). ${BUILDID}-container.tar

5. Add read permissions for the current user to the image by running the following command:

   $ sudo chmod a+r ${BUILDID}-container.tar

6. Bootstrap a server on port 8085 for the ostree container image to be consumed by the ISO build by completing the following steps:

   1. Get the IMAGEID variable result by running the following command:

      $ IMAGEID=$(cat < "./${BUILDID}-container.tar" | sudo podman load | grep -o -P '(?<=sha256[@:])[a-z0-9]*')

   2. Use the IMAGEID variable result to execute the podman command step by running the following command:

      $ sudo podman run -d --name=minimal-microshift-server -p 8085:8080 ${IMAGEID}

      This command also returns the ID of the container saved in the IMAGEID variable for monitoring.

7. Generate the installer blueprint file by running the following command:

   cat > microshift-installer.toml <<EOF
   name = "microshift-installer"
   
   description = ""
   version = "0.0.0"
   modules = []
   groups = []
   packages = []
   EOF

1. Add the blueprint to the image builder by running the following command:

   $ sudo composer-cli blueprints push microshift-installer.toml

2. Start the ostree ISO build by running the following command:

   $ BUILDID=$(sudo composer-cli compose start-ostree --url http://localhost:8085/repo/ --ref "rhel/9/$(uname -m)/edge" microshift-installer edge-installer | awk '{print $2}')

   This command also returns the identification (ID) of the build for monitoring.

3. You can check the status of the build periodically by running the following command:

   $ sudo composer-cli compose status

   Example output for a running build

   ID                                     Status     Time                      Blueprint              Version   Type               Size
   c793c24f-ca2c-4c79-b5b7-ba36f5078e8d   RUNNING    Wed Jun 7 13:22:20 2023   microshift-installer   0.0.0     edge-installer

   Example output for a completed build

   ID                                     Status     Time                      Blueprint              Version   Type               Size
   c793c24f-ca2c-4c79-b5b7-ba36f5078e8d   FINISHED   Wed Jun 7 13:34:49 2023   microshift-installer   0.0.0     edge-installer

1. Download the ISO using the ID by running the following command:

   $ sudo composer-cli compose image ${BUILDID}

2. Change the ownership of the downloaded container image to the current user by running the following command:

   $ sudo chown $(whoami). ${BUILDID}-installer.iso

3. Add read permissions for the current user to the image by running the following command:

   $ sudo chmod a+r ${BUILDID}-installer.iso

Prerequisites

1. You have created a RHEL for Edge Installer (ISO) image containing your RHEL for Edge commit with Red Hat build of MicroShift.

   1. This requirement includes the steps of composing an RFE Container image, creating the RFE Installer blueprint, starting the RFE container, and composing the RFE Installer image.

2. Create a Kickstart file or use an existing one. In the Kickstart file, you must include:

   1. Detailed instructions about how to create a user.
   2. How to fetch and deploy the RHEL for Edge image.

Procedure

1. In the main section of the Kickstart file, update the setup of the filesystem such that it contains an LVM volume group called rhel with at least 10GB system root. Leave free space for the LVMS CSI driver to use for storing the data for your workloads.

   Example kickstart snippet for configuring the filesystem

   # Partition disk such that it contains an LVM volume group called `rhel` with a
   # 10GB+ system root but leaving free space for the LVMS CSI driver for storing data.
   #
   # For example, a 20GB disk would be partitioned in the following way:
   #
   # NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
   # sda             8:0    0  20G  0 disk
   # ├─sda1          8:1    0 200M  0 part /boot/efi
   # ├─sda1          8:1    0 800M  0 part /boot
   # └─sda2          8:2    0  19G  0 part
   #  └─rhel-root  253:0    0  10G  0 lvm  /sysroot
   #
   ostreesetup --nogpg --osname=rhel --remote=edge \
   --url=file:///run/install/repo/ostree/repo --ref=rhel/<RHEL VERSION NUMBER>/x86_64/edge
   zerombr
   clearpart --all --initlabel
   part /boot/efi --fstype=efi --size=200
   part /boot --fstype=xfs --asprimary --size=800
   # Uncomment this line to add a SWAP partition of the recommended size
   #part swap --fstype=swap --recommended
   part pv.01 --grow
   volgroup rhel pv.01
   logvol / --vgname=rhel --fstype=xfs --size=10000 --name=root
   # To add users, use a line such as the following
   user --name=<YOUR_USER_NAME> \
   --password=<YOUR_HASHED_PASSWORD> \
   --iscrypted --groups=<YOUR_USER_GROUPS>

2. In the %post section of the Kickstart file, add your pull secret and the mandatory firewall rules.

   Example Kickstart snippet for adding the pull secret and firewall rules

   %post --log=/var/log/anaconda/post-install.log --erroronfail
   
   # Add the pull secret to CRI-O and set root user-only read/write permissions
   cat > /etc/crio/openshift-pull-secret << EOF
   YOUR_OPENSHIFT_PULL_SECRET_HERE
   EOF
   chmod 600 /etc/crio/openshift-pull-secret
   
   # Configure the firewall with the mandatory rules for MicroShift
   firewall-offline-cmd --zone=trusted --add-source=10.42.0.0/16
   firewall-offline-cmd --zone=trusted --add-source=169.254.169.1
   
   %end

3. Install the mkksiso tool by running the following command:

   $ sudo yum install -y lorax

4. Update the Kickstart file in the ISO with your new Kickstart file by running the following command:

   $ sudo mkksiso <your_kickstart>.ks <your_installer>.iso <updated_installer>.iso

Procedure

1. Access the list of container image references by using one of the following methods:

   • If the package is installed on the MicroShift host, get the location of the files by running the following command:

     $ rpm -ql microshift-release-info

     Example output

     /usr/share/microshift/release/release-x86_64.json

   • If the package is not installed on a MicroShift host, download and unpack the RPM package without installing it by running the following command:

     $ rpm2cpio microshift-release-info*.noarch.rpm | cpio -idmv

     Example output

     /usr/share/microshift/release/release-x86_64.json

2. Extract the list of container images into the microshift-container-refs.txt file by running the following commands:

   $ RELEASE_FILE=/usr/share/microshift/release/release-$(uname -m).json

   $ jq -r '.images | .[]' ${RELEASE_FILE} > microshift-container-refs.txt

Procedure

1. Install the skopeo tool used for copying the container images by running the following command:

   $ sudo dnf install -y skopeo

2. Set the environment variable that points to the pull secret file:

   $ PULL_SECRET_FILE=~/.pull-secret-mirror.json

3. Set the environment variable that points to the list of container images:

   $ IMAGE_LIST_FILE=~/microshift-container-refs.txt

4. Set the environment variable that points to the destination directory for storing the downloaded data:

   $ IMAGE_LOCAL_DIR=~/microshift-containers

5. Run the following script to download the container images to the ${IMAGE_LOCAL_DIR} directory:

   while read -r src_img ; do
      # Remove the source registry prefix
      dst_img=$(echo "${src_img}" | cut -d '/' -f 2-)
   
      # Run the image download command
      echo "Downloading '${src_img}' to '${IMAGE_LOCAL_DIR}'"
      mkdir -p "${IMAGE_LOCAL_DIR}/${dst_img}"
      skopeo copy --all --quiet \
         --preserve-digests \
         --authfile "${PULL_SECRET_FILE}" \
         docker://"${src_img}" dir://"${IMAGE_LOCAL_DIR}/${dst_img}"
   
   done < "${IMAGE_LIST_FILE}"

6. Transfer the image set to the target environment, such as air-gapped site. Then you can upload the image set into the mirror registry.

Procedure

1. Install the skopeo tool used for copying the container images by running the following command:

   $ sudo dnf install -y skopeo

2. Set the environment variables pointing to the pull secret file:

   $ IMAGE_PULL_FILE=~/.pull-secret-mirror.json

3. Set the environment variables pointing to the local container image directory:

   $ IMAGE_LOCAL_DIR=~/microshift-containers

4. Set the environment variables pointing to the mirror registry URL for uploading the container images:

   $ TARGET_REGISTRY=<registry_host>:<port> 1

   1

   Replace <registry_host>:<port> with the host name and port of your mirror registry server.

5. Run the following script to upload the container images to the ${TARGET_REGISTRY} mirror registry:

   image_tag=mirror-$(date +%y%m%d%H%M%S)
   image_cnt=1
      # Uses timestamp and counter as a tag on the target images to avoid
      # their overwrite by the 'latest' automatic tagging
   
   pushd "${IMAGE_LOCAL_DIR}" >/dev/null
   while read -r src_manifest ; do
      # Remove the manifest.json file name
      src_img=$(dirname "${src_manifest}")
      # Add the target registry prefix and remove SHA
      dst_img="${TARGET_REGISTRY}/${src_img}"
      dst_img=$(echo "${dst_img}" | awk -F'@' '{print $1}')
   
      # Run the image upload command
      echo "Uploading '${src_img}' to '${dst_img}'"
      skopeo copy --all --quiet \
         --preserve-digests \
         --authfile "${IMAGE_PULL_FILE}" \
         dir://"${IMAGE_LOCAL_DIR}/${src_img}" docker://"${dst_img}:${image_tag}-${image_cnt}"
      # Increment the counter
      (( image_cnt += 1 ))
   
   done < <(find . -type f -name manifest.json -printf '%P\n')
   popd >/dev/null

Procedure

1. Log into your MicroShift host.

2. Enable the SSL certificate trust on any host accessing the mirror registry by completing the following steps:

   1. Copy the rootCA.pem file from the mirror registry, for example, <registry_path>/quay-rootCA, to the MicroShift host at the /etc/pki/ca-trust/source/anchors directory.

   2. Enable the certificate in the system-wide trust store configuration by running the following command:

      $ sudo update-ca-trust

3. Create the /etc/containers/registries.conf.d/999-microshift-mirror.conf configuration file that maps the Red Hat registry host names to the mirror registry:

   Example mirror configuration file

   [[registry]]
       prefix = ""
       location = "<registry_host>:<port>" 1
       mirror-by-digest-only = true
       insecure = false
   
   [[registry]]
       prefix = ""
       location = "quay.io"
       mirror-by-digest-only = true
   [[registry.mirror]]
       location = "<registry_host>:<port>"
       insecure = false
   
   [[registry]]
       prefix = ""
       location = "registry.redhat.io"
       mirror-by-digest-only = true
   [[registry.mirror]]
       location = "<registry_host>:<port>"
       insecure = false
   
   [[registry]]
       prefix = ""
       location = "registry.access.redhat.com"
       mirror-by-digest-only = true
   [[registry.mirror]]
       location = "<registry_host>:<port>"
       insecure = false

   1

   Replace <registry_host>:<port> with the host name and port of your mirror registry server, for example, <microshift-quay:8443>.

4. Enable the MicroShift service by running the following command:

   $ sudo systemctl enable microshift

5. Reboot the host by running the following command:

   $ sudo reboot

Procedure

1. Get the exact list of container image references used by the MicroShift version you are deploying. You can either install the microshift-release-info RPM package by following step 2 or download and unpack the RPM by following step 3.

2. To install the microshift-release-info RPM package:

   1. Install the microshift-release-info RPM package by running the following command:

      $ sudo dnf install -y microshift-release-info-<release_version>

      Replace <release_version> with the numerical value of the release you are deploying, using the entire version number, such as 4.17.1.

   2. List the contents of the /usr/share/microshift/release directory to verify the presence of the release information files by running the following command:

      $ ls /usr/share/microshift/release

      Example output

      release-x86_64.json
      release-aarch64.json

      If you installed the microshift-release-info RPM, you can proceed to step 4.

3. If you did not complete step 2, download and unpack the microshift-release-info RPM without installing it:

   1. Download the RPM package by running the following command:

      $ sudo dnf download microshift-release-info-<release_version>

      Replace <release_version> with the numerical value of the release you are deploying, using the entire version number, such as 4.17.1.

      Example rpm

      microshift-release-info-4.17.1.*.el9.noarch.rpm 1

      1

      The * represents the date and commit ID. Your output should contain both, for example -202311101230.p0.g7dc6a00.assembly.4.17.1.

   2. Unpack the RPM package without installing it by running the following command:

      $ rpm2cpio <my_microshift_release_info> | cpio -idmv 1
      ./usr/share/microshift/release/release-aarch64.json
      ./usr/share/microshift/release/release-x86_64.json

      1

      Replace <my_microshift_release_info> with the name of the RPM package from the previous step.

4. Define the location of your JSON file, which contains the container reference information, by running the following command:

   $ RELEASE_FILE=</path/to/your/release-$(uname -m).json>

   Replace </path/to/your/release-$(uname -m).json> with the full path to your JSON file. Be sure to use the file needed for your architecture.

5. Define the location of your TOML file, which contains instructions for building the image, by running the following command:

   $ BLUEPRINT_FILE=</path/to/your/blueprint.toml>

   Replace </path/to/your/blueprint.toml> with the full path to your JSON file.

6. Generate and then embed the container image references in your blueprint TOML file by running the following command:

   $  jq -r '.images | .[] | ("[[containers]]\nsource = \"" + . + "\"\n")' "${RELEASE_FILE}" >> "${BLUEPRINT_FILE}"

   Example resulting <my_blueprint.toml> fragment showing container references

   [[containers]]
   source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82cfef91557f9a70cff5a90accba45841a37524e9b93f98a97b20f6b2b69e5db"
   
   [[containers]]
   source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82cfef91557f9a70cff5a90accba45841a37524e9b93f98a97b20f6b2b69e5db"

7. You can manually embed any container image by adding it to the image builder blueprint using the following example:

   Example section for manually embedding container image to image builder

   [[containers]]
   source = "<my_image_pullspec_with_tag_or_digest>"

   Replace <my_image_pullspec_with_tag_or_digest> with the exact reference to a container image used by the MicroShift version you are deploying.

Procedure

1. Add a pull secret for authenticating to the registry by setting the auth_file_path in the [containers] section of the /etc/osbuild-worker/osbuild-worker.toml osbuilder worker configuration file:

   [containers]
   auth_file_path = "/etc/osbuild-worker/pull-secret.json"

2. Restart the osbuild-worker to apply configuration changes by restarting the host. Restarting the host ensures that all osbuild-worker services currently running are restarted.