Running applications

Red Hat build of MicroShift 4.20

Running applications in MicroShift

Red Hat OpenShift Documentation Team

Abstract

This document provides details about how to run applications in MicroShift.

Chapter 1. Using Kustomize manifests to deploy applications
Copy link

You can use the kustomize configuration management tool with application manifests to deploy applications. Read through the following procedures for an example of how Kustomize works in MicroShift.

1.1. How Kustomize works with manifests to deploy applications
Copy link

The kustomize configuration management tool is integrated with MicroShift. You can use Kustomize and the OpenShift CLI (oc) together to apply customizations to your application manifests and deploy those applications to a MicroShift node.

A kustomization.yaml file is a specification of resources plus customizations.
Kustomize uses a kustomization.yaml file to load a resource, such as an application, then applies any changes you want to that application manifest and produces a copy of the manifest with the changes overlaid.
Using a manifest copy with an overlay keeps the original configuration file for your application intact, while enabling you to deploy iterations and customizations of your applications efficiently.
You can then deploy the application in your MicroShift node with an oc command.

Note

At each system start, MicroShift deletes the manifests found in the delete subdirectories and then applies the manifest files found in the manifest directories to the node.

1.1.1. How MicroShift uses manifests
Copy link

At every start, MicroShift searches the following manifest directories for Kustomize manifest files:

/etc/microshift/manifests
/etc/microshift/manifests.d/*
/usr/lib/microshift/
/usr/lib/microshift/manifests.d/*

MicroShift automatically runs the equivalent of the kubectl apply -k command to apply the manifests to the node if any of the following file types exists in the searched directories:

kustomization.yaml
kustomization.yml
Kustomization

This automatic loading from multiple directories means you can manage MicroShift workloads with the flexibility of having different workloads run independently of each other.

Expand

Table 1.1. MicroShift manifest directories
Location	Intent
`/etc/microshift/manifests`	Read-write location for configuration management systems or development.
`/etc/microshift/manifests.d/*`	Read-write location for configuration management systems or development.
`/usr/lib/microshift/manifests`	Read-only location for embedding configuration manifests on OSTree-based systems.
`/usr/lib/microshift/manifestsd./*`	Read-only location for embedding configuration manifests on OSTree-based systems.

1.2. Override the list of manifest paths
Copy link

You can override the list of default manifest paths by using a new single path, or by using a new glob pattern for multiple files. Use the following procedure to customize your manifest paths.

Procedure

Override the list of default paths by inserting your own values and running one of the following commands:
1. Set manifests.kustomizePaths to <"/opt/alternate/path"> in the configuration file for a single path.
2. Set kustomizePaths to ,"/opt/alternative/path.d/*". in the configuration file for a glob pattern.
  manifests: kustomizePaths: - <location>
  1
  Copy to Clipboard Toggle word wrap
  1
  Set each location entry to an exact path by using "/opt/alternate/path" or a glob pattern by using "/opt/alternative/path.d/*".
To disable loading manifests, set the configuration option to an empty list.
```
manifests:
    kustomizePaths: []
```
```
manifests:
    kustomizePaths: []
```
Copy to Clipboard Toggle word wrap
Note
The configuration file overrides the defaults entirely. If the kustomizePaths value is set, only the values in the configuration file are used. Setting the value to an empty list disables manifest loading.

1.3. Using manifests example
Copy link

This example demonstrates automatic deployment of a BusyBox container by using kustomize manifests in the /etc/microshift/manifests directory.

Procedure

Create the BusyBox manifest files by running the following commands:

Define the directory location:
```
MANIFEST_DIR=/etc/microshift/manifests
```
```
$ MANIFEST_DIR=/etc/microshift/manifests
```
Copy to Clipboard Toggle word wrap
Make the directory:
```
sudo mkdir -p ${MANIFEST_DIR}
```
```
$ sudo mkdir -p ${MANIFEST_DIR}
```
Copy to Clipboard Toggle word wrap

Place the YAML file in the directory:

sudo tee ${MANIFEST_DIR}/busybox.yaml &>/dev/null <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: busybox
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: busybox-deployment
spec:
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: BUSYBOX_IMAGE
        command: [ "/bin/sh", "-c", "while true ; do date; sleep 3600; done;" ]
EOF

sudo tee ${MANIFEST_DIR}/busybox.yaml &>/dev/null <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: busybox
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: busybox-deployment
spec:
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: BUSYBOX_IMAGE
        command: [ "/bin/sh", "-c", "while true ; do date; sleep 3600; done;" ]
EOF

Copy to Clipboard

Toggle word wrap

Next, create the kustomize manifest files by running the following commands:

Place the YAML file in the directory:

sudo tee ${MANIFEST_DIR}/kustomization.yaml &>/dev/null <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: busybox
resources:
  - busybox.yaml
images:
  - name: BUSYBOX_IMAGE
    newName: busybox:1.35
EOF

sudo tee ${MANIFEST_DIR}/kustomization.yaml &>/dev/null <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: busybox
resources:
  - busybox.yaml
images:
  - name: BUSYBOX_IMAGE
    newName: busybox:1.35
EOF

Copy to Clipboard

Toggle word wrap

Restart MicroShift to apply the manifests by running the following command:
```
sudo systemctl restart microshift
```
```
$ sudo systemctl restart microshift
```
Copy to Clipboard Toggle word wrap
Apply the manifests and start the busybox pod by running the following command:
```
oc get pods -n busybox
```
```
$ oc get pods -n busybox
```
Copy to Clipboard Toggle word wrap

Chapter 2. Deleting or updating Kustomize manifest resources
Copy link

MicroShift supports the deletion of manifest resources in the following situations:

Manifest removal: Manifests can be removed when you need to completely remove a resource from the node.
Manifest upgrade: During an application upgrade, some resources might need to be removed while others are retained to preserve data.

When creating new manifests, you can use manifest resource deletion to remove or update old objects, ensuring there are no conflicts or issues.

Important

Manifest files placed in the delete subdirectories are not automatically removed and require manual deletion. Only the resources listed in the manifest files placed in the delete subdirectories are deleted.

2.1. How manifest deletion works
Copy link

By default, MicroShift searches for deletion manifests in the delete subdirectories within the manifests path. When a user places a manifest in these subdirectories, MicroShift removes the manifests when the system is started. Read through the following to understand how manifests deletion works in MicroShift.

Each time the system starts, before applying the manifests, MicroShift scans the following delete subdirectories within the configured manifests directory to identify the manifests that need to be deleted:
- /usr/lib/microshift/manifests/delete
- /usr/lib/microshift/manifests.d/delete/*
- /etc/microshift/manifests/delete
- /etc/microshift/manifests.d/delete/*
MicroShift deletes the resources defined in the manifests found in the delete directories by running the equivalent of the kubectl delete --ignore-not-found -k command.

2.2. Use cases for manifest resource deletion
Copy link

The following sections explain the use case in which the manifest resource deletion is used.

2.2.1. Removing manifests for RPM systems
Copy link

Use the following procedure in the data removal scenario for RPM systems to completely delete the resource defined in the manifests.

Procedure

Identify the manifest that needs to be placed in the delete subdirectories.
Create the delete subdirectory in which the manifest will be placed by running the following command:
```
sudo mkdir -p <path_of_delete_directory>
```
```
$ sudo mkdir -p <path_of_delete_directory> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <path_of_delete_directory> with one of the following valid directory paths: /etc/microshift/manifests.d/delete, /etc/microshift/manifests/delete/, /usr/lib/microshift/manifests.d/delete, or /usr/lib/microshift/manifests/delete.
Move the manifest file into one of the delete subdirectories under the configured manifests directory by running the following command:
```
[sudo] mv <path_of_manifests> <path_of_delete_directory>
```
```
$ [sudo] mv <path_of_manifests> <path_of_delete_directory>
```
Copy to Clipboard Toggle word wrap
where: <path_of_manifests>:: Specifies the path of the manifest to be deleted, for example /etc/microshift/manifests.d/010-SOME-MANIFEST. <path_of_delete_directory>:: Specifies one of the following valid directory paths: /etc/microshift/manifests.d/delete, /etc/microshift/manifests/delete, /usr/lib/microshift/manifests.d/delete or /usr/lib/microshift/manifests/delete.
Restart MicroShift by running the following command:
```
sudo systemctl restart microshift
```
```
$ sudo systemctl restart microshift
```
Copy to Clipboard Toggle word wrap
MicroShift detects and removes the resource after the manifest file is placed in the delete subdirectories.

2.2.2. Removing manifests for OSTree systems
Copy link

Use the following procedure to completely delete the resource defined in the manifests.

Important

For OSTree installation, the delete subdirectories are read-only.

Procedure

Identify the manifest that needs to be placed in the delete subdirectories.
Package the manifest into an RPM. See Building the RPM package for the application for the procedure to package the manifest into an RPM.
Add the packaged RPM to the blueprint file to install it into correct location. See Adding application RPMs to a blueprint for the procedure to add an RPM to a blueprint.

2.2.3. Upgrading manifests for RPM systems
Copy link

Use the following procedure to remove some resources while retaining others to preserve data.

Procedure

Identify the manifest that requires updating.
Create new manifests to be applied in the manifest directories.
Create new manifests for resource deletion. It is not necessary to include the spec in these manifests. See Using manifests example to create new manifests using the example.
Use the procedure in "Removing manifests for RPM systems" to create delete subdirectories and place the manifests created for resource deletion in this path.

2.2.4. Upgrading manifests for OSTree systems
Copy link

Use the following procedure to remove some resources while retaining others to preserve data.

Important

For OSTree systems, the delete subdirectories are read-only.

Procedure

Identify the manifest that needs updating.
Create a new manifest to apply in the manifest directories. See Using manifests example to create new manifests using the example.
Create a new manifest for resource deletion to be placed in the delete subdirectories.
Use the procedure in "Removing manifests for OSTree systems" to remove the manifests.

Chapter 3. Using certificate manager on a MicroShift node
Copy link

The MicroShift certificate manager supports managing TLS certificates. This integration results in the issue, renewal, and management of certificate from certificate authorities.

3.1. MicroShift certificate manager functions
Copy link

With MicroShift certificate manager, you can complete the following tasks:

Automates certificate management: cert-manager creates or updates certificates and detects Kubernetes resources that are annotated with cert-manager.io/kind.
Supports multiple CAs: provides flexibility to select one that fits the security and operational needs.
Simplifies ingress certificates: cert-manager handles certificates for an ingress controller, which simplifies the configuration and management of secure communication channels.
Enhances security: certificate management is automated and the risk of error is reduced. Certificates are current and valid, which contribute to a secure environment.

3.2. Installing and enabling the cert-manager Operator using RPM
Copy link

The microshift-cert-manager RPM is an optional component that can be installed at any time. Follow these steps to install and verify the certificate manager:

Procedure

Install the cert-manager-operator using the microshift-cert-manager RPM by running the following command:
```
sudo dnf install microshift-cert-manager
```
```
$ sudo dnf install microshift-cert-manager
```
Copy to Clipboard Toggle word wrap
Verify the certificate manager versions that are used by running the following command:
```
rpm -qi microshift-cert-manager
```
```
$ rpm -qi microshift-cert-manager
```
Copy to Clipboard Toggle word wrap
Restart MicroShift by running the following command:
```
systemctl microshift restart
```
```
$ systemctl microshift restart
```
Copy to Clipboard Toggle word wrap

Verify that the microshift-cert-manager RPM is installed by running the following command:

oc get deployment -n  cert-manager-operator

$ oc get deployment -n  cert-manager-operator

Copy to Clipboard

Toggle word wrap

Example output

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager-operator-controller-manager   1/1     1            1           2d22h

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager-operator-controller-manager   1/1     1            1           2d22h

Copy to Clipboard

Toggle word wrap

Verify that the`cert-manager` deployments are in a ready state and are up-to-date in the cert-manager namespace by running the following command:

oc get deployment -n cert-manager

$ oc get deployment -n cert-manager

Copy to Clipboard

Toggle word wrap

Example output

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           2d22h
cert-manager-cainjector   1/1     1            1           2d22h
cert-manager-webhook      1/1     1            1           2d22h

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           2d22h
cert-manager-cainjector   1/1     1            1           2d22h
cert-manager-webhook      1/1     1            1           2d22h

Copy to Clipboard

Toggle word wrap

Verify that the pods are running in the cert-manager namespace by running the following command:

oc get pods -n cert-manager

$ oc get pods -n cert-manager

Copy to Clipboard

Toggle word wrap

Example output

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7cfb4fbb84-qdmk8              1/1     Running   2          2d22h
cert-manager-cainjector-854f669657-xzs8b   1/1     Running   2          2d22h
cert-manager-webhook-68fd6d5f5c-j942h      1/1     Running   2          2d22h

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7cfb4fbb84-qdmk8              1/1     Running   2          2d22h
cert-manager-cainjector-854f669657-xzs8b   1/1     Running   2          2d22h
cert-manager-webhook-68fd6d5f5c-j942h      1/1     Running   2          2d22h

Copy to Clipboard

Toggle word wrap

3.3. Installing and enabling the cert-manager Operator using OLM
Copy link

You can install the optional microshift-cert-manager by using OLM at any time. For more information, see Using Operator Lifecycle Manager with MicroShift and Installing the cert-manager Operator for Red Hat OpenShift.

Chapter 4. Using MicroShift Observability
Copy link

MicroShift Observability collects and transmits system data for monitoring and analysis. The data includes performance and usage metrics, and error reporting.

4.1. Installing and enabling MicroShift Observability
Copy link

You can install MicroShift Observability at any time, including during the initial MicroShift installation. Observability collects and transmits system data for monitoring and analysis, such as performance and usage metrics and error reporting.

Procedure

Install the microshift-observability RPM by entering the following command:
```
sudo dnf install microshift-observability
```
```
$ sudo dnf install microshift-observability
```
Copy to Clipboard Toggle word wrap
Enable the microshift-observability system service by entering the following command:
```
sudo systemctl enable microshift-observability
```
```
$ sudo systemctl enable microshift-observability
```
Copy to Clipboard Toggle word wrap
Start the microshift-observability system service by entering the following command:
```
sudo systemctl start microshift-observability
```
```
$ sudo systemctl start microshift-observability
```
Copy to Clipboard Toggle word wrap
Restart MicroShift after the initial installation.
```
sudo systemctl restart microshift-observability
```
```
$ sudo systemctl restart microshift-observability
```
Copy to Clipboard Toggle word wrap

The installation is successful if there is no output after you start the microshift-observability RPM.

4.2. Configuring MicroShift Observability
Copy link

You must configure MicroShift Observability after it is installed by specifying a valid endpoint. If an endpoint is not specified, MicroShift Observability does not start. You can specify any OpenTelemetry Protocol (OTLP)-compatible endpoint for each configuration before starting MicroShift.

Procedure

Update the /etc/microshift/observability/opentelemetry-collector.yaml file to specify an OTLP-compatible endpoint with the following information. The endpoint must link to an IP address or host name, and port number of an OTLP service.

OTLP-compatible endpoint configuration

# ...
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317 
    tls:
      insecure: true
# ...
service:
# ...
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318 
# ...

# ...
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317


    tls:
      insecure: true
# ...
service:
# ...
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318


# ...

Copy to Clipboard

Toggle word wrap

1: Replace ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. An unreachable endpoint is reported in the MicroShift service logs.
2: Replace ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. An unreachable endpoint is reported in the MicroShift service logs.

Each time that you update the opentelemetry-collector.yaml file, you must restart MicroShift Observability to apply the updates.
Restart MicroShift Observability by entering the following command:
```
sudo systemctl restart microshift-observability
```
```
$ sudo systemctl restart microshift-observability
```
Copy to Clipboard Toggle word wrap

4.3. Selecting a MicroShift Observability configuration
Copy link

The amount and complexity of the data depends on predefined configurations. These configurations determine the number of data sources and the amount of collected data that is transmitted. These configurations are defined as small, medium, and large (default).

The opentelemetry-collector.yaml file includes specific parameters that are used to collect data for monitoring the system resources. All warnings for node events are included in the collected data. MicroShift Observability collects and transmits data for the following resources:

CPU, memory, disk, and network metrics of containers, pods, and nodes
Kubernetes events
Host CPU, memory, disk, and network metrics
System journals for certain MicroShift services, and dependencies
Metrics exposed by pods that have the prometheus.io/scrape: true annotation

Replace the values of the exporters.otlp.endpoint and services.telemetry.metrics.readers[0].endpoint fields with the IP address or hostname of the remote back end. This IP address resolves to the local node’s host name. Any unreachable endpoint is reported in the MicroShift observability service logs.

4.4. Selecting a small configuration
Copy link

You can configure MicroShift Observability to collect the smallest amount of performance and resource information from various sources by updating the YAML file.

Procedure

Select a small configuration by adding the following information to the /etc/microshift/observability/opentelemetry-collector.yaml file:

 receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317 
    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318

 receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317


    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318

Copy to Clipboard

Toggle word wrap

1: Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the MicroShift service logs.
2: Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the MicroShift service logs.

Restart MicroShift Observability to complete the configuration selection by entering the following command:
```
sudo systemctl restart microshift-observability
```
```
$ sudo systemctl restart microshift-observability
```
Copy to Clipboard Toggle word wrap

4.5. Selecting a medium configuration
Copy link

You can configure MicroShift Observability to collect performance and resource information from various sources by updating the YAML file.

Procedure

Select a medium configuration by adding the following information to the /etc/microshift/observability/opentelemetry-collector.yaml file:

 receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
  journald:
    units:
      - microshift
    priority: info
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317 
    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/journald:
      receivers: [ journald ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318

 receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
  journald:
    units:
      - microshift
    priority: info
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317


    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/journald:
      receivers: [ journald ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318

Copy to Clipboard

Toggle word wrap

1: Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.
2: Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.

Restart MicroShift Observability to complete the configuration selection by entering the following command:
```
sudo systemctl restart microshift-observability
```
```
$ sudo systemctl restart microshift-observability
```
Copy to Clipboard Toggle word wrap

4.6. Selecting a large configuration
Copy link

You can configure MicroShift Observability to collect the maximum amount of performance and resource information, from the maximum number of sources, by updating the YAML file.

Procedure

Select a large configuration by adding the following information to the /etc/microshift/observability/opentelemetry-collector.yaml file. Large is the default configuration.

receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
  hostmetrics:
    root_path: /
    collection_interval: 10s
    scrapers:
      cpu:
      memory:
      network:
      disk:
      filesystem:
  journald:
    units:
      - microshift
      - microshift-observability
      - microshift-etcd
      - crio
      - openvswitch.service
      - ovsdb-server.service
      - ovs-vswitchd.service
    priority: info
  prometheus:
    config:
      scrape_configs:
        - job_name: k8s
          scrape_interval: 10s
          kubernetes_sd_configs:
            - kubeconfig_file: /var/lib/microshift/resources/observability-client/kubeconfig
              role: pod
          relabel_configs:
              # Only scrape Pods with annotation "prometheus.io/scrape": "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
              # Use value of "prometheus.io/path" annotation for scraping
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)
              # Use value of "prometheus.io/port" annotation for scraping
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              target_label: __address__
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317 
    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    metrics/hostmetrics:
      receivers: [ hostmetrics ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/host:
      receivers: [ hostmetrics ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
    logs/journald:
      receivers: [ journald ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
    metrics/pods:
      receivers: [ prometheus ]
      processors: [ batch ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318

receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
  hostmetrics:
    root_path: /
    collection_interval: 10s
    scrapers:
      cpu:
      memory:
      network:
      disk:
      filesystem:
  journald:
    units:
      - microshift
      - microshift-observability
      - microshift-etcd
      - crio
      - openvswitch.service
      - ovsdb-server.service
      - ovs-vswitchd.service
    priority: info
  prometheus:
    config:
      scrape_configs:
        - job_name: k8s
          scrape_interval: 10s
          kubernetes_sd_configs:
            - kubeconfig_file: /var/lib/microshift/resources/observability-client/kubeconfig
              role: pod
          relabel_configs:
              # Only scrape Pods with annotation "prometheus.io/scrape": "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
              # Use value of "prometheus.io/path" annotation for scraping
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)
              # Use value of "prometheus.io/port" annotation for scraping
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              target_label: __address__
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317


    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    metrics/hostmetrics:
      receivers: [ hostmetrics ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/host:
      receivers: [ hostmetrics ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
    logs/journald:
      receivers: [ journald ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
    metrics/pods:
      receivers: [ prometheus ]
      processors: [ batch ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318

Copy to Clipboard

Toggle word wrap

1: Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.
2: Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.

Restart MicroShift Observability to complete the configuration selection by entering the following command:
```
sudo systemctl restart microshift-observability
```
```
$ sudo systemctl restart microshift-observability
```
Copy to Clipboard Toggle word wrap

4.7. Verifying the MicroShift Observability state
Copy link

After MicroShift Observability starts, you can verify the state by using a systemd service. The MicroShift Observability service logs are available as journald logs.

Procedure

Check the MicroShift Observability status by entering the following command:
```
sudo systemctl status microshift-observability
```
```
$ sudo systemctl status microshift-observability
```
Copy to Clipboard Toggle word wrap
Check the MicroShift Observability logs by entering the following command:
```
sudo journalctl -u microshift-observability
```
```
$ sudo journalctl -u microshift-observability
```
Copy to Clipboard Toggle word wrap

Chapter 5. Options for embedding applications in a RHEL for Edge image
Copy link

You can embed microservices-based workloads and applications in a Red Hat Enterprise Linux for Edge (RHEL for Edge) image to run in a MicroShift node. Embedded applications can be installed directly on edge devices to run in disconnected or offline environments.

5.1. Adding application RPMs to an rpm-ostree image
Copy link

If you have an application that includes APIs, container images, and configuration files for deployment such as manifests, you can build application RPMs. You can then add the RPMs to your RHEL for Edge system image.

The following is an outline of the procedures to embed applications or workloads in a fully self-contained operating system image:

Build your own RPM that includes your application manifest.
Add the RPM to the blueprint you used to install Red Hat build of MicroShift.
Add the workload container images to the same blueprint.
Create a bootable ISO.

For a step-by-step tutorial about preparing and embedding applications in a RHEL for Edge image, use the following tutorial:

Embedding applications tutorial

5.2. Adding application manifests to an image for offline use
Copy link

If you have a simple application that includes a few files for deployment such as manifests, you can add those manifests directly to a RHEL for Edge system image.

See the "Create a custom file blueprint customization" section of the following RHEL for Edge documentation for an example:

Create a custom file blueprint customization

5.3. Embedding applications for offline use
Copy link

If you have an application that includes more than a few files, you can embed the application for offline use. See the following procedure:

Embedding applications for offline use

Chapter 6. Embedding applications for offline use
Copy link

You can embed microservices-based workloads and applications in a Red Hat Enterprise Linux for Edge (RHEL for Edge) image. Embedding means you can run a MicroShift node in air-gapped, disconnected, or offline environments.

6.1. Embedding workload container images for offline use
Copy link

To embed container images in devices at the edge that do not have any network connection, you must create a new container, mount the ISO, and then copy the contents into the file system.

Prerequisites

You have root access to the host.
Application RPMs have been added to a blueprint.
You installed the OpenShift CLI (oc).

Procedure

Render the manifests, extract all of the container image references, and translate the application image to blueprint container sources by running the following command:

oc kustomize ~/manifests | grep "image:" | grep -oE '[^ ]+$' | while read line; do echo -e "[[containers]]\nsource = \"${line}\"\n"; done >><my_blueprint>.toml

$ oc kustomize ~/manifests | grep "image:" | grep -oE '[^ ]+$' | while read line; do echo -e "[[containers]]\nsource = \"${line}\"\n"; done >><my_blueprint>.toml

Copy to Clipboard

Toggle word wrap

Push the updated blueprint to image builder by running the following command:
```
sudo composer-cli blueprints push <my_blueprint>.toml
```
```
$ sudo composer-cli blueprints push <my_blueprint>.toml
```
Copy to Clipboard Toggle word wrap
If your workload containers are located in a private repository, you must provide image builder with the necessary pull secrets:
1. Set the auth_file_path in the [containers] section in the /etc/osbuild-worker/osbuild-worker.toml configuration file to point to the pull secret.
2. If needed, create a directory and file for the pull secret, for example:
  Example directory and file
  [containers] auth_file_path = "/<path>/pull-secret.json"
  1
  
  Copy to Clipboard Toggle word wrap
  1
  Use the custom location previously set for copying and retrieving images.

Build the container image by running the following command:

sudo composer-cli compose start-ostree <my_blueprint> edge-commit

$ sudo composer-cli compose start-ostree <my_blueprint> edge-commit

Copy to Clipboard

Toggle word wrap

Proceed with your preferred rpm-ostree image flow, such as waiting for the build to complete, exporting the image and integrating it into your rpm-ostree repository or creating a bootable ISO.

Chapter 7. Embedding MicroShift applications tutorial
Copy link

The following tutorial gives a detailed example of how to embed applications in a RHEL for Edge image for use in a MicroShift node in various environments.

7.1. Embed application RPMs tutorial
Copy link

The following tutorial reviews the MicroShift installation steps and adds a description of the workflow for embedding applications. If you are already familiar with rpm-ostree systems such as Red Hat Enterprise Linux for Edge (RHEL for Edge) and MicroShift, you can go straight to the procedures.

7.1.1. Installation workflow review
Copy link

Embedding applications requires a similar workflow to embedding MicroShift into a RHEL for Edge image.

The following image shows how system artifacts such as RPMs, containers, and files are added to a blueprint and used by the image composer to create an ostree commit.
The ostree commit then can follow either the ISO path or the repository path to edge devices.
The ISO path can be used for disconnected environments, while the repository path is often used in places were the network is usually connected.

Embedding MicroShift workflow

468 RHbM install workflow 1023 1

Reviewing these steps can help you understand the steps needed to embed an application:

To embed MicroShift on RHEL for Edge, you added the MicroShift repositories to image builder.
You created a blueprint that declared all the RPMs, container images, files and customizations you needed, including the addition of MicroShift.
You added the blueprint to image builder and ran a build with the image builder CLI tool (composer-cli). This step created rpm-ostree commits, which were used to create the container image. This image contained RHEL for Edge.
You added the installer blueprint to image builder to create an rpm-ostree image (ISO) to boot from. This build contained both RHEL for Edge and MicroShift.
You downloaded the ISO with MicroShift embedded, prepared it for use, provisioned it, then installed it onto your edge devices.

7.1.2. Embed application RPMs workflow
Copy link

After you have set up a build host that meets the image builder requirements, you can add your application in the form of a directory of manifests to the image. After those steps, the simplest way to embed your application or workload into a new ISO is to create your own RPMs that include the manifests. Your application RPMs contain all of the configuration files describing your deployment.

The following "Embedding applications workflow" image shows how Kubernetes application manifests and RPM spec files are combined in a single application RPM build. This build becomes the RPM artifact included in the workflow for embedding MicroShift in an ostree commit.

Embedding applications workflow

468 RHbM install workflow 1023 2

The following procedures use the rpmbuild tool to create a specification file and local repository. The specification file defines how the package is built, moving your application manifests to the correct location inside the RPM package for MicroShift to pick them up. That RPM package is then embedded in the ISO.

7.1.3. Preparing to make application RPMs
Copy link

To build your own RPMs, choose a tool of your choice, such as the rpmbuild tool, and initialize the RPM build tree in your home directory. The following is an example procedure. If your RPMs are accessible to image builder, you can use the method you prefer to build the application RPMs.

Prerequisites

You have set up a Red Hat Enterprise Linux for Edge (RHEL for Edge) 9.6 build host that meets the image builder system requirements.
You have root access to the host.

Procedure

Install the rpmbuild tool and create the yum repository for it by running the following command:
```
sudo dnf install rpmdevtools rpmlint yum-utils createrepo
```
```
$ sudo dnf install rpmdevtools rpmlint yum-utils createrepo
```
Copy to Clipboard Toggle word wrap
Create the file tree you need to build RPM packages by running the following command:
```
rpmdev-setuptree
```
```
$ rpmdev-setuptree
```
Copy to Clipboard Toggle word wrap

Verification

List the directories to confirm creation by running the following command:
```
ls ~/rpmbuild/
```
```
$ ls ~/rpmbuild/
```
Copy to Clipboard Toggle word wrap
Example output
```
BUILD RPMS SOURCES SPECS SRPMS
```
```
BUILD RPMS SOURCES SPECS SRPMS
```
Copy to Clipboard Toggle word wrap

7.1.4. Building the RPM package for the application manifests
Copy link

To build your own RPMs, you must create a spec file that adds the application manifests to the RPM package. The following is an example procedure. As long as the application RPMs and other elements needed for image building are accessible to image builder, you can use the method that you prefer.

Prerequisites

You have set up a Red Hat Enterprise Linux for Edge (RHEL for Edge) 9.6 build host that meets the image builder system requirements.
You have root access to the host.
The file tree required to build RPM packages was created.

Procedure

In the ~/rpmbuild/SPECS directory, create a file such as <application_workload_manifests.spec> using the following template:

Example spec file

Name: <application_workload_manifests>
Version: 0.0.1
Release: 1%{?dist}
Summary: Adds workload manifests to microshift
BuildArch: noarch
License: GPL
Source0: %{name}-%{version}.tar.gz
#Requires: microshift
%description
Adds workload manifests to microshift
%prep
%autosetup
%install 
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT/%{_prefix}/lib/microshift/manifests
cp -pr ~/manifests $RPM_BUILD_ROOT/%{_prefix}/lib/microshift/
%clean
rm -rf $RPM_BUILD_ROOT

%files
%{_prefix}/lib/microshift/manifests/**
%changelog
* <DDD MM DD YYYY username@domain - V major.minor.patch>
- <your_change_log_comment>

Name: <application_workload_manifests>
Version: 0.0.1
Release: 1%{?dist}
Summary: Adds workload manifests to microshift
BuildArch: noarch
License: GPL
Source0: %{name}-%{version}.tar.gz
#Requires: microshift
%description
Adds workload manifests to microshift
%prep
%autosetup
%install


rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT/%{_prefix}/lib/microshift/manifests
cp -pr ~/manifests $RPM_BUILD_ROOT/%{_prefix}/lib/microshift/
%clean
rm -rf $RPM_BUILD_ROOT

%files
%{_prefix}/lib/microshift/manifests/**
%changelog
* <DDD MM DD YYYY username@domain - V major.minor.patch>
- <your_change_log_comment>

Copy to Clipboard

Toggle word wrap

1: The %install section creates the target directory inside the RPM package, /usr/lib/microshift/manifests/ and copies the manifests from the source home directory, ~/manifests.

Important

All of the required YAML files must be in the source home directory ~/manifests, including a kustomize.yaml file if you are using kustomize.

Build your RPM package in the ~/rpmbuild/RPMS directory by running the following command:
```
rpmbuild -bb ~/rpmbuild/SPECS/<application_workload_manifests.spec>
```
```
$ rpmbuild -bb ~/rpmbuild/SPECS/<application_workload_manifests.spec>
```
Copy to Clipboard Toggle word wrap

7.1.5. Adding application RPMs to a blueprint
Copy link

To add application RPMs to a blueprint, you must create a local repository that image builder can use to create the ISO. With this procedure, the required container images for your workload can be pulled over the network.

Prerequisites

You have root access to the host.
Workload or application RPMs exist in the ~/rpmbuild/RPMS directory.

Procedure

Create a local RPM repository by running the following command:
```
createrepo ~/rpmbuild/RPMS/
```
```
$ createrepo ~/rpmbuild/RPMS/
```
Copy to Clipboard Toggle word wrap
Give image builder access to the RPM repository by running the following command:
```
sudo chmod a+rx ~
```
```
$ sudo chmod a+rx ~
```
Copy to Clipboard Toggle word wrap
Note
You must ensure that image builder has all of the necessary permissions to access all of the files needed for image building, or the build cannot proceed.

Create the blueprint file, repo-local-rpmbuild.toml using the following template:

id = "local-rpm-build"
name = "RPMs build locally"
type = "yum-baseurl"
url = "file://<path>/rpmbuild/RPMS" 
check_gpg = false
check_ssl = false
system = false

id = "local-rpm-build"
name = "RPMs build locally"
type = "yum-baseurl"
url = "file://<path>/rpmbuild/RPMS"


check_gpg = false
check_ssl = false
system = false

Copy to Clipboard

Toggle word wrap

1: Specify part of the path to create a location that you choose. Use this path in the later commands to set up the repository and copy the RPMs.

Add the repository as a source for image builder by running the following command:
```
sudo composer-cli sources add repo-local-rpmbuild.toml
```
```
$ sudo composer-cli sources add repo-local-rpmbuild.toml
```
Copy to Clipboard Toggle word wrap

Add the RPM to your blueprint, by adding the following lines:

…
[[packages]]
name = "<application_workload_manifests>" 
version = "*"
…

…
[[packages]]
name = "<application_workload_manifests>"


version = "*"
…

Copy to Clipboard

Toggle word wrap

1: Add the name of your workload here.

Push the updated blueprint to image builder by running the following command:
```
sudo composer-cli blueprints push repo-local-rpmbuild.toml
```
```
$ sudo composer-cli blueprints push repo-local-rpmbuild.toml
```
Copy to Clipboard Toggle word wrap
At this point, you can either run image builder to create the ISO, or embed the container images for offline use.
1. To create the ISO, start image builder by running the following command:
  $ sudo composer-cli compose start-ostree repo-local-rpmbuild edge-commit
  Copy to Clipboard Toggle word wrap

In this scenario, the container images are pulled over the network by the edge device during startup.

Chapter 8. Using greenboot for application and workload health checks
Copy link

You can use greenboot health checks to assess the health of your workloads and applications.

8.1. How workload health checks work
Copy link

Greenboot health checks are helpful on edge devices where direct serviceability is either limited or non-existent. You can use greenboot health checks to assess the health of your workloads and applications. These additional health checks are useful for software problem detection and automatic system rollbacks.

Workload or application health checks can use the MicroShift basic health check functions already implemented for the MicroShift core services. Creating your own comprehensive scripts for your applications is recommended. For example, you can write one that verifies that a service has started.

You can also use the microshift healthcheck command, which can run checks that the basic functions of the workload are operating as expected.

Important

The following functions related to checking workload health in /usr/share/microshift/functions/greenboot.sh are deprecated and planned for removal in a future release:

wait_for
namespace_images_downloaded
namespace_deployment_ready
namespace_daemonset_ready
namespace_pods_ready
namespace_pods_not_restarting
print_failure_logs
log_failure_cmd
log_script_exit
lvmsDriverShouldExist
csiComponentShouldBeDeploy

8.2. How to use the MicroShift health check command
Copy link

The microshift healthcheck command checks whether a workload of the provided type exists and verifies its status for the specified timeout duration. The number of ready replicas, that is, pods, must match the expected amount.

To run the microshift healthcheck command successfully, use the following prerequisites:

Execute commands from a root user account.
Enable the MicroShift service.

You can add the following actions to the microshift healthcheck command:

-v=2 to increase verbosity of the output
--timeout="${WAIT_TIMEOUT_SECS}s" to override default 600s timeout value
--namespace `<namespace> to specify the namespace of the workloads

--deployments `<application-deployment> to check the readiness of a specific deployment

Example command

sudo microshift healthcheck -v=2 --timeout="300s" --namespace busybox --deployments busybox-deployment

$ sudo microshift healthcheck -v=2 --timeout="300s" --namespace busybox --deployments busybox-deployment

Copy to Clipboard

Toggle word wrap

Example output

??? I0410 08:54:03.766578    5898 service.go:29] microshift.service is enabled
??? I0410 08:54:03.766699    5898 service.go:31] Waiting 5m0s for microshift.service to be ready
??? I0410 08:54:03.768794    5898 service.go:38] microshift.service is ready
??? I0410 08:54:03.770585    5898 utils.go:34] Waiting for 1 goroutines
??? I0410 08:54:03.770955    5898 workloads.go:94] Waiting 5m0s for deployment/busybox-deployment in busybox
??? I0410 08:54:03.777830    5898 workloads.go:132] Deployment/busybox-deployment in busybox is ready
??? I0410 08:54:03.777858    5898 healthcheck.go:75] Workloads are ready

??? I0410 08:54:03.766578    5898 service.go:29] microshift.service is enabled
??? I0410 08:54:03.766699    5898 service.go:31] Waiting 5m0s for microshift.service to be ready
??? I0410 08:54:03.768794    5898 service.go:38] microshift.service is ready
??? I0410 08:54:03.770585    5898 utils.go:34] Waiting for 1 goroutines
??? I0410 08:54:03.770955    5898 workloads.go:94] Waiting 5m0s for deployment/busybox-deployment in busybox
??? I0410 08:54:03.777830    5898 workloads.go:132] Deployment/busybox-deployment in busybox is ready
??? I0410 08:54:03.777858    5898 healthcheck.go:75] Workloads are ready

Copy to Clipboard

Toggle word wrap

The microshift healthcheck command also accepts the following additional parameters to specify other kinds of workloads:

--daemonsets
--statefulsets
These options take a comma-delimited list of resources, for example, --daemonsets ovnkube-master,ovnkube-node.

Alternatively, a --custom option can be used with a JSON string, for example:

sudo microshift healthcheck --custom '{"openshift-storage":{"deployments":

$ sudo microshift healthcheck --custom '{"openshift-storage":{"deployments":
    ["lvms-operator"], "daemonsets": ["vg-manager"]}, "openshift-ovn-kubernetes":
    {"daemonsets": ["ovnkube-master", "ovnkube-node"]}}'

Copy to Clipboard

Toggle word wrap

Example output

??? I0410 08:54:25.291059    5979 service.go:29] microshift.service is enabled
??? I0410 08:54:25.291167    5979 service.go:31] Waiting 5m0s for microshift.service to be ready
??? I0410 08:54:25.293188    5979 service.go:38] microshift.service is ready
??? I0410 08:54:25.294331    5979 workloads.go:58] Waiting 5m0s for daemonset/ovnkube-node in openshift-ovn-kubernetes
??? I0410 08:54:25.294351    5979 workloads.go:58] Waiting 5m0s for daemonset/ovnkube-master in openshift-ovn-kubernetes
??? I0410 08:54:25.294331    5979 workloads.go:58] Waiting 5m0s for daemonset/vg-manager in openshift-storage
??? I0410 08:54:25.294341    5979 workloads.go:94] Waiting 5m0s for deployment/lvms-operator in openshift-storage
??? I0410 08:54:25.309739    5979 workloads.go:89] Daemonset/ovnkube-node in openshift-ovn-kubernetes is ready
??? I0410 08:54:25.310213    5979 workloads.go:89] Daemonset/vg-manager in openshift-storage is ready
??? I0410 08:54:25.310731    5979 workloads.go:132] Deployment/lvms-operator in openshift-storage is ready
??? I0410 08:54:25.311017    5979 workloads.go:89] Daemonset/ovnkube-master in openshift-ovn-kubernetes is ready
??? I0410 08:54:25.311189    5979 healthcheck.go:52] Workloads are ready

??? I0410 08:54:25.291059    5979 service.go:29] microshift.service is enabled
??? I0410 08:54:25.291167    5979 service.go:31] Waiting 5m0s for microshift.service to be ready
??? I0410 08:54:25.293188    5979 service.go:38] microshift.service is ready
??? I0410 08:54:25.294331    5979 workloads.go:58] Waiting 5m0s for daemonset/ovnkube-node in openshift-ovn-kubernetes
??? I0410 08:54:25.294351    5979 workloads.go:58] Waiting 5m0s for daemonset/ovnkube-master in openshift-ovn-kubernetes
??? I0410 08:54:25.294331    5979 workloads.go:58] Waiting 5m0s for daemonset/vg-manager in openshift-storage
??? I0410 08:54:25.294341    5979 workloads.go:94] Waiting 5m0s for deployment/lvms-operator in openshift-storage
??? I0410 08:54:25.309739    5979 workloads.go:89] Daemonset/ovnkube-node in openshift-ovn-kubernetes is ready
??? I0410 08:54:25.310213    5979 workloads.go:89] Daemonset/vg-manager in openshift-storage is ready
??? I0410 08:54:25.310731    5979 workloads.go:132] Deployment/lvms-operator in openshift-storage is ready
??? I0410 08:54:25.311017    5979 workloads.go:89] Daemonset/ovnkube-master in openshift-ovn-kubernetes is ready
??? I0410 08:54:25.311189    5979 healthcheck.go:52] Workloads are ready

Copy to Clipboard

Toggle word wrap

8.3. How to create a health check script for your application
Copy link

You can create workload or application health check scripts in the text editor of your choice. Save the scripts in the /etc/greenboot/check/required.d directory. When a script in the /etc/greenboot/check/required.d directory exits with an error, greenboot triggers a reboot in an attempt to heal the system.

Note

Any script in the /etc/greenboot/check/required.d directory triggers a reboot if it exits with an error.

If your health check logic requires any post-check steps, you can also create additional scripts and save them in the relevant greenboot directories. For example:

You can also place shell scripts you want to run after a boot has been declared successful in /etc/greenboot/green.d.
You can place shell scripts you want to run after a boot has been declared failed in /etc/greenboot/red.d. For example, if you have steps to heal the system before restarting, you can create scripts for your use case and place them in the /etc/greenboot/red.d directory.

8.3.1. Workload max duration or timeout script example
Copy link

The following example uses the MicroShift core services health check script as a template.

8.3.1.1. Basic prerequisites for creating a health check script
Copy link

The workload must be installed.
You must have root access.

8.3.1.2. Example and functional requirements
Copy link

You can start with the following example health check script. Add to it for your use case. In your custom workload health check script, you must define the relevant namespace, deployment, daemonset, and statefulset.

Important

Choose a name prefix for your application that ensures it runs after the 40_microshift_running_check.sh script, which implements the MicroShift health check procedure for its core services.

Example greenboot health check script

Load the workload health check functions library
Stop the script if the user running it is not 'root'
Set the wait timeout for the current check based on the boot counter

#!/bin/bash
set -e

SCRIPT_NAME=$(basename $0)

# Load the workload health check functions library
source /usr/share/microshift/functions/greenboot.sh

# Stop the script if the user running it is not 'root'
if [ $(id -u) -ne 0 ] ; then
    echo "The '${SCRIPT_NAME}' script must be run with the 'root' user privileges"
    exit 1
fi

echo "STARTED"

# Set the wait timeout for the current check based on the boot counter
WAIT_TIMEOUT_SECS=$(get_wait_timeout)

/usr/bin/microshift healthcheck -v=2 --timeout="${WAIT_TIMEOUT_SECS}s" --namespace busybox --deployments busybox-deployment

Copy to Clipboard

Toggle word wrap

Important

Functions related to checking workload health previously included in the /usr/share/microshift/functions/greenboot.sh script file are deprecated. You can write a custom script, or use the microshift healthcheck command with various options instead. See "How workload health check scripts work" for more information.

8.3.2. Testing a workload health check script
Copy link

The output of the greenboot workload health check script varies with the host system type. Example outputs for Red Hat Enterprise Linux (RHEL) system types are included for reference only.

Prerequisites

You have root access.
You installed a workload.
You created a health check script for the workload.
The MicroShift service is enabled.

Procedure

To test that greenboot is running a health check script file, reboot the host by running the following command:
```
sudo reboot
```
```
$ sudo reboot
```
Copy to Clipboard Toggle word wrap

Examine the output of greenboot health checks by running the following command:

sudo journalctl -o cat -u greenboot-healthcheck.service

$ sudo journalctl -o cat -u greenboot-healthcheck.service

Copy to Clipboard

Toggle word wrap

Note

MicroShift core service health checks run before the workload health checks.

Example output for an image mode for RHEL system

Starting greenboot Health Checks Runner...
Running Required Health Check Scripts...
Script '00_required_scripts_start.sh' SUCCESS
Running Wanted Health Check Scripts...
Script '00_wanted_scripts_start.sh' SUCCESS
Running Required Health Check Scripts...
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 11:54:30.526488     979 service.go:29] microshift.service is enabled
??? I0403 11:54:30.527145     979 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 11:58:52.530299     979 service.go:38] microshift.service is ready
??? I0403 11:58:52.532292     979 net.go:79] host gateway IP address: 192.168.112.125
??? I0403 11:58:52.555077     979 microshift_core_workloads.go:71] vgs reported: {"report":[{"vg":[{"vg_name":"rhel"}]}],"log":[]}
??? I0403 11:58:52.555138     979 microshift_core_workloads.go:93] Detected 1 volume group (rhel) - LVMS is expected
??? I0403 11:58:52.555143     979 microshift_core_workloads.go:126] Configured optional CSI components: []
??? I0403 11:58:52.555147     979 microshift_core_workloads.go:117] At least one CSI Component is enabled
??? I0403 11:58:52.555770     979 utils.go:34] Waiting for 9 goroutines
??? I0403 11:58:52.555791     979 workloads.go:94] Waiting 10m0s for deployment/service-ca in openshift-service-ca
??? I0403 11:58:52.555890     979 workloads.go:58] Waiting 10m0s for daemonset/ovnkube-master in openshift-ovn-kubernetes
??? I0403 11:58:52.555999     979 workloads.go:94] Waiting 10m0s for deployment/router-default in openshift-ingress
??? I0403 11:58:52.556096     979 workloads.go:58] Waiting 10m0s for daemonset/dns-default in openshift-dns
??? I0403 11:58:52.556244     979 workloads.go:58] Waiting 10m0s for daemonset/ovnkube-node in openshift-ovn-kubernetes
??? I0403 11:58:52.556330     979 workloads.go:94] Waiting 10m0s for deployment/lvms-operator in openshift-storage
??? I0403 11:58:52.556382     979 workloads.go:58] Waiting 10m0s for daemonset/vg-manager in openshift-storage
??? I0403 11:58:52.556425     979 workloads.go:94] Waiting 10m0s for deployment/csi-snapshot-controller in kube-system
??? I0403 11:58:52.556474     979 workloads.go:58] Waiting 10m0s for daemonset/node-resolver in openshift-dns
??? I0403 11:58:52.574284     979 workloads.go:89] Daemonset/ovnkube-node in openshift-ovn-kubernetes is ready
??? I0403 11:58:52.574344     979 workloads.go:89] Daemonset/dns-default in openshift-dns is ready
??? I0403 11:59:12.871058     979 workloads.go:89] Daemonset/node-resolver in openshift-dns is ready
??? I0403 11:59:12.871621     979 workloads.go:89] Daemonset/ovnkube-master in openshift-ovn-kubernetes is ready
??? I0403 11:59:12.871748     979 workloads.go:132] Deployment/csi-snapshot-controller in kube-system is ready
??? I0403 11:59:25.175015     979 workloads.go:132] Deployment/service-ca in openshift-service-ca is ready
??? I0403 11:59:42.559264     979 workloads.go:132] Deployment/lvms-operator in openshift-storage is ready
??? I0403 11:59:52.557786     979 workloads.go:132] Deployment/router-default in openshift-ingress is ready
??? I0403 11:59:52.558489     979 workloads.go:89] Daemonset/vg-manager in openshift-storage is ready
??? I0403 11:59:52.558505     979 healthcheck.go:28] MicroShift is ready
Script '40_microshift_running_check.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 11:59:52.750474    4059 service.go:29] microshift.service is enabled
??? I0403 11:59:52.750873    4059 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 11:59:52.752273    4059 service.go:38] microshift.service is ready
??? I0403 11:59:52.753263    4059 utils.go:34] Waiting for 1 goroutines
??? I0403 11:59:52.753393    4059 workloads.go:94] Waiting 10m0s for deployment/kserve-controller-manager in redhat-ods-applications
??? I0403 12:00:02.755475    4059 workloads.go:132] Deployment/kserve-controller-manager in redhat-ods-applications is ready
??? I0403 12:00:02.755605    4059 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_ai_model_serving.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:00:02.896949    4128 service.go:29] microshift.service is enabled
??? I0403 12:00:02.897208    4128 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:00:02.899492    4128 service.go:38] microshift.service is ready
??? I0403 12:00:02.900279    4128 utils.go:34] Waiting for 2 goroutines
??? I0403 12:00:02.900363    4128 workloads.go:94] Waiting 10m0s for deployment/istiod-openshift-gateway-api in openshift-gateway-api
??? I0403 12:00:02.900948    4128 workloads.go:94] Waiting 10m0s for deployment/servicemesh-operator3 in openshift-gateway-api
??? I0403 12:00:42.913338    4128 workloads.go:132] Deployment/servicemesh-operator3 in openshift-gateway-api is ready
??? I0403 12:01:12.902297    4128 workloads.go:132] Deployment/istiod-openshift-gateway-api in openshift-gateway-api is ready
??? I0403 12:01:12.902418    4128 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_gateway_api.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:01:13.057998    4772 service.go:29] microshift.service is enabled
??? I0403 12:01:13.058107    4772 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:01:13.059839    4772 service.go:38] microshift.service is ready
??? I0403 12:01:13.060617    4772 utils.go:34] Waiting for 2 goroutines
??? I0403 12:01:13.060644    4772 workloads.go:58] Waiting 10m0s for daemonset/dhcp-daemon in openshift-multus
??? I0403 12:01:13.060686    4772 workloads.go:58] Waiting 10m0s for daemonset/multus in openshift-multus
??? I0403 12:01:13.069341    4772 workloads.go:89] Daemonset/multus in openshift-multus is ready
??? I0403 12:01:13.069450    4772 workloads.go:89] Daemonset/dhcp-daemon in openshift-multus is ready
??? I0403 12:01:13.069503    4772 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_multus.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:01:13.206381    4804 service.go:29] microshift.service is enabled
??? I0403 12:01:13.206583    4804 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:01:13.207979    4804 service.go:38] microshift.service is ready
??? I0403 12:01:13.208717    4804 utils.go:34] Waiting for 2 goroutines
??? I0403 12:01:13.208779    4804 workloads.go:94] Waiting 10m0s for deployment/catalog-operator in openshift-operator-lifecycle-manager
??? I0403 12:01:13.209285    4804 workloads.go:94] Waiting 10m0s for deployment/olm-operator in openshift-operator-lifecycle-manager
??? I0403 12:01:13.215578    4804 workloads.go:132] Deployment/catalog-operator in openshift-operator-lifecycle-manager is ready
??? I0403 12:01:13.215673    4804 workloads.go:132] Deployment/olm-operator in openshift-operator-lifecycle-manager is ready
??? I0403 12:01:13.215684    4804 healthcheck.go:75] Workloads are ready
Script '50_microshift_running_check_olm.sh' SUCCESS
Running Wanted Health Check Scripts...
Finished greenboot Health Checks Runner.

Starting greenboot Health Checks Runner...
Running Required Health Check Scripts...
Script '00_required_scripts_start.sh' SUCCESS
Running Wanted Health Check Scripts...
Script '00_wanted_scripts_start.sh' SUCCESS
Running Required Health Check Scripts...
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 11:54:30.526488     979 service.go:29] microshift.service is enabled
??? I0403 11:54:30.527145     979 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 11:58:52.530299     979 service.go:38] microshift.service is ready
??? I0403 11:58:52.532292     979 net.go:79] host gateway IP address: 192.168.112.125
??? I0403 11:58:52.555077     979 microshift_core_workloads.go:71] vgs reported: {"report":[{"vg":[{"vg_name":"rhel"}]}],"log":[]}
??? I0403 11:58:52.555138     979 microshift_core_workloads.go:93] Detected 1 volume group (rhel) - LVMS is expected
??? I0403 11:58:52.555143     979 microshift_core_workloads.go:126] Configured optional CSI components: []
??? I0403 11:58:52.555147     979 microshift_core_workloads.go:117] At least one CSI Component is enabled
??? I0403 11:58:52.555770     979 utils.go:34] Waiting for 9 goroutines
??? I0403 11:58:52.555791     979 workloads.go:94] Waiting 10m0s for deployment/service-ca in openshift-service-ca
??? I0403 11:58:52.555890     979 workloads.go:58] Waiting 10m0s for daemonset/ovnkube-master in openshift-ovn-kubernetes
??? I0403 11:58:52.555999     979 workloads.go:94] Waiting 10m0s for deployment/router-default in openshift-ingress
??? I0403 11:58:52.556096     979 workloads.go:58] Waiting 10m0s for daemonset/dns-default in openshift-dns
??? I0403 11:58:52.556244     979 workloads.go:58] Waiting 10m0s for daemonset/ovnkube-node in openshift-ovn-kubernetes
??? I0403 11:58:52.556330     979 workloads.go:94] Waiting 10m0s for deployment/lvms-operator in openshift-storage
??? I0403 11:58:52.556382     979 workloads.go:58] Waiting 10m0s for daemonset/vg-manager in openshift-storage
??? I0403 11:58:52.556425     979 workloads.go:94] Waiting 10m0s for deployment/csi-snapshot-controller in kube-system
??? I0403 11:58:52.556474     979 workloads.go:58] Waiting 10m0s for daemonset/node-resolver in openshift-dns
??? I0403 11:58:52.574284     979 workloads.go:89] Daemonset/ovnkube-node in openshift-ovn-kubernetes is ready
??? I0403 11:58:52.574344     979 workloads.go:89] Daemonset/dns-default in openshift-dns is ready
??? I0403 11:59:12.871058     979 workloads.go:89] Daemonset/node-resolver in openshift-dns is ready
??? I0403 11:59:12.871621     979 workloads.go:89] Daemonset/ovnkube-master in openshift-ovn-kubernetes is ready
??? I0403 11:59:12.871748     979 workloads.go:132] Deployment/csi-snapshot-controller in kube-system is ready
??? I0403 11:59:25.175015     979 workloads.go:132] Deployment/service-ca in openshift-service-ca is ready
??? I0403 11:59:42.559264     979 workloads.go:132] Deployment/lvms-operator in openshift-storage is ready
??? I0403 11:59:52.557786     979 workloads.go:132] Deployment/router-default in openshift-ingress is ready
??? I0403 11:59:52.558489     979 workloads.go:89] Daemonset/vg-manager in openshift-storage is ready
??? I0403 11:59:52.558505     979 healthcheck.go:28] MicroShift is ready
Script '40_microshift_running_check.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 11:59:52.750474    4059 service.go:29] microshift.service is enabled
??? I0403 11:59:52.750873    4059 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 11:59:52.752273    4059 service.go:38] microshift.service is ready
??? I0403 11:59:52.753263    4059 utils.go:34] Waiting for 1 goroutines
??? I0403 11:59:52.753393    4059 workloads.go:94] Waiting 10m0s for deployment/kserve-controller-manager in redhat-ods-applications
??? I0403 12:00:02.755475    4059 workloads.go:132] Deployment/kserve-controller-manager in redhat-ods-applications is ready
??? I0403 12:00:02.755605    4059 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_ai_model_serving.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:00:02.896949    4128 service.go:29] microshift.service is enabled
??? I0403 12:00:02.897208    4128 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:00:02.899492    4128 service.go:38] microshift.service is ready
??? I0403 12:00:02.900279    4128 utils.go:34] Waiting for 2 goroutines
??? I0403 12:00:02.900363    4128 workloads.go:94] Waiting 10m0s for deployment/istiod-openshift-gateway-api in openshift-gateway-api
??? I0403 12:00:02.900948    4128 workloads.go:94] Waiting 10m0s for deployment/servicemesh-operator3 in openshift-gateway-api
??? I0403 12:00:42.913338    4128 workloads.go:132] Deployment/servicemesh-operator3 in openshift-gateway-api is ready
??? I0403 12:01:12.902297    4128 workloads.go:132] Deployment/istiod-openshift-gateway-api in openshift-gateway-api is ready
??? I0403 12:01:12.902418    4128 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_gateway_api.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:01:13.057998    4772 service.go:29] microshift.service is enabled
??? I0403 12:01:13.058107    4772 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:01:13.059839    4772 service.go:38] microshift.service is ready
??? I0403 12:01:13.060617    4772 utils.go:34] Waiting for 2 goroutines
??? I0403 12:01:13.060644    4772 workloads.go:58] Waiting 10m0s for daemonset/dhcp-daemon in openshift-multus
??? I0403 12:01:13.060686    4772 workloads.go:58] Waiting 10m0s for daemonset/multus in openshift-multus
??? I0403 12:01:13.069341    4772 workloads.go:89] Daemonset/multus in openshift-multus is ready
??? I0403 12:01:13.069450    4772 workloads.go:89] Daemonset/dhcp-daemon in openshift-multus is ready
??? I0403 12:01:13.069503    4772 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_multus.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:01:13.206381    4804 service.go:29] microshift.service is enabled
??? I0403 12:01:13.206583    4804 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:01:13.207979    4804 service.go:38] microshift.service is ready
??? I0403 12:01:13.208717    4804 utils.go:34] Waiting for 2 goroutines
??? I0403 12:01:13.208779    4804 workloads.go:94] Waiting 10m0s for deployment/catalog-operator in openshift-operator-lifecycle-manager
??? I0403 12:01:13.209285    4804 workloads.go:94] Waiting 10m0s for deployment/olm-operator in openshift-operator-lifecycle-manager
??? I0403 12:01:13.215578    4804 workloads.go:132] Deployment/catalog-operator in openshift-operator-lifecycle-manager is ready
??? I0403 12:01:13.215673    4804 workloads.go:132] Deployment/olm-operator in openshift-operator-lifecycle-manager is ready
??? I0403 12:01:13.215684    4804 healthcheck.go:75] Workloads are ready
Script '50_microshift_running_check_olm.sh' SUCCESS
Running Wanted Health Check Scripts...
Finished greenboot Health Checks Runner.

Copy to Clipboard

Toggle word wrap

Example partial output for a RHEL for Edge system

#...
GRUB boot variables:
boot_success=0
boot_indeterminate=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_WAIT_TIMEOUT_SEC=600
System installation type:
ostree
System installation status:
* rhel 19619bd269094510180c845c44d0944fd9aa15925376f249c4d680a3355e51ae.0
    Version: 9.6
    origin refspec: edge:rhel-9.6-microshift-4.20
#...

#...
GRUB boot variables:
boot_success=0
boot_indeterminate=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_WAIT_TIMEOUT_SEC=600
System installation type:
ostree
System installation status:
* rhel 19619bd269094510180c845c44d0944fd9aa15925376f249c4d680a3355e51ae.0
    Version: 9.6
    origin refspec: edge:rhel-9.6-microshift-4.20
#...

Copy to Clipboard

Toggle word wrap

Example partial output for an RPM system

#...
GRUB boot variables:
boot_success=1
boot_indeterminate=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
System installation type:
RPM
System installation status:
Not an ostree / bootc system
#...

#...
GRUB boot variables:
boot_success=1
boot_indeterminate=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
System installation type:
RPM
System installation status:
Not an ostree / bootc system
#...

Copy to Clipboard

Toggle word wrap

Chapter 9. Automating application management with the GitOps controller
Copy link

GitOps with Argo CD for MicroShift is a lightweight, optional add-on controller derived from the Red Hat OpenShift GitOps Operator. GitOps for MicroShift uses the command-line interface (CLI) of Argo CD to interact with the GitOps controller that acts as the declarative GitOps engine. You can consistently configure and deploy Kubernetes-based infrastructure and applications across node and development lifecycles.

9.1. What you can do with the GitOps agent
Copy link

By using the GitOps with Argo CD agent with MicroShift, you can utilize the following principles:

Implement application lifecycle management.
- Create and manage your node and application configuration files using the core principles of developing and maintaining software in a Git repository.
- You can update the single repository and GitOps automates the deployment of new applications or updates to existing ones.
- For example, if you have 1,000 edge devices, each using MicroShift and a local GitOps agent, you can easily add or update an application on all 1,000 devices with just one change in your central Git repository.
The Git repository contains a declarative description of the infrastructure you need in your specified environment and contains an automated process to make your environment match the described state.
You can also use the Git repository as an audit trail of changes so that you can create processes based on Git flows such as review and approval for merging pull requests that implement configuration changes.

9.2. Creating GitOps applications on MicroShift
Copy link

You can create a custom YAML configuration to deploy and manage applications in your MicroShift service. To install the necessary packages to run GitOps applications, follow the documentation in "Installing the GitOps Argo CD manifests from an RPM package".

Prerequisites

You installed the microshift-gitops packages.
The Argo CD pods are running in the openshift-gitops namespace.

Procedure

Create a YAML file and add your customized configurations for the application:

Example YAML for a spring-petclinic application

kind: AppProject
apiVersion: argoproj.io/v1alpha1
metadata:
  name: default
  namespace: openshift-gitops
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  destinations:
  - namespace: '*'
    server: '*'
  sourceRepos:
  - '*'
---
kind: Application
apiVersion: argoproj.io/v1alpha1
metadata:
  name: spring-petclinic
  namespace: openshift-gitops
spec:
  destination:
    namespace: spring-petclinic
    server: https://kubernetes.default.svc
  project: default
  source:
    directory:
      recurse: true
    path: app
    repoURL: https://github.com/siamaksade/openshift-gitops-getting-started
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true

kind: AppProject
apiVersion: argoproj.io/v1alpha1
metadata:
  name: default
  namespace: openshift-gitops
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  destinations:
  - namespace: '*'
    server: '*'
  sourceRepos:
  - '*'
---
kind: Application
apiVersion: argoproj.io/v1alpha1
metadata:
  name: spring-petclinic
  namespace: openshift-gitops
spec:
  destination:
    namespace: spring-petclinic
    server: https://kubernetes.default.svc
  project: default
  source:
    directory:
      recurse: true
    path: app
    repoURL: https://github.com/siamaksade/openshift-gitops-getting-started
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true

Copy to Clipboard

Toggle word wrap

To deploy the applications defined in the YAML file, run the following command:
```
oc apply -f <my_app.yaml>
```
```
$ oc apply -f <my_app.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <my_app.yaml> with the name of your application YAML.

Verification

To verify your application is deployed and synced, run the following command:

oc get applications -A

$ oc get applications -A

Copy to Clipboard

Toggle word wrap

It might take a few minutes for the application to show the Healthy status.

Example output

NAMESPACE          NAME               SYNC STATUS   HEALTH STATUS
openshift-gitops   spring-petclinic   Synced        Healthy

NAMESPACE          NAME               SYNC STATUS   HEALTH STATUS
openshift-gitops   spring-petclinic   Synced        Healthy

Copy to Clipboard

Toggle word wrap

9.3. Limitations of using the GitOps agent with MicroShift
Copy link

GitOps with Argo CD for MicroShift has the following differences from the Red Hat OpenShift GitOps Operator:

The gitops-operator component is not used with MicroShift.
To maintain the small resource use of MicroShift, the Argo CD web console is not available. You can use the Argo CD CLI.
Because MicroShift is single-node, there is no multi-node support. Each instance of MicroShift is paired with a local GitOps agent.
The oc adm must-gather command is not available in MicroShift.

9.4. Troubleshooting GitOps
Copy link

If you have problems with your GitOps controller, you can use the OpenShift CLI (oc) tool.

9.4.1. Debugging GitOps with oc adm inspect
Copy link

You can debug GitOps by using the OpenShift CLI (oc).

Prerequisites

The oc command-line tool is installed.

Procedure

Run the oc adm inspect command when in the GitOps namespace:

oc adm inspect ns/openshift-gitops

$ oc adm inspect ns/openshift-gitops

Copy to Clipboard

Toggle word wrap

Example output

Gathering data for ns/openshift-gitops...
W0501 20:34:35.978508 57625 util.go:118] the server doesn't have a resource type egressfirewalls, skipping the inspection
W0501 20:34:35.980881 57625 util.go:118] the server doesn't have a resource type egressqoses, skipping the inspection
W0501 20:34:36.040664 57625 util.go:118] the server doesn't have a resource type servicemonitors, skipping the inspection
Wrote inspect data to inspect.local.2673575938140296280.

Gathering data for ns/openshift-gitops...
W0501 20:34:35.978508 57625 util.go:118] the server doesn't have a resource type egressfirewalls, skipping the inspection
W0501 20:34:35.980881 57625 util.go:118] the server doesn't have a resource type egressqoses, skipping the inspection
W0501 20:34:36.040664 57625 util.go:118] the server doesn't have a resource type servicemonitors, skipping the inspection
Wrote inspect data to inspect.local.2673575938140296280.

Copy to Clipboard

Toggle word wrap

Next steps

If oc adm inspect did not provide the information you need, you can run an sos report.

Chapter 10. Pod security authentication and authorization with SCC
Copy link

Pod security admission is an implementation of the Kubernetes pod security standards. Use security content constraints (SCC) for pod security admission to restrict pod behavior.

10.1. Security context constraint synchronization with pod security standards
Copy link

MicroShift includes Kubernetes pod security admission.

In addition to the global pod security admission control configuration, a controller exists that applies pod security admission control warn and audit labels to namespaces according to the security context constraint (SCC) permissions of the service accounts that are in a given namespace.

Important

Namespaces that are defined as part of the node payload have pod security admission synchronization disabled permanently. You can enable pod security admission synchronization on other namespaces as necessary. If an Operator is installed in a user-created openshift-* namespace, synchronization is turned on by default after a cluster service version (CSV) is created in the namespace.

The controller examines ServiceAccount object permissions to use security context constraints in each namespace. Security context constraints (SCCs) are mapped to pod security profiles based on their field values; the controller uses these translated profiles. Pod security admission warn and audit labels are set to the most privileged pod security profile found in the namespace to prevent warnings and audit logging as pods are created.

Namespace labeling is based on consideration of namespace-local service account privileges.

Applying pods directly might use the SCC privileges of the user who runs the pod. However, user privileges are not considered during automatic labeling.

10.1.1. Viewing security context constraints in a namespace
Copy link

You can view the security context constraints (SCC) permissions in a given namespace.

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

To view the security context constraints in your namespace, run the following command:
```
oc get --show-labels namespace <namespace>
```
```
oc get --show-labels namespace <namespace>
```
Copy to Clipboard Toggle word wrap

10.2. Controlling pod security admission synchronization
Copy link

You can enable automatic pod security admission synchronization for most namespaces.

System defaults are not enforced when the security.openshift.io/scc.podSecurityLabelSync field is empty or set to false. You must set the label to true for synchronization to occur.

Important

Namespaces that are defined as part of the node payload have pod security admission synchronization disabled permanently. These namespaces include:

default
kube-node-lease
kube-system
kube-public
openshift
All system-created namespaces that are prefixed with openshift-, except for openshift-operators By default, all namespaces that have an openshift- prefix are not synchronized. You can enable synchronization for any user-created openshift-* namespaces. You cannot enable synchronization for any system-created openshift-* namespaces, except for openshift-operators.

If an Operator is installed in a user-created openshift-* namespace, synchronization is turned on by default after a node service version (CSV) is created in the namespace. The synchronized label inherits the permissions of the service accounts in the namespace.

Procedure

To enable pod security admission label synchronization in a namespace, set the value of the security.openshift.io/scc.podSecurityLabelSync label to true.
Run the following command:
```
oc label namespace <namespace> security.openshift.io/scc.podSecurityLabelSync=true
```
```
$ oc label namespace <namespace> security.openshift.io/scc.podSecurityLabelSync=true
```
Copy to Clipboard Toggle word wrap

Note

You can use the --overwrite flag to reverse the effects of the pod security label synchronization in a namespace.

Chapter 11. Operators
Copy link

11.1. Using Operators with MicroShift
Copy link

You can use Operators with MicroShift to create applications that monitor the running services in your node. Operators can manage applications and their resources, such as deploying a database or message bus. As customized software running inside your node, Operators can be used to implement and automate common operations.

Operators offer a more localized configuration experience and integrate with Kubernetes APIs and CLI tools such as kubectl and oc. Operators are designed specifically for your applications. Operators enable you to configure components instead of modifying a global configuration file.

MicroShift applications are generally expected to be deployed in static environments. However, Operators are available if helpful in your use case. To determine the compatibility of an Operator with MicroShift, check the Operator documentation.

11.1.1. How to use Operators with a MicroShift node
Copy link

There are two ways to use Operators for your MicroShift node:

11.1.1.1. Manifests for Operators
Copy link

Operators can be installed and managed directly by using manifests. You can use the kustomize configuration management tool with MicroShift to deploy an application. Use the same steps to install Operators with manifests.

See Using Kustomize manifests to deploy applications and Using manifests example for details.

11.1.1.2. Operator Lifecycle Manager for Operators
Copy link

You can also install add-on Operators to a MicroShift node by using Operator Lifecycle Manager (OLM). OLM can be used to manage both custom Operators and Operators that are widely available. Building catalogs is required to use OLM with MicroShift.

For details, see Using Operator Lifecycle Manager with MicroShift.

11.2. Using Operator Lifecycle Manager with MicroShift
Copy link

Operator Lifecycle Manager (OLM) is used in MicroShift for installing and running optional add-on Operators. See the following link for more information:

Operator Lifecycle Manager

11.2.1. Considerations for using OLM with MicroShift
Copy link

Cluster Operators as applied in OpenShift Container Platform are not used in MicroShift.
You must create your own catalogs for the add-on Operators you want to use with your applications. Catalogs are not provided by default.
- Each catalog must have an accessible CatalogSource added to a node, so that the OLM catalog Operator can use the catalog for content.
You must use the CLI to conduct OLM activities with MicroShift. The console and OperatorHub GUIs are not available.
- Use the Operator Package Manager opm CLI with a network-connected node, or for building catalogs for custom Operators that use an internal registry.
- To mirror your catalogs and Operators for disconnected or offline nodes, install the oc-mirror OpenShift CLI plugin.

Important

Before using an Operator, verify with the provider that the Operator is supported on Red Hat build of MicroShift.

11.2.2. Determining your OLM installation type
Copy link

You can install the OLM package manager for use with MicroShift 4.15 or newer versions. There are different ways to install OLM for a MicroShift node, depending on your use case.

You can install the microshift-olm RPM at the same time you install the MicroShift RPM on Red Hat Enterprise Linux (RHEL).
You can install the microshift-olm on an existing MicroShift 4.20. Restart the MicroShift service after installing OLM for the changes to apply. See Installing the Operator Lifecycle Manager (OLM) from an RPM package.
You can embed OLM in a Red Hat Enterprise Linux for Edge (RHEL for Edge) image. See Adding the Operator Lifecycle Manager (OLM) service to a blueprint.

11.2.3. Namespace use in MicroShift
Copy link

The microshift-olm RPM creates the three default namespaces: one for running OLM, and two for catalog and Operator installation. You can create additional namespaces as needed for your use case.

11.2.3.1. Default namespaces
Copy link

The following table lists the default namespaces and a brief description of how each namespace works.

Expand

Table 11.1. Default namespaces created by OLM for MicroShift
Default Namespace	Details
`openshift-operator-lifecycle-manager`	The OLM package manager runs in this namespace.
`openshift-marketplace`	The global namespace. Empty by default. To make the catalog source to be available globally to users in all namespaces, set the `openshift-marketplace` namespace in the catalog-source YAML.
`openshift-operators`	The default namespace where Operators run in MicroShift. Operators that reference catalogs in the `openshift-operators` namespace must have the AllNamespaces watch scope.

11.2.3.2. Custom namespaces
Copy link

If you want to use a catalog and Operator together in a single namespace, then you must create a custom namespace. After you create the namespace, you must create the catalog in that namespace. All Operators running in the custom namespace must have the same single-namespace watch scope.

11.2.4. About building Operator catalogs
Copy link

To use Operator Lifecycle Manager (OLM) with MicroShift, you must build custom Operator catalogs that you can then manage with OLM. The standard catalogs that are included with OpenShift Container Platform are not included with MicroShift.

11.2.4.1. File-based Operator catalogs
Copy link

You can create catalogs for your custom Operators or filter catalogs of widely available Operators. You can combine both methods to create the catalogs needed for your specific use case. To run MicroShift with your own Operators and OLM, make a catalog by using the file-based catalog structure.

For details, see Managing custom catalogs and Example catalog.
See also opm CLI reference.

Important

When adding a catalog source to a cluster, set the securityContextConfig value to restricted in the catalogSource.yaml file. Ensure that your catalog can run with restricted permissions.

11.2.5. How to deploy Operators using OLM
Copy link

After you create and deploy your custom catalog, you must create a Subscription custom resource (CR) that can access the catalog and install the Operators you choose. Where Operators run depends on the namespace in which you create the Subscription CR.

Important

Operators in OLM have a watch scope. For example, some Operators only support watching their own namespace, while others support watching every namespace in the node. All Operators installed in a given namespace must have the same watch scope.

11.2.5.1. Connectivity and OLM Operator deployment
Copy link

Operators can be deployed anywhere a catalog is running.

For a node that is connected to the internet, mirroring images is not required. Images can be pulled over the network.
For restricted networks in which MicroShift has access to an internal network only, images must be mirrored to an internal registry.
For use cases in which a MicroShift node is completely offline, all images must be embedded into an osbuild blueprint.

11.2.5.2. Adding OLM-based Operators to a networked node using the global namespace
Copy link

To deploy different operators to different namespaces, use this procedure. For a MicroShift node that has network connectivity, Operator Lifecycle Manager (OLM) can access sources hosted on remote registries. The following procedure lists the basic steps of using configuration files to install an Operator that uses the global namespace.

Note

To use an Operator installed in a different namespace, or in more than one namespace, make sure that the catalog source and the Subscription CR that references the Operator are running in the openshift-marketplace namespace.

Prerequisites

The OpenShift CLI (oc) is installed.
Operator Lifecycle Manager (OLM) is installed.
You have created a custom catalog in the global namespace.

Procedure

Confirm that OLM is running by using the following command:

oc -n openshift-operator-lifecycle-manager get pod -l app=olm-operator

$ oc -n openshift-operator-lifecycle-manager get pod -l app=olm-operator

Copy to Clipboard

Toggle word wrap

Example output

NAME                            READY   STATUS    RESTARTS   AGE
olm-operator-85b5c6786-n6kbc    1/1     Running   0          2m24s

NAME                            READY   STATUS    RESTARTS   AGE
olm-operator-85b5c6786-n6kbc    1/1     Running   0          2m24s

Copy to Clipboard

Toggle word wrap

Confirm that the OLM catalog Operator is running by using the following command:

oc -n openshift-operator-lifecycle-manager get pod -l app=catalog-operator

$ oc -n openshift-operator-lifecycle-manager get pod -l app=catalog-operator

Copy to Clipboard

Toggle word wrap

Example output

NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-5fc7f857b6-tj8cf   1/1     Running   0          2m33s

NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-5fc7f857b6-tj8cf   1/1     Running   0          2m33s

Copy to Clipboard

Toggle word wrap

Note

The following steps assume you are using the global namespace, openshift-marketplace. The catalog must run in the same namespace as the Operator. The Operator must support the AllNamespaces mode.

Create the CatalogSource object by using the following example YAML:

Example catalog source YAML

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: operatorhubio-catalog
  namespace: openshift-marketplace 
spec:
  sourceType: grpc
  image: quay.io/operatorhubio/catalog:latest
  displayName: Community Operators 
  publisher: OperatorHub.io
  grpcPodConfig:
    securityContextConfig: restricted 
  updateStrategy:
    registryPoll:
      interval: 60m

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: operatorhubio-catalog
  namespace: openshift-marketplace


spec:
  sourceType: grpc
  image: quay.io/operatorhubio/catalog:latest
  displayName: Community Operators


  publisher: OperatorHub.io
  grpcPodConfig:
    securityContextConfig: restricted


  updateStrategy:
    registryPoll:
      interval: 60m

Copy to Clipboard

Toggle word wrap

1: The global namespace. Setting the metadata.namespace to openshift-marketplace enables the catalog to run in all namespaces. Subscriptions in any namespace can reference catalogs created in the openshift-marketplace namespace.
2: Community Operators are not installed by default with OLM for MicroShift. Listed here for example only.
3: The value of securityContextConfig must be set to restricted for MicroShift.

Apply the CatalogSource configuration by running the following command:
```
oc apply -f <catalog_source.yaml>
```
```
$ oc apply -f <catalog_source.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <catalog-source.yaml> with your catalog source configuration file name. In this example, catalogsource.yaml is used.
Example output
```
catalogsource.operators.coreos.com/operatorhubio-catalog created
```
```
catalogsource.operators.coreos.com/operatorhubio-catalog created
```
Copy to Clipboard Toggle word wrap

To verify that the catalog source is applied, check for the READY state by using the following command:

oc describe catalogsources.operators.coreos.com -n openshift-marketplace operatorhubio-catalog

$ oc describe catalogsources.operators.coreos.com -n openshift-marketplace operatorhubio-catalog

Copy to Clipboard

Toggle word wrap

Example output

Name:         operatorhubio-catalog
Namespace:    openshift-marketplace
Labels:       <none>
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         CatalogSource
Metadata:
  Creation Timestamp:  2024-01-31T09:55:31Z
  Generation:          1
  Resource Version:    1212
  UID:                 4edc1a96-83cd-4de9-ac8c-c269ca895f3e
Spec:
  Display Name:  Community Operators
  Grpc Pod Config:
    Security Context Config:  restricted
  Image:                      quay.io/operatorhubio/catalog:latest
  Publisher:                  OperatorHub.io
  Source Type:                grpc
  Update Strategy:
    Registry Poll:
      Interval:  60m
Status:
  Connection State:
    Address:              operatorhubio-catalog.openshift-marketplace.svc:50051
    Last Connect:         2024-01-31T09:55:57Z
    Last Observed State:  READY 
  Registry Service:
    Created At:         2024-01-31T09:55:31Z
    Port:               50051
    Protocol:           grpc
    Service Name:       operatorhubio-catalog
    Service Namespace:  openshift-marketplace
Events:                 <none>

Name:         operatorhubio-catalog
Namespace:    openshift-marketplace
Labels:       <none>
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         CatalogSource
Metadata:
  Creation Timestamp:  2024-01-31T09:55:31Z
  Generation:          1
  Resource Version:    1212
  UID:                 4edc1a96-83cd-4de9-ac8c-c269ca895f3e
Spec:
  Display Name:  Community Operators
  Grpc Pod Config:
    Security Context Config:  restricted
  Image:                      quay.io/operatorhubio/catalog:latest
  Publisher:                  OperatorHub.io
  Source Type:                grpc
  Update Strategy:
    Registry Poll:
      Interval:  60m
Status:
  Connection State:
    Address:              operatorhubio-catalog.openshift-marketplace.svc:50051
    Last Connect:         2024-01-31T09:55:57Z
    Last Observed State:  READY


  Registry Service:
    Created At:         2024-01-31T09:55:31Z
    Port:               50051
    Protocol:           grpc
    Service Name:       operatorhubio-catalog
    Service Namespace:  openshift-marketplace
Events:                 <none>

Copy to Clipboard

Toggle word wrap

1: The status is reported as READY.

Confirm that the catalog source is running by using the following command:

oc get pods -n openshift-marketplace -l olm.catalogSource=operatorhubio-catalog

$ oc get pods -n openshift-marketplace -l olm.catalogSource=operatorhubio-catalog

Copy to Clipboard

Toggle word wrap

Example output

NAME                          READY   STATUS    RESTARTS   AGE
operatorhubio-catalog-x24nh   1/1     Running   0          59s

NAME                          READY   STATUS    RESTARTS   AGE
operatorhubio-catalog-x24nh   1/1     Running   0          59s

Copy to Clipboard

Toggle word wrap

Create a Subscription CR configuration file by using the following example YAML:

Example Subscription custom resource YAML

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-cert-manager
  namespace: openshift-operators
spec:
  channel: stable
  name: cert-manager
  source: operatorhubio-catalog
  sourceNamespace: openshift-marketplace

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-cert-manager
  namespace: openshift-operators
spec:
  channel: stable
  name: cert-manager
  source: operatorhubio-catalog
  sourceNamespace: openshift-marketplace

Copy to Clipboard

Toggle word wrap

1: The global namespace. Setting the sourceNamespace value to openshift-marketplace enables Operators to run in multiple namespaces if the catalog also runs in the openshift-marketplace namespace.

Apply the Subscription CR configuration by running the following command:
```
oc apply -f <subscription_cr.yaml>
```
```
$ oc apply -f <subscription_cr.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <subscription_cr.yaml> with your Subscription CR filename.
Example output
```
subscription.operators.coreos.com/my-cert-manager created
```
```
subscription.operators.coreos.com/my-cert-manager created
```
Copy to Clipboard Toggle word wrap
You can create a configuration file for the specific Operand you want to use and apply it now.

Verification

Verify that your Operator is running by using the following command:

oc get pods -n openshift-operators

$ oc get pods -n openshift-operators

Copy to Clipboard

Toggle word wrap

1: The namespace from the Subscription CR is used.

Note

Allow a minute or two for the Operator start.

Example output

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7df8994ddb-4vrkr              1/1     Running   0          19s
cert-manager-cainjector-5746db8fd7-69442   1/1     Running   0          18s
cert-manager-webhook-f858bf58b-748nt       1/1     Running   0          18s

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7df8994ddb-4vrkr              1/1     Running   0          19s
cert-manager-cainjector-5746db8fd7-69442   1/1     Running   0          18s
cert-manager-webhook-f858bf58b-748nt       1/1     Running   0          18s

Copy to Clipboard

Toggle word wrap

11.2.5.3. Adding OLM-based Operators to a networked node in a specific namespace
Copy link

Use this procedure if you want to specify a namespace for an Operator, for example, olm-microshift. In this example, the catalog is scoped and available in the global openshift-marketplace namespace. The Operator uses content from the global namespace, but runs only in the olm-microshift namespace. For a MicroShift node that has network connectivity, Operator Lifecycle Manager (OLM) can access sources hosted on remote registries.

Important

All of the Operators installed in a specific namespace must have the same watch scope. In this case, the watch scope is OwnNamespace.

Prerequisites

The OpenShift CLI (oc) is installed.
Operator Lifecycle Manager (OLM) is installed.
You have created a custom catalog that is running in the global namespace.

Procedure

Confirm that OLM is running by using the following command:

oc -n openshift-operator-lifecycle-manager get pod -l app=olm-operator

$ oc -n openshift-operator-lifecycle-manager get pod -l app=olm-operator

Copy to Clipboard

Toggle word wrap

Example output

NAME                           READY   STATUS    RESTARTS   AGE
olm-operator-85b5c6786-n6kbc   1/1     Running   0          16m

NAME                           READY   STATUS    RESTARTS   AGE
olm-operator-85b5c6786-n6kbc   1/1     Running   0          16m

Copy to Clipboard

Toggle word wrap

Confirm that the OLM catalog Operator is running by using the following command:

oc -n openshift-operator-lifecycle-manager get pod -l app=catalog-operator

$ oc -n openshift-operator-lifecycle-manager get pod -l app=catalog-operator

Copy to Clipboard

Toggle word wrap

Example output

NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-5fc7f857b6-tj8cf   1/1     Running   0          16m

NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-5fc7f857b6-tj8cf   1/1     Running   0          16m

Copy to Clipboard

Toggle word wrap

Create a namespace by using the following example YAML:

Example namespace YAML

apiVersion: v1
kind: Namespace
metadata:
  name: olm-microshift

apiVersion: v1
kind: Namespace
metadata:
  name: olm-microshift

Copy to Clipboard

Toggle word wrap

Apply the namespace configuration using the following command:
```
oc apply -f <ns.yaml>
```
```
$ oc apply -f <ns.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <ns.yaml> with the name of your namespace configuration file. In this example, olm-microshift is used.
Example output
```
namespace/olm-microshift created
```
```
namespace/olm-microshift created
```
Copy to Clipboard Toggle word wrap

Create the Operator group YAML by using the following example YAML:

Example Operator group YAML

kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og
  namespace: olm-microshift
spec: 
  targetNamespaces:
  - olm-microshift

kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og
  namespace: olm-microshift
spec:


  targetNamespaces:
  - olm-microshift

Copy to Clipboard

Toggle word wrap

1: For Operators using the global namespace, omit the spec.targetNamespaces field and values.

Apply the Operator group configuration by running the following command:
```
oc apply -f <og.yaml>
```
```
$ oc apply -f <og.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <og.yaml> with the name of your operator group configuration file.
Example output
```
operatorgroup.operators.coreos.com/og created
```
```
operatorgroup.operators.coreos.com/og created
```
Copy to Clipboard Toggle word wrap

Create the CatalogSource object by using the following example YAML:

Example catalog source YAML

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: operatorhubio-catalog
  namespace: openshift-marketplace 
spec:
  sourceType: grpc
  image: quay.io/operatorhubio/catalog:latest
  displayName: Community Operators 
  publisher: OperatorHub.io
  grpcPodConfig:
    securityContextConfig: restricted 
  updateStrategy:
    registryPoll:
      interval: 60m

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: operatorhubio-catalog
  namespace: openshift-marketplace


spec:
  sourceType: grpc
  image: quay.io/operatorhubio/catalog:latest
  displayName: Community Operators


  publisher: OperatorHub.io
  grpcPodConfig:
    securityContextConfig: restricted


  updateStrategy:
    registryPoll:
      interval: 60m

Copy to Clipboard

Toggle word wrap

1: The global namespace. Setting the metadata.namespace to openshift-marketplace enables the catalog to run in all namespaces. Subscriptions CRs in any namespace can reference catalogs created in the openshift-marketplace namespace.
2: Community Operators are not installed by default with OLM for MicroShift. Listed here for example only.
3: The value of securityContextConfig must be set to restricted for MicroShift.

Apply the CatalogSource configuration by running the following command:
```
oc apply -f <catalog_source.yaml>
```
```
$ oc apply -f <catalog_source.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <catalog_source.yaml> with your catalog source configuration file name.

To verify that the catalog source is applied, check for the READY state by using the following command:

oc describe catalogsources.operators.coreos.com -n openshift-marketplace operatorhubio-catalog

$ oc describe catalogsources.operators.coreos.com -n openshift-marketplace operatorhubio-catalog

Copy to Clipboard

Toggle word wrap

Example output

Name:         operatorhubio-catalog
Namespace:    openshift-marketplace
Labels:       <none>
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         CatalogSource
Metadata:
  Creation Timestamp:  2024-01-31T10:09:46Z
  Generation:          1
  Resource Version:    2811
  UID:                 60ce4a36-86d3-4921-b9fc-84d67c28df48
Spec:
  Display Name:  Community Operators
  Grpc Pod Config:
    Security Context Config:  restricted
  Image:                      quay.io/operatorhubio/catalog:latest
  Publisher:                  OperatorHub.io
  Source Type:                grpc
  Update Strategy:
    Registry Poll:
      Interval:  60m
Status:
  Connection State:
    Address:              operatorhubio-catalog.openshift-marketplace.svc:50051
    Last Connect:         2024-01-31T10:10:04Z
    Last Observed State:  READY 
  Registry Service:
    Created At:         2024-01-31T10:09:46Z
    Port:               50051
    Protocol:           grpc
    Service Name:       operatorhubio-catalog
    Service Namespace:  openshift-marketplace
Events:                 <none>

Name:         operatorhubio-catalog
Namespace:    openshift-marketplace
Labels:       <none>
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         CatalogSource
Metadata:
  Creation Timestamp:  2024-01-31T10:09:46Z
  Generation:          1
  Resource Version:    2811
  UID:                 60ce4a36-86d3-4921-b9fc-84d67c28df48
Spec:
  Display Name:  Community Operators
  Grpc Pod Config:
    Security Context Config:  restricted
  Image:                      quay.io/operatorhubio/catalog:latest
  Publisher:                  OperatorHub.io
  Source Type:                grpc
  Update Strategy:
    Registry Poll:
      Interval:  60m
Status:
  Connection State:
    Address:              operatorhubio-catalog.openshift-marketplace.svc:50051
    Last Connect:         2024-01-31T10:10:04Z
    Last Observed State:  READY


  Registry Service:
    Created At:         2024-01-31T10:09:46Z
    Port:               50051
    Protocol:           grpc
    Service Name:       operatorhubio-catalog
    Service Namespace:  openshift-marketplace
Events:                 <none>

Copy to Clipboard

Toggle word wrap

1: The status is reported as READY.

Confirm that the catalog source is running by using the following command:

oc get pods -n openshift-marketplace -l olm.catalogSource=operatorhubio-catalog

$ oc get pods -n openshift-marketplace -l olm.catalogSource=operatorhubio-catalog

Copy to Clipboard

Toggle word wrap

Example output

NAME                          READY   STATUS    RESTARTS   AGE
operatorhubio-catalog-j7sc8   1/1     Running   0          43s

NAME                          READY   STATUS    RESTARTS   AGE
operatorhubio-catalog-j7sc8   1/1     Running   0          43s

Copy to Clipboard

Toggle word wrap

Create a Subscription CR configuration file by using the following example YAML:

Example Subscription custom resource YAML

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-gitlab-operator-kubernetes
  namespace: olm-microshift 
spec:
  channel: stable
  name: gitlab-operator-kubernetes
  source: operatorhubio-catalog
  sourceNamespace: openshift-marketplace

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-gitlab-operator-kubernetes
  namespace: olm-microshift


spec:
  channel: stable
  name: gitlab-operator-kubernetes
  source: operatorhubio-catalog
  sourceNamespace: openshift-marketplace

Copy to Clipboard

Toggle word wrap

1: The specific namespace. Operators reference the global namespace for content, but run in the olm-microshift namespace.
2: The global namespace. Subscriptions CRs in any namespace can reference catalogs created in the openshift-marketplace namespace.

Apply the Subscription CR configuration by running the following command:
```
oc apply -f <subscription_cr.yaml>
```
```
$ oc apply -f <subscription_cr.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <subscription_cr.yaml> with the name of the Subscription CR configuration file.
Example output
```
subscription.operators.coreos.com/my-gitlab-operator-kubernetes
```
```
subscription.operators.coreos.com/my-gitlab-operator-kubernetes
```
Copy to Clipboard Toggle word wrap
You can create a configuration file for the specific Operand you want to use and apply it now.

Verification

Verify that your Operator is running by using the following command:

oc get pods -n olm-microshift

$ oc get pods -n olm-microshift

Copy to Clipboard

Toggle word wrap

1: The namespace from the Subscription CR is used.

Note

Allow a minute or two for the Operator start.

Example output

NAME                                         READY   STATUS    RESTARTS   AGE
gitlab-controller-manager-69bb6df7d6-g7ntx   2/2     Running   0          3m24s

NAME                                         READY   STATUS    RESTARTS   AGE
gitlab-controller-manager-69bb6df7d6-g7ntx   2/2     Running   0          3m24s

Copy to Clipboard

Toggle word wrap

11.3. Creating custom Operator catalogs using the oc-mirror plugin
Copy link

You can create custom catalogs with widely available Operators and mirror them by using the oc-mirror OpenShift CLI (oc) plugin.

11.3.1. Using Red Hat-provided Operator catalogs and mirror registries
Copy link

You can filter catalogs and delete images to get specific Operators and mirror them by using the oc-mirror OpenShift CLI (oc) plugin. You can also use Operators in disconnected settings or embedded in a Red Hat Enterprise Linux (RHEL) image.

To understand more about how to configure your systems for mirroring, follow the links in the following "Additional resources" section.
If you are ready to deploy Operators from Red Hat-provided Operator catalogs, mirror them, or to embed them in a RHEL image, start with the following section, "Inspecting catalog contents by using the oc-mirror plugin."

11.3.2. About the oc-mirror plugin for creating a mirror registry
Copy link

You can use the oc-mirror OpenShift CLI (oc) plugin with MicroShift to filter and delete images from Operator catalogs. You can then mirror the filtered catalog contents to a mirror registry or use the container images in disconnected or offline deployments.

The procedure to mirror content from Red Hat-hosted registries connected to the internet to a disconnected image registry is the same, independent of the registry you select. After you mirror the contents of your catalog, configure each node to retrieve this content from your mirror registry.

11.3.2.1. Connectivity considerations when populating a mirror registry
Copy link

When you populate your registry, you can use one of following connectivity scenarios:

Connected mirroring: If you have a host that can access both the internet and your mirror registry, but not your node, you can directly mirror the content from that machine.
Disconnected mirroring: If you do not have a host that can access both the internet and your mirror registry, you must mirror the images to a file system and then bring that host or removable media into your disconnected environment.
Important
A container registry must be reachable by every machine in the node that you provision. Installing, updating, and other operations, such as relocating workloads, fail if the registry is unreachable.

To avoid problems caused by an unreachable registry, use the following standard practices:

Run mirror registries in a highly available way.
Ensure that the mirror registry at least matches the production availability of your node.

11.3.3. Inspecting catalog contents by using the oc-mirror plugin
Copy link

Use the following example procedure to select a catalog and list OpenShift Container Platform Operators to add to your oc-mirror plugin image set configuration file. You must use oc mirror v1 to selecting a catalog and listing Operators.

Note

If you use your own catalogs and Operators, you can push the images directly to your internal registry.

Prerequisites

You uninstalled OpenShift CLI (oc).
You installed the Operator Lifecycle Manager (OLM).
You installed the oc-mirror plugin.

Procedure

Get a list of available Red Hat-provided Operator catalogs to filter by running the following command:
```
oc mirror list operators --version 4.20 --catalogs
```
```
$ oc mirror list operators --version 4.20 --catalogs
```
Copy to Clipboard Toggle word wrap
Get a list of Operators in the Red Hat Operators catalog by running the following command:
```
oc mirror list operators <--catalog=<catalog_source>>
```
```
$ oc mirror list operators <--catalog=<catalog_source>> 
```
1
Copy to Clipboard Toggle word wrap
1
Specifies your catalog source, such as registry.redhat.io/redhat/redhat-operator-index:v4.20 or quay.io/operatorhubio/catalog:latest.
Select an Operator. This example uses the amq-broker-rhel9 Operator.

Optional: To inspect the channels and versions of the Operator you want to filter, enter the following commands:

Get a list of channels by running the following command:

oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.20 --package=amq-broker-rhel9

$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.20 --package=amq-broker-rhel9

Copy to Clipboard

Toggle word wrap

Get a list of versions within a channel by running the following command:

oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.20 --package=amq-broker-rhel9 --channel=7.13.x

$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.20 --package=amq-broker-rhel9 --channel=7.13.x

Copy to Clipboard

Toggle word wrap

Next steps

Create and edit an image set configuration file using the information gathered in this procedure.
Mirror the images from the transformed image set configuration file to a mirror registry or disk.

11.3.4. Creating an image set configuration file
Copy link

You must create an ImageSetConfiguration YAML file. This image set configuration file specifies both the Operators to mirror and the configuration settings for the oc-mirror plugin. Edit the contents of the image set configuration file so that the entries are compatible with both MicroShift and the Operator you plan to use.

Note

oc mirror v2 uses a cache system instead of metadata. The cache system prevents the need to start the entire mirroring process over when a single step fails. Instead, you can troubleshoot the failed step and the process does not re-mirror images that existed before the failure.

Prerequisites

You created a container image registry credentials file. For more information, see the following reference:
- Configuring credentials that allow images to be mirrored

Procedure

Create and edit the ImageSetConfiguration YAML for MicroShift by using the following example as a guide:
Example edited MicroShift image set configuration file
```
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.20 
    packages:
    - name: amq-broker-rhel9 
      channels:
      - name: 7.13.x 
  additionalImages: 
   - name: quay.io/rh_ee_aguidi/multi-platform-container:latest
   - name: quay.io/rh_ee_aguidi/empty-image:latest
```
```
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.20 
```
1
```
    packages:
    - name: amq-broker-rhel9 
```
2
```
      channels:
      - name: 7.13.x 
```
3
```
  additionalImages: 
```
4
```
   - name: quay.io/rh_ee_aguidi/multi-platform-container:latest
   - name: quay.io/rh_ee_aguidi/empty-image:latest
```
Copy to Clipboard Toggle word wrap
1
Set the Operator catalog to retrieve images from.
2
Specify the Operator packages to include in the image set. Remove this field to retrieve all packages in the catalog.
3
Specify only certain channels of the Operator packages to include in the image set. You must always include the default channel for the Operator package even if you do not use the bundles in that channel. You can find the default channel by running the following command: oc mirror list operators --catalog=<catalog_name> --package=<package_name>.
4
Specify any additional images to include in the image set. If you do not need to specify additional images, delete this field.
Important
The platform field, related fields, and Helm are not supported by MicroShift.
Save the updated file as ImageSetConfiguration.yaml.

Next steps

Use the oc-mirror plugin to mirror an image set directly to a target mirror registry.
Configure CRI-O.
Apply the catalog sources to your node.

11.3.4.1. ImageSet configuration parameters for oc-mirror plugin v2
Copy link

The oc-mirror plugin v2 requires an image set configuration file that defines what images to mirror. The following table lists the available parameters for the ImageSetConfiguration resource.

Note

Using the minVersion and maxVersion properties to filter for a specific Operator version range can result in a multiple channel heads error. The error message states that there are multiple channel heads. This is because when the filter is applied, the update graph of the Operator is truncated.

OLM requires that every Operator channel contains versions that form an update graph with exactly one end point, that is, the latest version of the Operator. When the filter range is applied, that graph can turn into two or more separate graphs or a graph that has more than one end point.

To avoid this error, do not filter out the latest version of an Operator. If you still run into the error, depending on the Operator, either the maxVersion property must be increased or the minVersion property must be decreased. Because every Operator graph can be different, you might need to adjust these values until the error resolves.

Expand

Table 11.2. ImageSetConfiguration parameters
Parameter	Description	Values
`apiVersion`	The API version of the `ImageSetConfiguration` content.	String Example: `mirror.openshift.io/v2alpha1`
`mirror`	The configuration of the image set.	Object
`mirror.additionalImages`	The additional images configuration of the image set.	Array of objects Example: `additionalImages: - name: registry.redhat.io/ubi8/ubi:latest` Copy to Clipboard Toggle word wrap
`mirror.additionalImages.name`	The tag or digest of the image to mirror.	String Example: `registry.redhat.io/ubi8/ubi:latest`
`mirror.blockedImages`	List of images with a tag or digest (SHA) to block from mirroring.	Array of strings Example: `docker.io/library/alpine`
`mirror.operators`	The Operators configuration of the image set.	Array of objects Example: `operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:4.20 packages: - name: elasticsearch-operator minVersion: '2.4.0'` Copy to Clipboard Toggle word wrap
`mirror.operators.catalog`	The Operator catalog to include in the image set.	String Example: `registry.redhat.io/redhat/redhat-operator-index:v4.15`
`mirror.operators.full`	When `true`, downloads the full catalog, Operator package, or Operator channel.	Boolean The default value is `false`.
`mirror.operators.packages`	The Operator packages configuration.	Array of objects Example: `operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:4.20 packages: - name: elasticsearch-operator minVersion: '5.2.3-31'` Copy to Clipboard Toggle word wrap
`mirror.operators.packages.name`	The Operator package name to include in the image set.	String Example: `elasticsearch-operator`
`mirror.operators.packages.channels`	Operator package channel configuration	Object
`mirror.operators.packages.channels.name`	The Operator channel name, unique within a package, to include in the image set.	String Eample: `fast` or `stable-v4.15`
`mirror.operators.packages.channels.maxVersion`	The highest version of the Operator mirror across all channels in which it exists.	String Example: `5.2.3-31`
`mirror.operators.packages.channels.minVersion`	The lowest version of the Operator to mirror across all channels in which it exists	String Example: `5.2.3-31`
`mirror.operators.packages.maxVersion`	The highest version of the Operator to mirror across all channels in which it exists.	String Example: `5.2.3-31`
`mirror.operators.packages.minVersion`	The lowest version of the Operator to mirror across all channels in which it exists.	String Example: `5.2.3-31`
`mirror.operators.targetCatalog`	An alternative name and optional namespace hierarchy to mirror the referenced catalog as	String Example: `my-namespace/my-operator-catalog`
`mirror.operators.targetCatalogSourceTemplate`	Path on disk for a template to use to complete catalogSource custom resource generated by oc-mirror plugin v2.	String Example: `/tmp/catalog-source_template.yaml` Example of a template file: `apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: discarded namespace: openshift-marketplace spec: image: discarded sourceType: grpc updateStrategy: registryPoll: interval: 30m0s` Copy to Clipboard Toggle word wrap
`mirror.operators.targetTag`	An alternative tag to append to the `targetName` or `targetCatalog`.	String Example: `v1`

11.3.4.1.1. DeleteImageSetConfiguration parameters
Copy link

To use remove images with the oc-mirror plugin v2, you must use a DeleteImageSetConfiguration.yaml configuration file that defines which images to delete from the mirror registry. The following table lists the available parameters for the DeleteImageSetConfiguration resource.

Expand

Table 11.3. DeleteImageSetConfiguration parameters
Parameter	Description	Values
`apiVersion`	The API version for the `DeleteImageSetConfiguration` content.	String Example: `mirror.openshift.io/v2alpha1`
`delete`	The configuration of the image set to delete.	Object
`delete.additionalImages`	The additional images configuration of the delete image set.	Array of objects Example: `additionalImages: - name: registry.redhat.io/ubi8/ubi:latest` Copy to Clipboard Toggle word wrap
`delete.additionalImages.name`	The tag or digest of the image to delete.	String Example: `registry.redhat.io/ubi8/ubi:latest`
`delete.operators`	The Operators configuration of the delete image set.	Array of objects Example: `operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:{product-version} packages: - name: elasticsearch-operator minVersion: '2.4.0'` Copy to Clipboard Toggle word wrap
`delete.operators.catalog`	The Operator catalog to include in the delete image set.	String Example: `registry.redhat.io/redhat/redhat-operator-index:v4.15`
`delete.operators.full`	When true, deletes the full catalog, Operator package, or Operator channel.	Boolean The default value is `false`
`delete.operators.packages`	Operator packages configuration	Array of objects Example: `operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:{product-version} packages: - name: elasticsearch-operator minVersion: '5.2.3-31'` Copy to Clipboard Toggle word wrap
`delete.operators.packages.name`	The Operator package name to include in the delete image set.	String Example: `elasticsearch-operator`
`delete.operators.packages.channels`	Operator package channel configuration	Object
`delete.operators.packages.channels.name`	The Operator channel name, unique within a package, to include in the delete image set.	String Example: `fast` or `stable-v4.15`
`delete.operators.packages.channels.maxVersion`	The highest version of the Operator to delete within the selected channel.	String Example: `5.2.3-31`
`delete.operators.packages.channels.minVersion`	The lowest version of the Operator to delete within the selection in which it exists.	String Example: `5.2.3-31`
`delete.operators.packages.maxVersion`	The highest version of the Operator to delete across all channels in which it exists.	String Example: `5.2.3-31`
`delete.operators.packages.minVersion`	The lowest version of the Operator to delete across all channels in which it exists.	String Example: `5.2.3-31`

11.3.5. Mirroring from mirror to mirror
Copy link

You can use the oc-mirror plugin to mirror an image set directly to a target mirror registry that is accessible during image set creation.

Prerequisites

You have access to the internet to get the required container images.
You installed the OpenShift CLI (oc).
You installed the oc-mirror CLI plugin.
You created the image set configuration file.

Procedure

Mirror the images from the specified image set configuration to a specified registry by running the following command:
```
oc-mirror --config imageset-config.yaml --workspace file://<emphasis><v2_workspace></emphasis> \
  docker://<emphasis><remote_registry></emphasis> --v2
```
```
$ oc-mirror --config imageset-config.yaml --workspace file://<emphasis><v2_workspace></emphasis> \
```
1
```
  docker://<emphasis><remote_registry></emphasis> --v2 
```
2
Copy to Clipboard Toggle word wrap
1
You must use the --workspace flag for the mirror-to-mirror process. Replace <v2_workspace> with the directory you want to use to store custom resources for the mirroring process.
2
Replace <remote_registry> with the name of the registry to mirror the image set file to. The registry must start with docker://. If you specify a top-level namespace for the mirror registry, you must also use this same namespace on later executions.
Example output
```
Rendering catalog image "registry.example.com/redhat/redhat-operator-index:v{ocp-version}" with file-based catalog
```
```
Rendering catalog image "registry.example.com/redhat/redhat-operator-index:v{ocp-version}" with file-based catalog
```
Copy to Clipboard Toggle word wrap
Important
You must use the ImageDigestMirrorSet YAML file as reference content for manual configuration of CRI-O in MicroShift. You cannot apply the resource directly into a MicroShift node.

Verification

List the contents of the cluster-resources subdirectory by running the following command:
```
ls <v2_workspace>/working-dir/cluster-resources/
```
```
$ ls <v2_workspace>/working-dir/cluster-resources/ 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <v2_workspace> with the directory you used to store custom resources for the mirroring process.

Next steps

Convert the ImageDigestMirrorSet YAML content for use in manually configuring CRI-O.
If required, mirror the images from mirror to disk for disconnected or offline use.

Troubleshooting

Unable to retrieve source image.

11.3.6. Configuring CRI-O for using a registry mirror for Operators
Copy link

You must transform the ImageDigestMirrorSet YAML file created with the oc-mirror plugin into a format that is compatible with the CRI-O container runtime configuration used by MicroShift.

Prerequisites

The OpenShift CLI (oc) is installed.
You installed Operator Lifecycle Manager (OLM).
You installed the oc-mirror plugin.
You installed the yq binary.
The ImageDigestMirrorSet and CatalogSource YAML files are available in the cluster-resources subdirectory.

Procedure

Confirm the contents of the ImageDigestMirrorSet YAML file by running the following command:

cat <v2_workspace>/working-dir/cluster-resources/imagedigestmirrorset.yaml

$ cat <v2_workspace>/working-dir/cluster-resources/imagedigestmirrorset.yaml

Copy to Clipboard

Toggle word wrap

1: Replace <v2_workspace> with the directory name that you used when you generated mirroring resources.

Example output

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  labels:
    operators.openshift.org/catalog: "true"
  name: operator-0
spec:
  imageDigestMirrors:
  - mirrors:
    - registry.example.com/amq7
    source: registry.redhat.io/amq7

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  labels:
    operators.openshift.org/catalog: "true"
  name: operator-0
spec:
  imageDigestMirrors:
  - mirrors:
    - registry.example.com/amq7
    source: registry.redhat.io/amq7

Copy to Clipboard

Toggle word wrap

Transform the imagedigestmirrorset.yaml into a format ready for CRI-O configuration by running the following command:

yq '.spec.imageDigestMirrors[] as $item ireduce([]; . + [{"mirror": $item.mirrors[], "source": ($item | .source)}]) | .[] |
  "[[registry]]
      prefix = \"" + .source + "\"
      location = \"" + .mirror + "\"
      mirror-by-digest-only = true
      insecure = true
      "' ./mirror1/working-dir/cluster-resources/imagedigestmirrorset.yaml

yq '.spec.imageDigestMirrors[] as $item ireduce([]; . + [{"mirror": $item.mirrors[], "source": ($item | .source)}]) | .[] |
  "[[registry]]
      prefix = \"" + .source + "\"
      location = \"" + .mirror + "\"
      mirror-by-digest-only = true
      insecure = true
      "' ./mirror1/working-dir/cluster-resources/imagedigestmirrorset.yaml

Copy to Clipboard

Toggle word wrap

Example output

[[registry]]
      prefix = "registry.redhat.io/amq7"
      location = "registry.example.com/amq7"
      mirror-by-digest-only = true
      insecure = true

[[registry]]
      prefix = "registry.redhat.io/amq7"
      location = "registry.example.com/amq7"
      mirror-by-digest-only = true
      insecure = true

Copy to Clipboard

Toggle word wrap

Add the output to the CRI-O configuration file in the /etc/containers/registries.conf.d/ directory:

Example crio-config.yaml mirror configuration file

[[registry]]
      prefix = "registry.redhat.io/amq7"
      location = "registry.example.com/amq7"
      mirror-by-digest-only = true
      insecure = true

[[registry]]
    prefix = ""
    location = "quay.io"
    mirror-by-digest-only = true
[[registry.mirror]]
    location = "<registry_host>:<port>" 
    insecure = false

[[registry]]
      prefix = "registry.redhat.io/amq7"
      location = "registry.example.com/amq7"
      mirror-by-digest-only = true
      insecure = true

[[registry]]
    prefix = ""
    location = "quay.io"
    mirror-by-digest-only = true
[[registry.mirror]]
    location = "<registry_host>:<port>"


    insecure = false

Copy to Clipboard

Toggle word wrap

1: Specify the hostname and port of your mirror registry server, for example microshift-quay:8443.

Apply the CRI-O configuration changes by restarting MicroShift with the following command:
```
sudo systemctl restart crio
```
```
$ sudo systemctl restart crio
```
Copy to Clipboard Toggle word wrap

11.3.7. Installing a custom catalog created with the oc-mirror plugin
Copy link

After you mirror your image set to the mirror registry, you must apply the generated CatalogSource custom resource (CR) into the node. Operator Lifecycle Manager (OLM) uses the CatalogSource CR to retrieve information about the available Operators in the mirror registry. You must then create and apply a subscription CR to subscribe to your custom catalog.

Prerequisites

You mirrored the image set to your registry mirror.
You added image reference information to the CRI-O container runtime configuration.

Procedure

Apply the catalog source configuration file from the results directory to create the catalog source object by running the following command:
```
oc apply -f ./<v2_workspace>/working-dir/cluster-resources/catalogSource-cs-redhat-catalog.yaml
```
```
$ oc apply -f ./<v2_workspace>/working-dir/cluster-resources/catalogSource-cs-redhat-catalog.yaml 
```
1
Copy to Clipboard Toggle word wrap
1
Replace <v2_workspace> with the directory you used to store custom resources for the mirroring process.
Example output
```
catalogsource.operators.coreos.com/cs-redhat-catalog created
```
```
catalogsource.operators.coreos.com/cs-redhat-catalog created
```
Copy to Clipboard Toggle word wrap

For reference, see the following example file:

Example catalog source configuration file

apiVersion: operators.coreos.com/v2alpha1
kind: CatalogSource
metadata:
  name: redhat-catalog
  namespace: openshift-marketplace 
spec:
  sourceType: grpc
  image: registry.example.com/redhat/redhat-catalog:v4.20
  updateStrategy:
    registryPoll:
      interval: 60m

apiVersion: operators.coreos.com/v2alpha1
kind: CatalogSource
metadata:
  name: redhat-catalog
  namespace: openshift-marketplace


spec:
  sourceType: grpc
  image: registry.example.com/redhat/redhat-catalog:v4.20
  updateStrategy:
    registryPoll:
      interval: 60m

Copy to Clipboard

Toggle word wrap

1: Specifies the global namespace. Setting the metadata.namespace to openshift-marketplace enables the catalog to reference catalogs in all namespaces. Subscriptions in any namespace can reference catalogs created in the openshift-marketplace namespace.

Verify that the CatalogSource resources were successfully installed by running the following command:

oc get catalogsource --all-namespaces

$ oc get catalogsource --all-namespaces

Copy to Clipboard

Toggle word wrap

Example output

NAMESPACE               NAME                  DISPLAY               TYPE   PUBLISHER   AGE
openshift-marketplace   certified-operators   Certified Operators   grpc   Red Hat     37m
openshift-marketplace   community-operators   Community Operators   grpc   Red Hat     37m
openshift-marketplace   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     37m
openshift-marketplace   redhat-catalog        Red Hat Catalog     grpc   Red Hat     37m

NAMESPACE               NAME                  DISPLAY               TYPE   PUBLISHER   AGE
openshift-marketplace   certified-operators   Certified Operators   grpc   Red Hat     37m
openshift-marketplace   community-operators   Community Operators   grpc   Red Hat     37m
openshift-marketplace   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     37m
openshift-marketplace   redhat-catalog        Red Hat Catalog     grpc   Red Hat     37m

Copy to Clipboard

Toggle word wrap

Verify that the catalog source is running by using the following command:

oc get pods -n openshift-marketplace

$ oc get pods -n openshift-marketplace

Copy to Clipboard

Toggle word wrap

Example output

NAME                             READY   STATUS    RESTARTS   AGE
cs-redhat-catalog-4227b   2/2     Running   0          2m5s

NAME                             READY   STATUS    RESTARTS   AGE
cs-redhat-catalog-4227b   2/2     Running   0          2m5s

Copy to Clipboard

Toggle word wrap

Create a Subscription CR, similar to the following example:

Example Subscription CR

apiVersion: operators.coreos.com/v2alpha1
kind: Subscription
metadata:
  name: amq-broker
  namespace: openshift-operators
spec:
  channel: 7.13.x
  name: amq-broker-rhel9
  source: cs-redhat-catalog
  sourceNamespace: openshift-marketplace

apiVersion: operators.coreos.com/v2alpha1
kind: Subscription
metadata:
  name: amq-broker
  namespace: openshift-operators
spec:
  channel: 7.13.x
  name: amq-broker-rhel9
  source: cs-redhat-catalog
  sourceNamespace: openshift-marketplace

Copy to Clipboard

Toggle word wrap

Apply the Subscription CR configuration by running the following command:
```
oc apply -f ./<subscription_cr.yaml>
```
```
$ oc apply -f ./<subscription_cr.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Specify the name of your subscription in <subscription_cr.yaml>, for example amq—broker-subscription-cr.yaml.
Example output
```
subscription.operators.coreos.com/amq-broker created
```
```
subscription.operators.coreos.com/amq-broker created
```
Copy to Clipboard Toggle word wrap

11.4. Adding OLM-based Operators to a disconnected node
Copy link

You can use OLM-based Operators in disconnected situations by embedding them in a Red Hat Enterprise Linux for Edge (RHEL for Edge) image.

11.4.1. About adding OLM-based Operators to a disconnected node
Copy link

For Operators that are installed on disconnected nodes, Operator Lifecycle Manager (OLM) by default cannot access sources hosted on remote registries because those remote sources require full internet connectivity. Therefore, you must mirror the remote registries to a highly available container registry.

The following steps are required to use OLM-based Operators in disconnected situations:

Include OLM in the container image list for your mirror registry.
Configure the system to use your mirror registry by updating your CRI-O configuration directly. ImageContentSourcePolicy is not supported in MicroShift.
Add a CatalogSource object to the node so that the OLM catalog Operator can use the local catalog on the mirror registry.
Ensure that MicroShift is installed to run in a disconnected capacity.
Ensure that the network settings are configured to run in disconnected mode.

After enabling OLM in a disconnected node, you can continue to use your internet-connected workstation to keep your local catalog sources updated as newer versions of Operators are released.

11.4.1.1. Performing a dry run
Copy link

You can use oc-mirror to perform a dry run, without actually mirroring any images. A dry run means you can review the list of images to be mirrored. You can catch any errors with your image set configuration early by using a dry run, or use the generated list of images with other tools to conduct mirroring.

Prerequisites

You have access to the internet to obtain the necessary container images.
You installed the OpenShift CLI (oc).
You installed the oc-mirror CLI plugin.
You created the image set configuration file.

Procedure

Run the oc mirror command with the --dry-run flag to perform a dry run:

oc-mirror --config <ImageSetConfig.yaml> docker://localhost:5000 --workspace file://<outm2m> --dry-run --v2

$ oc-mirror --config <ImageSetConfig.yaml> docker://localhost:5000 --workspace file://<outm2m> --dry-run --v2

Copy to Clipboard

Toggle word wrap

where:

ImageSetConfig.yaml

Specifies the name of the image set configuration file that you created.

docker://localhost:5000

Specifies the mirror registry. Nothing is mirrored to this registry when you use the --dry-run flag.

--workspace file://<outm2m>

Insert the address of the workspace path.

--dry-run

The dry run flag generates the dry run artifacts and not an actual image set file.

--v2

Specifies oc mirror v2.

Example output

2025/08/25 15:50:44  [INFO]   : 👋 Hello, welcome to oc-mirror
2025/08/25 15:50:44  [INFO]   : ⚙  setting up the environment for you...
2025/08/25 15:50:44  [INFO]   : 🔀 workflow mode: mirrorToMirror
2025/08/25 15:50:44  [INFO]   : 🕵  going to discover the necessary images...
2025/08/25 15:50:44  [INFO]   : 🔍 collecting release images...
2025/08/25 15:50:44  [INFO]   : 🔍 collecting operator images...
 ✓   (1m30s) Collecting catalog registry.redhat.io/redhat/redhat-operator-index:v4.20
2025/08/25 15:52:14  [INFO]   : 🔍 collecting additional images...
2025/08/25 15:52:14  [INFO]   : 📄 list of all images for mirroring in : wspace/working-dir/dry-run/mapping.txt
2025/08/25 15:52:14  [INFO]   : mirror time     : 1m30.399585837s
2025/08/25 15:52:14  [INFO]   : 👋 Goodbye, thank you for using oc-mirror

2025/08/25 15:50:44  [INFO]   : 👋 Hello, welcome to oc-mirror
2025/08/25 15:50:44  [INFO]   : ⚙  setting up the environment for you...
2025/08/25 15:50:44  [INFO]   : 🔀 workflow mode: mirrorToMirror
2025/08/25 15:50:44  [INFO]   : 🕵  going to discover the necessary images...
2025/08/25 15:50:44  [INFO]   : 🔍 collecting release images...
2025/08/25 15:50:44  [INFO]   : 🔍 collecting operator images...
 ✓   (1m30s) Collecting catalog registry.redhat.io/redhat/redhat-operator-index:v4.20
2025/08/25 15:52:14  [INFO]   : 🔍 collecting additional images...
2025/08/25 15:52:14  [INFO]   : 📄 list of all images for mirroring in : wspace/working-dir/dry-run/mapping.txt
2025/08/25 15:52:14  [INFO]   : mirror time     : 1m30.399585837s
2025/08/25 15:52:14  [INFO]   : 👋 Goodbye, thank you for using oc-mirror

Copy to Clipboard

Toggle word wrap

Review the mapping.txt file that was generated by running the following command:

cat wspace/working-dir/dry-run/mapping.txt

$ cat wspace/working-dir/dry-run/mapping.txt

Copy to Clipboard

Toggle word wrap

Example output

docker://registry.redhat.io/amq8/amq-broker-rhel9@sha256:47fd4ce2533496828aba37bd1f9715e2164d5c90bd0fc6b25e7e0786d723bf01=docker://mirror.com/amq8/amq-broker-rhel9:sha256-47fd4ce2533496828aba37bd1f9715e2164d5c90bd0fc6b25e7e0786d723bf01
docker://registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:9cc48eecf1442ae04b8543fa5d4381a13bc2831390850828834d387006d1342b=docker://mirror.com/amq7/amq-broker-init-rhel9:sha256-9cc48eecf1442ae04b8543fa5d4381a13bc2831390850828834d387006d1342b
docker://registry.redhat.io/amq8/amq-broker-rhel9@sha256:bb6fbd68475a7852b4d99eea6c4ab313f9267da7963162f0d75375d7063409e7=docker://mirror.com/amq8/amq-broker-rhel9:sha256-bb6fbd68475a7852b4d99eea6c4ab313f9267da7963162f0d75375d7063409e7
docker://registry.redhat.io/amq8/amq-broker-rhel9@sha256:d42d713da0ce6806fdc6492b6342586783e6865a82a8647d3c4288439b1751ee=docker://mirror.com/amq8/amq-broker-rhel9:sha256-d42d713da0ce6806fdc6492b6342586783e6865a82a8647d3c4288439b1751ee
docker://registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:ffffa9875f0379e9373f89f05eb06e5a193273bb04bc3aa5f85b044357b79098=docker://mirror.com/amq8/amq-broker-init-rhel9:sha256-ffffa9875f0379e9373f89f05eb06e5a193273bb04bc3aa5f85b044357b79098

docker://registry.redhat.io/amq8/amq-broker-rhel9@sha256:47fd4ce2533496828aba37bd1f9715e2164d5c90bd0fc6b25e7e0786d723bf01=docker://mirror.com/amq8/amq-broker-rhel9:sha256-47fd4ce2533496828aba37bd1f9715e2164d5c90bd0fc6b25e7e0786d723bf01
docker://registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:9cc48eecf1442ae04b8543fa5d4381a13bc2831390850828834d387006d1342b=docker://mirror.com/amq7/amq-broker-init-rhel9:sha256-9cc48eecf1442ae04b8543fa5d4381a13bc2831390850828834d387006d1342b
docker://registry.redhat.io/amq8/amq-broker-rhel9@sha256:bb6fbd68475a7852b4d99eea6c4ab313f9267da7963162f0d75375d7063409e7=docker://mirror.com/amq8/amq-broker-rhel9:sha256-bb6fbd68475a7852b4d99eea6c4ab313f9267da7963162f0d75375d7063409e7
docker://registry.redhat.io/amq8/amq-broker-rhel9@sha256:d42d713da0ce6806fdc6492b6342586783e6865a82a8647d3c4288439b1751ee=docker://mirror.com/amq8/amq-broker-rhel9:sha256-d42d713da0ce6806fdc6492b6342586783e6865a82a8647d3c4288439b1751ee
docker://registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:ffffa9875f0379e9373f89f05eb06e5a193273bb04bc3aa5f85b044357b79098=docker://mirror.com/amq8/amq-broker-init-rhel9:sha256-ffffa9875f0379e9373f89f05eb06e5a193273bb04bc3aa5f85b044357b79098

Copy to Clipboard

Toggle word wrap

11.4.1.2. Getting catalogs and Operator container image references
Copy link

After performing a dry run with the oc-mirror plugin to review the list of images that you want to mirror, you must get all of the container image references, then format the output for adding to an image builder blueprint.

Note

For catalogs made for proprietary Operators, you can format image references for the image builder blueprint without using the following procedure.

Prerequisites

You have a catalog index for the Operators you want to use.
You have installed the jq CLI tool.
You are familiar with image builder blueprint files.
You have an image builder blueprint TOML file.

Procedure

Parse the catalog index.json file to get the image references that you need to include in the image builder blueprint. You can use either the unfiltered catalog or you can filter out images that cannot be mirrored:

Parse the unfiltered catalog index.json file to get the image references by running the following command:

jq -r --slurp '.[] | select(.relatedImages != null) | "[[containers]]\nsource = \"" + .relatedImages[].image + "\"\n"'   ./oc-mirror-workspace/src/catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.20/index/index.json

jq -r --slurp '.[] | select(.relatedImages != null) | "[[containers]]\nsource = \"" + .relatedImages[].image + "\"\n"'   ./oc-mirror-workspace/src/catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.20/index/index.json

Copy to Clipboard

Toggle word wrap

If you want to filter out images that cannot be mirrored, filter and parse the catalog index.json file by running the following command:

jq -r --slurp '.[] | select(.relatedImages != null) | .relatedImages[] | select(.name |  contains("ppc") or contains("s390x") | not) | "[[containers]]\\nsource = \\"" + .image + "\\"\\n"' ./oc-mirror-workspace/src/catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.20/index/index.json

$ jq -r --slurp '.[] | select(.relatedImages != null) | .relatedImages[] | select(.name |  contains("ppc") or contains("s390x") | not) | "[[containers]]\\nsource = \\"" + .image + "\\"\\n"' ./oc-mirror-workspace/src/catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.20/index/index.json

Copy to Clipboard

Toggle word wrap

Note

This step uses the AMQ Broker Operator as an example. You can add other criteria to the jq command for further filtering as required by your use case.

Example image-reference output

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:0b2126cfb6054fdf428c1f43b69e36e93a09a49ce15350e9273c98cc08c6598b"

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:0dde839c2dce7cb684094bf26523c8e16677de03149a0fff468b8c3f106e1f4f"
...
...

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:e8fa2a00e576ecb95561ffbdbf87b1c82d479c8791ab2c6ce741dd0d0b496d15"

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:ff6fefad518a6c997d4c5a6e475ba89640260167f0bc27715daf3cc30116fad1"
…
EOF

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:0b2126cfb6054fdf428c1f43b69e36e93a09a49ce15350e9273c98cc08c6598b"

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:0dde839c2dce7cb684094bf26523c8e16677de03149a0fff468b8c3f106e1f4f"
...
...

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:e8fa2a00e576ecb95561ffbdbf87b1c82d479c8791ab2c6ce741dd0d0b496d15"

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:ff6fefad518a6c997d4c5a6e475ba89640260167f0bc27715daf3cc30116fad1"
…
EOF

Copy to Clipboard

Toggle word wrap

Important

For mirrored and disconnected use cases, ensure that all of the sources filtered from your catalog index.json file are digests. If any of the sources use tags instead of digests, the Operator installation fails. Tags require an internet connection.

View the imageset-config.yaml to get the catalog image reference for the CatalogSource custom resource (CR) by running the following command:

cat imageset-config.yaml

$ cat imageset-config.yaml

Copy to Clipboard

Toggle word wrap

Example output

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.20 
    packages:
    - name: amq-broker-rhel9
      channels:
      - name: 7.13.x

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.20


    packages:
    - name: amq-broker-rhel9
      channels:
      - name: 7.13.x

Copy to Clipboard

Toggle word wrap

1: Use the value in the mirror.catalog catalog image reference for the following jq command to get the image digest. In this example, <registry.redhat.io/redhat/redhat-operator-index:v4.20>.

Get the SHA of the catalog index image by running the following command:
```
skopeo inspect docker://<registry.redhat.io/redhat/redhat-operator-index:v{product-version}> | jq .Digest
```
```
$ skopeo inspect docker://<registry.redhat.io/redhat/redhat-operator-index:v{product-version}> | jq .Digest 
```
1
Copy to Clipboard Toggle word wrap
1
Use the value in the mirror.catalog catalog image reference for the jq command to get the image digest. In this example, <registry.redhat.io/redhat/redhat-operator-index:v4.20>.
Example output
```
"sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6"
```
```
"sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6"
```
Copy to Clipboard Toggle word wrap

To get ready to add the image references to your image builder blueprint file, format the catalog image reference by using the following example:

[[containers]]
source = "registry.redhat.io/redhat/redhat-operator-index@sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6"

[[containers]]
source = "registry.redhat.io/redhat/redhat-operator-index@sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6"

Copy to Clipboard

Toggle word wrap

Add the image references from all of the previous steps to the image builder blueprint.

Generated image builder blueprint example snippet

name = "microshift_blueprint"
description = "MicroShift 4.20.1 on x86_64 platform"
version = "0.0.1"
modules = []
groups = []

[[packages]] 
name = "microshift"
version = "4.20.1"
...
...

[customizations.services] 
enabled = ["microshift"]

[customizations.firewall]
ports = ["22:tcp", "80:tcp", "443:tcp", "5353:udp", "6443:tcp", "30000-32767:tcp", "30000-32767:udp"]
...
...

[[containers]] 
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41e79c17e8b41f1b0a5a32c3e2dd7cd15b8274554d3f1ba12b2598a347475f4"

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbc65f1fba7d92b36cf7514cd130fe83a9bd211005ddb23a8dc479e0eea645fd"
...
...

[[containers]] 
source = "registry.redhat.io/redhat/redhat-operator-index@sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6"
...
...

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:0dde839c2dce7cb684094bf26523c8e16677de03149a0fff468b8c3f106e1f4f"
...
...

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:e8fa2a00e576ecb95561ffbdbf87b1c82d479c8791ab2c6ce741dd0d0b496d15"

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:ff6fefad518a6c997d4c5a6e475ba89640260167f0bc27715daf3cc30116fad1"
…
EOF

name = "microshift_blueprint"
description = "MicroShift 4.20.1 on x86_64 platform"
version = "0.0.1"
modules = []
groups = []

[[packages]]


name = "microshift"
version = "4.20.1"
...
...

[customizations.services]


enabled = ["microshift"]

[customizations.firewall]
ports = ["22:tcp", "80:tcp", "443:tcp", "5353:udp", "6443:tcp", "30000-32767:tcp", "30000-32767:udp"]
...
...

[[containers]]


source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41e79c17e8b41f1b0a5a32c3e2dd7cd15b8274554d3f1ba12b2598a347475f4"

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbc65f1fba7d92b36cf7514cd130fe83a9bd211005ddb23a8dc479e0eea645fd"
...
...

[[containers]]


source = "registry.redhat.io/redhat/redhat-operator-index@sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6"
...
...

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-init-rhel9@sha256:0dde839c2dce7cb684094bf26523c8e16677de03149a0fff468b8c3f106e1f4f"
...
...

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:e8fa2a00e576ecb95561ffbdbf87b1c82d479c8791ab2c6ce741dd0d0b496d15"

[[containers]]
source = "registry.redhat.io/amq8/amq-broker-rhel9@sha256:ff6fefad518a6c997d4c5a6e475ba89640260167f0bc27715daf3cc30116fad1"
…
EOF

Copy to Clipboard

Toggle word wrap

1: References for all non-optional MicroShift RPM packages using the same version compatible with the microshift-release-info RPM.
2: References for automatically enabling MicroShift on system startup and applying default networking settings.
3: References for all non-optional MicroShift container images necessary for a disconnected deployment.
4: References for the catalog index.

11.4.1.3. Applying catalogs and Operators in a disconnected-deployment RHEL for Edge image
Copy link

After you have created a RHEL for Edge image for a disconnected environment and configured MicroShift networking settings for disconnected use, you can configure the namespace and create catalog and Operator custom resources (CR) for running your Operators.

Prerequisites

You have a RHEL for Edge image.
Networking is configured for disconnected use.
You completed the oc-mirror plugin dry run procedure.

Procedure

Create a CatalogSource custom resource (CR), similar to the following example:
Example my-catalog-source-cr.yaml file
```
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: cs-redhat-operator-index
  namespace: openshift-marketplace 
spec:
  image: registry.example.com/redhat/redhat-operator-index:v4.17
  sourceType: grpc
  displayName:
  publisher:
  updateStrategy:
    registryPoll:
      interval: 60m
```
```
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: cs-redhat-operator-index
  namespace: openshift-marketplace 
```
1
```
spec:
  image: registry.example.com/redhat/redhat-operator-index:v4.17
  sourceType: grpc
  displayName:
  publisher:
  updateStrategy:
    registryPoll:
      interval: 60m
```
Copy to Clipboard Toggle word wrap
1
The global namespace. Setting the metadata.namespace to openshift-marketplace enables the catalog to run in all namespaces. Subscriptions in any namespace can reference catalogs created in the openshift-marketplace namespace.
Note
The default pod security admission definition for openshift-marketplace is baseline, therefore a catalog source custom resource (CR) created in that namespace does not require a spec.grpcPodConfig.securityContextConfig value to be set. You can set a legacy or restricted value if required for the namespace and Operators you want to use.

Add the SHA of the catalog index commit to the Catalog Source (CR), similar to the following example:

Example namespace spec.image configuration

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: cs-redhat-operator-index
  namespace: openshift-marketplace
spec:
  image: registry.example.com/redhat/redhat-operator-index@sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6 
  sourceType: grpc
  displayName:
  publisher:
  updateStrategy:
    registryPoll:
      interval: 60m

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: cs-redhat-operator-index
  namespace: openshift-marketplace
spec:
  image: registry.example.com/redhat/redhat-operator-index@sha256:7a76c0880a839035eb6e896d54ebd63668bb37b82040692141ba39ab4c539bc6


  sourceType: grpc
  displayName:
  publisher:
  updateStrategy:
    registryPoll:
      interval: 60m

Copy to Clipboard

Toggle word wrap

1: The SHA of the image commit. Use the same SHA you added to the image builder blueprint.

Important

You must use the SHA instead of a tag in your catalog CR or the pod fails to start.

Apply the YAML file from the oc-mirror plugin dry run results directory to the node by running the following command:

oc apply -f ./oc-mirror-workspace/results-1708508014/catalogSource-cs-redhat-operator-index.yaml

$ oc apply -f ./oc-mirror-workspace/results-1708508014/catalogSource-cs-redhat-operator-index.yaml

Copy to Clipboard

Toggle word wrap

Example output

catalogsource.operators.coreos.com/cs-redhat-operator-index created

catalogsource.operators.coreos.com/cs-redhat-operator-index created

Copy to Clipboard

Toggle word wrap

Verify that the CatalogSource resources were successfully installed by running the following command:
```
oc get catalogsource --all-namespaces
```
```
$ oc get catalogsource --all-namespaces
```
Copy to Clipboard Toggle word wrap

Verify that the catalog source is running by using the following command:

oc get pods -n openshift-marketplace

$ oc get pods -n openshift-marketplace

Copy to Clipboard

Toggle word wrap

Example output

NAME                             READY   STATUS    RESTARTS   AGE
cs-redhat-operator-index-4227b   2/2     Running   0          2m5s

NAME                             READY   STATUS    RESTARTS   AGE
cs-redhat-operator-index-4227b   2/2     Running   0          2m5s

Copy to Clipboard

Toggle word wrap

Create a Subscription CR, similar to the following example:

Example my-subscription-cr.yaml file

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-broker
  namespace: openshift-operators
spec:
  channel: 7.11.x
  name: amq-broker-rhel8
  source: cs-redhat-operator-index
  sourceNamespace: openshift-marketplace

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-broker
  namespace: openshift-operators
spec:
  channel: 7.11.x
  name: amq-broker-rhel8
  source: cs-redhat-operator-index
  sourceNamespace: openshift-marketplace

Copy to Clipboard

Toggle word wrap

Apply the Subscription CR by running the following command:
```
oc apply -f ./<my-subscription-cr.yaml>
```
```
$ oc apply -f ./<my-subscription-cr.yaml> 
```
1
Copy to Clipboard Toggle word wrap
1
Specify the name of your Subscription CR, such as my-subscription-cr.yaml.
Example output
```
subscription.operators.coreos.com/amq-broker created
```
```
subscription.operators.coreos.com/amq-broker created
```
Copy to Clipboard Toggle word wrap

Legal Notice
Copy link

The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Running applications