Install Ansible automation portal in air-gapped OpenShift Container Platform environments

You can install Ansible automation portal in a disconnected OpenShift Container Platform environment.

Prerequisites for disconnected installation
Copy link

Review the mandatory subscriptions, permissions, and platform access required before starting the disconnected installation of the Ansible automation portal.

Fulfilling these prerequisites helps ensure a successful deployment.

  • You have a valid subscription to Red Hat Ansible Automation Platform.
  • You have access to an instance of Red Hat Ansible Automation Platform with the appropriate permissions to create an OAuth application.
  • You have access to a Red Hat OpenShift Container Platform instance with the appropriate permissions within your project to create an application.
  • You have installed oc, the OpenShift command-line interface (CLI) tool, on your local machine.
  • You have installed Helm 3.10 or newer.
  • You have installed skopeo and podman for mirroring container images and plug-in artifacts.
  • You have internet access to pull images and charts from public repositories, including registry.redhat.io.
  • A Red Hat pull secret that allows you to pull images from registry.redhat.io.
  • You have a method to provide the Ansible plug-ins in the disconnected environment:
    • For OCI delivery: A method to mirror the OCI artifacts image referenced by imageTagInfo.
    • For HTTP plug-in registry: The ability to host the plug-in tarball files.
  • You have registry credentials for the registry endpoint used by the dynamic plug-in installer.
Important

The image versions and compatibility requirements vary between Helm chart releases. Before you begin, consult the Ansible Automation Portal Lifecycle page for version mappings between the Helm chart, Red Hat Developer Hub, and PostgreSQL.

Prepare for air-gapped installation
Copy link

Before you can install Ansible automation portal in a disconnected OpenShift Container Platform environment, you must complete some processes on a connected bastion host.

Mirror container images
Copy link

Use skopeo copy to mirror the required container images from the Red Hat registry to your disconnected registry for installing the Ansible automation portal in an isolated environment.

Before you begin
Copy link

  • skopeo is installed.
  • You have authenticated to registry.redhat.io: 
    $ skopeo login registry.redhat.io
  • You have authenticated to your disconnected registry: 
    $ skopeo login <disconnected_registry_url>

About this task
Copy link

skopeo copy preserves SHA256 digests, so the Helm chart's default digest-based image references work without additional tag overrides.

If you mirror registry.redhat.io content to a different registry host (or to a registry prefix such as quay.io/<org>), you can set redhat-developer-hub.global.imageRegistry so the Helm chart pulls all of its images from that mirrored location. imageRegistry is a single override that controls the registry for the base application image, PostgreSQL image, OCI plug-in artifacts, and Ansible Dev Tools sidecar.

The dynamic plug-in init container does not use cluster-level image mirror configuration (for example, ImageDigestMirrorSet or ImageTagMirrorSet). You must set imageRegistry even if your cluster redirects registry.redhat.io pulls.

Expand
Table 1. Required container images
Image Source registry Purpose
rhdh/rhdh-hub-rhel9:<rhdh_version> registry.redhat.io Red Hat Developer Hub application and install-dynamic-plugins init container
rhel9/postgresql-<version>:<tag> registry.redhat.io Built-in PostgreSQL database (skip if using an external database)
ansible-automation-platform-25/ansible-dev-tools-rhel8:latest or ansible-automation-platform-26/ansible-dev-tools-rhel9:latest registry.redhat.io Ansible Dev Tools sidecar (base image varies by Ansible Automation Platform version)
ansible-automation-platform/automation-portal:<plugin-version> registry.redhat.io Ansible plug-in OCI artifacts (OCI delivery only)
rhdh/rhdh-plugin-catalog-index-rhel9:<rhdh_version> registry.redhat.io Plug-in catalog index (rebuilt by mirror-plugins.sh)

Replace version placeholders with the versions bundled with your Helm chart. See the Ansible Automation Portal Lifecycle page for version mappings.

Procedure
Copy link

  1. Copy the Red Hat Developer Hub hub image to your disconnected registry: 
    $ skopeo copy \
    docker://registry.redhat.io/rhdh/rhdh-hub-rhel9:<rhdh_version> \
    docker://<disconnected_registry_url>/rhdh/rhdh-hub-rhel9:<rhdh_version>

    This image is used for both the main application container and the install-dynamic-plugins init container. Replace <rhdh_version> with the Red Hat Developer Hub version bundled with your Helm chart.

  2. If you use the built-in PostgreSQL database, copy the PostgreSQL image.

    An external database is the supported production architecture and does not require this step.

    $ skopeo copy \
    docker://registry.redhat.io/rhel9/postgresql-<version>:<tag> \
    docker://<disconnected_registry_url>/rhel9/postgresql-<version>:<tag>

    Replace <version> with the PostgreSQL major version and <tag> with the image tag bundled with your Helm chart. See the Ansible Automation Portal Lifecycle page for version mappings.

  3. Copy the plug-in catalog index image: 
    $ skopeo copy \
    docker://registry.redhat.io/rhdh/rhdh-plugin-catalog-index-rhel9:<rhdh_version> \
    docker://<disconnected_registry_url>/rhdh/rhdh-plugin-catalog-index-rhel9:<rhdh_version>

    The Mirror dynamic plug-in artifacts procedure rebuilds this image with updated registry references. You must still mirror the original image because the script uses it as its source.

  4. If you use OCI container delivery, copy the Ansible plug-ins OCI artifacts image: 
    $ skopeo copy \
    docker://registry.redhat.io/ansible-automation-platform/automation-portal:<plugin-version> \
    docker://<disconnected_registry_url>/ansible-automation-platform/automation-portal:<plugin-version>
  5. Copy the Ansible Dev Tools sidecar image, using the path that matches your Ansible Automation Platform version.

    Ansible Automation Platform 2.5:

    $ skopeo copy \
    docker://registry.redhat.io/ansible-automation-platform-25/ansible-dev-tools-rhel8:latest \
    docker://<disconnected_registry_url>/ansible-automation-platform-25/ansible-dev-tools-rhel8:latest

    Ansible Automation Platform 2.6:

    $ skopeo copy \
    docker://registry.redhat.io/ansible-automation-platform-26/ansible-dev-tools-rhel9:latest \
    docker://<disconnected_registry_url>/ansible-automation-platform-26/ansible-dev-tools-rhel9:latest

Results
Copy link

Verify that each image is accessible in your disconnected registry. For example:

$ skopeo inspect docker://<disconnected_registry_url>/rhdh/rhdh-hub-rhel9:<rhdh_version>

A successful response returns the image manifest metadata. An error indicates the image was not mirrored correctly.

Download the helm chart package
Copy link

Download the Helm chart package and modify the internal image references to point to your disconnected registry. This prepares the installation package for the air-gapped environment.

Procedure
Copy link

  1. Add the OpenShift Helm charts repository and update your local cache: 
    helm repo add openshift-helm-charts https://charts.openshift.io/
helm repo update
  2. Pull the required version of the chart: 
    helm pull openshift-helm-charts/redhat-rhaap-portal --version x.y.z

    The chart is saved as a .tgz file (for example, redhat-rhaap-portal-1.0.1.tgz).

  3. Extract the chart files: 
    tar -xvf redhat-rhaap-portal-x.y.z.tgz

    This creates a directory with a name similar to redhat-rhaap-portal-1.0.1/.

  4. In the redhat-rhaap-portal/values.yaml file, replace all image: references with the full path to the images in your disconnected registry.
  5. Repack the chart with your modifications: 
    helm package redhat-rhaap-portal-x.y.z

    A new .tgz file is created containing your changes.

Transfer assets to the disconnected environment
Copy link

Transfer the modified Helm chart package from the connected bastion host to a machine inside your disconnected network. This action stages the installation assets for deployment within the isolated OpenShift environment.

Procedure
Copy link

  1. Copy the modified Helm chart .tgz file or files (for example, redhat-rhaap-portal-1.0.1.tgz) from your connected bastion host to a machine or jump box within your disconnected OpenShift network.
  2. If you use the HTTP plug-in registry method, transfer the plug-in tarball files to the disconnected environment.

Install the Helm chart in the disconnected OpenShift environment
Copy link

You can install the modified Helm chart using the helm install command in your disconnected OpenShift environment. This deploys the Ansible automation portal using the locally available assets.

After preparing the disconnected environment with mirrored images and transferred assets, install the Helm chart to deploy the Ansible automation portal.

Access the disconnected OpenShift environment
Copy link

Ensure your disconnected OpenShift cluster is configured to trust the private registry containing the mirrored container images. This step is crucial for successful image pulling during installation.

Before you begin
Copy link

  • You have the necessary kubeconfig and permissions for setting up image pull secrets or insecure registries.

Procedure
Copy link

  1. In a terminal, log in to your disconnected OpenShift cluster using the oc CLI. 
    oc login --token=<your_token> --server=<your_openshift_api_url>

    Use the following command if you have a kubeconfig:

    export KUBECONFIG=/path/to/your/kubeconfig
oc login
  2. Ensure that your OpenShift cluster is configured to trust your disconnected registry.

Configure plug-in delivery for disconnected environments
Copy link

Create the registry authentication secret so the dynamic plug-in init container can pull OCI artifacts from your disconnected registry.

Before you begin
Copy link

  • You have mirrored the required container images and plug-in artifacts to your disconnected registry.
  • You have credentials for your disconnected registry.

Procedure
Copy link

  1. Create an auth.json file with credentials for your disconnected registry: 
    {
  "auths": {
    "<disconnected_registry_url>": {
      "auth": "<base64-encoded-registry-credentials>"
    }
  }
}

    Generate the base64-encoded value:

    $ printf '%s' '<registry_username>:<registry_password>' | base64 -w0
  2. Create the registry authentication secret: 
    $ oc create secret generic <release-name>-dynamic-plugins-registry-auth \
  --from-file=auth.json=./auth.json \
  -n <project_name>

    Replace <release-name> with your Helm release name. If you use the default release name from the OpenShift catalog, the secret name is redhat-rhaap-portal-dynamic-plugins-registry-auth.

Results
Copy link

Verify that the secret exists in the project:

$ oc get secret <release-name>-dynamic-plugins-registry-auth -n <project_name>

Install the Helm chart
Copy link

Install the Ansible automation portal by using the helm install command, referencing the local Helm chart file and required configuration values.

Procedure
Copy link

  1. On the machine within your disconnected environment, navigate to the directory where you placed the transferred Helm chart .tgz file: 
    $ cd /path/to/your/transferred/charts/
  2. Define your namespace and cluster router base as environment variables: 
    $ export MY_NAMESPACE="<your_namespace_name>"
$ export MY_CLUSTER_ROUTER_BASE="<your_cluster_router_base>"
  3. If the namespace does not exist, create it: 
    $ oc new-project "${MY_NAMESPACE}"
  4. Create a custom values file and reuse it for install and upgrade.

    Choose the example that matches your plug-in delivery method.

    OCI delivery (values-oci.yaml):

    redhat-developer-hub:
  global:
    clusterRouterBase: <your_cluster_router_base>
    pluginMode: oci
    imageTagInfo: "<plugin-version>"
    # Registry host only. Replaces registry.redhat.io for all images.
    imageRegistry: "<disconnected_registry_url>"
    # Optional. Full OCI plugin image path for mirrors with non-standard
    # repository paths. Overrides imageRegistry for OCI plugin artifacts only.
    # Use when your mirror does not preserve the default repository path
    # (ansible-automation-platform/automation-portal).
    # ociPluginImage: "<disconnected_registry_url>/custom-path/automation-portal"
  catalogIndex:
    image:
      registry: "<disconnected_registry_url>"
  upstream:
    backstage:
      # The Helm chart pins images by SHA256 digest by default.
      # Override with tag-based references if your mirroring method
      # does not preserve digests.
      image:
        repository: rhdh/rhdh-hub-rhel9
        tag: "<rhdh_version>"
      appConfig:
        ansible:
          rhaap:
            checkSSL: true  # Set to false if using self-signed certificates.
        catalog:
          providers:
            rhaap:
              '{{- include "catalog.providers.env" . }}':
                orgs: "<your-aap-organization-name>"
    postgresql:
      image:
        repository: rhel9/postgresql-<version>
        tag: "<postgresql_tag>"
    Note

    The Helm chart pins the PostgreSQL image by SHA256 digest. If your mirroring method does not preserve digests, set postgresql.image.tag to the tag you used when mirroring the image.

    HTTP plug-in registry delivery (values-tarball.yaml):

    redhat-developer-hub:
  global:
    clusterRouterBase: <your_cluster_router_base>
    pluginMode: tarball
    # Registry host only. Replaces registry.redhat.io for the images.
    imageRegistry: "<disconnected_registry_url>"
  catalogIndex:
    image:
      registry: "<disconnected_registry_url>"
  upstream:
    backstage:
      image:
        repository: rhdh/rhdh-hub-rhel9
        tag: "<rhdh_version>"
      appConfig:
        ansible:
          rhaap:
            checkSSL: true  # Set to false if using self-signed certificates.
        catalog:
          providers:
            rhaap:
              '{{- include "catalog.providers.env" . }}':
                orgs: "<your-aap-organization-name>"
    postgresql:
      image:
        repository: rhel9/postgresql-<version>
        tag: "<postgresql_tag>"
    Note

    Replace <rhdh_version> and <version> with the versions bundled with your Helm chart. See the Ansible Automation Portal Lifecycle page for version mappings.

     Important

    Set imageRegistry to the registry host only (for example, mirror.example.com). Do not include repository paths. The chart appends the default repository path ansible-automation-platform/automation-portal automatically. If your mirror uses a different repository structure, set ociPluginImage to the full image path instead.

     Note

    The secrets-scm secret is only required if you use private SCM repositories for importing catalog entities or running software templates that access SCM. If you do not use SCM integration, you can skip creating this secret.
  5. Install the chart using the helm install command: 
    $ helm install redhat-rhaap-portal \
  redhat-rhaap-portal-x.y.z.tgz \
  --namespace "${MY_NAMESPACE}" \
  -f /path/to/values-oci.yaml

    To apply changes after deployment, upgrade the Helm release. If you install from the OpenShift web console, use Developer -> Helm -> select the release -> Actions -> Upgrade -> YAML view.

    To upgrade from the command line:

    $ helm upgrade redhat-rhaap-portal \
  redhat-rhaap-portal-x.y.z.tgz \
  --namespace "${MY_NAMESPACE}" \
  -f /path/to/values-oci.yaml

    Alternatively, you can pass values using --set flags.

    OCI delivery:

    $ helm install redhat-rhaap-portal \
  redhat-rhaap-portal-x.y.z.tgz \
  --namespace "${MY_NAMESPACE}" \
  --set redhat-developer-hub.global.clusterRouterBase="${MY_CLUSTER_ROUTER_BASE}" \
  --set redhat-developer-hub.global.imageRegistry="<disconnected_registry_url>" \
  --set redhat-developer-hub.global.pluginMode=oci

    To also set ociPluginImage via --set:

      --set redhat-developer-hub.global.ociPluginImage="<disconnected_registry_url>/custom-path/automation-portal"

    HTTP plug-in registry delivery:

    $ helm install redhat-rhaap-portal \
  redhat-rhaap-portal-x.y.z.tgz \
  --namespace "${MY_NAMESPACE}" \
  --set redhat-developer-hub.global.clusterRouterBase="${MY_CLUSTER_ROUTER_BASE}" \
  --set redhat-developer-hub.global.imageRegistry="<disconnected_registry_url>" \
  --set redhat-developer-hub.global.pluginMode=tarball

Configure CA certificates for private registries
Copy link

If your private registry uses a certificate signed by an internal or self-signed CA, mount the CA certificate into the install-dynamic-plugins init container so that skopeo trusts the registry.

Procedure
Copy link

  1. Obtain the CA certificate chain that signed your mirror registry's TLS certificate.

    If the registry uses a certificate chain, include the full chain:

    $ cat registry.crt intermediate.crt root.crt > ca-bundle.crt
  2. Create a ConfigMap from the CA certificate: 
    $ oc create configmap registry-ca-crt \
  --from-file=ca.crt=ca-bundle.crt -n <namespace>
  3. Add the volume and mount to your values file.
    Important

    Helm replaces arrays entirely rather than merging them. You must include all chart-default volumes, mounts, and init container configuration in your override. If you specify only the CA certificate entries, the chart-default values are removed and the deployment fails.

    To inspect the current chart defaults, run:

    $ helm show values redhat-rhaap-portal-x.y.z.tgz

    Add the registry-ca-crt volume to extraVolumes alongside the chart defaults, and add the CA certificate volume mount to the install-dynamic-plugins init container:

    redhat-developer-hub:
  upstream:
    backstage:
      extraVolumes:
        # -- Chart defaults (do not remove) --
        - name: dynamic-plugins-root
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 2Gi
        - name: audit-log-data
          persistentVolumeClaim:
            claimName: '{{ printf "%s-audit-log" .Release.Name }}'
        - name: dynamic-plugins
          configMap:
            defaultMode: 420
            name: '{{ printf "%s-dynamic-plugins" .Release.Name }}'
            optional: true
        - name: dynamic-plugins-npmrc
          secret:
            defaultMode: 420
            optional: true
            secretName: dynamic-plugins-npmrc
        - configMap:
            defaultMode: 420
            name: rbac-policy
          name: rbac-policy
        - name: npmcacache
          emptyDir: {}
        # -- Custom addition: private registry CA certificate --
        - name: registry-ca-crt
          configMap:
            name: registry-ca-crt
      initContainers:
        - name: install-dynamic-plugins
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1000m
              memory: 2.5Gi
              ephemeral-storage: 5Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            seccompProfile:
              type: "RuntimeDefault"
          image: '{{ include "backstage.image" . }}'
          command:
            - ./install-dynamic-plugins.sh
            - /dynamic-plugins-root
          env:
            - name: NPM_CONFIG_USERCONFIG
              value: /opt/app-root/src/.npmrc.dynamic-plugins
          imagePullPolicy: Always
          volumeMounts:
            # -- Chart defaults (do not remove) --
            - mountPath: /dynamic-plugins-root
              name: dynamic-plugins-root
            - mountPath: /opt/app-root/src/dynamic-plugins.yaml
              name: dynamic-plugins
              readOnly: true
              subPath: dynamic-plugins.yaml
            - mountPath: /opt/app-root/src/.npmrc.dynamic-plugins
              name: dynamic-plugins-npmrc
              readOnly: true
              subPath: .npmrc
            - mountPath: /opt/app-root/src/.npm/_cacache
              name: npmcacache
            # -- Custom addition: private registry CA certificate --
            - name: registry-ca-crt
              mountPath: /etc/containers/certs.d/<registry-host>
              readOnly: true
          workingDir: /opt/app-root/src
    Note

    The mountPath for the CA certificate must be the registry hostname only (for example, /etc/containers/certs.d/mirror.example.com). Do not include repository paths. If the registry uses a non-standard port, include it in the path (for example, /etc/containers/certs.d/mirror.example.com:5000).

Results
Copy link

After the deployment restarts, check the install-dynamic-plugins init container logs for certificate errors:

$ oc logs <pod-name> -c install-dynamic-plugins -n <namespace> | grep -i "x509\|certificate"

If the CA certificate is mounted correctly, there are no x509: certificate signed by unknown authority errors.

Verify the disconnected installation
Copy link

Verify the successful installation of the Helm chart in the disconnected environment. Check the Helm release status, monitor the pods, and verify that the application routes are accessible.

Procedure
Copy link

  1. Check the Helm release status: 
    $ helm list -n ${MY_NAMESPACE}
  2. Monitor the pods in your namespace to ensure they are running: 
    $ oc get pods -n ${MY_NAMESPACE}
  3. Check for ImagePullBackOff or other errors in pod events: 
    $ oc describe pod <pod_name> -n ${MY_NAMESPACE}
  4. If the chart uses routes to expose services, verify that the routes are created and accessible: 
    $ oc get route -n ${MY_NAMESPACE}

Troubleshooting disconnected installations
Copy link

Use this reference to troubleshoot common issues that occur during disconnected Ansible automation portal installations.

Expand
Symptom Cause Solution
authentication required or unauthorized in install-dynamic-plugins init container logs Auth secret missing or malformed. The init container uses skopeo and does not use cluster pull secrets. Create <release-name>-dynamic-plugins-registry-auth secret. Use base64 -w0 to avoid multiline values that corrupt auth.json.
Duplicate path in OCI URL (for example, .../ansible-automation-platform/ansible-automation-platform/...) imageRegistry includes a repository path instead of the registry host only. Set imageRegistry to the registry host only. If your mirror uses a different repository structure, use ociPluginImage to set the full image path.
x509: certificate signed by unknown authority in init container logs Private registry uses a self-signed or internal CA certificate. Mount the CA certificate into the init container. See Configure CA certificates for private registries.