Upgrade Ansible automation portal

Plug-in not found or install tries HTTP plug-in registry

Copy link

Symptom: The install-dynamic-plugins init container logs reference http://plugin-registry:8080/... or report that a plug-in was not found in the in-cluster registry.

Cause: pluginMode is still set to tarball (the chart default) or you did not set pluginMode: oci in the values file used for the upgrade.

Solution:

1. Check effective values:

   helm get values <release_name> -n <namespace>

   To include defaults, add the --all flag.

2. Edit backup-values.yaml: set pluginMode: oci and set imageTagInfo to the plug-in tag from the Ansible automation portal lifecycle page.
3. Upgrade again:

   helm upgrade <release_name> openshift-helm-charts/redhat-rhaap-portal \
     --version <plugin-version> \
     -f backup-values.yaml \
     -n <namespace>

4. Confirm that the init container logs show oci://...automation-portal:... URLs. For more information, see the procedure in the upgrade guide for migrating from tarball to OCI during an upgrade.

Plug-in or OCI pull errors

Copy link

Symptom: The init container fails during OCI pull or install.

Cause: The imageTagInfo value is wrong for the chart version, registry authentication is missing, or pluginMode is not set to oci.

Solution:

1. Confirm that backup-values.yaml sets imageTagInfo to the plug-in tag from the Ansible automation portal lifecycle page for your chart version, and sets pluginMode: oci.
2. Verify that the registry auth secret exists and matches your Helm release name:

   oc get secret <release_name>-dynamic-plugins-registry-auth -n <namespace>

   A secret named redhat-rhaap-portal-dynamic-plugins-registry-auth is incorrect if your release name is not redhat-rhaap-portal. Recreate the secret by following the OCI container delivery procedure in the upgrade guide.

3. Inspect init container logs on the <release_name>-rhaap-portal pod:

   oc logs <pod_name> -c install-dynamic-plugins -n <namespace>

4. Re-upgrade with -f backup-values.yaml.

Plug-in version mismatch (HTTP plug-in registry, deprecated)

Copy link

Symptom: Plug-in load errors occur when pluginMode is set to tarball.

Cause: The plug-in bundle version does not match the Helm chart version.

Solution: Refresh the bundle and update the plug-in registry, or migrate to OCI plug-in delivery. For more information, see the procedures in the upgrade guide for updating the plug-in registry and migrating from tarball to OCI during an upgrade.

Pods stuck in CrashLoopBackOff after upgrade

Copy link

Symptom: Pods restart with a status of CrashLoopBackOff.

Solution:

1. Check the install-dynamic-plugins init container logs and main pod logs:

   oc logs -n <namespace> <pod_name> --previous

2. If logs show database errors, verify database connectivity and secrets.
3. Re-upgrade with -f backup-values.yaml after you fix the values or secrets.

Helm upgrade fails with a release not found error

Copy link

Symptom: Running helm upgrade returns an error stating that the release cannot be found.

Solution:

1. List all Helm releases in your cluster:

   helm list --all-namespaces

2. Identify the correct release name and namespace for your Ansible automation portal deployment.
3. Run the upgrade command with the correct parameters:

   helm upgrade <release_name> openshift-helm-charts/redhat-rhaap-portal \
     --version <plugin-version> \
     -f backup-values.yaml \
     -n <namespace>

Custom values lost after upgrade

Copy link

Symptom: OAuth, RBAC, or certificate settings reverted after the upgrade.

Cause: The upgrade ran without your values file. This commonly occurs when you upgrade from the console without exporting values first.

Solution:

1. If you created backup-values.yaml before the upgrade, re-run the upgrade with the values file:

   helm upgrade <release_name> openshift-helm-charts/redhat-rhaap-portal \
     --version <plugin-version> \
     -f backup-values.yaml \
     -n <namespace>

2. If you did not export values before the upgrade, export what remains and restore missing settings manually:

   helm get values <release_name> -n <namespace> > recovered-values.yaml

   Review recovered-values.yaml, restore any missing settings, and upgrade again with -f recovered-values.yaml.

Authentication failures

Copy link

Symptom: Init container logs show authentication required or unauthorized: access denied.

Cause: The <release-name>-dynamic-plugins-registry-auth secret is missing, malformed, or has incorrect credentials.

Resolution:

1. Verify the secret exists:

   $ oc get secret <release-name>-dynamic-plugins-registry-auth -n <namespace>

2. Check the secret contents:

   $ oc get secret <release-name>-dynamic-plugins-registry-auth \
     -o jsonpath='{.data.auth\.json}' -n <namespace> | base64 -d

   The output should be valid JSON with your credentials.

3. Ensure the secret name matches your Helm release name exactly. For example, if your release is redhat-rhaap-portal, the secret must be redhat-rhaap-portal-dynamic-plugins-registry-auth.
4. Recreate the secret with correct credentials if needed:

   $ oc delete secret <release-name>-dynamic-plugins-registry-auth -n <namespace>
   $ oc create secret generic <release-name>-dynamic-plugins-registry-auth \
     --from-file=auth.json=./auth.json -n <namespace>

Duplicate registry path in OCI URLs

Copy link

Symptom: Init container logs or Helm values show duplicate paths, for example: oci://yb-artifactory/ansible-automation-platform/ansible-automation-platform/automation-portal:2.2.

Cause: The imageRegistry value includes a repository path instead of being the registry host only.

Resolution: Edit your backup-values.yaml and set imageRegistry to the registry host only, without the repository path:

# Incorrect (includes repository path):
imageRegistry: "yb-artifactory/ansible-automation-platform"

# Correct (host only):
imageRegistry: "yb-artifactory"

Then re-run helm upgrade with the corrected values file.

x509 certificate errors for private registries

Copy link

Symptom: Init container logs show x509: certificate signed by unknown authority or x509: certificate has expired.

Cause: Your mirror registry uses a self-signed or internal CA certificate that the skopeo utility cannot verify.

Resolution: Mount your CA certificate into the init container at the per-registry trust path. Obtain your CA certificate bundle (including the full chain), create a ConfigMap, and update your Helm values to mount it. For detailed instructions, see the RHDH documentation: Install plugins from OCI registries by using custom certificates.

No such image error

Copy link

Symptom: Init container logs show Error: no such image or manifest not found.

Cause: The OCI image does not exist in the specified registry, or the imageTagInfo version does not match what is available.

Resolution:

1. Verify the imageTagInfo value in your Helm release matches an available version. Check the Ansible automation portal lifecycle page.
2. If using a mirror registry, ensure the image was mirrored to the correct path. Run:

   $ podman search <your-mirror-registry>/ansible-automation-platform/automation-portal

3. Confirm that imageRegistry is set correctly and does not include a duplicate repository path (see the Duplicate registry path section above).

Integrity check errors

Copy link

Symptom: Init container logs show integrity check failed or digest mismatch.

Cause: The OCI image in your registry does not match the expected digest, or the image was corrupted during mirroring.

Resolution:

1. Re-mirror the image from registry.redhat.io using skopeo copy instead of podman tag/push. The skopeo copy command preserves the original manifest digest:

   $ skopeo copy \
     docker://registry.redhat.io/ansible-automation-platform/automation-portal:<plugin-version> \
     docker://<your-mirror-registry>/ansible-automation-platform/automation-portal:<plugin-version>

2. If the error persists, verify the image in your mirror registry:

   $ skopeo inspect docker://<your-mirror-registry>/ansible-automation-platform/automation-portal:<plugin-version>

3. Ensure you are not using a stale local image cache. Delete and re-pull if needed.

Accessing init container logs

Copy link

To view the logs of the install-dynamic-plugins init container:

$ oc get pods -n <namespace> -l app.kubernetes.io/component=backstage
$ oc logs <pod-name> -c install-dynamic-plugins -n <namespace>

If the pod has already completed (e.g., if the init container succeeded), you can view the previous logs:

$ oc logs <pod-name> -c install-dynamic-plugins --previous -n <namespace>

If the pod is still running, stream logs in real time:

$ oc logs -f <pod-name> -c install-dynamic-plugins -n <namespace>