About Ansible automation portal RHEL appliances

Replace the SSL certificates

Copy link

Prerequisites:

• A TLS certificate and private key in PEM format, issued by a certificate authority trusted by your organization.
• SSH access to the appliance.

Procedure:

1. Copy your certificates to the appliance:

   $ sudo cp certificate-file.pem /etc/portal/ssl/cert.pem
   $ sudo cp private-key-file.pem /etc/portal/ssl/key.pem
   $ sudo chmod 644 /etc/portal/ssl/cert.pem
   $ sudo chmod 600 /etc/portal/ssl/key.pem

2. Restart the Ansible automation portal service:

   $ sudo systemctl restart portal

Verification:

• Verify that the Ansible automation portal is using the new certificate:

  $ curl -vk https://localhost 2>&1 | grep -i "issuer"

  The output displays the issuer from your certificate, not the self-signed issuer.

Trust a custom certificate authority

Copy link

If your organization uses a custom certificate authority (CA) or a self-signed certificate for your Ansible Automation Platform instance, configure the Ansible automation portal RHEL appliance to trust it.

Prerequisites:

• The CA certificate or self-signed certificate in PEM format.
• SSH access to the appliance.

Procedure:

1. Copy the CA certificate or self-signed certificate to the appliance:

   $ sudo cp ca-certificate-file.pem /etc/portal/ssl/ca-bundle.crt

   If you have multiple CA certificates to trust, concatenate them into a single bundle file before copying it to the appliance.

2. Create a Quadlet drop-in to set the NODE_EXTRA_CA_CERTS environment variable:

   $ sudo mkdir -p /etc/containers/systemd/portal.container.d
   $ echo -e '[Container]\nEnvironment=NODE_EXTRA_CA_CERTS=/etc/portal/ssl/ca-bundle.crt' \
     | sudo tee /etc/containers/systemd/portal.container.d/ca-certs.conf

3. Reload systemd and restart the Ansible automation portal service:

   $ sudo systemctl daemon-reload
   $ sudo systemctl restart portal

Verification:

• Verify that the environment variable is set in the portal container:

  $ sudo podman exec portal env | grep NODE_EXTRA_CA_CERTS

  The output displays NODE_EXTRA_CA_CERTS=/etc/portal/ssl/ca-bundle.crt.

When you upgrade, the entire system image is replaced atomically. If an upgrade causes issues, you can roll back to the previous image in one command.

Bootc downloads only the layers that changed between the current and new image, making incremental upgrades fast and bandwidth-efficient.

Bootc divides the filesystem into three categories that determine what happens during an upgrade:

Your configuration files in /etc are preserved through upgrades using a three-way merge: bootc compares the original file from the old image, your modified version, and the new file from the new image. Your changes take precedence. Files in /var (database, backups) are never modified by bootc.

For more information about RHEL image mode, see Managing RHEL bootc images.

Prerequisites:

• Back up and restore data before upgrading.
• For connected upgrades, authenticate to registry.redhat.io (see the Authenticate to the container registry section).
• For disconnected upgrades, Upgrade the Ansible automation portal RHEL appliance in a disconnected environment.

Authenticate to the container registry

Copy link

To pull new appliance images from registry.redhat.io, authenticate to the registry and save the credentials where bootc can find them.

Procedure:

1. SSH into the Ansible automation portal RHEL appliance and log in to the container registry:

   $ sudo podman login --authfile /etc/ostree/auth.json registry.redhat.io

Upgrade the appliance

Copy link

Procedure:

1. Back up and restore data.
2. Run the upgrade:

   $ sudo bootc upgrade

   The upgrade is staged and does not take effect until you reboot. The current version continues running. You can schedule the reboot for a convenient maintenance window.

   bootc upgrade pulls the latest image for the current tag. To upgrade to a specific version, use bootc switch with a version tag:

   $ sudo bootc switch registry.redhat.io/ansible-automation-platform/bootc-automation-portal-rhel9:version

3. Reboot to activate the new image:

   $ sudo systemctl reboot

Verification:

• Verify that the new image is booted:

  $ sudo bootc status

  The booted digest matches the digest from the upgrade output. The previous version is retained as a rollback target.

• Check the portal service logs:

  $ sudo journalctl -u portal -f

• Verify that all services are running:

  $ sudo systemctl status portal postgres devtools

  All three services (portal, postgres, devtools) show active (running).

Roll back an upgrade

Copy link

Bootc maintains two image slots: the booted image and one rollback image. After an upgrade, the previous version becomes the rollback target. After a rollback, the upgraded version becomes the rollback target. You can switch between the two versions as needed.

If the new version has issues, roll back to the previous image.

Procedure:

1. Roll back to the previous image:

   $ sudo bootc rollback

2. Reboot to activate the rollback image:

   $ sudo systemctl reboot

Verification:

• Confirm the rollback was applied:

  $ sudo bootc status

  The booted image shows the previous digest. The upgraded image is now listed as the rollback target.

• Verify that all services are running:

  $ sudo systemctl status portal postgres devtools

Post-upgrade reconciliation

Copy link

After reboot, the appliance automatically reconciles Ansible plugins and configuration with the new image version:

1. Clears the dynamic plugins directory so that plugins are regenerated from the new image.
2. Clears generated configuration files (regenerated on portal start).
3. Creates any new directories required by the new image version.
4. Fixes file ownership for portal and PostgreSQL directories.

SSH access (standard cloud-init)

Copy link

Ansible Automation Platform connection

Copy link

Database

Copy link

Security

Copy link

Network

Copy link

Source control integrations

Copy link

Accessing the appliance

Copy link

You can access the appliance using SSH with the key you provided during initial configuration:

ssh -i /path/to/ssh-key/id_ed25519 -p port_number admin@VM_IP

Replace the following placeholders:

• /path/to/ssh-key/id_ed25519 with the path to your SSH private key.
• port_number with the SSH port number (default is 22).
• VM_IP with the IP address or hostname of your appliance.

The ansible-portal CLI

Copy link

The ansible-portal command is the primary management interface for the appliance, following Red Hat Ansible tooling conventions (ansible-navigator, ansible-builder, ansible-lint).

ansible-portal <command> [options]

Run ansible-portal command --help for command-specific options.

ansible-portal status

Copy link

Displays the current status of all Ansible automation portal services.

Usage:

sudo ansible-portal status            # One-shot status display
sudo ansible-portal status --watch    # Refresh status every 5 seconds

Description:

Displays the current status of all Ansible automation portal services, including:

• Setup completion status
• PostgreSQL database state and connectivity
• Devtools service state
• Portal service state and plug-in installation status
• Scheduled backup status
• Disk and memory resource usage

Use this command to verify that all services are running correctly after installation or troubleshooting. Use --watch to continuously monitor service status.

ansible-portal backup

Copy link

Creates a backup of the portal configuration and data.

Usage:

sudo ansible-portal backup                          # Interactive backup (select content)
sudo ansible-portal backup --full                   # Full backup (all items)
sudo ansible-portal backup --minimal                # Required items only
sudo ansible-portal backup --list                   # List existing backups
sudo ansible-portal backup --export /path/to/dir/   # Copy latest backup to a directory

Description:

Creates a backup archive containing the portal configuration, Podman secrets, and data. Use this command before making significant configuration changes or for disaster recovery planning.

• backup without options starts an interactive backup where you select the content to include.
• --full creates a complete backup of all configuration and data.
• --minimal backs up only the required configuration items.
• --list lists existing backups.
• --export copies the latest backup to the specified directory. If no backup exists, one is created first.

gpg --symmetric --cipher-algo AES256 /var/lib/portal/backups/backup-file.tar.gz

Example output:

Portal Backup Tool
==================
1) Full backup (config + database + secrets) [recommended]
2) Config only (config + secrets, no database)
3) Database only

Select backup type [1]: 1

Creating backup...
  Backing up configuration... done
  Backing up secrets... done
  Backing up database... done

Backup created: /var/lib/portal/backups/portal-backup-2026-05-07-143022.tar.gz

ansible-portal restore

Copy link

Restores the portal from a backup.

Usage:

sudo ansible-portal restore --list                  # List available backups
sudo ansible-portal restore --latest                # Restore the newest backup
sudo ansible-portal restore --latest --dry-run      # Preview restore without changes
sudo ansible-portal restore /path/to/backup.tar.gz  # Restore a specific backup file

Description:

Restores the portal configuration, Podman secrets, Quadlet drop-in files, and data from a backup created by ansible-portal backup.

• --list lists all available backups.
• --latest restores from the most recent backup.
• --dry-run previews what would be restored without making changes.
• Specify a file path to restore from a specific backup archive.

Example output:

Portal Restore Tool
===================
Restoring from: /var/lib/portal/backups/portal-backup-2026-05-07-143022.tar.gz

Stopping services...
  Restoring configuration... done
  Restoring secrets... done
  Restoring database... done
Starting services...

Restore complete.

Services stop during restore and restart automatically.

ansible-portal registry-login

Copy link

Manages container registry credentials for pulling portal images.

Usage:

sudo ansible-portal registry-login                 # Log in to registry.redhat.io (default)
sudo ansible-portal registry-login registry_host # Log in to a specific registry
sudo ansible-portal registry-login --test          # Test stored credentials
sudo ansible-portal registry-login --logout        # Remove stored credentials

Description:

Manages authentication to container registries used for pulling Ansible automation portal images and updates.

• registry-login without arguments prompts for credentials for registry.redhat.io.
• Specify a registry hostname to authenticate to a different registry, such as a private mirror.
• --test verifies that stored credentials are valid by inspecting the registry without downloading images.
• --logout clears stored credentials for the registry.

Use this command to configure registry authentication for bootc upgrade operations. In disconnected environments, use this command to authenticate to your private registry mirror.

In disconnected environments where no registry is available, you can bypass the registry authentication gate by creating an override marker:

sudo touch /etc/portal/.registry-auth-override

Configuration management

Copy link

The appliance uses two YAML configuration files at /etc/portal/configs/app-config/:

app-config.yaml

Infrastructure configuration (base URL, database connection, TLS, system defaults). This file is set during initial deployment and is not typically modified.

app-config.production.yaml

Application configuration (Ansible Automation Platform connection, OAuth settings, SCM integrations, catalog sync). Edit this file for day-to-day configuration changes.

Sensitive values such as tokens and passwords are stored as Podman secrets and injected into the portal container through environment variable interpolation (${VAR} syntax in the YAML files).

Updating configuration

To update the portal configuration:

1. Edit the application configuration file:

   sudo vi /etc/portal/configs/app-config/app-config.production.yaml

2. Restart the portal service to apply the changes:

   sudo systemctl restart portal

Managing secrets

Copy link

The portal stores sensitive values as Podman secrets, separate from the configuration files.

Listing secrets

sudo podman secret ls

Updating a secret

To update a secret value, remove the existing secret and create a new one. Write the value to a temporary file on tmpfs to avoid exposing it in the process list:

sudo podman secret rm portal_aap_token
temp_file=$(mktemp -p /dev/shm) && chmod 600 "$temp_file"
printf '%s' "new_token_value" > "$temp_file"
sudo podman secret create portal_aap_token "$temp_file"
rm -f "$temp_file"
sudo systemctl restart portal

View logs and service status

Copy link

Start troubleshooting by checking the status of all services and reviewing recent logs.

Check the status of all services:

$ sudo systemctl status portal postgres devtools

A service that failed to start shows active (failed) or inactive (dead).

View recent portal logs:

$ sudo journalctl -u portal -n 200 --no-pager

Check the bootc image status:

$ sudo bootc status

Portal services do not start

Copy link

Symptom: The Ansible automation portal service is not running after boot.

Cause: The Ansible automation portal RHEL appliance requires Ansible Automation Platform credentials in the cloud-init user-data to complete setup. If you provided SSH keys but no aap: section, you can SSH in but Ansible automation portal services do not start.

Resolution:

1. Check the portal service logs for errors:

   $ sudo journalctl -u portal -b --no-pager

2. Verify that your cloud-init user-data includes the required aap: section with host_url, token, client_id, and client_secret.
3. If the configuration is missing or incorrect, edit the configuration file directly:

   $ sudo vi /etc/portal/configs/app-config/app-config.production.yaml

   Update ansible.rhaap.baseUrl, auth.providers.rhaap.production.clientId, and other required values.

4. If Ansible Automation Platform credentials need to be updated, use Podman secrets:

   $ temp_file=$(mktemp -p /dev/shm) && chmod 600 "$temp_file"
   $ printf '%s' "aap-token" > "$temp_file"
   $ sudo podman secret rm portal_aap_token
   $ sudo podman secret create portal_aap_token "$temp_file"
   $ rm -f "$temp_file"

   $ temp_file=$(mktemp -p /dev/shm) && chmod 600 "$temp_file"
   $ printf '%s' "oauth-client-secret" > "$temp_file"
   $ sudo podman secret rm portal_aap_oauth_client_secret
   $ sudo podman secret create portal_aap_oauth_client_secret "$temp_file"
   $ rm -f "$temp_file"

5. Restart the Ansible automation portal service:

   $ sudo systemctl restart portal

OAuth login fails

Copy link

Symptom: After clicking Sign in with RHAAP, the browser redirects back to the login page or Ansible Automation Platform returns "Invalid redirect_uri."

Cause: The OAuth redirect URI in Ansible Automation Platform does not match the Ansible automation portal user-accessible URL, or the app.baseUrl, backend.baseUrl, and backend.cors.origin values in the configuration file are not consistent with each other.

Resolution:

1. Check the current user-accessible URL and CORS configuration:

   $ grep -E 'baseUrl|origin' /etc/portal/configs/app-config/app-config.production.yaml

2. Verify that app.baseUrl, backend.baseUrl, and backend.cors.origin all use the same address.
3. Verify that the OAuth application redirect URI in Ansible Automation Platform matches the format https://portal-address/api/auth/rhaap/handler/frame.
4. If the values do not match, update either:
   • The OAuth redirect URI in Ansible Automation Platform (if the Ansible automation portal address is correct), or
   • The Ansible automation portal user-accessible URL and CORS origin in app-config.production.yaml (if the OAuth URI is correct).
5. Restart the Ansible automation portal service after any configuration change:

   $ sudo systemctl restart portal

GitHub login appears instead of RHAAP

Copy link

Symptom: The login page shows a "Sign in with GitHub" option instead of "Sign in with RHAAP," or both options appear.

Cause: The app.baseUrl, backend.baseUrl, and backend.cors.origin values in the configuration file do not match the URL used to access the Ansible automation portal RHEL appliance. This commonly happens when accessing the VM by IP address but the configuration uses a domain name, or vice versa. The CORS mismatch can cause the RHAAP provider to fail silently, falling back to a GitHub provider if one is configured.

Resolution:

1. Check the configured authentication providers:

   $ grep -A 5 'auth:' /etc/portal/configs/app-config/app-config.production.yaml

2. Verify that the auth.providers section contains rhaap and does not contain github. If a github provider is present, remove it.
3. Verify that app.baseUrl, backend.baseUrl, and backend.cors.origin all match the URL you use to access the Ansible automation portal RHEL appliance in your browser:

   $ grep -E 'baseUrl|origin' /etc/portal/configs/app-config/app-config.production.yaml

4. Restart the Ansible automation portal service after any configuration change:

   $ sudo systemctl restart portal

Login fails with a 500 error

Copy link

Symptom: After clicking Sign in with RHAAP, the browser shows a 500 Internal Server Error or redirects to an error page instead of completing the login.

Cause: A 500 error during login typically indicates that the Ansible Automation Platform API token or OAuth client secret is invalid, expired, or missing. The Ansible automation portal cannot authenticate with the Ansible Automation Platform instance.

Resolution:

1. Check the portal logs for authentication errors:

   $ sudo journalctl -u portal -n 100 --no-pager | grep -i -E 'auth|oauth|401|403|500|error'

2. Verify that the required Podman secrets exist:

   $ sudo podman secret ls | grep -E 'aap_token|oauth'

   You should see both portal_aap_token and portal_aap_oauth_client_secret listed. If either is missing or the values need to be updated, recreate them:

   $ temp_file=$(mktemp -p /dev/shm) && chmod 600 "$temp_file"
   $ printf '%s' "new-aap-token" > "$temp_file"
   $ sudo podman secret rm portal_aap_token
   $ sudo podman secret create portal_aap_token "$temp_file"
   $ rm -f "$temp_file"

3. Verify that the Ansible Automation Platform instance is reachable from the appliance:

   $ curl -sk https://aap-host/api/v2/ping/

4. Verify that the OAuth client ID in the configuration matches the OAuth application in Ansible Automation Platform:

   $ grep clientId /etc/portal/configs/app-config/app-config.production.yaml

5. Restart the Ansible automation portal service after any changes:

   $ sudo systemctl restart portal

Cannot SSH to the VM

Copy link

Symptom: SSH connection refused or permission denied.

Cause: The appliance enforces SSH key-only authentication. Password login is disabled.

Resolution:

• Verify that your SSH private key matches the public key you provided in the cloud-init user-data.
• Verify that you are using the correct username from your cloud-init configuration.

If you have lost access to your SSH key, use platform-specific recovery:

• KVM: Access the VM console with virsh console vm-name and add a new SSH key.
• Red Hat OpenShift Virtualization: Access the VM console with virtctl console vm-name -n namespace.
• VMware vSphere: Use the vSphere console with systemd.unit=rescue.target boot parameter to add a new SSH key.

VM is inaccessible

Copy link

Symptom: The VM was deployed without cloud-init user-data. No SSH keys or Ansible Automation Platform credentials are configured.

Cause: The appliance has no default password. Without cloud-init configuration, the VM is not accessible.

Resolution: Redeploy the VM with a cloud-init configuration attached. See Configure the appliance at first boot.

Red Hat OpenShift Virtualization issues

Copy link

Symptom: Upload, boot, or scheduling failures when deploying on Red Hat OpenShift Virtualization.

Resolution:

Upload fails with "effective image size is larger than available storage"

Increase --size (for example, from 40 Gi to 50 Gi). Check the virtual size with qemu-img info disk.qcow2.

Upload hangs at "Waiting for PVC upload pod to be ready"

The storage class uses WaitForFirstConsumer binding. Add --force-bind to the virtctl command.

VM shows "Boot failed: not a bootable disk"

The disk image was not properly converted during upload. Verify the upload completed with "Processing completed successfully." If processing was interrupted, delete the DataVolume and PVC and re-upload.

VM stuck in Scheduling with "Insufficient memory"

Reduce the VM memory request or add cluster resources.

VMware vSphere troubleshooting

Copy link

Disk conversion fails with vmkfstools

• Verify that you have SSH access to the ESXi host.
• Verify that the disk.vmdk file was uploaded successfully to the datastore.
• Check that you have sufficient permissions to run vmkfstools on the ESXi host.
• Verify that there is sufficient free space on the datastore for the converted disk.

Virtual machine fails to boot after attaching the disk

• Verify that you selected the correct disk file.
• Verify that the disk is attached to the correct SCSI controller and bus.
• Check the virtual machine boot order settings and ensure the hard disk is the first boot device.

Cannot upload disk image to datastore

• Verify that your user account has permissions to upload files to the datastore.
• Check that there is sufficient free space on the datastore.
• Verify that the vSphere web client is properly connected.

Collect diagnostic information

Copy link

If an issue persists, collect the following information before contacting support:

$ sudo systemctl status portal postgres devtools > service-status.txt
$ sudo journalctl -u portal -n 200 > portal-logs.txt
$ sudo journalctl -u postgres -n 100 > postgres-logs.txt
$ sudo journalctl -u devtools -n 100 > devtools-logs.txt
$ sudo bootc status > bootc-status.txt
$ cat /etc/os-release > os-release.txt

SSH key recovery

Copy link

After initial configuration, the admin user account is locked and console login is disabled. If you lose SSH access, use one of the following recovery methods for your deployment environment.

Recovery on Red Hat OpenShift Virtualization

Copy link

1. Access the virtual machine console through the Red Hat OpenShift Container Platform web console.
2. Reboot the virtual machine.
3. At the GRUB boot menu, press e to edit boot parameters.
4. Add init=/bin/bash to the end of the kernel line.
5. Press Ctrl+X to boot into an emergency shell.
6. Remount the filesystem and add a new SSH key:

   mount -o remount,rw /
   passwd -u admin
   mkdir -p /home/admin/.ssh
   echo "your_ssh_public_key" >> /home/admin/.ssh/authorized_keys
   chown -R admin:admin /home/admin/.ssh
   chmod 600 /home/admin/.ssh/authorized_keys
   restorecon -R /home/admin/.ssh
   passwd -l admin
   exec /sbin/init

Recovery on VMware vSphere

Copy link

1. Open the virtual machine console in vCenter.
2. Reboot the virtual machine.
3. At the GRUB boot menu, press e to edit boot parameters.
4. Add init=/bin/bash to the end of the kernel line.
5. Press Ctrl+X to boot into an emergency shell.
6. Remount the filesystem and add a new SSH key:

   mount -o remount,rw /
   passwd -u admin
   mkdir -p /home/admin/.ssh
   echo "your_ssh_public_key" >> /home/admin/.ssh/authorized_keys
   chown -R admin:admin /home/admin/.ssh
   chmod 600 /home/admin/.ssh/authorized_keys
   restorecon -R /home/admin/.ssh
   passwd -l admin
   exec /sbin/init

Recovery on KVM

Copy link

Shut down the virtual machine and mount the QCOW2 image directly to add a new SSH key:

$ sudo modprobe nbd max_part=8
$ sudo qemu-nbd --connect=/dev/nbd0 disk.qcow2
$ sudo fdisk -l /dev/nbd0
$ sudo mount /dev/nbd0pN /mnt
$ sudo mkdir -p /mnt/home/admin/.ssh
$ echo "your_ssh_public_key" | sudo tee -a /mnt/home/admin/.ssh/authorized_keys
$ sudo chown 1000:1000 /mnt/home/admin/.ssh/authorized_keys
$ sudo chmod 600 /mnt/home/admin/.ssh/authorized_keys
$ sudo umount /mnt
$ sudo qemu-nbd --disconnect /dev/nbd0

Where:

• fdisk -l /dev/nbd0 lists partitions to identify the root filesystem. The root partition is typically the largest partition.
• Replace N in /dev/nbd0pN with the correct root partition number from the fdisk output.

Account locking behavior

Copy link

When the admin account is locked with passwd -l, the system adds a ! prefix to the password hash in /etc/shadow. This disables all password-based authentication (console login, SSH password auth) while SSH key authentication continues to work.

To temporarily unlock the account for recovery, use passwd -u admin. After adding a new SSH key, lock the account again with passwd -l admin for security.