Red Hat Ansible Automation Platform operator backup and recovery guide
Safeguard against data loss with backup and recovery of Ansible Automation Platform operator on OpenShift Container Platform
Abstract
Preface
Thank you for your interest in Red Hat Ansible Automation Platform. Ansible Automation Platform is a commercial offering that helps teams manage complex multi-tier deployments by adding control, knowledge, and delegation to Ansible-powered environments.
Use the procedures in this guide to create backup resources that can be used for recovering your Red Hat Ansible Automation Platform deployment in the event of a failure.
Providing feedback on Red Hat documentation
If you have a suggestion to improve this documentation, or find an error, you can contact technical support at https://access.redhat.com to open a request.
Chapter 1. Backup and recovery of Red Hat Ansible Automation Platform
To safeguard against unexpected data loss and application errors, it is critical that you perform periodic backups of your Red Hat Ansible Automation Platform deployment. In addition to data loss prevention, backups allow you to fall back to a different deployment state.
1.1. About backup and recovery
Red Hat recommends backing up deployments of Red Hat Ansible Automation Platform in your Red Hat OpenShift Container Platform environment to prevent data loss.
A backup resource of your Red Hat Ansible Automation Platform deployment includes the following:
-
Custom deployment of specific values in the
spec
section of the Ansible Automation Platform custom resource object -
Back up of the
postgresql
database -
secret_key
,admin_password
, andbroadcast_websocket
secrets - Database configuration
Be sure to secure your backup resources because they can include sensitive information.
1.1.1. Backup recommendations
Recovering from data loss requires that you plan for and create backup resources of your Red Hat Ansible Automation Platform deployments on a regular basis. At a minimum, Red Hat recommends backing up deployments of Red Hat Ansible Automation Platform under the following circumstances:
- Before upgrading your Red Hat Ansible Automation Platform deployments
- Before upgrading your Openshift cluster
- Once per week. This is particularly important if your environment is configured for automatic upgrades.
Chapter 2. Creating Red Hat Ansible Automation Platform backup resources
Backing up your Red Hat Ansible Automation Platform deployment involves creating backup resources for your deployed automation hub and automation controller instances. Use these procedures to create backup resources for your Red Hat Ansible Automation Platform deployment.
2.1. Backing up the Automation controller deployment
Use this procedure to back up a deployment of the controller, including jobs, inventories, and credentials.
Prerequisites
- You must be authenticated with an Openshift cluster.
- The Ansible Automation Platform Operator has been installed to the cluster.
- The automation controller is deployed to using the Ansible Automation Platform Operator.
Procedure
- Log in to Red Hat OpenShift Container Platform.
- Navigate to → .
- Select the Ansible Automation Platform Operator installed on your project namespace.
- Select the Automation Controller Backup tab.
- Click .
- Enter a Name for the backup.
-
Enter the Deployment name of the deployed Ansible Automation Platform instance being backed up. For example, if your automation controller must be backed up and the deployment name is
aap-controller
, enter 'aap-controller' in the Deployment name field. If you want to use a custom, pre-created pvc:
- Optionally enter the name of the Backup persistant volume claim.
Optionally enter the Backup PVC storage requirements, and Backup PVC storage class.
NoteIf no pvc or storage class is provided, the cluster’s default storage class is used to create the pvc.
If you have a large database, specify your storage requests accordingly under Backup management pod resource requirements.
NoteYou can check the size of the existing postgres database data directory by running the following command inside the postgres pod.
$ df -h | grep "/var/lib/pgsql/data"
Click
.A backup tarball of the specified deployment is created and available for data recovery or deployment rollback. Future backups are stored in separate tar files on the same pvc.
Verification
- Log in to Red Hat Red Hat OpenShift Container Platform
- Navigate to → .
- Select the Ansible Automation Platform Operator installed on your project namespace.
- Select the AutomationControllerBackup tab.
- Select the backup resource you want to verify.
Scroll to Conditions and check that the Successful status is
True
.NoteIf Successful is
False
, the backup has failed. Check the automation controller operator logs for the error to fix the issue.
2.2. Using YAML to back up the Automation controller deployment
See the following procedure for how to back up a deployment of the automation controller using YAML.
Prerequisites
- You must be authenticated with an OpenShift cluster.
- The Ansible Automation Platform Operator has been installed to the cluster.
- The automation controller is deployed to using the Ansible Automation Platform Operator.
Procedure
Create a file named "backup-awx.yml" with the following contents:
--- apiVersion: automationcontroller.ansible.com/v1beta1 kind: AWXBackup metadata: name: awxbackup-2024-07-15 namespace: my-namespace spec: deployment_name: controller
NoteThe "deployment_name" above is the name of the automation controller deployment you intend to backup from. The namespace above is the one containing the automation controller deployment you intend to back up.
-
Use the
oc apply
command to create the backup object in your cluster:
$ oc apply -f backup-awx.yml
2.3. Backing up the Automation hub deployment
Use this procedure to back up a deployment of the hub, including all hosted Ansible content.
Prerequisites
- You must be authenticated with an Openshift cluster.
- The Ansible Automation Platform Operator has been installed to the cluster.
- The automation hub is deployed to using the Ansible Automation Platform Operator.
Procedure
- Log in to Red Hat OpenShift Container Platform.
- Navigate to → .
- Select the Ansible Automation Platform Operator installed on your project namespace.
- Select the Automation Hub Backup tab.
- Click .
- Enter a Name for the backup.
-
Enter the Deployment name of the deployed Ansible Automation Platform instance being backed up. For example, if your automation hub must be backed up and the deployment name is
aap-hub
, enter 'aap-hub' in the Deployment name field. If you want to use a custom, pre-created pvc:
- Optionally, enter the name of the Backup persistent volume claim, Backup persistent volume claim namespace, Backup PVC storage requirements, and Backup PVC storage class.
Click
.A backup of the specified deployment is created and available for data recovery or deployment rollback.
Chapter 3. Recovering a Red Hat Ansible Automation Platform deployment
If you lose information on your system or issues with an upgrade, you can use the backup resources of your deployment instances. Use these procedures to recover your automation controller and automation hub deployment files.
3.1. Recovering the Automation controller deployment
Use this procedure to restore a previous controller deployment from an AutomationControllerBackup. The deployment name you provide will be the name of the new AutomationController custom resource that will be created.
The name specified for the new AutomationController custom resource must not match an existing deployment.
If the backup custom resource being restored is a backup of a currently running AutomationController custom resource the recovery process will fail. See Troubleshooting for steps to resolve the issue.
Prerequisites
- You must be authenticated with an Openshift cluster.
- The automation controller has been deployed to the cluster.
- An AutomationControllerBackup is available on a PVC in your cluster.
Procedure
- Log in to Red Hat OpenShift Container Platform.
- Navigate to → .
- Select the Ansible Automation Platform Operator installed on your project namespace.
- Select the Automation Controller Restore tab.
- Click .
- Enter a Name for the recovery deployment.
Enter a New Deployment name for the restored deployment.
NoteThis should be different from the original deployment name.
- Select the Backup source to restore from. Backup CR is recommended.
- Enter the Backup Name of the AutomationControllerBackup object.
Click
.A new deployment is created and your backup is restored to it. This can take approximately 5 to 15 minutes depending on the size of your database.
Verification
- Log in to Red Hat Red Hat OpenShift Container Platform
- Navigate to → .
- Select the Ansible Automation Platform Operator installed on your project namespace.
- Select the AutomationControllerRestore tab.
- Select the restore resource you want to verify.
Scroll to Conditions and check that the Successful status is
True
.NoteIf Successful is
False
, the recovery has failed. Check the automation controller operator logs for the error to fix the issue.
3.2. Using YAML to recover the Automation controller deployment
See the following procedure for how to restore a deployment of the automation controller using YAML.
Prerequisite
The external database must be a PostgreSQL database that is the version supported by the current release of Ansible Automation Platform.
Ansible Automation Platform 2.4 supports PostgreSQL 13.
Procedure
The external postgres instance credentials and connection information must be stored in a secret, which is then set on the automation controller spec.
Create a
external-postgres-configuration-secret
YAML file, following the template below:apiVersion: v1 kind: Secret metadata: name: external-restore-postgres-configuration namespace: <target_namespace> 1 stringData: host: "<external_ip_or_url_resolvable_by_the_cluster>" 2 port: "<external_port>" 3 database: "<desired_database_name>" username: "<username_to_connect_as>" password: "<password_to_connect_with>" 4 sslmode: "prefer" 5 type: "unmanaged" type: Opaque
- 1
- Namespace to create the secret in. This should be the same namespace you wish to deploy to.
- 2
- The resolvable hostname for your database node.
- 3
- External port defaults to
5432
. - 4
- Value for variable
password
should not contain single or double quotes (', ") or backslashes (\) to avoid any issues during deployment, backup or restoration. - 5
- The variable
sslmode
is valid forexternal
databases only. The allowed values are:prefer
,disable
,allow
,require
,verify-ca
, andverify-full
.
Apply
external-postgres-configuration-secret.yml
to your cluster using theoc create
command.$ oc create -f external-postgres-configuration-secret.yml
When creating your
AutomationControllerRestore
custom resource object, specify the secret on your spec, following the example below:kind: AutomationControllerRestore apiVersion: automationcontroller.ansible.com/v1beta1 metadata: namespace: my-namespace name: awxrestore-2024-07-15 spec: deployment_name: restored_controller backup_name: awxbackup-2024-07-15 postgres_configuration_secret: 'external-restore-postgres-configuration'
3.3. Recovering the Automation hub deployment
Use this procedure to restore a previous hub deployment into the namespace. The deployment name you provide will be the name of the new AutomationHub custom resource that will be created.
The name specified for the new AutomationHub custom resource must not match an existing deployment or the recovery process will fail.
Prerequisites
- You must be authenticated with an Openshift cluster.
- The automation hub has been deployed to the cluster.
- An AutomationHubBackup is available on a PVC in your cluster.
Procedure
- Log in to Red Hat OpenShift Container Platform.
- Navigate to → .
- Select the Ansible Automation Platform Operator installed on your project namespace.
- Select the Automation Hub Restore tab.
- Click .
- Enter a Name for the recovery deployment.
- Select the Backup source to restore from. Backup CR is recommended.
- Enter the Backup Name of the AutomationHubBackup object.
Click
.A new deployment is created and your backup is restored to it.
Chapter 4. Troubleshooting
Use this information to diagnose and resolve issues during backup and recovery.
4.1. Automation controller custom resource has the same name as an existing deployment
The name specified for the new AutomationController custom resource must not match an existing deployment or the recovery process will fail.
If your AutomationController customer resource matches an existing deployment, perform the following steps to resolve the issue.
Procedure
Delete the existing AutomationController and the associated postgres PVC:
oc delete automationcontroller <YOUR_DEPLOYMENT_NAME> -n <YOUR_NAMESPACE> oc delete pvc postgres-13-<YOUR_DEPLOYMENT_NAME>-13-0 -n <YOUR_NAMESPACE>
Use AutomationControllerRestore with the same deployment_name in it:
oc apply -f restore.yaml