Chapter 2. Automation mesh for operator-based Red Hat Ansible Automation Platform
Scaling your automation mesh is available on OpenShift deployments of Red Hat Ansible Automation Platform and is possible through adding or removing nodes from your cluster dynamically, using the Instances resource of the Ansible Automation Platform UI, without running the installation script.
Instances serve as nodes in your mesh topology. Automation mesh enables you to extend the footprint of your automation. The location where you launch a job can be different from the location where the ansible-playbook runs.
To manage instances from the Ansible Automation Platform UI, you must have System Administrator or System Auditor permissions.
In general, the more processor cores (CPU) and memory (RAM) a node has, the more jobs that can be scheduled to run on that node at once.
For more information, see Automation controller capacity determination and job impact.
2.1. Prerequisites Copy linkLink copied to clipboard!
The automation mesh is dependent on hop and execution nodes running on Red Hat Enterprise Linux (RHEL). Your Red Hat Ansible Automation Platform subscription grants you ten Red Hat Enterprise Linux licenses that can be used for running components of Ansible Automation Platform.
For more information about Red Hat Enterprise Linux subscriptions, see Registering the system and managing subscriptions in the Red Hat Enterprise Linux documentation.
The following steps prepare the RHEL instances for deployment of the automation mesh.
- You require a Red Hat Enterprise Linux operating system. Each node in the mesh requires a static IP address, or a resolvable DNS hostname that Ansible Automation Platform can access.
- Ensure that you have the minimum requirements for the RHEL virtual machine before proceeding. For more information, see the System requirements.
Deploy the RHEL instances within the remote networks where communication is required. For information about creating virtual machines, see Creating Virtual Machines in the Red Hat Enterprise Linux documentation. Remember to scale the capacity of your virtual machines sufficiently so that your proposed tasks can run on them.
- RHEL ISOs can be obtained from access.redhat.com.
- RHEL cloud images can be built using Image Builder from console.redhat.com.
2.2. Setting up Virtual Machines for use in an automation mesh Copy linkLink copied to clipboard!
Procedure
SSH into each of the RHEL instances and perform the following steps. Depending on your network access and controls, SSH proxies or other access models might be required.
Use the following command:
ssh [username]@[host_ip_address]
ssh [username]@[host_ip_address]Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example, for an Ansible Automation Platform instance running on Amazon Web Services.
ssh ec2-user@10.0.0.6
ssh ec2-user@10.0.0.6Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Create or copy an SSH key that can be used to connect from the hop node to the execution node in later steps. This can be a temporary key used just for the automation mesh configuration, or a long-lived key. The SSH user and key are used in later steps.
Enable your RHEL subscription with
baseosandappstreamrepositories. Ansible Automation Platform RPM repositories are only available through subscription-manager, not the Red Hat Update Infrastructure (RHUI). If you attempt to use any other Linux footprint, including RHEL with RHUI, this causes errors.sudo subscription-manager register --auto-attach
sudo subscription-manager register --auto-attachCopy to Clipboard Copied! Toggle word wrap Toggle overflow If Simple Content Access is enabled for your account, use:
sudo subscription-manager register
sudo subscription-manager registerCopy to Clipboard Copied! Toggle word wrap Toggle overflow For more information about Simple Content Access, see Getting started with simple content access.
Enable Ansible Automation Platform subscriptions and the proper Red Hat Ansible Automation Platform channel:
For RHEL 8
subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-8-x86_64-rpms
# subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-8-x86_64-rpmsCopy to Clipboard Copied! Toggle word wrap Toggle overflow For RHEL 9
subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-9-x86_64-rpms
# subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-9-x86_64-rpmsCopy to Clipboard Copied! Toggle word wrap Toggle overflow For ARM
subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-aarch64-rpms
# subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-aarch64-rpmsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Ensure the packages are up to date:
sudo dnf upgrade -y
sudo dnf upgrade -yCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install the ansible-core packages on the machine where the downloaded bundle is to run:
sudo dnf install -y ansible-core
sudo dnf install -y ansible-coreCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteAnsible core is required on the machine that runs the automation mesh configuration bundle playbooks. This document assumes that happens on the execution node. However, this step can be omitted if you run the playbook from a different machine. You cannot run directly from the control node, this is not currently supported, but future support expects that the control node has direct connectivity to the execution node.
2.3. Defining automation mesh node types Copy linkLink copied to clipboard!
To expand job capacity, create a standalone execution node that can be added to run alongside a deployment of automation controller. These execution nodes are not part of the automation controller Kubernetes cluster.
The control nodes run in the cluster connect and submit work to the execution nodes through Receptor.
These execution nodes are registered in automation controller as type execution instances, meaning they are only used to run jobs, not dispatch work or handle web requests as control nodes do.
Hop nodes can be added to sit between the control plane of automation controller and standalone execution nodes. These hop nodes are not part of the Kubernetes cluster and are registered in automation controller as an instance of type hop, meaning they only handle inbound and outbound traffic for otherwise unreachable nodes in different or more strict networks.
The following procedure demonstrates how to set the node type for the hosts.
By default, Red Hat Ansible Automation Platform Service on AWS includes two hop nodes that you can peer execution nodes to.
Procedure
-
From the navigation panel, select
. On the Instances list page, click . The Add Instance window opens.
An instance requires the following attributes:
Host name: (required) Enter a fully qualified domain name (public DNS) or IP address for your instance. This field is equivalent to
hostnamefor installer-based deployments.NoteIf the instance uses private DNS that cannot be resolved from the control cluster, DNS lookup routing fails, and the generated SSL certificates is invalid. Use the IP address instead.
- Optional: Description: Enter a description for the instance.
- Instance state: This field is auto-populated, indicating that it is being installed, and cannot be modified.
-
Listener port: This port is used for the receptor to listen on for incoming connections. You can set the port to one that is appropriate for your configuration. This field is equivalent to
listener_portin the API. The default value is 27199, though you can set your own port value. Instance type: Only
executionandhopnodes can be created. Operator based deployments do not support Control or Hybrid nodes.Options:
- Enable instance: Check this box to make it available for jobs to run on an execution node.
- Check the Managed by policy box to enable policy to dictate how the instance is assigned.
Peers from control nodes:
If you are configuring a hop node:
- If the hop node needs to have requests pushed directly from automation controller, then check the Peers from Control box.
- If the hop node is peered to another hop node, then make sure Peers from Control is not checked.
If you are configuring an execution node:
- If the execution node needs to have requests pushed directly from automation controller, then check the Peers from Control box.
- If the execution node is peered to a hop node, then make sure that Peers from Control is not checked.
- Click .
To verify peering configuration and the direction of traffic, you can use the topology view to view a graphical representation of your updated topology. This can help to determine where your firewall rules might need to be updated. For more information, see Topology view.
NoteComplete the following steps from any computer that has SSH access to the newly created instance.
Click the
icon next to Download Bundle to download the tar file that includes this new instance and the files necessary to install the created node into the automation mesh.
The install bundle has TLS certificates and keys, a certificate authority, and a proper Receptor configuration file.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Extract the downloaded
tar.gzInstall Bundle from the location where you downloaded it. To ensure that these files are in the correct location on the remote machine, the install bundle includes theinstall_receptor.ymlplaybook. Before running the
ansible-playbookcommand, edit the following fields in theinventory.ymlfile:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Ensure
ansible_hostis set to the IP address or DNS of the node. -
Set
ansible_userto the username running the installation. -
Set
ansible_ssh_private_key_fileto contain the filename of the private key used to connect to the instance. -
The content of the
inventory.ymlfile serves as a template and contains variables for roles that are applied during the installation and configuration of a receptor node in a mesh topology. You can modify some of the other fields, or replace the file in its entirety for advanced scenarios. For more information, see Role Variables.
-
Ensure
For a node that uses a private DNS, add the following line to
inventory.yml:ansible_ssh_common_args: <your ssh ProxyCommand setting>
ansible_ssh_common_args: <your ssh ProxyCommand setting>Copy to Clipboard Copied! Toggle word wrap Toggle overflow This instructs the
install-receptor.ymlplaybook to use the proxy command to connect through the local DNS node to the private node.- When the attributes are configured, click . The Details page of the created instance opens.
- Save the file to continue.
The system that is going to run the install bundle to setup the remote node and run
ansible-playbookrequires theansible.receptorcollection to be installed:ansible-galaxy collection install ansible.receptor
ansible-galaxy collection install ansible.receptorCopy to Clipboard Copied! Toggle word wrap Toggle overflow or
ansible-galaxy install -r requirements.yml
ansible-galaxy install -r requirements.ymlCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Installing the receptor collection dependency from the
requirements.ymlfile consistently retrieves the receptor version specified there. Additionally, it retrieves any other collection dependencies that might be needed in the future. - Install the receptor collection on all nodes where your playbook will run, otherwise an error occurs.
-
Installing the receptor collection dependency from the
If
receptor_listener_portis defined, the machine also requires an available open port on which to establish inbound TCP connections, for example, 27199. Run the following command to open port 27199 for receptor communication (Make sure you have port 27199 open in your firewall):sudo firewall-cmd --permanent --zone=public --add-port=27199/tcp
sudo firewall-cmd --permanent --zone=public --add-port=27199/tcpCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Run the following playbook on the machine where you want to update your automation mesh:
ansible-playbook -i inventory.yml install_receptor.yml
ansible-playbook -i inventory.yml install_receptor.yml
+
OpenSSL is required for this playbook. You can install it by running the following command:
openssl -v
openssl -v
If it returns then a version OpenSSL is installed. Otherwise you need to install OpenSSL with:
sudo dnf install -y openssl
sudo dnf install -y openssl
+ After this playbook runs, your automation mesh is configured.
It might be the case that some servers do not listen on the receptor port (the default is 27199)
Suppose you have a Control plane with nodes A, B, and C
The following is a peering set up for three controller nodes:
Controller node A -→ Controller node B
Controller node A -→ Controller node C
Controller node B -→ Controller node C
You can force the listener by setting
receptor_listener=True
However, a connection Controller B -→ A is likely to be rejected as that connection already exists.
This means that nothing connects to Controller A as Controller A is creating the connections to the other nodes, and the following command does not return anything on Controller A:
[root@controller1 ~]# ss -ntlp | grep 27199 [root@controller1 ~]#
The RPM installer creates a strongly connected peering between the control plane nodes with a least privileged approach and opens the tcp listener only on those nodes where it is required. All the receptor connections are bidirectional, so once the connection is created, the receptor can communicate in both directions.
To remove an instance from the mesh, see Removing instances.
r
2.4. Creating an instance group Copy linkLink copied to clipboard!
Use the following procedure to create a new instance group.
Procedure
-
From the navigation panel, select
. - Click and select Create instance group from the list.
Enter the appropriate details into the following fields:
- Name: Names must be unique and must not be named "controller".
- Policy instance minimum: Enter the minimum number of instances to automatically assign to this group when new instances come online.
Policy instance percentage: Use the slider to select a minimum percentage of instances to automatically assign to this group when new instances come online.
NotePolicy instance fields are not required to create a new instance group. If you do not specify values, then the Policy instance minimum and Policy instance percentage default to 0.
- Max concurrent jobs: Specify the maximum number of forks that can be run for any given job.
Max forks: Specify the maximum number of concurrent jobs that can be run for any given job.
NoteThe default value of 0 for Max concurrent jobs and Max forks denotes no limit. For more information, see Instance group capacity limits.
- Click , or, if you have edited an existing Instance Group click
Next steps
When you have successfully created the instance group the Details tab of the newly created instance group enables you to review and edit your instance group information.
You can also edit Instances and review Jobs associated with this instance group:
2.5. Associating instances to an instance group Copy linkLink copied to clipboard!
Procedure
- Select the Instances tab on the Details page of an Instance Group.
- Click .
- Click the checkbox next to one or more available instances from the list to select the instances you want to associate with the instance group and click
2.6. Running jobs on execution nodes Copy linkLink copied to clipboard!
You must specify where jobs are run, or they default to running in the control cluster.
To do this, set up a Job Template.
For more information on Job Templates, see Job templates in Using automation execution.
Procedure
-
The Templates list view shows job templates that are currently available. From this screen you can launch
, edit
, and duplicate
a workflow job template.
-
Select the job you want and click the
icon.
- Select the Instance Group on which you want to run the job. Note that a System Administrator must grant you or your team permissions to be able to use an instance group in a job template. If you select multiple jobs templates, the order in which you select them sets the execution precedence.
- Click .
- Click .
2.7. Connecting nodes through mesh ingress Copy linkLink copied to clipboard!
If you are using a network which restricts or does not permit inbound connections, using an instance to set up a hop node peered to the control plane can cause problems. Creating a hop node instance also requires that the hop node has a 'listener_port' set up for internal connections. There is, however, an alternative method of setting up an automation mesh, using mesh ingress.
When you instantiate mesh ingress you set up a pod, or receptor hop node inside the kubernetes control cluster, registered to the database through the operator. It also creates a service, and a route URL that is used by the control plane to connect to the hop node, and automation controller.
Prerequisites
- Create node instances within the remote networks for execution nodes in the automation mesh.
Use the following procedure to set up mesh nodes.
Procedure
Create a YAML file (in this case named
oc_meshingress.yml) to set up the mesh ingress node.Your file should resemble the following:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Where:
- apiVersion: defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and might reject unrecognized values. This value is static.
kind: Is the type of node to create.
Use the value
AutomationControllerMeshIngress.AutomationControllerMeshIngresscontrols the deployment and configuration of mesh ingress on automation controller.- name: enter a name for the mesh ingress node.
namespace: enter a name for the Kubernetes namespace to deploy the mesh ingress into.
This must be in the same namespace as the automation controller that the mesh is connected to.
- deployment_name: is the automation controller instance that this mesh ingress is attached to. Provide the name of your automation controller instance.
Apply this YAML file using:
oc apply -f oc_meshingress.yml
oc apply -f oc_meshingress.ymlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run this playbook to creates the
AutomationControllerMeshIngressresource. The operator creates a hop node in automation controller with thenameyou supplied.When the MeshIngress instance has been created, it appears in the Instances page.
ImportantAny instance that is to function as a remote execution node in "pull" mode need to be created after this procedure and must be configured as follows:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Associate this new instance with the hop node you created using the procedure in this paragraph
Download the tarball.
NoteAssociation with the hop node must be done before creating the tarball.
2.8. Pulling the secret for OpenShift Container Platform deployments Copy linkLink copied to clipboard!
This does not apply to Ansible Automation Platform on Microsoft Azure.
If you are using the default execution environment provided with automation controller to run on remote execution nodes, you must add a pull secret in automation controller that contains the credential for pulling the execution environment image.
To do this, create a pull secret on the automation controller namespace and configure the ee_pull_credentials_secret parameter in the Operator as follows:
Procedure
Create a secret using the following command:
oc create secret generic ee-pull-secret \ --from-literal=username=<username> \ --from-literal=password=<password> \ --from-literal=url=registry.redhat.iooc create secret generic ee-pull-secret \ --from-literal=username=<username> \ --from-literal=password=<password> \ --from-literal=url=registry.redhat.ioCopy to Clipboard Copied! Toggle word wrap Toggle overflow Add
ee_pull_credentials_secretandee-pull-secretto the specification by editing the deployment specification:oc edit automationcontrollers aap-controller-o yaml
oc edit automationcontrollers aap-controller-o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow and add the following:
spec ee_pull_credentials_secret=ee-pull-secret
spec ee_pull_credentials_secret=ee-pull-secretCopy to Clipboard Copied! Toggle word wrap Toggle overflow - To manage instances from the automation controller UI, you must have System Administrator or System Auditor permissions.
2.9. Removing Instances Copy linkLink copied to clipboard!
From the Instances page, you can add, remove or run health checks on your nodes.
You must follow the procedures for installing RHEL packages for any additional nodes you create. If you peer this additional node to an existing hop node, you must also install the Install Bundle on each node.
Use the check boxes next to an instance to select it to remove it, or run a health check against it.
- If a node is removed using the UI, then the node is "removed" and does not show a status. If you delete the VM of the node before it is removed in the UI, it will show an error.
- You only need to reinstall the Install Bundle if the topology changes the communication pattern, that is, hop nodes change or you add nodes.
When a button is disabled, you do not have permission for that particular action. Contact your Administrator to grant you the required level of access.
If you are able to remove an instance, you receive a prompt for confirmation.
You can still remove an instance even if it is active and jobs are running on it. Automation controller waits for jobs running on this node to complete before removing it.
2.10. Upgrading receptors Copy linkLink copied to clipboard!
A software update addresses any issues or bugs to provide a better experience of working with the technology. Anyone with administrative rights can update the receptor on an execution node.
Red Hat recommends performing updates to the receptor after any Ansible Automation Platform control plane updates. This ensures you are using the latest version. The best practice is to perform regular updates outside of any updates to the control plane.
Procedure
Check the current receptor version:
receptor --version
receptor --versionCopy to Clipboard Copied! Toggle word wrap Toggle overflow Update the receptor:
sudo dnf update ansible-runner receptor -y
sudo dnf update ansible-runner receptor -yCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteTo upgrade all packages (not just the receptor), use
dnf update, then reboot withreboot.Verify the installation. After the update is complete, check the receptor version again to verify the update:
receptor --version
receptor --versionCopy to Clipboard Copied! Toggle word wrap Toggle overflow Restart the receptor service:
sudo systemctl restart receptor
sudo systemctl restart receptorCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Ensure the receptor is working correctly and is properly connected to the controller or other nodes in the system.