Chapter 5. Running Jobs on Hosts
Red Hat Satellite supports the ability to run arbitrary commands on hosts. This is referred to as remote execution. Remote execution is enabled by default on the Satellite Server, but must be enabled manually on all desired Capsule Servers. Communication occurs through the Capsule Server which means that the Satellite Server does not require direct access to the target host, and can scale to control many hosts. Remote execution uses the SSH service which must be enabled and running on the target host. Ensure the Capsule has access to port 22 on the target hosts.
Commands can be customized in a similar fashion to provisioning templates or partition tables. Several job templates are included by default, that you can use to run commands. See Section 5.4.1, “Setting up Job Templates”.
Any Capsule Server’s base system is a client of Satellite Server’s internal Capsule, and therefore this section applies to any type of host connected to Satellite Server, including Capsule Servers.
You can execute commands on multiple hosts at once, and you can use variables in your commands to suit your deployment. Variable values can be filled by host fact, Smart Class Parameter, Smart Variable, or even host parameter. In addition, you can specify custom values for templates when you run the command. See Section 5.4.2, “Executing Jobs”.
The following list provides some examples of how you can use remote execution:
- Install, update, or remove software packages
- Bootstrap a configuration management agent
- Trigger a Puppet, Salt, or Chef run
By default, each Capsule is installed with the remote execution feature disabled. To use remote execution on a Capsule Server you need to enable it. To enable, run the following command:
# satellite-installer --scenario capsule \ --enable-foreman-proxy-plugin-remote-execution-ssh
To verify that remote execution is running on the Capsule Server and in the web UI navigate to Infrastructure > Capsules. The Capsule Server should now list in the Features column that SSH is running.
By default, Satellite Server is configured to use remote execution rather than Katello Agent. If required, these settings can be changed by first creating custom job templates and then selecting these new templates in the web UI by going to Administer > Remote Execution Features. For each action you want to change, select the label and then select the job template to use.
5.1. Establishing a Secure Connection for Remote Commands
The SSH keys used for remote execution are created automatically when installing a Capsule and the settings are in the /etc/foreman-proxy/settings.d/remote_execution_ssh.yml file. They include the following options:
- ssh_identity_file
- File to load the SSH key from. By default, set to /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.
- local_working_dir
- Directory used on the Satellite or Capsule to run the scripts necessary for remote execution. By default, set to /var/tmp.
- remote_working_dir
- Directory on the client system that is used to execute the remote execution jobs. By default, set to /var/tmp.
If the client system has noexec set for the /var/ volume or file system, change the remote_working_dir as otherwise the remote execution job will fail since the script cannot be executed.
If required to use an alternative directory, create the new directory, for example new_place, and then copy the SELinux context from the default directory. For example:
# chcon --reference=/var new_placeSee Maintaining SELinux Labels in the SELinux User’s and Administrator’s Guide for more information on working with SELinux labels.
Distributing the SSH Keys for Remote Execution
To enable remote execution, distribute the public SSH key from a Capsule to the hosts that you want to manage. Ensure the SSH service is enabled and running on the hosts. Configure any network or host-based firewalls to enable access to port 22.
There are three ways to distribute the public key from a Capsule to target hosts:
To distribute keys manually, execute the following command on the Capsule:
# ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub root@target.example.comHere target.example.com is the host name of the target host. Repeat for each target host you want to manage.
To confirm the key was successfully copied to the target host, execute the following command on the Capsule:
# ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@target.example.comTo use the Satellite API to download the public key directly from the Capsule, execute the following command on each target host:
# curl https://myproxy.example.com:9090/ssh/pubkey >> ~/.ssh/authorized_keysHere myproxy.example.com stands for the host name of the Capsule.
To include the public key in newly-provisioned hosts, modify for example the Kickstart default finish template to include the following line:
<%= snippet 'remote_execution_ssh_keys' %>
5.2. Configuring a keytab for Kerberos Ticket Granting Tickets
Use this procedure to configure Satellite to use a keytab to obtain Kerberos ticket granting tickets. If you do not set up a keytab, you must manually retrieve tickets.
To ensure that the foreman-proxy user on Satellite can obtain Kerberos ticket granting tickets, complete the following steps:
Find the ID of the
foreman-proxyuser:# id -u foreman-proxy
Modify the
umaskvalue so that new files have the permissions600:# umask 077
Create the directory for the keytab:
# mkdir -p "/var/kerberos/krb5/user/USER_ID"Create a keytab or copy an existing keytab to the directory:
# cp your_client.keytab /var/kerberos/krb5/user/USER_ID/client.keytab
Change the directory owner to the
foreman-proxyuser:# chown -R foreman-proxy:foreman-proxy "/var/kerberos/krb5/user/USER_ID"Ensure that the keytab file is read-only:
# chmod -wx "/var/kerberos/krb5/user/USER_ID/client.keytab"Restore the SELinux context:
# restorecon -RvF /var/kerberos/krb5
5.3. Setting Up Kerberos Authentication for Remote Execution
From Satellite 6.3, you can use Kerberos authentication to establish an SSH connection for remote execution on Satellite hosts.
Prerequisites
Before you can use Kerberos authentication for remote execution on Red Hat Satellite, you must set up a Kerberos server for identity management and ensure that you complete the following prerequisites:
- Enroll Satellite Server on the Kerberos server
- Enroll the Satellite target host on the Kerberos server
- Configure and initialize a Kerberos user account for remote execution
-
Ensure that the
foreman-proxyuser on Satellite has a valid Kerberos ticket granting ticket
To set up Satellite to use Kerberos authentication for remote execution on hosts, complete the following steps:
Before installing the
tfm-rubygem-net-ssh-krbpackage, you must temporarily set SELinux topermissiveuntil Red Hat bug 1541481 is resolved:# setenforce 0
To install the
tfm-rubygem-net-ssh-krbpackage, enter the following command:# yum install tfm-rubygem-net-ssh-krb
To install and enable Kerberos authentication for remote execution, enter the following command:
# satellite-installer --scenario satellite \ --foreman-proxy-plugin-remote-execution-ssh-ssh-kerberos-auth true
Set SELinux to
enforcing:# setenforce 1
- To edit the default user for remote execution, in the Satellite web UI, navigate to Administer > Settings and click the RemoteExecution tab. In the remote_execution_ssh_user row, edit the second column and add the user name for the Kerberos account.
- Navigate to remote_execution_effective_user and edit the second column to add the user name for the Kerberos account.
To confirm that Kerberos authentication is ready to use, run a remote job on the host.
5.4. Configuring and Running Remote Commands
Any command that you want to execute on a remote host must be defined as a job template. After you have defined a job template you can execute it multiple times.
5.4.1. Setting up Job Templates
Satellite provides a number of default job templates that you can use for executing jobs, find them under Hosts > Job templates. If you find a template fitting your needs amongst the default templates, proceed to Section 5.4.2, “Executing Jobs”.
You can also use default templates as a basis for developing your own. Default job templates are locked for editing, therefore you have to first clone the template to be able to modify it. Job templates use the Embedded Ruby (ERB) syntax, for more information see Appendix A, Template Writing Reference.
To Create a Job Template:
- Navigate to Hosts > Job templates.
- Click New Job Template. As an alternative, you can modify an existing template – in the Actions column, select Clone from the drop-down menu.
Configure the job template:
- On the Template tab, enter a unique name for your job template. Select Default to make the template available for all organizations and locations. You can insert the template manually using Template editor or upload it from a text file by clicking Browse. Templates use Embedded Ruby (ERB) template syntax, see ] for more information. An advanced template is required, for example, for executing jobs that perform power actions; see xref:exam-Managing_Hosts-Including_Power_Actions_in_Templates[ for information on how to include the Power Action - SSH Default template in a custom template.
- On the Job tab, you can define the job category (define your own or select from the default categories listed in ]) as well as the effective user; these settings can be configured also when invoking the job (see xref:proc-Managing_Hosts-Executing_a_Remote_Job[). You can also define input parameters for template commands. These parameters are then requested when executing the job.
- Remaining tabs enable setting the template type, organizations and locations as well as viewing the template history.
- Click Submit. When the page refreshes, your new template should appear in the list of job templates.
Note that only the parameters visible on the Parameters tab at the host’s edit page can be used as input parameters for job templates.
| Job template category | Description |
|---|---|
| Packages | Templates for performing package related actions. Install, update, and remove actions are included by default. |
| Puppet | Templates for executing Puppet runs on target hosts. |
| Power | Templates for performing power related actions. Restart and shutdown actions are included by default. |
| Commands | Templates for executing custom commands on remote hosts. |
| Services | Templates for performing service related actions. Start, stop, restart, and status actions are included by default. |
| Katello | Templates for performing content related actions. These templates are used mainly from different parts of the Satellite web UI (for example bulk actions UI for content hosts), but can be used separately to perform operations such as errata installation. |
Example 5.1. Creating a restorecon Template
This example shows how to create a template called Run Command - restorecon that will restore the default SELinux context for all files in the selected directory on target hosts.
- Navigate to Hosts > Job templates. Click New Job Template.
Insert Run Command - restorecon in the Name field. Select Default to make the template available to all organizations. Add the following text to the Template editor:
restorecon -RvF <%= input("directory") %>The
<%= input("directory") %>string will be replaced by a user-defined directory during job invocation.On the Job tab, perform the following actions:
-
Set Job category to
Commands. -
Click Add Input to allow job customization. Insert
directoryto the Name field. The input name must match the value specified in the Template editor. - Click Required so that the command cannot be executed without the user specified parameter.
-
Select
User inputfrom the Input type drop-down list. Also provide a Description to be shown during job invocation, for example Target directory for restorecon.
-
Set Job category to
- Click Submit.
See Example 5.2, “Executing a restorecon Template on Multiple Hosts” for information on how to execute a job based on this template.
5.4.2. Executing Jobs
This section shows how to run a job based on a job template against one or more hosts.
To Execute a Remote Job:
- Navigate to Hosts > All hosts and select the target hosts for your job. You can use the search field to narrow down the host list.
- From the Select Action menu at the upper right of the screen select Schedule Remote Job. This will take you to the Job invocation page. Alternatively, if you target just one host, click its name and click Schedule Remote Job on the host information page. Note that you can invoke jobs also from the Job Templates page by using the Run button.
On the Job invocation page, define the main job settings:
- Select the Job category and the Job template you want to use. These settings are required.
- Optionally, select a stored search string in the Bookmark list to specify the target hosts.
- Optionally, further limit the targeted hosts by inserting a Search query. The Resolves to line displays the number of hosts affected by your query. Use the refresh button to recalculate the number after changing the query. The preview icon will list the targeted hosts.
- The remaining settings depend on the selected job template. See To Create a Job Template: for information on adding custom parameters to a template.
Clicking Display advanced fields will show advanced setting for the job. Some of the advanced settings depend on the job template, the following settings are general:
- Effective user defines the user for executing the job, by default it is the SSH user.
- Concurrency level defines maximum number of jobs executed at once, which can prevent overload of systems' resources in a case of executing the job on a large number of hosts.
- Time span defines time interval in seconds after which the job should be killed, if it is not finished already. A task which could not be started during the defined interval, for example, if the previous task took too long to finish, is canceled.
Type of query defines when the search query is evaluated. This helps to keep the query up to date for scheduled tasks.
Concurrency level and Time span settings enable you to tailor job execution to fit your infrastructure hardware and needs.
-
If you want to execute the job immediately, ensure that Schedule is set to
Execute now. You can also define a one-time future job, or set up a recurring job. For recurring tasks, you can define start and end dates, number and frequency of runs. You can also use cron syntax to define repetition. For more information about cron, see the Automating System Tasks section of the Red Hat Enterprise Linux 7 System Administrator’s Guide. - Click Submit. This displays the Job Overview page, and when the job completes, also displays the status of the job.
Example 5.2. Executing a restorecon Template on Multiple Hosts
This example shows how to execute a job based on the template created in Example 5.1, “Creating a restorecon Template” on multiple hosts. The job will restore the SELinux context in all files under the /home/ directory.
- Navigate to Hosts > All hosts and select target hosts. Select Schedule Remote Job from the Select Action drop-down list.
-
In the Job invocation page, select the
Commandsjob category and theRun Command - restoreconjob template. -
Type
/homein the directory field. -
Set Schedule to
Execute now. - Click Submit. You are taken to the Job invocation page where you can monitor the status of job execution.
5.4.3. Monitoring Jobs
You can monitor the progress of the job while it is running. This can help in any troubleshooting that may be required.
To Monitor a Job:
-
Navigate to the Job page. This page is automatically displayed if you triggered the job with the
Execute nowsetting. To monitor scheduled jobs, navigate to Monitor > Jobs and select the job run you wish to inspect. - On the Job page, click the Hosts tab. This displays the list of hosts on which the job is running.
- In the Host column, click the name of the host that you want to inspect. This displays the Detail of Commands page where you can monitor the job execution in real time.
- Click Back to Job at any time to return to the Job Details page.
5.4.4. Creating Advanced Templates
When creating a job template, you can import an existing template in the Template editor field – this is referred to as rendering. This way you can combine templates, or create more specific templates from the general ones.
The following template combines default templates to install and start the httpd service on Red Hat Enterprise Linux systems:
<%= render_template 'Package Action - SSH Default', :action => 'install', :package => 'httpd' %> <%= render_template 'Service Action - SSH Default', :action => 'start', :service_name => 'httpd' %>
The above template specifies parameter values for the rendered template directly. It is also possible to use the input() method to allow users to define input for the rendered template on job execution. For example, you can use the following syntax:
<%= render_template 'Package Action - SSH Default', :action => 'install', :package => input("package") %>With the above template, you have to import the parameter definition from the rendered template. To do so, navigate to the Jobs tab, click Add Foreign Input Set, and select the rendered template from the Target template drop-down list. You can import all parameters or specify a comma separated list.
Example 5.3. Rendering a restorecon Template
This example shows how to create a template derived from the Run command - restorecon template created in Example 5.1, “Creating a restorecon Template”. This template does not require user input on job execution, it will restore the SELinux context in all files under the /home/ directory on target hosts.
Create a new template as described in Section 5.4.1, “Setting up Job Templates”, and specify the following string in the Template editor screen:
<%= render_template("Run Command - restorecon", :directory => "/home") %>Example 5.4. Including Power Actions in Templates
This example shows how to set up a job template for performing power actions, such as reboot. This procedure prevents Satellite from interpreting the disconnect exception upon reboot as an error, and consequently, remote execution of the job works correctly.
Create a new template as described in Section 5.4.1, “Setting up Job Templates”, and specify the following string in the Template editor screen:
<%= render_template("Power Action - SSH Default", :action => "restart") %>5.5. Configuring Global Settings
The Satellite remote execution feature provides numerous global settings that you can use to configure its behavior. These are listed in Table 5.2, “Global Settings for Remote Execution”. To review and update these settings, navigate to Administer > Settings and click the Remote Execution tab.
| Parameter Name | Description |
|---|---|
| remote_execution_effective_user | This is the default effective user for any job. When the job is executed the effective user of the process is changed accordingly (for example, by sudo). This option can be overridden per job template and job invocation. |
| remote_execution_effective_user_method | Specifies which method to use to set the effective user on the target host. Currently only su and sudo are supported. |
| remote_execution_fallback_proxy | Search the host for any Capsule with remote execution configured. This is useful when the host has no subnet or if the subnet does not have a Capsule with remote execution enabled. |
| remote_execution_global_proxy | Search for a remote execution Capsule outside of the Capsules assigned to the host. If Locations or Organizations are enabled, the search will be limited to the host’s Organization or Location. |
| remote_execution_ssh_user |
The default user to use while the Capsule connects to the target using SSH. You can set the
You can set this by Host, Host Group, Operating System, Domain, Location, or Organization. This can also be a different user from the |
| remote_execution_sync_templates | Defines whether job templates should be synchronized from disk when seeding a database. |
It is possible to set global parameters in the /etc/foreman/settings.yaml configuration file, but any manual changes that you make to this file are overwritten the next time you run satellite-installer. Consequently, Red Hat recommends that you modify these parameters in the web UI. Alternatively, use the foreman-rake config command from a console.
5.5.1. Choosing a Capsule for Remote Execution
Remote execution requires a Capsule Server to perform any specified job on a host. By default, any Capsule within the host’s organization and location with the remote execution provider feature enabled is considered available to perform these jobs. You can set the remote_execution_global_proxy variable to false to disable this behavior. This may be necessary in more complex environments, where not all Capsules can be used due to possible network isolation. In this configuration, you can assign a pool of Capsules to each subnet, and jobs are load balanced across them.
Alternatively, you can set the remote_execution_fallback_proxy variable to true to enable fallback mode. In this configuration, remote execution will use any Capsule associated with the host, such as its Puppet Master, provided that Capsule also has remote execution configured.
5.6. Delegating Permissions for Remote Execution
You can control which users can run which jobs within your infrastructure, including which hosts they can target. The remote execution feature provides two built-in roles:
- Remote Execution Manager: This role allows access to all remote execution features and functionality.
- Remote Execution User: This role only allows running jobs; it does not provide permission to modify job templates.
You can clone the Remote Execution User role and customize its filter for increased granularity. If you adjust the filter with the view_job_templates permission, the user can only see and trigger jobs based on matching job templates. You can use the view_hosts and view_smart_proxies permissions to limit which hosts or Capsules are visible to the role.
The execute_template_invocation permission is a special permission that is checked immediately before execution of a job begins. This permission defines which job template you can run on a particular host. This allows for even more granularity when specifying permissions. For more information on working with roles and permissions see Creating and Managing Roles in the Administering Red Hat Satellite.
The following example shows filters for the execute_template_invocation permission:
name = Reboot and host.name = staging.example.com name = Reboot and host.name ~ *.staging.example.com name = "Restart service" and host_group.name = webservers
The first line in the above example permits the user to execute the Reboot template on one selected host. The second line defines a pool of hosts with names ending with .staging.example.com. The third line binds the template with a host group.
Permissions assigned to users can change over time. If a user has already scheduled some jobs to run in the future, and the permissions have changed, this can result in execution failure because the permissions are checked immediately before job execution.
