Chapter 9. Maintaining Satellite Server
This chapter provides information on how to maintain a Red Hat Satellite Server, including information on relevant log files, how to enable debug logging, how to open a support case and attach the relevant log tar files, and how to use Red Hat Insights to proactively diagnose systems.
9.1. Logging and Reporting
Red Hat Satellite provides system information in the form of notifications and log files.
Log File | Description of Log File Content |
---|---|
| Subscription management |
| Foreman |
| Foreman proxy |
| Apache HTTP server |
| Satellite installer |
| Capsule Server installer |
| Virtualization API |
| Satellite database |
|
Celerybeat and Celery startup request messages. After startup is complete, messages are logged to |
| Configuration management |
| Subscription management |
| Apache web server messages for Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7, respectively. |
| Various other log messages related to pulp, rhsm, and goferd. |
You can also use the foreman-tail
command to follow many of the log files related to Satellite. You can run foreman-tail -l
to list the processes and services that it follows.
On Red Hat Enterprise Linux 7, you can use the journal for more extensive logging information. See Using the Journal[1] for more information.
9.2. Enabling Debug Logging
This section describes how to enable debug logging to provide detailed debugging information for Satellite 6.3. Debug logging provides the most detailed log information and can help with troubleshooting issues that can arise with Satellite 6.3 and its components. It is also possible to enable or disable individual loggers for selective logging.
To enable debug logging, modify the /etc/foreman/settings.yaml
file.
Set the Logging Level to "debug"
By default, the logging level is set to
info
, as in the following::logging: :level: info
Alter these lines so that they look like this:
:logging: :level: debug
Select Individual Logging Types
By default, the end of
/etc/foreman/settings.yaml
looks like this:# Individual logging types can be toggled on/off here :loggers:
Alter the
/etc/foreman/settings.yaml
file so that it looks like this::loggers: :ldap: :enabled: true :permissions: :enabled: true :sql: :enabled: true
Restart Katello services:
# katello-service restart
Complete List of Loggers with their Default Values
:app: :enabled: true :ldap: :enabled: false :permissions: :enabled: false :sql: :enabled: false
9.3. Collecting Information from Log Files
There are two utilies available to collect information from log files.
Command | Description |
---|---|
|
The
Additionally, the
For more information, run There is no timeout when running this command. |
|
The
By default, the output tar file is located at
The |
Both foreman-debug
and sosreport
remove security information such as passwords, tokens, and keys while collecting information. However, the tar files can still contain sensitive information about the Red Hat Satellite Server. Red Hat recommends that you send this information directly to the intended recipient and not to a public target.
9.4. Using Log Files in Support Cases
You can use the log files and other information described in this chapter to do your own troubleshooting, or you can capture these and many more files, as well as diagnostic and configuration information, to send to Red Hat Support if you need further assistance.
There are two methods to open a support case with Red Hat Support. You can open a support case directly from the Satellite web UI or from the Customer Portal.
- Section 9.8.5, “Creating Support Cases Using the Red Hat Access Plug-in”: How to open a support case from the Satellite web UI
- https://access.redhat.com/articles/38363: How to open and manage a support case on the Customer Portal
9.5. Cleaning Unused Tasks
Cleaning unused tasks reduces disk space in the database and limits the rate of disk growth. When you perform regular cleaning, Satellite backup completes faster and overall performance is higher.
To clean unused tasks
The installer has a feature to enable a cron job to automatically remove old tasks. This feature is not enabled by default to avoid unwanted tasks cleanup.
Enable the cron job:
# satellite-installer --foreman-plugin-tasks-automatic-cleanup true
By default, the cron job is scheduled to run every day at 19:45. To adjust to the time, change the value of the
--foreman-plugin-tasks-cron-line
parameter:# satellite-installer --foreman-plugin-tasks-cron-line "00 15 * * *"
The previous command schedules the cron job to run every day at 15:00, see
man 5 crontab
for more details on cron format.
To change the period after which to delete all the tasks and to configure further advanced settings of the cron job, change the content of the /etc/foreman/plugins/foreman-tasks.yaml file
.
9.6. Recovering from a Full Disk
The following procedure describes how to resolve the situation when a logical volume (LV) with the Pulp database on it has no free space.
To recover from a full disk
- Let running Pulp tasks finish but do not trigger any new ones as they can fail due to the full disk.
Ensure that the LV with the
/var/lib/pulp
directory on it has sufficient free space. Here are some ways to achieve that:Remove orphaned content:
# foreman-rake katello:delete_orphaned_content RAILS_ENV=production
This is run weekly so it will not free much space.
- Change the download policy from Immediate to On Demand for as many repositories as possible and remove already downloaded packages. See the Red Hat Knowledgebase solution How to change syncing policy for Repositories on Satellite 6.2 from "Immediate" to "On-Demand" on the Red Hat Customer Portal for instructions.
Grow the file system on the LV with the
/var/lib/pulp
directory on it. For more information, see Growing a File System on a Logical Volume in the Red Hat Enterprise Linux 7 Logical Volume Manager Administration Guide.NoteIf you use an untypical file system (other than for example ext3, ext4, or xfs), you might need to unmount the file system so that it is not in use. In that case:
- Stop Katello services:
# katello-service stop
- Grow the file system on the LV.
- Start Katello services:
# katello-service start
- If some Pulp tasks failed due to the full disk, run them again.
9.7. Reclaiming Disk Space from MongoDB
The MongoDB database can use a large amount of disk space especially in heavily loaded deployments. The following procedure describes how to reclaim some of this disk space.
Prerequisites
- A backup of the MongoDB database. For instructions on creating a backup, see Section 8.1.3, “Performing a Backup without Pulp Content”.
Pulp services are stopped:
# systemctl stop goferd httpd pulp_workers pulp_celerybeat \ pulp_resource_manager pulp_streamer
To reclaim disk space from MongoDB
Access the MongoDB shell:
# mongo pulp_database
Check the amount of disk space used by MongoDB before a repair:
> db.stats()
- Ensure that you have free disk space equal to the size of your current MongoDB database plus 2 GB. If the volume containing the MongoDB database lacks sufficient space, you can mount a separate volume and use that for the repair.
Enter the repair command:
> db.repairDatabase()
Note that the repair command blocks all other operations and can take a long time to complete, depending on the size of the database.
Check the amount of disk space used by MongoDB after a repair:
> db.stats()
Start Pulp services:
# systemctl start goferd httpd pulp_workers pulp_celerybeat \ pulp_resource_manager pulp_streamer
9.8. Accessing Customer Portal Services from Red Hat Satellite
The Red Hat Access pre-installed plug-in lets you access several Red Hat Customer Portal services from within the Satellite web UI.
The Red Hat Access plug-in provides the following services:
- Search: Search solutions in the Customer Portal from within the Satellite web UI.
- Logs: Send specific parts (snippets) of the log files to assist in problem solving. Send these log snippets to the Red Hat Customer Portal diagnostic tool chain.
- Support: Access your open support cases, modify an open support case and open a new support case from within the Satellite web UI.
To access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.
9.8.1. Searching for Solutions in the Red Hat Access Plug-in
The Red Hat Access plug-in provides search capabilities that look through the solutions database available in the Red Hat Customer Portal.
To Search for Solutions from the Red Hat Satellite Server:
- In the upper right, click Red Hat Access > Search.
If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.
NoteTo access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.
- In the Red Hat Search field, enter your search query. Search results display in the left-hand Recommendations list.
- In the Recommendations list, click a solution. The solution article displays in the main panel.
9.8.2. Using Logs in the Red Hat Access Plug-in
The log file viewer lets you view the log files and isolate log snippets. You can also send the log snippets through the Customer Portal diagnostic tool chain to assist with problem solving.
To Use the Logs Diagnostic Tool from the Red Hat Satellite Server:
- In the upper right, click Red Hat Access > Logs.
If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.
NoteTo access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.
- In the left file tree, select a log file and click the file name.
- Click Select File. A pop-up window displays the log file contents.
- In the log file, highlight any text sections you want diagnosed. The Red Hat Diagnose button displays.
- Click Red Hat Diagnose. The system sends the highlighted information to the Red Hat Customer Portal, and provides solutions that closely match the provided log information.
If a solution does the following:
- Matches the problem, click the solution and follow the required steps to troubleshoot the issue.
- Does not match the problem, click Open a New Support Case. The support case is populated with the highlighted text from the log file. See Section 9.8.5, “Creating Support Cases Using the Red Hat Access Plug-in”.
9.8.3. Viewing Existing Support Cases Using the Red Hat Access Plug-in
You can view your existing support case from your Red Hat Satellite Server using the Red Hat Access Plug-in.
To View Existing Support Cases from the Red Hat Satellite Server:
- In the upper right, click Red Hat Access > Support > My Cases.
If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.
NoteTo access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.
To search for a specific support case from existing cases, do any of the following:
- In the Search field, provide a key word or phrase.
- From the drop-down list, choose a specific Case Group. Your organization has defined Case Groups inside the Red Hat Customer Portal.
- Choose a Case Status.
- From the results, choose a specific support case and click the Case ID. The support case is ready to view.
9.8.4. Modifying Support Cases Using the Red Hat Access Plug-in
You can update your existing support cases from your Red Hat Satellite Server using the Red Hat Access Plug-in.
To Update Support Cases from the Red Hat Satellite Server Web UI:
- Complete the instructions from Section 9.8.3, “Viewing Existing Support Cases Using the Red Hat Access Plug-in”
In the support case, scroll down to the marked sections to do the following:
Attachments: - Attach a local file from the system. Add a file name to make it easier to identify.
NoteFile names must be less than 80 characters and the maximum file size for attachments uploaded using the web UI is 250 MB. Use FTP for larger files.
- Case Discussion: - Add any updated information about the case you wish to discuss with Global Support Services. After adding information, click Add Comment.
9.8.5. Creating Support Cases Using the Red Hat Access Plug-in
You can create a new support case from your Red Hat Satellite Server using the Red Hat Access Plug-in.
To Create a New Support Case Using the Red Hat Satellite Server:
- In the upper right, click Red Hat Access > Support > New Case.
If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.
NoteTo access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.
The Product and Product Version fields are automatically populated. Complete the other relevant fields, as follows:
- Summary — Provide a brief summary of the issue.
Description — Write a detailed description of the issue.
Based on the summary provided, recommendations for possible solutions display in the main panel.
- Click Next.
Choose the appropriate options, as follows:
- Severity — Select the ticket urgency as 4 (low), 3 (normal), 2 (high), or 1 (urgent).
- Case Group — Based on who needs to be notified, create case groups associated with the support case. Select Case Groups in Red Hat Satellite. Create Case Groups within the Customer Portal.
Attach the output of
sosreport
and any required files. Add a file description and click Attach.Note-
If you have large log files or many Satellite tasks, it is recommended to also attach the output of
foreman-debug
. - File names must be less than 80 characters and the maximum file size for attachments uploaded using the web UI is 250 MB. Use FTP for larger files.
-
If you have large log files or many Satellite tasks, it is recommended to also attach the output of
- Click Submit. The system uploads the case to the Customer Portal, and provides a case number for your reference.
The Red Hat Knowledgebase article https://access.redhat.com/articles/445443: Red Hat Access: Red Hat Support Tool has additional information, examples, and video tutorials.
9.9. Using Red Hat Insights with Satellite Server
Red Hat Insights enables you to proactively diagnose systems and downtime related to security exploits, performance degradation and stability failures. You can use the dashboard to quickly identify key risks to stability, security, or performance. You can sort by category, view details of the impact and resolution, and then determine what systems are affected.
Red Hat Insights is installed by default on Satellite Server. Before using Insights with Satellite Server, go to Red Hat Insights and click Satellite 6 for the pre-installation checks and to register your Satellite Servers.
9.10. Monitoring Satellite Server in the Web UI
From the About page in the Satellite Server web UI, you can find an overview of the following:
- System Status, including Capsules, Available Providers, Compute Resources, and Plug-ins
- Support information
- System Information
- Backend System Status
- Installed packages
To navigate to the About page:
- In the upper right corner of the Satellite Server web UI, click Administer > About.
After Pulp failure, the status of Pulp might show OK instead of Error for up to 10 minutes due to synchronization delay.