SupportConsoleDevelopersStart a trialContact

Chapter 10. Troubleshooting

This chapter provides tips for determining the cause of and resolving the most common errors associated with Red Hat Satellite. If you need additional help, contact Red Hat Network support at https://access.redhat.com/support/. Log in using your Satellite-entitled account to see the full list of options.
To begin troubleshooting general problems, examine the log file or files related to the component exhibiting failures. A useful exercise is to issue the tail -f command for all log files and then run yum list. You should then examine all new log entries for potential clues.
10.1. Disk Space
Q: My disk space filled up fast. What happened and what should I do?
10.2. Installing and Updating
Q: SELinux keeps giving me messages when I'm trying to install. Why?
Q: I changed /var/satellite to an NFS mount, and now SELinux is stopping it from working properly. What do I need to do?
Q: My Satellite is failing. Any idea why?
10.3. Services
Q: Why isn't the Apache Web server running?
Q: How do I find out what the status of the Red Hat Network Task Engine is?
Q: How do I find out what the status of the Satellite's Embedded Database is?
Q: What do I do if the push capability of the Red Hat Satellite stops working?
10.4. Connectivity
Q: I can't connect! How do I work out what is wrong?
Q: What do I do if importing or synchronizing a channel fails and I can't recover it?
Q: I'm getting "SSL_CONNECT" errors. What do I do now?
10.5. Logging and Reporting
Q: What are the different log files?
Q: How do I use spacewalk-report?
Q: How do I work out what version of the database schema I have?
Q: How do I work out what character set types I have?
Q: Why isn't the administrator getting email?
Q: How do I change the sender of the traceback mail?
10.6. Errors
Q: I'm getting an "Error validating satellite certificate" error during a Red Hat Satellite installation. How do I fix it?
Q: I'm getting an "ERROR: server.mount_point not set in the configuration file" error when I try to activate or synchronize the Red Hat Satellite. How do I fix it?
Q: Why does cobbler check give an error saying that it needs a different version of yum-utils?
Q: I'm getting an "unsupported version" error when I try to activate the Red Hat Satellite certificate. How do I fix it?
Q: I'm getting an "Internal Server Error" complaining about ASCII when I try to edit the kickstart profile. What's going on?
Q: I'm getting "Host Not Found" or "Could Not Determine FQDN" errors. What do I do now?
Q: I'm getting a "This server is not an entitled Satellite" when I try to synchronize the Red Hat Satellite server. How do fix it?
10.7. Web Interface
Q: I'm having problems with the Red Hat Satellite user interface. Which log files should I check?
10.8. Anaconda
Q: I'm getting an error that says Error downloading kickstart file. What is the problem and how do I fix it?
Q: I'm getting a package installation error that says The file chkconfig-1.3.30.1-2.i386.rpm cannot be opened. What is the problem and how do I fix it?
10.9. Tracebacks
Q: I'm getting emails with "WEB TRACEBACK" in the subject. What should I do about them?
10.10. Registration
Q: The rhnreg_ks command is failing when I run it, saying ERROR: unable to read system id. What is the problem?
10.11. Kickstarts and Snippets
Q: What is the directory structure for kickstarts?
Q: What is the directory structure for Cobbler snippets?
10.12. Multi-Organization Satellites and Satellite Certificate
Q: How do I register my systems in a Multiple Organization environment when I do not have enough entitlements in my Satellite Certificate?
Q: I have extra entitlements on my Satellite Certificate that are not being used. What happens to these entitlements?
10.13. Proxy Installation and Configuration
Q: After configuring the Red Hat Network Package Manager how can I determine if the local packages were successfully added to the private Red Hat Network channel?
Q: How can I determine whether the clients are connecting to the Squid server?
Q: The Red Hat Update Agent on the client systems does not connect through the Red Hat Satellite Proxy. How can I resolve this error?
Q: My Red Hat Satellite Proxy configuration does not work. Where do I begin troubleshooting it?
Q: How do I troubleshoot general problems in the Red Hat Satellite Proxy?
Q: My Red Hat Satellite Proxy encountered the error "Host Not Found"/"Could not Determine FQDN". What should I do?
Q: I am having issues with Red Hat Satellite Proxy and network connection errors. What should I do?
Q: I am having issues with package delivery errors and object corruption. What should I check for?

10.1. Disk Space

Q:
My disk space filled up fast. What happened and what should I do?
A:
A common issue is full disk space. An almost sure sign of this is the appearance of halted writing in the log files. If logging stopped during a write, such as mid-word, the hard disks may be full. To confirm this, run this command and check the percentages in the Use% column: 
# df -h
In addition to log files, you can obtain valuable information by retrieving the status of your Red Hat Satellite and its various components. This can be done with the command: 
# /usr/sbin/rhn-satellite status
In addition, you can obtain the status of components such as the Apache Web server and the Red Hat Network Task Engine individually. For instance, to view the status of the Apache Web server, run the command: 
# service httpd status

10.2. Installing and Updating

Q:
SELinux keeps giving me messages when I'm trying to install. Why?
A:
If you encounter any issues with SELinux messages (such as AVC denial messages) while installing Red Hat Satellite, be sure to have the audit.log files available so that Red Hat Support personnel can assist you. You can find the file in /var/log/audit/audit.log and can attach the file to your Support ticket for engineers to assist you.
Q:
I changed /var/satellite to an NFS mount, and now SELinux is stopping it from working properly. What do I need to do?
A:
SELinux parameters need to be changed based on the new NFS mount in order for SELinux to allow that traffic. Do this with the command: 
# /usr/sbin/setsebool -P spacewalk_nfs_mountpoint on
If you are using Red Hat Enterprise Linux 6, you will also need to run the command: 
# /usr/sbin/setsebool -P cobbler_use_nfs on
Q:
My Satellite is failing. Any idea why?
A:
Do not subscribe the Red Hat Satellite to any of the following child channels available from Red Hat Network's central servers:
  • Red Hat Developer Suite
  • Red Hat Application Server
  • Red Hat Extras
  • JBoss product channels
Subscribing to these channels and updating the Satellite might install newer, incompatible versions of critical software components, causing the Satellite to fail.

10.3. Services

Q:
Why isn't the Apache Web server running?
A:
If the Apache Web server isn't running, entries in your /etc/hosts file may be incorrect.
Q:
How do I find out what the status of the Red Hat Network Task Engine is?
A:
To obtain the status of the Red Hat Network Task Engine, run the command: 
# service taskomatic status
Q:
How do I find out what the status of the Satellite's Embedded Database is?
A:
To view the status of the Satellite's Embedded Database, if it exists, run the command: 
# db-control status
Q:
What do I do if the push capability of the Red Hat Satellite stops working?
A:
If the push capability of the Red Hat Satellite ceases to function, it is possible that old log files may be at fault. Stop the jabberd daemon before removing these files. To do so, issue the following commands as root: 
# service jabberd stop
# rm -f /var/lib/jabberd/db/_db*
# service jabberd start

10.4. Connectivity

Q:
I can't connect! How do I work out what is wrong?
A:
The following measures can be used to troubleshoot general connection errors:
  • Attempt to connect to the Red Hat Satellite's database at the command line using the correct connection string as found in /etc/rhn/rhn.conf: 
    # sqlplus username/password@sid
  • Make sure that Red Hat Satellite is using Network Time Protocol (NTP) and set to the appropriate time zone. This also applies to all client systems and the separate database machine in Red Hat Satellite with Stand-Alone Database.
  • Confirm the correct package: 
    rhn-org-httpd-ssl-key-pair-MACHINE_NAME-VER-REL.noarch.rpm
    is installed on the Red Hat Satellite and the corresponding rhn-org-trusted-ssl-cert-*.noarch.rpm or raw CA SSL public (client) certificate is installed on all client systems.
  • Verify the client systems are configured to use the appropriate certificate.
  • If also using one or more Red Hat Satellite Proxy Servers, ensure each Proxy's SSL certificates are prepared correctly. The Proxy should have both its own server SSL key-pair and CA SSL public (client) certificate installed, since it will serve in both capacities. See the SSL Certificates chapter of the Red Hat Satellite Client Configuration Guide for specific instructions.
  • Make sure client systems are not using firewalls of their own, blocking required ports as identified in the Red Hat Satellite Installation Guide's Additional Requirements section.
Q:
What do I do if importing or synchronizing a channel fails and I can't recover it?
A:
If importing/synchronizing a channel fails and you can't recover it in any other way, run this command to delete the cache: 
# rm -rf temporary-directory

Note

The Red Hat Satellite Installation Guide section on Preparing for Import from Local Media specifies /var/rhn-sat-import/ as the temporary directory.
Next, restart the importation or synchronization.
Q:
I'm getting "SSL_CONNECT" errors. What do I do now?
A:
A common connection problem, indicated by SSL_CONNECT errors, is the result of a Satellite being installed on a machine whose time had been improperly set. During the Satellite installation process, SSL certificates are created with inaccurate times. If the Satellite's time is then corrected, the certificate start date and time may be set in the future, making it invalid.
To troubleshoot this, check the date and time on the clients and the Satellite with the following command: 
# date
The results should be nearly identical for all machines and within the "notBefore" and "notAfter" validity windows of the certificates. Check the client certificate dates and times with the following command: 
# openssl x509 -dates -noout -in /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
Check the Satellite server certificate dates and times with the following command: 
# openssl x509 -dates -noout -in /etc/httpd/conf/ssl.crt/server.crt
By default, the server certificate has a one-year life while client certificates are good for 10 years. If you find the certificates are incorrect, you can either wait for the valid start time, if possible, or create new certificates, preferably with all system times set to GMT.

10.5. Logging and Reporting

Q:
What are the different log files?
A:
Virtually every troubleshooting step should start with a look at the associated log file or files. These provide invaluable information about the activity that has taken place on the device or within the application that can be used to monitor performance and ensure proper configuration. See Table 10.1, “Log Files” for the paths to all relevant log files:
There may be numbered log files (such as /var/log/rhn/rhn_satellite_install.log.1, /var/log/rhn/rhn_satellite_install.log.2, etc.) within the /var/log/rhn/ directory. These are rotated logs, which are log files created with a .<NUMBER> extension when the current rhn_satellite_install.log file fills up to a size as specified by the logrotate(8) daemon and the contents written to a rotated log file. For example, the rhn_satellite_install.log.1 contains the oldest rotated log file, while rhn_satellite_install.log.4 contains the most recently rotated log.
Table 10.1. Log Files
Component/Task Log File Location
Apache Web server /var/log/httpd/ directory
Red Hat Satellite /var/log/rhn/ directory
Red Hat Satellite Installation Program /var/log/rhn/rhn_satellite_install.log
Database installation - Embedded Database /var/log/rhn/install_db.log
Database population /var/log/rhn/populate_db.log
Red Hat Satellite Synchronization Tool /var/log/rhn/rhn_server_satellite.log
Red Hat Network DB Control - Embedded Database /var/log/rhn/rhn_database.log
Red Hat Network Task Engine (taskomatic) /var/log/messages
yum /var/log/yum.log
XML-RPC transactions /var/log/rhn/rhn_server_xmlrpc.log
Q:
How do I use spacewalk-report?
A:
There are instances where administrators may need a concise, formatted summary of their Red Hat Satellite resources, whether it is to take inventory of their entitlements, subscribed systems, or users and organizations. Rather than gathering such information manually from the Satellite interface, Red Hat Satellite includes the spacewalk-report command to gather and display vital Satellite information at once.

Note

To use spacewalk-report you must have the spacewalk-reports package installed.
The logging feature of Satellite is added by default in fresh installations of Satellite version 5.6 and above. If the Satellite is upgraded from a version below 5.6 the logging feature will be turned on at the time of upgrade and from that point all events will be audited. This means that all users created before the upgrade will get logged from the time of upgrade. The past creation of a user and any past events will not appear in the log but all future events will be logged.
spacewalk-report allows administrators to organize and display reports about content, errata, systems, system event history, and user resources across the Satellite. The spacewalk-report command is used to generate reports on:
  • System Inventory - Lists all of the systems registered to the Satellite.
  • Entitlements - Lists all organizations on the Satellite, sorted by system or channel entitlements.
  • Errata - Lists all the errata relevant to the registered systems, sorts errata by severity as well as the systems that apply to a particular erratum.
  • Users - Lists all the users registered to the Satellite, and lists any systems associated with a particular user.
  • System History - Lists all, or a subset, of the system events that have occurred.
To get a report in CSV format, run the following at the command prompt of your Satellite server. 
# spacewalk-report report_name
The following reports are available:
Table 10.2. spacewalk-report Reports
Report Invoked as Description
Group Audit audit-server-groups Audit of user changes in group
Server Audit audit-servers Audit of server changes
User Audit audit-users Audit of user changes
Packages Report channel-packages Lists the packages, as well as the channels they are in
Channels channels Lists the channels available on the server
Cloned Channels cloned-channels Lists channels that have been cloned
Custom Information custom-info Displays any custom information about the system
Entitlements entitlements Lists all organizations on the Satellite with their system or channel entitlements
Errata in channels errata-channels Lists errata in channels
Errata Compliance errata-list Lists the details of errata out of compliance information
All Errata errata-list-all Complete list of all errata
Errata for systems errata-systems Lists applicable errata and any registered systems that are affected
Relationship Mapping host-guests Provides host-guest mapping details
Inactive Systems inactive-systems
System Inventory inventory List of systems registered to the server, together with hardware and software information
Kickstart Trees kickstartable-trees Lists trees able to be kickstarted
Package Update packages-updates-all List of all packages that can be updated
Newest Package Update package-updates-newest Lists the newest updates to packages
SCAP Scans
scap-scan
scap-scan-results
Displays the results of an OpenSCAP xccdf evaluation
Splice Reporting splice-export Displays system data needed for splice integration for enhanced reporting
Crash Count system-crash-count Displays the number of times systems have crashed
Crash Details system-crash-details Lists the systems' crash details
System Currency system-currency Lists system currency values
System Groups system-groups Lists system groups in the Satellite server
Group Activation keys system-groups-keys Lists all existing activation keys for the system groups
Systems in System Groups system-groups-systems Lists all system groups and systems within each group
Users in System Groups system-groups-users Lists all system groups and their affiliated users
System history system-history Lists system event history
System history channels system-history-channels Lists system event history
System history configuration system-history-configuration Lists system configuration event history
System history entitlements system-history-entitlements Lists system entitlement event history
System history errata system-history-errata Lists system errata event history
System history kickstart system-history-kickstart Lists system kickstart and provisioning event history
System history packages system-history-packages Lists system package event history
SCAP Event History system-history-scap Lists systems' OpenSCAP event history
Installed Packages system-packages-installed Lists all packages installed on the systems
Users in the system users Lists all users registered to the Satellite
Systems administered users-systems Lists systems that can be administered by individual users
For more information about an individual report, run spacewalk-report with the --info or --list-fields-info and the report name. The description and list of possible fields in the report will be shown.
For further information, the spacewalk-report(8) manpage as well as the --help parameter of the spacewalk-report program can be used to get additional information about the program invocations and their options.
Q:
How do I work out what version of the database schema I have?
A:
To determine the version of your database schema, run the command: 
# rhn-schema-version
Q:
How do I work out what character set types I have?
A:
To derive the character set types of your Satellite's database, run the command: 
# rhn-charsets
Q:
Why isn't the administrator getting email?
A:
If the administrator is not getting email from the Red Hat Satellite, confirm the correct email addresses have been set for traceback_mail in /etc/rhn/rhn.conf.
Q:
How do I change the sender of the traceback mail?
A:
If the traceback mail is marked from dev-null@rhn.redhat.com and you would like the address to be valid for your organization, include the web.default_mail_from option and appropriate value in /etc/rhn/rhn.conf.

10.6. Errors

Q:
I'm getting an "Error validating satellite certificate" error during a Red Hat Satellite installation. How do I fix it?
A:
An "Error validating satellite certificate" error message during a Red Hat Satellite installation is caused by having an HTTP proxy in the environment. This can be confirmed by looking at the install.log file, and locating the following error: 
ERROR: unhandled exception occurred:
Traceback (most recent call last):
  File "/usr/bin/rhn-satellite-activate", line 45, in ?
    sys.exit(abs(mod.main() or 0))
  File "/usr/share/rhn/satellite_tools/rhn_satellite_activate.py", line 585, in main
    activateSatellite_remote(options)
  File "/usr/share/rhn/satellite_tools/rhn_satellite_activate.py", line 291, in activateSatellite_remote
    ret = s.satellite.deactivate_satellite(systemid, rhn_cert)
  File "/usr/lib/python2.4/site-packages/rhn/rpclib.py", line 603, in __call__
    return self._send(self._name, args)
  File "/usr/lib/python2.4/site-packages/rhn/rpclib.py", line 326, in _request
    self._handler, request, verbose=self._verbose)
  File "/usr/lib/python2.4/site-packages/rhn/transports.py", line 171, in request
    headers, fd = req.send_http(host, handler)
  File "/usr/lib/python2.4/site-packages/rhn/transports.py", line 698, in send_http
    self._connection.connect()
  File "/usr/lib/python2.4/site-packages/rhn/connections.py", line 193, in connect
    sock.connect((self.host, self.port))
  File "<string>", line 1, in connect
socket.timeout: timed out
To resolve the issue:
  1. Run the install script in disconnected mode, and skip the database installation which has already been done: 
    # ./install.pl --disconnected --skip-db-install
  2. Open /etc/rhn/rhn.conf with your preferred text editor, and add or modify the following line: 
    server.satellite.rhn_parent = satellite.rhn.redhat.com
    Remove the following line: 
    disconnected=1
    If you are using a proxy for the connection to Red Hat Network, you will also need to add or modify the following lines to reflect the proxy settings. 
    server.satellite.http_proxy = <hostname>:<port>
server.satellite.http_proxy_username = <username>
server.satellite.http_proxy_password = <password>
  3. Re-activate Satellite in connected mode, using the rhn-satellite-activate command as the root user, including the path and filename of the Satellite manifest: 
    # rhn-satellite-activate --manifest=manifest.zip --verbose
Alternatively, try running the install.pl script in connected mode, but with the --answer-file=answer file option. Ensure the answer file has the HTTP proxy information specified as follows: 
rhn-http-proxy = <hostname>:<port>
rhn-http-proxy-username = <username>
rhn-http-proxy-password = <password>
Q:
I'm getting an "ERROR: server.mount_point not set in the configuration file" error when I try to activate or synchronize the Red Hat Satellite. How do I fix it?
A:
An "ERROR: server.mount_point not set in the configuration file" error during Red Hat Satellite activation or synchronization can occur if the mount_point configuration parameter in /etc/rhn/rhn.conf does not point to a directory path, or the directory path it points to is not present or does not have permission to access the directory.
To resolve the issue, check the value of the mount_point configuration parameter in /etc/rhn/rhn.conf. If it set to the default value of /var/satellite, verify that the /var/satellite and /var/satellite/redhat directories exist. For all values, check that path to the file is accurate, and that the permissions are set correctly.
Q:
Why does cobbler check give an error saying that it needs a different version of yum-utils?
A:
Sometimes, running the cobbler check command can give an error similar to the following: 
# cobbler check
The following potential problems were detected:
#0: yum-utils need to be at least version 1.1.17 for reposync -l, current version is 1.1.16
This is a known issue in Cobbler's reposync package. The error is spurious and can be safely ignored. This error will be resolved in future versions of Red Hat Satellite.
Q:
I'm getting an "unsupported version" error when I try to activate the Red Hat Satellite certificate. How do I fix it?
A:
If your Red Hat Satellite certificate has become corrupted, you could get one of the following errors: 
ERROR: <Fault -2: 'unhandled internal exception: unsupported version: 96'>

RHN_PARENT: satellite.rhn.redhat.com
     Error reported from RHN: <Fault -2: 'unhandled internal exception: unsupported version: 115'>
     ERROR: unhandled XMLRPC fault upon remote activation: <Fault -2: 'unhandled internal exception: unsupported version: 115'>
     ERROR: <Fault -2: 'unhandled internal exception: unsupported version: 115'>

Invalid satellite certificate
To resolve this issue, contact Red Hat support services for a new certificate.
Q:
I'm getting an "Internal Server Error" complaining about ASCII when I try to edit the kickstart profile. What's going on?
A:
If you have recently added some kernel parameters to your kickstart profile, you might find that when you attempt to View a List of Kickstart Profiles that you get the following Internal Server Error: 
'ascii' codec can't encode character u'\u2013'
This error occurs because some text in the profile is not being recognized correctly.
To resolve the issue:
  1. Ssh directly onto the Satellite server as the root user: 
    # ssh root@satellite.fqdn.com
  2. Find the kickstart profile that is causing the problem by looking at the dates of the files in /var/lib/cobbler/config/profiles.d and locating the one that was edited most recently: 
    # ls -l /var/lib/cobbler/config/profiles.d/
  3. Open the profile in your preferred text editor, and locate the following text: 
    \u2013hostname
    Change the entry to read: 
    --hostname
  4. Save changes to the profile and close the file.
  5. Restart the Red Hat Satellite services to pick up the updated profile: 
    # rhn-satellite restart
Shutting down rhn-satellite...
Stopping RHN Taskomatic...
Stopped RHN Taskomatic.
Stopping cobbler daemon:                                   [  OK  ]
Stopping rhn-search...
Stopped rhn-search.
Stopping MonitoringScout ...                               [  OK  ]
Stopping Monitoring ...                                    [  OK  ]
Stopping httpd:                                            [  OK  ]
Stopping tomcat5:                                          [  OK  ]
Shutting down osa-dispatcher:                              [  OK  ]
Shutting down Oracle Net Listener ...                      [  OK  ]
Shutting down Oracle DB instance "rhnsat" ...              [  OK  ]
Shutting down Jabber router:                               [  OK  ]
Done.
Starting rhn-satellite...
Starting Jabber services                                   [  OK  ]
Starting Oracle Net Listener ...                           [  OK  ]
Starting Oracle DB instance "rhnsat" ...                   [  OK  ]
Starting osa-dispatcher:                                   [  OK  ]
Starting tomcat5:                                          [  OK  ]
Starting httpd:                                            [  OK  ]
Starting Monitoring ...                                    [  OK  ]
Starting MonitoringScout ...                               [  OK  ]
Starting rhn-search...
Starting cobbler daemon:                                   [  OK  ]
Starting RHN Taskomatic...
Done.
  6. Return to the web interface. Note that the interface can take some time to resolve the services. It should return to normal after some time.
Q:
I'm getting "Host Not Found" or "Could Not Determine FQDN" errors. What do I do now?
A:
Because Red Hat Network configuration files rely exclusively on fully qualified domain names (FQDNs), it is imperative that key applications are able to resolve the name of the Red Hat Satellite into an IP address. Red Hat Update Agent, Red Hat Network Registration Client, and the Apache Web server are particularly prone to this problem with the Red Hat Network applications issuing errors of "host not found" and the Web server stating "Could not determine the server's fully qualified domain name" upon failing to start.
This problem typically originates from the /etc/hosts file. You may confirm this by examining /etc/nsswitch.conf, which defines the methods and the order by which domain names are resolved. Usually, the /etc/hosts file is checked first, followed by Network Information Service (NIS) if used, followed by DNS. One of these has to succeed for the Apache Web server to start and the Red Hat Network client applications to work.
To resolve this problem, identify the contents of the /etc/hosts file. It may look like this: 
127.0.0.1 this_machine.example.com this_machine localhost.localdomain \ localhost
First, in a text editor, remove the offending machine information, like so: 
127.0.0.1 localhost.localdomain.com localhost
Then, save the file and attempt to re-run the Red Hat Network client applications or the Apache Web server. If they still fail, explicitly identify the IP address of the Satellite in the file, such as: 
127.0.0.1 localhost.localdomain.com localhost
123.45.67.8 this_machine.example.com this_machine
Replace the value here with the actual IP address of the Satellite. This should resolve the problem. Keep in mind, if the specific IP address is stipulated, the file will need to be updated when the machine obtains a new address.
Q:
I'm getting a "This server is not an entitled Satellite" when I try to synchronize the Red Hat Satellite server. How do fix it?
A:
If cdn-sync reports that the server is not activated as a Red Hat Satellite, it isn't subscribed to the respective Red Hat Satellite channel. If this is a newly installed system, make sure that the satellite manifest is activated on the system. If it was activated earlier, then it has become deactivated.
Check the system's child channels to discover if it is subscribed to any Red Hat Network Red Hat Satellite channel. View subscribed channels with the following command: 
# yum repolist
Activate the same Satellite manifest again on your Satellite using this command as the root user: 
# rhn-satellite-activate --manifest=manifest.zip --verbose

10.7. Web Interface

Q:
I'm having problems with the Red Hat Satellite user interface. Which log files should I check?
A:
If you experience errors viewing, scheduling, or working with kickstarts in the Red Hat Satellite user interface, check the /var/log/tomcat6/catalina.out log file.
For all other user interface errors, check the /var/log/httpd/error_log log file.

10.8. Anaconda

Q:
I'm getting an error that says Error downloading kickstart file. What is the problem and how do I fix it?
A:
This error is usually the result of a network issue. To locate the problem, run the cobbler check command, and read the output, which should look something like this: 
# cobbler check
The following potential problems were detected:
#0: reposync is not installed, need for cobbler reposync, install/upgrade yum-utils?
#1: yumdownloader is not installed, needed for cobbler repo add with --rpm-list parameter, install/upgrade yum-utils?
#2: The default password used by the sample templates for newly installed machines (default_password_crypted in /etc/cobbler/settings) is still set to 'cobbler' and should be changed
#3: fencing tools were not found, and are required to use the (optional) power management features. install cman to use them
If cobbler check does not provide any answers, check the following:
  • Verify httpd is running: service httpd status
  • Verify cobblerd is running: service cobblerd status
  • Verify that you can fetch the kickstart file using wget from a different host: 
    wget http://satellite.example.com/cblr/svc/op/ks/profile/rhel5-i386-u3:1:Example-Org
Q:
I'm getting a package installation error that says The file chkconfig-1.3.30.1-2.i386.rpm cannot be opened. What is the problem and how do I fix it?
A:
Clients will fetch content from Red Hat Satellite based on the --url parameter in the kickstart. For example: 
url --url http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3
If you receive errors from Anaconda stating it can't find images or packages, check that the URL in the kickstart will generate a 200 OK response. You can do this by attempting to wget the file located at that URL: 
wget http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3
--2011-08-19 15:06:55--  http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3
Resolving satellite.example.com... 10.10.77.131
Connecting to satellite.example.com|10.10.77.131|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: `ks-rhel-i386-server-5-u3.1'
2011-08-19 15:06:55 (0.00 B/s) - `ks-rhel-i386-server-5-u3.1' saved [0/0]
If you get a response other than 200 OK, check the error logs to find out what the problem is. You can also check the actual file Anaconda tried to download by searching the access_log file: 
# grep chkconfig /var/log/httpd/access_log
10.10.77.131 - - [19/Aug/2011:15:12:36 -0400] "GET /rhn/common/DownloadFile.do?url=/ks/dist/ks-rhel-i386-server-
5-u3/Server  /chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1" 206 24744 "-" "urlgrabber/3.1.0 yum/3.2.19"
10.10.76.143 - - [19/Aug/2011:15:12:36 -0400] "GET /ks/dist/ks-rhel-i386-server-5-u3/Server/chkconfig-
1.3.30.1-2.i386.rpm HTTP/1.1" 206 24744 "-" "urlgrabber/3.1.0 yum/3.2.19"
10.10.76.143 - - [19/Aug/2011:15:14:20 -0400] "GET /ks/dist/ks-rhel-i386-server-5-u3/Server/chkconfig-
1.3.30.1-2.i386.rpm HTTP/1.1" 200 162580 "-" "urlgrabber/3.1.0 yum/3.2.19"
10.10.77.131 - - [19/Aug/2011:15:14:20 -0400] "GET /rhn/common/DownloadFile.do?url=/ks/dist/ks-rhel-i386-server-
5-u3/Server/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1" 200 162580 "-" "urlgrabber/3.1.0 yum/3.2.19"
If the requests are not appearing in the access_log file, the system might be having trouble with the networking setup. If the requests are appearing but are generating errors, check the error logs.
You can also try manually downloading the files to see if the package is available: 
wget http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3/Server/chkconfig-1.3.30.1-2.i386.rpm

10.9. Tracebacks

Q:
I'm getting emails with "WEB TRACEBACK" in the subject. What should I do about them?
A:
A typical traceback email might look something like this: 
Subject: WEB TRACEBACK from satellite.example.com
Date: Wed, 19 Aug 2011 20:28:01 -0400
From:Red Hat Satellite <dev-null@redhat.com>
To: admin@example.com

java.lang.RuntimeException: XmlRpcException calling cobbler.
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:72)
	at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:76)
	at com.redhat.rhn.taskomatic.task.SingleThreadedTestableTask.execute(SingleThreadedTestableTask.java:54)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
	at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:434)
	at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
	at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
	... 4 more
Caused by: java.io.IOException: Server returned HTTP response code: 503 for URL: http://someserver.example.com:80/cobbler_api
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1236)
	at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:420)
	... 7 more
This indicates that there has been a problem with Cobbler communicating with the taskomatic service. Try checking the following:
  • Verify httpd is running: # service httpd status
  • Verify cobblerd is running: # service cobblerd status
  • Verify that there are no firewall rules that would prevent localhost connections

10.10. Registration

Q:
The rhnreg_ks command is failing when I run it, saying ERROR: unable to read system id. What is the problem?
A:
At the end of the kickstart file, there is a %post section that registers the machine to the Red Hat Satellite: 
# begin Red Hat management server registration
mkdir -p /usr/share/rhn/
wget http://satellite.example.com/pub/RHN-ORG-TRUSTED-SSL-CERT -O /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
perl -npe 's/RHNS-CA-CERT/RHN-ORG-TRUSTED-SSL-CERT/g' -i /etc/sysconfig/rhn/*
rhnreg_ks --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=1-c8d01e2f23c6bbaedd0f6507e9ac079d
# end Red Hat management server registration
Interpreting this in the order it was added in, this will:
  • Create a directory to house the custom SSL cert used by the Red Hat Satellite.
  • Fetch the SSL certificate to use during registration.
  • Search and replace the SSL certificate strings from the rhn_register configuration files, and then register to the Red Hat Satellite using the SSL certificate and an activation key. Every kickstart profile includes an activation key that assures that the system is assigned the correct base and child channels, and gets the correct system entitlements. If it is a reprovisioning of an existing system, the activation key will also ensure it is associated with the previous system profile.
If the rhnreg_ks command fails, you might see errors like this in the ks-post.log log file: 
ERROR: unable to read system id.
These errors will also occur if an attempt is made to perform an rhn_check and the system has not registered to the Red Hat Satellite.
The best way to troubleshoot this is to view the kickstart file and copy and paste the four steps directly at the command prompt after the kickstart has completed. This will produce error messages that are more detailed to help locate the problem.

10.11. Kickstarts and Snippets

Q:
What is the directory structure for kickstarts?
A:
The base path where the kickstart files are stored is /var/lib/rhn/kickstarts/. Within this directory, raw kickstarts are in the upload subdirectory, and wizard-generated kickstarts are in the wizard subdirectory: 
Raw Kickstarts: /var/lib/rhn/kickstarts/upload/$profile_name--$org_id.cfg
Wizard Kickstarts: /var/lib/rhn/kickstarts/wizard/$profile_name--$org_id.cfg
Q:
What is the directory structure for Cobbler snippets?
A:
Cobbler snippets are stored in /var/lib/rhn/kickstarts/snippets. Cobbler accesses snippets using the symbolic link /var/lib/cobbler/snippets/spacewalk. 
Snippets:  /var/lib/rhn/kickstarts/snippets/$org_id/$snippet_name

Important

Red Hat Satellite RPMs expect the Cobbler kickstart and snippet directories to be in their default locations, do not change them.

10.12. Multi-Organization Satellites and Satellite Certificate

Q:
How do I register my systems in a Multiple Organization environment when I do not have enough entitlements in my Satellite Certificate?
A:
There are some situations in which you need to free entitlements and do not have a lot of time to do so, and may not have access to each organization in order to do this yourself. There is an option in Multi-Org Satellites that allows the Satellite administrator to reduce an organization's entitlement count below their usage. This method must be done logged into the administrative organization.
For example, logged into the administrative organization, if your certificate is 5 system management entitlements shy of being able to cover all registered systems on your Satellite, the 5 systems that were most recently registered to that organization will be unentitled. This process is described below:
  1. In the /etc/rhn/rhn.conf file, set web.force_unentitlement to 1.
  2. Restart the Satellite.
  3. Reduce the allocated entitlements to the desired organizations either via each organization's Subscriptions tab or via individual entitlement's Organizations tabs.
  4. A number of systems in the organization should now be in an unentitled state. The number of systems unentitled in the organization will be equal to the difference between the total number of entitlements you removed from the organization and the number of entitlements the organization did not have applied to the systems.
    For example, if you removed 10 entitlements from the organization in step 3, and the organization has 4 entitlements that were not in use by systems, then 6 systems in the organization will be unentitled.
After you have the sufficient number of entitlements required, you should then be able to activate your new Satellite certificate. Note that modifying the web.force_unentitlement variable is only necessary to reduce an organization's allocated entitlements below what they are using. If an organization has more entitlements than are being actively used, you do not need to set this variable to remove them.
Q:
I have extra entitlements on my Satellite Certificate that are not being used. What happens to these entitlements?
A:
If you are issued a new Satellite certificate and it has more entitlements than are being consumed on your Satellite, any extra entitlements will be assigned to the administrative organization. If you log into the web interface as the Satellite administrator, you will be able to allocate these entitlements to other organizations. The previously-allocated entitlements to other organizations will be unaffected.

10.13. Proxy Installation and Configuration

Q:
After configuring the Red Hat Network Package Manager how can I determine if the local packages were successfully added to the private Red Hat Network channel?
A:
Use the command rhn_package_manager -l -c "name_of_private_channel" to list the private channel packages known to the Satellite. Or visit the Satellite interface.
After subscribing a registered system to the private channel, you can also execute the command yum --disablerepo="*" --enablerepo="your_repo_name" list available on the registered system and look for the packages from the private Satellite channel.
Q:
How can I determine whether the clients are connecting to the Squid server?
A:
The /var/log/squid/access.log file logs all connections to the Squid server.
Q:
The Red Hat Update Agent on the client systems does not connect through the Red Hat Satellite Proxy. How can I resolve this error?
A:
Make sure that the latest version of the Red Hat Update Agent is installed on the client systems. The latest version contains features necessary to connect through a Red Hat Satellite Proxy. The latest version can be obtained through the Red Hat Network by issuing the command yum update yum as root or from http://www.redhat.com/support/errata/.
The Red Hat Satellite Proxy is an extension of Apache. See the Log Files section of the Red Hat Satellite Proxy Installation Guide for its log file location.
Q:
My Red Hat Satellite Proxy configuration does not work. Where do I begin troubleshooting it?
A:
Make sure /etc/sysconfig/rhn/systemid is owned by root.apache with the permissions 0640.
Read the log files. A list is available on the Log Files section of the Red Hat Satellite Proxy Installation Guide.
Q:
How do I troubleshoot general problems in the Red Hat Satellite Proxy?
A:
To begin troubleshooting general problems, examine the log file or files related to the component exhibiting failures.
A common issue is full disk space. An almost sure sign of this is the appearance of halted writing in the log files. If logging stops during a write, such as mid-word, you likely have filled disks. To confirm this, run this command and check the percentages in the Use% column: 
df -h
In addition to log files, you can obtain valuable information by retrieving the status of your various components. This can be done for the Apache Web server and Squid.
To obtain the status of the Apache Web server, run the command: 
service httpd status
To obtain the status of Squid, run the command: 
service squid status
If the administrator is not getting email from the Red Hat Satellite Proxy, confirm the correct email addresses have been set for traceback_mail in /etc/rhn/rhn.conf.
Q:
My Red Hat Satellite Proxy encountered the error "Host Not Found"/"Could not Determine FQDN". What should I do?
A:
Because Red Hat Network configuration files rely exclusively on fully qualified domain names (FQDN), it is imperative that key applications are able to resolve the name of the Red Hat Satellite Proxy into an IP address. Red Hat Update Agent, Red Hat Network Registration Client, and the Apache Web server are particularly prone to this problem with the Red Hat Network applications issuing errors of "host not found" and the Web server stating "Could not determine the server's fully qualified domain name" upon failing to start.
This problem originates from the /etc/hosts file. Confirm this by examining the /etc/nsswitch.conf file, which defines the methods and the order by which domain names are resolved. Usually, the /etc/hosts file is checked first, followed by Network Information Service (NIS) if it is being used, followed by DNS. One of these has to succeed for the Apache Web server to start and the Red Hat Network client applications to work.
To resolve this problem, identify the contents of the /etc/hosts file. It may look like this: 
127.0.0.1 this_machine.example.com this_machine localhost.localdomain \ localhost
In a text editor, remove the machine host information from the file, it should look like so: 
127.0.0.1 localhost.localdomain.com localhost
Save the file and attempt to re-run the Red Hat Network client applications or the Apache Web server. If they still fail, explicitly identify the IP address of the Proxy in the file, such as: 
127.0.0.1 localhost.localdomain.com localhost
123.45.67.8 this_machine.example.com this_machine
Replace the value here with the actual IP address of the Proxy. This should resolve the problem. Keep in mind, if the specific IP address is stipulated, the file will need to be updated when the machine obtains a new address.
Q:
I am having issues with Red Hat Satellite Proxy and network connection errors. What should I do?
A:
If you are experiencing problems that you believe to be related to failed connections, follow these measures:
  • Confirm the correct package: 
     rhn-org-httpd-ssl-key-pair-MACHINE_NAME-VER-REL.noarch.rpm
    is installed on the Red Hat Satellite Proxy and the corresponding rhn-org-trusted-ssl-cert-*.noarch.rpm or raw CA SSL public (client) certificate is installed on all client systems.
  • Verify the client systems are configured to use the appropriate certificate.
  • If using one or more Red Hat Satellite Proxies, ensure each Proxy's SSL certificate is prepared correctly. If using the Red Hat Satellite Proxy in conjunction with a Red Hat Satellite, the Proxy should have both its own server SSL key-pair and CA SSL public (client) certificate installed, since it will serve in both capacities. See the SSL Certificates chapter of the Red Hat Satellite Client Configuration Guide for specific instructions.
  • If the Red Hat Satellite Proxy is connecting through an HTTP Proxy, make sure the URL listed is valid. For instance, the HTTP Proxy URL field should not contain references to protocols, such as http:// or https://. Only the hostname and port should be included in the form hostname:port, such as your-gateway.example.com:8080.
  • Make sure client systems are not using firewalls of their own, blocking required ports, as identified in the Additional Requirements section of the Red Hat Satellite Proxy Installation Guide.
Q:
I am having issues with package delivery errors and object corruption. What should I check for?
A:
If package delivery fails or an object appears to be corrupt, and it is not related to connection errors, you should consider clearing the caches. The Red Hat Satellite Proxy has two caches you should be concerned with: one for Squid and the other for authentication.
The Squid cache is located in /var/spool/squid/. To clear it:
  1. Stop the Apache Web server: service httpd stop
  2. Stop the Squid server: service squid stop
  3. Delete the contents of that directory: rm -fv /var/spool/squid/*
  4. Restart both services: 
    service squid start
service httpd start
The same task can be accomplished quicker by just clearing the directory and restarting squid, but this method will most likely result in a number of Red Hat Network traceback messages.
The internal caching mechanism used for authentication by the Proxy may also need its cache cleared. To do this, issue the following command: 
 rm -fv /var/spool/squid/*

Note

If you have exhausted these troubleshooting steps or want to defer them to Red Hat Network professionals, Red Hat recommends you take advantage of the strong support that comes with Red Hat Satellite. The most efficient way to do this is to aggregate your Satellite's configuration parameters, log files, and database information and send this package directly to Red Hat.
Red Hat Network provides a command line tool explicitly for this purpose: The Satellite Diagnostic Info Gatherer, commonly known by its command satellite-debug. To use this tool, issue the command as root. You will see the pieces of information collected and the single tarball created, like so: 
# satellite-debug
Collecting and packaging relevant diagnostic information.
Warning: this may take some time...
    * copying configuration information
    * copying logs
    * querying RPM database (versioning of Red Hat Satellite, etc.)
    * querying schema version and database character sets
    * get diskspace available
    * timestamping
    * creating tarball (may take some time): /tmp/satellite-debug.tar.bz2
    * removing temporary debug tree

Debug dump created, stored in /tmp/satellite-debug.tar.bz2
Deliver the generated tarball to your Red Hat Network contact or support channel.
Once finished, email the new file from the /tmp/ directory to your Red Hat representative for immediate diagnosis.
Additionally, Red Hat provides a command line tool called the SoS Report, commonly known by its command sosreport. This tool collects your Proxy's configuration parameters, log files, and database information and sends it directly to Red Hat.
To use this tool for Red Hat Satellite information, you must have the sos package installed. Type sosreport -o satellite as root on the Satellite server to create a report. For example: 
[root@satserver ~]# sosreport -o satellite

sosreport (version 3.2)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/tmp and may be provided to a Red Hat support representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.
You are then prompted for your first initial and last name, then a support case number.
It may take several minutes for the system to generate and archive the report to a compressed file. Once finished, email the new file from the /tmp/ directory to your Red Hat representative for immediate diagnosis.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.