红帽全球峰会此内容没有您所选择的语言版本。
Chapter 2. Log Files and Validation Scripts
This chapter helps you locate various log files and other information that can aid in troubleshooting some of the common issues with OpenShift Enterprise.
OpenShift Enterprise uses mostly standard Linux components such as networking, httpd, SELinux, and others. Any prior administration experience with these components will be helpful in troubleshooting common issues with your OpenShift Enterprise deployment.
2.1.1. General Information
复制链接链接已复制到粘贴板!
General information can be found in the
/var/log/messages file. This serves as a good starting point to investigate issues that might not be logged anywhere else.
The
/var/log/httpd/access_log file shows whether your web request was received by the host.
The
/var/log/httpd/error_log file can be helpful in troubleshooting certain problems on broker and node hosts.
The
/var/log/audit/audit.log file is useful for finding problems that might be caused by SELinux violations.
The
/var/log/secure file logs user and SSH interactions. Because users can SSH into their gears, and all Git requests also authenticate using SSH, this file is useful for checking interaction with gears on node hosts.
2.1.2. Networking
复制链接链接已复制到粘贴板!
DNS
The best place for Linux operators to begin troubleshooting DNS problems on broker, node, or client hosts is the /etc/resolv.conf file. On client hosts running other operating systems, look in the appropriate network configuration file.
If your OpenShift Enterprise installation uses a BIND server, this should be listed in the
/etc/resolv.conf file as the first nameserver.
On client hosts, the first nameserver listed in the
/etc/resolv.conf file should point to your OpenShift Enterprise installation, either receiving updates from it, or delegating the domain to the nameserver of your installation.
If a hostname of your OpenShift Enterprise installation does not resolve correctly, use the following command to find the server the response is actually coming from:
The application hostname is a CNAME for the node host DNS record. However, for a scaled application, this command will only show which node host contains the HAProxy gear; other gears could reside on different node hosts.
# dig hostnameBIND
If you are running a BIND server on the broker (or supporting) host, the configuration information is contained in the /var/named/dynamic directory. The zone file syntax is domain.com.db.zone; so if the domain of your OpenShift Enterprise installation is example.com, the zone file name would be example.com.db.zone. However, not all changes will be in the zone file. Recent changes can be contained in a binary journal file.
Use the following command to view the entire zone according to the nameserver:
# dig domain axfrDHCP
For broker and node hosts, DHCP is currently only supported if the host IPs are pinned, meaning they do not change during lease renewal. This also applies to nameservers, in that they should also not change if pinned.
If DHCP is in use, networking parameters will update at boot time, or at lease renewal. If DNS resolution on an OpenShift Enterprise host stops working after initial installation, look in the
/etc/dhcp/dhclient-network-interface.conf file to verify the nameservers provided by the DHCP service are being overwritten when a new lease is obtained.
If your configuration in the
/etc/resolv.conf file is overwritten with incorrect values, check your configuration in the dhclient-network-interface.conf file.
2.1.3. SELinux
复制链接链接已复制到粘贴板!
When error messages indicate access denials but standard Linux file permissions appear to allow access, it could be due to SELinux policy. Use the following method for troubleshooting SELinux issues or verify whether the problem is SELinux related. Red Hat recommends this procedure for finding all SELinux related problems.
Procedure 2.1. To Troubleshoot SELinux Issues:
- As root, run the following command to set SELinux to permissive mode:
# setenforce 0 - Retry the failing action. If the action succeeds then the issue is SELinux related.
- Run the following command to set SELinux back to enforcing mode:
# setenforce 1 - Check the
/var/log/audit/audit.logfile for any SELinux violations.
This allows the offending action to proceed and log everything that otherwise would have been denied with SELinux enforced. In enforcing mode, not all denials are logged in the audit log, and the first denial generally blocks the action from proceeding to what might be further denials.
2.1.4. Control Groups on Node Hosts
复制链接链接已复制到粘贴板!
Control groups (cgroups) enable you to allocate resources such as CPU time, system memory, and network bandwidth among user-defined groups of tasks (processes) running on a system. When the
cgconfig service is running correctly on a node host, you see the following:
- The
/etc/cgconfig.conffile exists with the SELinux label:system_u:object_r:cgconfig_etc_t:s0. - The
/etc/cgconfig.conffile joins CPU, cpuacct, memory, freezer, and net_cls in the/croup/alldirectory. - The
/cgroupdirectory exists, with the SELinux label:system_u:object_r:cgroup_t:s0. - The cgconfig service is running.
When the cgred service is running correctly, you see the following:
- The
/etc/cgrules.conffile exists with the SELinux label:system_u:object_r:cgrules_etc_t:s0 - The cgred service is running.
If there are gears running on the node host, you also see the following:
- A line for each gear in the
/etc/cgrules.conffile. - A directory for each gear in the
/cgroup/all/openshiftdirectory. - All processes with the gear UUID are listed in the gear's
cgroup.procsfile. This file is located in the/cgroup/all/openshift/gear_UUIDdirectory.
Important
If you created the configuration files interactively as a root user, the SELinux user label would be
unconfined_u and not system_u. For example, the SELinux label in /etc/cgconfig.conf would be unconfined_u:object_r:cgconfig_etc_t:s0.
2.1.5. Pluggable Authentication Modules
复制链接链接已复制到粘贴板!
The pam_limits module controls access to system resources. Specifically, OpenShift Enterprise uses the
nproc value to control the number of processes a given account can create.
The default value for a new gear is configured in the
/etc/openshift/resource_limits.conf file on the node host:
limits_nproc=2048
When a new gear is created, a
84-gear_UUID.conf file is created on the node host, in the /etc/security/limits.d directory. Replace gear_UUID with the UNIX account name for the gear. This contains a rule set that defines the limits for that UNIX account. The first field of each line in the file is the gear UUID.
The
nproc limit for an individual gear is increased by changing the value in the 84-gear_UUID.conf file:
# PAM process limits for guest
# see limits.conf(5) for details
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
32ec916eeaa04032b1481af5037a6dfb hard nproc 250
If a gear process is failing, check the
nproc limit.
2.1.6. Disk Quotas
复制链接链接已复制到粘贴板!
Verify that the mount point for the
/var/lib/openshift directory has the usrquota option enabled in the /etc/fstab file, and has been mounted. Remount the directory if necessary using the command shown below, and check the output.
# mount -o remount filesystem
Use the following command to verify that the quotas are configured correctly:
# repquota -a2.1.7. iptables
复制链接链接已复制到粘贴板!
Network firewalls are implemented with iptables. Use the following command to view the current policies:
# iptables -L
The screen outputs for the
iptables -L command for both a broker host and a node host are shown below.
iptables Policy for Broker Host
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:domain
ACCEPT udp -- anywhere anywhere state NEW udp dpt:domain
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:https
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:61613
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
iptables Policy for Node Host
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:https
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http
ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:35531:65535
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}