SupportConsoleDevelopersStart a trialContact

Chapter 2. Log Files and Validation Scripts

This chapter helps you locate various log files and other information that can aid in troubleshooting some of the common issues with OpenShift Enterprise.

2.1. Configuration and Log Files for Standard Linux Components

OpenShift Enterprise uses mostly standard Linux components such as networking, httpd, SELinux, and others. Any prior administration experience with these components will be helpful in troubleshooting common issues with your OpenShift Enterprise deployment.

2.1.1. General Information

General information can be found in the /var/log/messages file. This serves as a good starting point to investigate issues that might not be logged anywhere else.
The /var/log/httpd/access_log file shows whether your web request was received by the host.
The /var/log/httpd/error_log file can be helpful in troubleshooting certain problems on broker and node hosts.
The /var/log/audit/audit.log file is useful for finding problems that might be caused by SELinux violations.
The /var/log/secure file logs user and SSH interactions. Because users can SSH into their gears, and all Git requests also authenticate using SSH, this file is useful for checking interaction with gears on node hosts.

2.1.2. Networking

DNS

The best place for Linux operators to begin troubleshooting DNS problems on broker, node, or client hosts is the /etc/resolv.conf file. On client hosts running other operating systems, look in the appropriate network configuration file.

If your OpenShift Enterprise installation uses a BIND server, this should be listed in the /etc/resolv.conf file as the first nameserver.
On client hosts, the first nameserver listed in the /etc/resolv.conf file should point to your OpenShift Enterprise installation, either receiving updates from it, or delegating the domain to the nameserver of your installation.
If a hostname of your OpenShift Enterprise installation does not resolve correctly, use the following command to find the server the response is actually coming from: 
# dig hostname
The application hostname is a CNAME for the node host DNS record. However, for a scaled application, this command will only show which node host contains the HAProxy gear; other gears could reside on different node hosts.
BIND

If you are running a BIND server on the broker (or supporting) host, the configuration information is contained in the /var/named/dynamic directory. The zone file syntax is domain.com.db.zone; so if the domain of your OpenShift Enterprise installation is example.com, the zone file name would be example.com.db.zone. However, not all changes will be in the zone file. Recent changes can be contained in a binary journal file.

Use the following command to view the entire zone according to the nameserver: 
# dig domain axfr
DHCP

For broker and node hosts, DHCP is currently only supported if the host IPs are pinned, meaning they do not change during lease renewal. This also applies to nameservers, in that they should also not change if pinned.

If DHCP is in use, networking parameters will update at boot time, or at lease renewal. If DNS resolution on an OpenShift Enterprise host stops working after initial installation, look in the /etc/dhcp/dhclient-network-interface.conf file to verify the nameservers provided by the DHCP service are being overwritten when a new lease is obtained.
If your configuration in the /etc/resolv.conf file is overwritten with incorrect values, check your configuration in the dhclient-network-interface.conf file.

2.1.3. SELinux

When error messages indicate access denials but standard Linux file permissions appear to allow access, it could be due to SELinux policy. Use the following method for troubleshooting SELinux issues or verify whether the problem is SELinux related. Red Hat recommends this procedure for finding all SELinux related problems.

Procedure 2.1. To Troubleshoot SELinux Issues:

  1. As root, run the following command to set SELinux to permissive mode: 
    # setenforce 0
  2. Retry the failing action. If the action succeeds then the issue is SELinux related.
  3. Run the following command to set SELinux back to enforcing mode: 
    # setenforce 1
  4. Check the /var/log/audit/audit.log file for any SELinux violations.
This allows the offending action to proceed and log everything that otherwise would have been denied with SELinux enforced. In enforcing mode, not all denials are logged in the audit log, and the first denial generally blocks the action from proceeding to what might be further denials.

2.1.4. Control Groups on Node Hosts

Control groups (cgroups) enable you to allocate resources such as CPU time, system memory, and network bandwidth among user-defined groups of tasks (processes) running on a system. When the cgconfig service is running correctly on a node host, you see the following:
  • The /etc/cgconfig.conf file exists with the SELinux label: system_u:object_r:cgconfig_etc_t:s0.
  • The /etc/cgconfig.conf file joins CPU, cpuacct, memory, freezer, and net_cls in the /croup/all directory.
  • The /cgroup directory exists, with the SELinux label: system_u:object_r:cgroup_t:s0.
  • The cgconfig service is running.
When the cgred service is running correctly, you see the following:
  • The /etc/cgrules.conf file exists with the SELinux label: system_u:object_r:cgrules_etc_t:s0
  • The cgred service is running.
If there are gears running on the node host, you also see the following:
  • A line for each gear in the /etc/cgrules.conf file.
  • A directory for each gear in the /cgroup/all/openshift directory.
  • All processes with the gear UUID are listed in the gear's cgroup.procs file. This file is located in the /cgroup/all/openshift/gear_UUID directory.

Important

If you created the configuration files interactively as a root user, the SELinux user label would be unconfined_u and not system_u. For example, the SELinux label in /etc/cgconfig.conf would be unconfined_u:object_r:cgconfig_etc_t:s0.

2.1.5. Pluggable Authentication Modules

The pam_limits module controls access to system resources. Specifically, OpenShift Enterprise uses the nproc value to control the number of processes a given account can create.
The default value for a new gear is configured in the /etc/openshift/resource_limits.conf file on the node host: 
limits_nproc=2048
When a new gear is created, a 84-gear_UUID.conf file is created on the node host, in the /etc/security/limits.d directory. Replace gear_UUID with the UNIX account name for the gear. This contains a rule set that defines the limits for that UNIX account. The first field of each line in the file is the gear UUID.
The nproc limit for an individual gear is increased by changing the value in the 84-gear_UUID.conf file: 
# PAM process limits for guest 
# see limits.conf(5) for details
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
32ec916eeaa04032b1481af5037a6dfb hard nproc 250
If a gear process is failing, check the nproc limit.

2.1.6. Disk Quotas

Verify that the mount point for the /var/lib/openshift directory has the usrquota option enabled in the /etc/fstab file, and has been mounted. Remount the directory if necessary using the command shown below, and check the output. 
# mount -o remount filesystem
Use the following command to verify that the quotas are configured correctly: 
# repquota -a

2.1.7. iptables

Network firewalls are implemented with iptables. Use the following command to view the current policies: 
# iptables -L
The screen outputs for the iptables -L command for both a broker host and a node host are shown below.
iptables Policy for Broker Host

Chain INPUT (policy ACCEPT)
target     prot opt source               destination          
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     icmp --  anywhere             anywhere             
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:domain 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:domain 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:61613 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

iptables Policy for Node Host

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpts:35531:65535
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.