Chapter 5. Tuning

The Apache HTTP Server is a core component of Satellite. Passenger and Pulp, which are core components of Satellite, depend upon Apache HTTP Server to serve incoming requests. Requests that arrive through the web UI or the Satellite API are received by Apache HTTP Server and then forwarded to the components of Satellite that operate on them.

The version of Apache HTTP Server that ships with Red Hat Satellite 6.3 uses a process-based model of request handling. The Apache HTTP Server spawns a new process each time there is a new incoming request. Each new process takes up space in memory. The maximum number of processes that Apache HTTP Server can spawn is usually governed by its configuration. It is possible to change this number so that Apache HTTP Server can handle a larger number of requests.

When there is a burst of requests at the same time, this can exceed the capacity of Apache HTTP Server to handle requests, which causes new incoming requests to receive HTTP 503 responses.

The two settings that govern how quickly Apache HTTP Server is able to respond to incoming requests during bursts are found in /etc/httpd/conf.modules.d/prefork.conf. These two settings are:

ServerLimit 512
StartServers 10

The ServerLimit parameter defines the maximum number of child processes that Apache HTTP Server is able to spawn to handle new incoming requests. As this setting is increased, the amount of memory used by Apache HTTP Server increases each time a burst of requests arrives at the Satellite and Capsule servers.

The StartServer parameter defines the number of child processes launched by Apache HTTP Server when it starts. Increase this number to improve response times for new incoming requests. Increasing this number creates a situation in which requests do not have to wait for new child processes to be spawned before they are responded to.

Apache HTTP Server uses the KeepAlive directive to manage TCP connections. When KeepAlive is set to On, two variables control how TCP connections are used.

KeepAliveTimeout specifies how long a connection should be kept open. Because establishing and re-establishing connections consumes resources, keeping frequently used connections alive improves performance. Conversely, keeping connections alive for too long ties up resources as well, and can cause performance to drop.

Turn on Apache HTTP Server’s KeepAlive tunable and set values for KeepAliveTimeout and MakeKeepAliveRequests to reduce the number of TCP connections and usage of the CPU. Red Hat recommends that you set KeepAliveTimeout to between 2 and 5 seconds unless there is latency between Red Hat Satellite and the Capsules or hosts. Red Hat recommends that MaxKeepAliveRequests be set to 0 to allow each connection to request all its content over the same TCP connection. The time that it takes to load a page on the web server is lower if KeepAlive is on. The default configuration for KeepAlive for Red Hat Satellite is found in /etc/httpd/conf.d/05-foreman-ssl.conf:

<VirtualHost>
KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 5
</VirtualHost>

Increase the maximum number of files that can be open when doing many registrations or when increasing the scale of Capsules, Content Hosts, and Content Views.

Edit configuration file /etc/systemd/system/httpd.service.d/limits.conf, setting the value for LimitNOFILE.

# cat /etc/systemd/system/httpd.service.d/limits.conf
[Service]
LimitNOFILE=640000
# systemctl daemon-reload
# katello-service restart

The maximum open files limit can be validated with the following command:

# systemctl status httpd | grep 'Main PID:'

Main PID: 13888 (httpd)
# grep -e 'Limit' -e 'Max open files' /proc/13888/limits
Limit             Soft Limit           Hard Limit            Units
Max open files    1000000              1000000               files

Passenger is a web server and a core component of Red Hat Satellite. Satellite uses Passenger to run Ruby applications such as Foreman and Puppet 3. Passenger integrates with Apache HTTP Server to capture incoming requests and redirects them to the respective components that handle them.

Passenger is involved in Satellite when the GUI is accessed, when the APIs are accessed, and when content hosts are registered. Each request that is serviced by Passenger consumes an Apache HTTP Server process. Passenger queues requests into an application-specific wait queue. The maximum number of requests that can be queued by Passenger is defined in the Passenger configuration. When running at scale, it might be desirable to increase the number of requests that Passenger can handle concurrently. It might also be desirable to increase the size of the wait queue to accommodate bursts of requests.

Passenger is configured within the Apache HTTP Server configuration files. It can be used to control the performance, scaling, and behavior of Foreman and Puppet.

This section applies to instances where your system has 12 or more VCPUs.

For deployments that consist of workloads that do not require large scale and do not require much use of Puppet, increase the pool size for Passenger. The pool size defines how many application processes passenger is able to launch to handle incoming requests.

Set the pool size to two times the number of VCPUs in the system. For example, if your system has 24 VCPUs, set the PassengerMaxPoolSize to 48.

Increasing the value of PassengerMaxPoolSize to a value greater than two times the number of VCPUs in the system can cause processes to contend with one another for CPU resources, and this can cause an increase in request timeouts.

This section applies to instances where your system has 12 or more VCPUs.

For deployments that consist of workloads that are heavily dependent on Puppet 3, increase the pool size for Passenger. The pool size defines how many application processes Passenger is able to launch to handle incoming requests.

Set the pool size to the same number as the number of VCPUs in your system. For example, if your system has 24 VCPUs, set PassengerMaxPoolSize to 24.

This section applies to instances where your system has 12 or more VCPUs.

For deployments that consist of workloads that are heavily dependent on Puppet 4, increase the pool size for Passenger. The pool size defines how many application processes Passenger is able to launch to handle incoming requests.

Puppet 4 does not require Passenger to launch its server processes, so the entire Passenger process pool can be dedicated to launching Foreman processes.

Set the pool size to 1.5 multiplied by the number of VCPUs in your system. For example, if your system has 24 VCPUs, set PassengerMaxPoolSize to 36.

In larger, scaled-out setups in which large numbers of requests can arrive in bursts, set the value of PassengerMaxRequestQueueSize to 400.

By using the passenger-status command, the Foreman and Puppet processes spawned by Passenger can be obtained to confirm the PassengerMaxPoolSize.

Mount the Pulp directory onto a large local partition that you can easily scale. Use Logical Volume Manager (LVM) to create this partition.

To adjust the number of tasks that run in parallel, change the value of PULP_CONCURRENCY in the /etc/default/pulp_workers file. As Pulp synchronizes more repositories simultaneously, more workers are able to consume Satellite 6.3 resources. However, such configurations can starve other components of Satellite. It is important to experiment with the concurrency level of Pulp in an environment that has a concurrent workload such as Puppet. By default, on a system with less than 8 CPUs, PULP_CONCURRENCY is set to the number of CPUs. On a system with more than 8 CPUs, it is set to 8.

Transparent Huge Pages is a memory management technique used by the Linux kernel which reduces the overhead of using Translation Lookaside Buffer (TLB) by using larger sized memory pages. Due to databases having Sparse Memory Access patterns instead of Contiguous Memory access patterns, database workloads often perform poorly when Transparent Huge Pages is enabled.

To improve performance of MongoDB, Red Hat recommends Transparent Huge Pages be disabled. For details on disabling Transparent Huge Pages, see Red Hat Solution 1320153.

Red Hat Satellite 6.3 uses /var/lib/pulp to store and manage repository content. Red Hat does not recommend that you run Pulp on NFS. Red Hat recommends instead that you use high-bandwidth, low-latency storage for the /var/lib/pulp filesystem. Red Hat Satellite has many operations that are IO-intensive, and that means that the use of high-latency, low-bandwidth storage could degrade the performance of Satellite 6.3.

Pulp uses MongoDB. Red Hat recommends that you never use NFS with MongoDB.

The value PassengerMinInstances specifies the minimum active instances of Foreman. The remaining slots are used to spawn Puppet.

For a scaled setup involving many operations, set the following executor memory values:

EXECUTOR_MEMORY_LIMIT: 3gb
EXECUTOR_MEMORY_MONITOR_DELAY: 3600
EXECUTOR_MEMORY_MONITOR_INTERVAL: 120

The following tunings in /var/lib/pgsql/data/postgresql.conf provide performance gains, but also increase memory usage:

max_connections = 500
shared_buffers = 512MB
work_mem = 8MB
checkpoint_segments = 32
checkpoint_completion_target = 0.9

A non-deterministic run-interval that does not distribute the load throughout the interval causes scaling problems and errors in a Puppet environment. Evenly distributing the load allows a system to scale reliably and to handle more requests with fewer spikes. Run-interval can be distributed in the following ways:

Measure other Puppet interactions that are performed in your environment. These interactions place a load on Red Hat Satellite Server and its Capsules, and these interactions include submitting facts, serving files, and submitting reports. Each of these interactions imposes a cost on the system.

Determining when to use an external Capsule versus an integrated Capsule depends on hardware, configuration, and workload. Plan against the Puppet requirements, because a number of variables in the Puppet workload will directly affect the scalability of Red Hat Satellite. Raising the runinterval directly increases the capacity at the cost of increasing the interval between which Puppet applies the configuration. Reducing the runinterval reduces the capacity. If the clients are not spread evenly, a large group of clients can fill the Passenger queue and block other requests while leaving Red Hat Satellite Server underutilized at other times. The amount of work that each Puppet client has to perform in order to complete a Puppet run changes scalability. Raising the configured number of Puppet processes improves scalability if physical hardware resources are available. The nature of these variables means that it is not constructive to provide a universally-applicable recommendation regarding when it is wise to move to an external Capsule. Benchmark a specific Puppet workload to determine its scalability.

Apply the hardware considerations that are listed in this document to Capsules. Virtualized Capsules make it possible to tune the number of vCPUs and available memory as long as the Capsule is not colocated on a machine that hosts virtual machines that overcommit the host’s resources. Apache HTTP Server and Passenger configuration considerations also apply to the Capsule in the context of Puppet.

KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 5

/etc/httpd/conf.d/passenger.conf:

LoadModule passenger_module modules/mod_passenger.so
<IfModule mod_passenger.c>
    PassengerRoot
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini
    PassengerRuby /usr/bin/ruby
    PassengerMaxPoolSize 6
    PassengerStatThrottleRate 120
</IfModule>

/etc/httpd/conf.d/25-puppet.conf:

PassengerMinInstances 2
PassengerStartTimeout 90
PassengerMaxPreloaderIdleTime 0
PassengerMaxRequests 10000
PassengerPreStart https://example-capsule.com:8140

To scale up Satellite and Capsules so that they support larger numbers of hosts, it is essential that processes should be able to open the required number of descriptors.

The following tunings are applicable both to Apache HTTP Server and qrouterd:

[Service]
LimitNOFILE=640000

If your deployment uses katello-agent, you must tune the file limits for qpidd and qrouterd.

[Service]
LimitNOFILE=301000

[Service]
LimitNOFILE=40500

Increase the maximum number of allowable concurrent AIO requests by increasing the kernel parameter fs.aio-max-nr.

This number should be bigger than 33 multiplied by the maximum number of the content hosts planned to be registered to Satellite.

See also Red Hat Solution 1425893 for more detail.

Plan to have enough storage capacity for directory /var/lib/qpidd in advance when you are planning an installation that will use katello-agent extensively. In Red Hat Satellite 6.3, /var/lib/qpidd requires 2MB disk space per content host (see Bug 1366323).

The following line is sufficient in new installations:

cat /usr/share/katello-installer-base/modules/qpid/templates/qpidc.conf.erb efp-file-size=256

You might see the following error in /var/log/journal in Red Hat Enterprise Linux 7:

satellite.example.com qpidd[92464]: [Broker] error Channel exception: not-attached: Channel 2 is not attached
(/builddir/build/BUILD/qpid-cpp-0.30/src/qpid/amqp_0_10/SessionHandler.cpp: 39)

satellite.example.com qpidd[92464]: [Protocol] error Connection qpid.10.1.10.1:5671-10.1.10.1:53790 timed out: closing

This error message appears because qpid maintains management objects for queues, sessions, and connections and recycles them every ten seconds by default. The same object with the same ID is created, deleted, and created again. The old management object is not yet purged, which is why qpid throws this error. Here’s a workaround: lower the mgmt-pub-interval parameter from the default 10 seconds to something lower. Add it to /etc/qpid/qpidd.conf and restart the qpidd service. See also Bug 1335694 comment 7.