25.5.3. Managing Queues
All types of queues can be further configured to match your requirements. You can use several directives to modify both action queues and the main message queue. Currently, there are more than 20 queue parameters available, see the section called “Online Documentation”. Some of these settings are used commonly, others, such as worker thread management, provide closer control over the queue behavior and are reserved for advanced users. With advanced settings, you can optimize rsyslog's performance, schedule queuing, or modify the behavior of a queue on system shutdown.
Limiting Queue Size
You can limit the number of messages that queue can contain with the following setting:
$objectQueueHighWatermark number
Replace object with
MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Replace number with a number of enqueued messages. You can set the queue size only as the number of messages, not as their actual memory size. The default queue size is 10,000 messages for the main message queue and ruleset queues, and 1000 for action queues.
Disk assisted queues are unlimited by default and can not be restricted with this directive, but you can reserve them physical disk space in bytes with the following settings:
$objectQueueMaxDiscSpace number
Replace object with
MainMsg or with Action. When the size limit specified by number is hit, messages are discarded until sufficient amount of space is freed by dequeued messages.
Discarding Messages
When a queue reaches a certain number of messages, you can discard less important messages in order to save space in the queue for entries of higher priority. The threshold that launches the discarding process can be set with the so-called discard mark:
$objectQueueDiscardMark number
Replace object with
MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Here, number stands for a number of messages that have to be in the queue to start the discarding process. To define which messages to discard, use:
$objectQueueDiscardSeverity priority
Replace priority with one of the following keywords (or with a number):
debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0). With this setting, both newly incoming and already queued messages with lower than defined priority are erased from the queue immediately after the discard mark is reached.
Using Timeframes
You can configure rsyslog to process queues during a specific time period. With this option you can, for example, transfer some processing into off-peak hours. To define a time frame, use the following syntax:
$objectQueueDequeueTimeBegin hour
$objectQueueDequeueTimeEnd hour
With hour you can specify hours that bound your time frame. Use the 24-hour format without minutes.
Configuring Worker Threads
A worker thread performs a specified action on the enqueued message. For example, in the main message queue, a worker task is to apply filter logic to each incoming message and enqueue them to the relevant action queues. When a message arrives, a worker thread is started automatically. When the number of messages reaches a certain number, another worker thread is turned on. To specify this number, use:
$objectQueueWorkerThreadMinimumMessages number
Replace number with a number of messages that will trigger a supplemental worker thread. For example, with number set to 100, a new worker thread is started when more than 100 messages arrive. When more than 200 messages arrive, the third worker thread starts and so on. However, too many working threads running in parallel becomes ineffective, so you can limit the maximum number of them by using:
$objectQueueWorkerThreads number
where number stands for a maximum number of working threads that can run in parallel. For the main message queue, the default limit is 1 thread. Once a working thread has been started, it keeps running until an inactivity timeout appears. To set the length of timeout, type:
$objectQueueWorkerTimeoutThreadShutdown time
Replace time with the duration set in milliseconds. Without this setting, a zero timeout is applied and a worker thread is terminated immediately when it runs out of messages. If you specify time as
-1, no thread will be closed.
Batch Dequeuing
To increase performance, you can configure rsyslog to dequeue multiple messages at once. To set the upper limit for such dequeueing, use:
$objectQueueDequeueBatchSize number
Replace number with the maximum number of messages that can be dequeued at once. Note that a higher setting combined with a higher number of permitted working threads results in greater memory consumption.
Terminating Queues
When terminating a queue that still contains messages, you can try to minimize the data loss by specifying a time interval for worker threads to finish the queue processing:
$objectQueueTimeoutShutdown time
Specify time in milliseconds. If after that period there are still some enqueued messages, workers finish the current data element and then terminate. Unprocessed messages are therefore lost. Another time interval can be set for workers to finish the final element:
$objectQueueTimeoutActionCompletion time
In case this timeout expires, any remaining workers are shut down. To save data at shutdown, use:
$objectQueueTimeoutSaveOnShutdown time
If set, all queue elements are saved to disk before rsyslog terminates.
