Chapter 5. Managing Load-balancing service instance logs
You can enable tenant flow logging or suppress logging to the amphora local file system. You can also forward administrative or tenant flow logs to syslog receivers at a set of containers or to other syslog receivers at endpoints of your choosing.
In addition, you can control a range of other logging features such as, setting the syslog facility value, changing the tenant flow log format, and widening the scope of administrative logging to include logs from sources like the kernel and from cron.
- Section 5.1, “Basics of offloading Load-balancing service instance (amphora) logs”
- Section 5.2, “Enabling Load-balancing service instance administrative log offloading”
- Section 5.3, “Enabling tenant flow log offloading for Load-balancing service instances”
- Section 5.4, “Disabling Load-balancing service instance tenant flow logging”
- Section 5.5, “Disabling Load-balancing service instance local log storage”
- Section 5.6, “Heat parameters for Load-balancing service instance logging”
- Section 5.7, “Load-balancing service instance tenant flow log format”
5.1. Basics of offloading Load-balancing service instance (amphora) logs
By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. However, you can specify that amphorae offload logs to syslog receivers to aggregate both administrative and tenant traffic flow logs. Log offloading enables administrators to go to one location for logs, and retain logs when amphorae are rotated.
5.2. Enabling Load-balancing service instance administrative log offloading
By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. However, you can specify that the amphorae offload logs to syslog receivers to aggregate administrative logs. Log offloading enables administrators to go to one location for logs, and retain logs when the amphorae are rotated.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the undercloud credentials file:
$ source ~/stackrc
Create a custom YAML environment file.
Example
$ vi /home/stack/templates/my-octavia-environment.yaml
In the YAML environment file under
parameter_defaults
, setOctaviaLogOffload
totrue
.parameter_defaults: OctaviaLogOffload: true ...
NoteThe amphorae offload administrative logs use the syslog facility value of
local1
, by default, unless you specify another value with theOctaviaAdminLogFacility
parameter.Example
parameter_defaults: OctaviaLogOffload: true OctaviaAdminLogFacility: 2 ...
The amphorae forward only load balancer-related administrative logs, such as the haproxy admin logs, keepalived, and amphora agent logs. If you want to configure the amphorae to send all of the administrative logs from the amphorae, such as the kernel, system, and security logs, set
OctaviaForwardAllLogs
totrue
.Example
parameter_defaults: OctaviaLogOffload: true OctaviaForwardAllLogs: true ...
The amphorae use a set of default containers defined by the Orchestration service (heat) that contain syslog receivers listening for log messages. If you want to use a different set of endpoints, you can specify those with the
OctaviaAdminLogTargets
parameter:OctaviaAdminLogTargets: <ip_address>:<port>[, <ip_address>:<port>]
Example
parameter_defaults: OctaviaLogOffload: true OctaviaAdminLogTargets: 192.0.2.1:10514, 2001:db8:1::10:10514 ...
By default, when you enable log offloading, tenant flow logs are also offloaded.
If you want to disable tenant flow log offloading, set the
OctaviaConnectionLogging
tofalse
.Example
parameter_defaults: OctaviaLogOffload: true OctaviaConnectionLogging: false ...
Run the deployment command and include the core heat templates, environment files, and this new custom environment file.
ImportantThe order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence.
Example
$ openstack overcloud deploy --templates \ -e [your-environment-files] \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \ -e /home/stack/templates/my-octavia-environment.yaml
Verification
-
Unless you specified specific endpoints with the
OctaviaAdminLogTargets
orOctaviaTenantLogTargets
, the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs (/var/log/containers/octavia/
). Check the appropriate location for the presence of the following log files:
-
octavia-amphora.log
-- Log file for the administrative log. -
(if enabled)
octavia-tenant-traffic.log
-- Log file for the tenant traffic flow log.
-
Additional resources
- Section 5.6, “Heat parameters for Load-balancing service instance logging”
- Environment files in the Advanced Overcloud Customization guide
- Including environment files in overcloud creation in the Advanced Overcloud Customization guide
5.3. Enabling tenant flow log offloading for Load-balancing service instances
By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. You can specify that the amphorae offload logs to syslog receivers on endpoints that contain disk space sufficient for tenant flow logs that can grow in size depending on the number of tenant connections.
Tenant flow log offloading for Load-balancing service instances is automatically enabled when administrative log offloading is enabled. The only case when administrative log offloading is on and tenant flow log offloading is off, is when the OctaviaConnectionLogging
parameter is set to false
.
Tenant flow logging can produce a large number of syslog messages depending on how many connections the load balancers are receiving. Tenant flow logging produces one log entry for each connection to the load balancer. Monitor log volume and configure your syslog receivers appropriately based on the expected number of connections that your load balancers manage.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the undercloud credentials file:
$ source ~/stackrc
Locate the environment file in which the
OctaviaConnectionLogging
parameter is set:$ grep -rl OctaviaConnectionLogging /home/stack/templates/
If you do not find the file, create an environment file:
$ vi /home/stack/templates/my-octavia-environment.yaml
Add the
OctaviaLogOffload
andOctaviaConnectionLogging
parameters to theparameter_defaults
section of the environment file and set the values totrue
:parameter_defaults: OctaviaLogOffload: true OctaviaConnectionLogging: true ...
NoteThe amphorae use the syslog facility default value of
local0
to offload tenant flow logs unless you use theOctaviaTenantLogFacility
parameter to specify another value.Optional: The amphorae use a set of default containers that contain syslog receivers that listen for log messages. You can change the admin and tenant log endpoints using the parameters
OctaviaAdminLogTargets
andOctaviaTenantLogTargets
.OctaviaAdminLogTargets: <ip-address>:<port>[, <ip-address>:<port>] OctaviaTenantLogTargets: <ip-address>:<port>[, <ip-address>:<port>]
Run the deployment command and include the core heat templates, environment files, and the custom environment file you modified.
ImportantThe order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence.
$ openstack overcloud deploy --templates \ -e <your_environment_files> \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \ -e /home/stack/templates/my-octavia-environment.yaml
Verification
-
Unless you specified specific endpoints with the
OctaviaAdminLogTargets
orOctaviaTenantLogTargets
, the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs (/var/log/containers/octavia/
). Check the appropriate location for the presence of the following log files:
-
octavia-amphora.log
-- Log file for the administrative log. -
octavia-tenant-traffic.log
-- Log file for the tenant traffic flow log.
-
Additional resources
- Section 5.6, “Heat parameters for Load-balancing service instance logging”
- Environment files in the Advanced Overcloud Customization guide
- Including environment files in overcloud creation in the Advanced Overcloud Customization guide
- Section 5.7, “Load-balancing service instance tenant flow log format”
5.4. Disabling Load-balancing service instance tenant flow logging
Tenant flow log offloading for Load-balancing service instances (amphorae) is automatically enabled when you enable administrative log offloading.
To keep administrative log offloading enabled and to disable tenant flow logging, you must set the OctaviaConnectionLogging
parameter to false
.
When the OctaviaConnectionLogging
parameter is false
, the amphorae do not write tenant flow logs to the disk inside the amphorae, nor offload any logs to syslog receivers listening elsewhere.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the undercloud credentials file:
$ source ~/stackrc
Locate the YAML custom environment file in which amphora logging is configured.
Example
$ grep -rl OctaviaLogOffload /home/stack/templates/
In the custom environment file, under
parameter_defaults
, setOctaviaConnectionLogging
tofalse
.Example
parameter_defaults: OctaviaLogOffload: true OctaviaConnectionLogging: false ...
Run the deployment command and include the core heat templates, environment files, and the custom environment file in which you set
OctaviaConnectionLogging
totrue
.ImportantThe order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence.
Example
$ openstack overcloud deploy --templates \ -e [your-environment-files] \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \ -e /home/stack/templates/my-octavia-environment.yaml
Verification
-
Unless you specified specific endpoints with the
OctaviaAdminLogTargets
orOctaviaTenantLogTargets
, the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs (/var/log/containers/octavia/
). -
Check the appropriate location for the absence of
octavia-tenant-traffic.log
.
Additional resources
- Environment files in the Advanced Overcloud Customization guide
- Including environment files in overcloud creation in the Advanced Overcloud Customization guide
5.5. Disabling Load-balancing service instance local log storage
Even when you configure Load-balancing service instances (amphorae) to offload administrative and tenant flow logs, the amphorae continue to write these logs to the disk inside the amphorae. To improve the performance of the load balancer, you can stop logging locally.
If you disable logging locally, you also disable all log storage in the amphora, including kernel, system, and security logging.
If you disable local log storage and the OctaviaLogOffload
parameter is set to false, ensure that you set OctaviaConnectionLogging
to false for improved load balancing performance.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the undercloud credentials file:
$ source ~/stackrc
Create a custom YAML environment file.
Example
$ vi /home/stack/templates/my-octavia-environment.yaml
In the environment file under
parameter_defaults
, setOctaviaDisableLocalLogStorage
totrue
.parameter_defaults: OctaviaDisableLocalLogStorage: true ...
Run the deployment command and include the core heat templates, environment files, and this new custom environment file.
ImportantThe order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence.
Example
$ openstack overcloud deploy --templates \ -e <your_environment_files> \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \ -e /home/stack/templates/my-octavia-environment.yaml
Verification
- On the amphora instance, check the location where log files are written, and verify that no new log files are being written.
Additional resources
- Environment files in the Advanced Overcloud Customization guide
- Including environment files in overcloud creation in the Advanced Overcloud Customization guide
5.6. Heat parameters for Load-balancing service instance logging
When you want to configure Load-balancing service instance (amphora) logging, you set values for one or more Orchestration service (heat) parameters that control logging and run the openstack overcloud deploy
command.
These heat parameters for amphora logging enable you to control features such as turning on log offloading, defining custom endpoints to offload logs to, setting the syslog facility value for logs, and so on.
Parameter | Default | Description |
---|---|---|
|
|
When |
|
|
When |
|
|
When
For instances to recognize |
Parameter | Default | Description |
---|---|---|
| No value. | A comma-delimited list of syslog endpoints (<host>:<port>) to receive administrative log messages. These endpoints can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port.
When |
|
| A number between 0 and 7 that is the syslog "LOG_LOCAL" facility to use for the administrative log messages. |
Parameter | Default | Description |
---|---|---|
|
|
When
When |
| No value. | A comma-delimited list of syslog endpoints (<host>:<port>) to receive tenant traffic flow log messages. These endpoints can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port.
When |
|
| A number between 0 and 7 that is the syslog "LOG_LOCAL" facility to use for the tenant traffic flow log messages. |
|
| The format for the tenant traffic flow log. The alphanumerics represent specific octavia fields, and the curly braces ({}) are substitution variables. |
Additional resources
- Including environment files in overcloud creation in the Advanced Overcloud Customization guide
5.7. Load-balancing service instance tenant flow log format
The log format that the tenant flow logs for Load-balancing service instances (amphorae) follows is the HAProxy log format. The two exceptions are the project_id
and lb_id
variables whose values are provided by the amphora provider driver.
Sample
Here is a sample log entry with rsyslog as the syslog receiver:
Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]: 5408b89aa45b48c69a53dca1aaec58db fd8f23df-960b-4b12-ba62-2b1dff661ee7 261ecfc2-9e8e-4bba-9ec2-3c903459a895 172.24.4.1 41152 12/Jun/2019:00:44:13.030 "GET / HTTP/1.1" 200 76 73 - "" e37e0e04-68a3-435b-876c-cffe4f2138a4 6f2720b3-27dc-4496-9039-1aafe2fee105 4 --
Notes
- A hyphen (-) indicates any field that is unknown or not applicable to the connection.
The prefix in the earlier sample log entry originates from the rsyslog receiver, and is not part of the syslog message from the amphora:
Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]:”
Default
The default amphora tenant flow log format is:
`"{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc"`
Refer to the table that follows for a description of the format.
Variable | Type | Field name |
---|---|---|
| UUID | Project ID (substitution variable from the amphora provider driver) |
| UUID | Load balancer ID (substitution variable from the amphora provider driver) |
| string |
|
| IP address |
|
| numeric |
|
| date |
|
| numeric |
|
| numeric |
|
| numeric |
|
| Boolean |
|
| string |
|
| string |
|
| string |
|
| numeric |
|
| string |
|
Additional resources
- Custom log format in HAProxy Documentation