第 6 章 Forwarding logs to third party systems
By default, OpenShift Container Platform cluster logging sends logs to the default internal Elasticsearch log store, defined in the ClusterLogging custom resource (CR).
You can configure cluster logging to send logs to destinations outside of your OpenShift Container Platform cluster instead of the default Elasticsearch log store using the following methods:
- Sending logs using the Fluentd forward protocol. You can create a config map to use the Fluentd forward protocol to securely send logs to an external logging aggregator that accepts the Fluent forward protocol.
- Sending logs using syslog. You can create a config map to use the syslog protocol to send logs to an external syslog (RFC 3164) server.
Alternatively, you can use the Log Forwarding API, currently in Technology Preview. The Log Forwarding API, which is easier to configure than the Fluentd protocol and syslog, exposes configuration for sending logs to the internal Elasticsearch log store and to external Fluentd log aggregation solutions.
You cannot use the config map methods and the Log Forwarding API in the same cluster.
The Log Forwarding API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
The methods for forwarding logs using a config map are deprecated and will be replaced by the Log Forwarding API in a future release.
6.1. Forwarding logs using the Fluentd forward protocol
You can use the Fluentd forward protocol to send a copy of your logs to an external log aggregator, instead of the default Elasticsearch log store. On the OpenShift Container Platform cluster, you use the Fluentd forward protocol to send logs to a server configured to accept the protocol. You are responsible to configure the external log aggregator to receive the logs from OpenShift Container Platform.
This method for forwarding logs is deprecated in OpenShift Container Platform and will be replaced by the Log Forwarding API in a future release.
To configure OpenShift Container Platform to send logs using the Fluentd forward protocol, create a ConfigMap called secure-forward in the openshift-logging namespace that points to an external log aggregator.
Starting with the OpenShift Container Platform 4.3, the process for using the Fluentd forward protocol has changed. You now need to create a ConfigMap, as described below.
Additionally, you can add any certificates required by your configuration to a secret named secure-forward that will be mounted to the Fluentd Pods.
Sample secure-forward.conf
<store>
@type forward
<security>
self_hostname ${hostname} # ${hostname} is a placeholder.
shared_key "fluent-receiver"
</security>
transport tls
tls_verify_hostname false # Set false to ignore server cert hostname.
tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
<buffer>
@type file
path '/var/lib/fluentd/secureforwardlegacy'
queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }"
chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }"
flush_interval "#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}"
flush_at_shutdown "#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}"
flush_thread_count "#{ENV['FLUSH_THREAD_COUNT'] || 2}"
retry_max_interval "#{ENV['FORWARD_RETRY_WAIT'] || '300'}"
retry_forever true
# the systemd journald 0.0.8 input plugin will just throw away records if the buffer
# queue limit is hit - 'block' will halt further reads and keep retrying to flush the
# buffer to the remote - default is 'exception' because in_tail handles that case
overflow_action "#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}"
</buffer>
<server>
host fluent-receiver.openshift-logging.svc # or IP
port 24224
</server>
</store>
Sample secure-forward ConfigMap based on the configuration
apiVersion: v1
data:
secure-forward.conf: "<store>
\ @type forward
\ <security>
\ self_hostname ${hostname} # ${hostname} is a placeholder.
\ shared_key \"fluent-receiver\"
\ </security>
\ transport tls
\ tls_verify_hostname false # Set false to ignore server cert hostname.
\ tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
\ <buffer>
\ @type file
\ path '/var/lib/fluentd/secureforwardlegacy'
\ queued_chunks_limit_size \"#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }\"
\ chunk_limit_size \"#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }\"
\ flush_interval \"#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}\"
\ flush_at_shutdown \"#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}\"
\ flush_thread_count \"#{ENV['FLUSH_THREAD_COUNT'] || 2}\"
\ retry_max_interval \"#{ENV['FORWARD_RETRY_WAIT'] || '300'}\"
\ retry_forever true
\ # the systemd journald 0.0.8 input plugin will just throw away records if the buffer
\ # queue limit is hit - 'block' will halt further reads and keep retrying to flush the
\ # buffer to the remote - default is 'exception' because in_tail handles that case
\ overflow_action \"#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}\"
\ </buffer>
\ <server>
\ host fluent-receiver.openshift-logging.svc # or IP
\ port 24224
\ </server>
</store>"
kind: ConfigMap
metadata:
creationTimestamp: "2020-01-15T18:56:04Z"
name: secure-forward
namespace: openshift-logging
resourceVersion: "19148"
selfLink: /api/v1/namespaces/openshift-logging/configmaps/secure-forward
uid: 6fd83202-93ab-d851b1d0f3e8
Procedure
To configure OpenShift Container Platform to forward logs using the Fluentd forward protocol:
Create a configuration file named
secure-forward.conffor the forward parameters:Configure the secrets and TLS information:
<store> @type forward self_hostname ${hostname} 1 shared_key <SECRET_STRING> 2 transport tls 3 tls_verify_hostname true 4 tls_cert_path <path_to_file> 5- 1
- Specify the default value of the auto-generated certificate common name (CN).
- 2
- Enter the Shared key between nodes
- 3
- Specify
tlsto enable TLS validation. - 4
- Set to
trueto verify the server cert hostname. Set tofalseto ignore server cert hostname. - 5
- Specify the path to private CA certificate file as
/etc/ocp-forward/ca_cert.pem.
To use mTLS, see the Fluentd documentation for information about client certificate, key parameters, and other settings.
Configure the name, host, and port for your external Fluentd server:
<server> name 1 host 2 hostlabel 3 port 4 </server> <server> 5 name host </server>For example:
<server> name externalserver1 host 192.168.1.1 hostlabel externalserver1.example.com port 24224 </server> <server> name externalserver2 host externalserver2.example.com port 24224 </server> </store>
Create a ConfigMap named
secure-forwardin theopenshift-loggingnamespace from the configuration file:$ oc create configmap secure-forward --from-file=secure-forward.conf -n openshift-logging
Optional: Import any secrets required for the receiver:
$ oc create secret generic secure-forward --from-file=<arbitrary-name-of-key1>=cert_file_from_fluentd_receiver --from-literal=shared_key=value_from_fluentd_receiver
For example:
$ oc create secret generic secure-forward --from-file=ca-bundle.crt=ca-for-fluentd-receiver/ca.crt --from-literal=shared_key=fluentd-receiver
Refresh the
fluentdPods to apply thesecure-forwardsecret andsecure-forwardConfigMap:$ oc delete pod --selector logging-infra=fluentd
- Configure the external log aggregator to accept messages securely from OpenShift Container Platform.
