Red Hat Summit Red Hat SummitApoyoConsolaDesarrolladoresIniciar una pruebaContacto

Este contenido no está disponible en el idioma seleccionado.

Chapter 8. Enabling JSON logging

You can configure the Log Forwarding API to parse JSON strings into a structured object.

8.1. Parsing JSON logs
Copiar enlace

Logs including JSON logs are usually represented as a string inside the 

message
field. That makes it hard for users to query specific fields inside a JSON document. OpenShift Logging’s Log Forwarding API enables you to parse JSON logs into a structured object and forward them to either OpenShift Logging-managed Elasticsearch or any other third-party system supported by the Log Forwarding API.

To illustrate how this works, suppose that you have the following structured JSON log entry.

Example structured JSON log entry

{"level":"info","name":"fred","home":"bedrock"}

Normally, the 

ClusterLogForwarder
custom resource (CR) forwards that log entry in the 
message
field. The 
message
field contains the JSON-quoted string equivalent of the JSON log entry, as shown in the following example.

Example message field

{"message":"{\"level\":\"info\",\"name\":\"fred\",\"home\":\"bedrock\"",
 "more fields..."}

To enable parsing JSON log, you add 

parse: json
to a pipeline in the 
ClusterLogForwarder
CR, as shown in the following example.

Example snippet showing parse: json

pipelines:
- inputRefs: [ application ]
  outputRefs: myFluentd
  parse: json

When you enable parsing JSON logs by using 

parse: json
, the CR copies the JSON-structured log entry in a 
structured
field, as shown in the following example. This does not modify the original 
message
field.

Example structured output containing the structured JSON log entry

{"structured": { "level": "info", "name": "fred", "home": "bedrock" },
 "more fields..."}

Important

If the log entry does not contain valid structured JSON, the 

structured
field will be absent.

To enable parsing JSON logs for specific logging platforms, see Forwarding logs to third-party systems.

8.2. Configuring JSON log data for Elasticsearch
Copiar enlace

If your JSON logs follow more than one schema, storing them in a single index might cause type conflicts and cardinality problems. To avoid that, you must configure the 

ClusterLogForwarder
custom resource (CR) to group each schema into a single output definition. This way, each schema is forwarded to a separate index.

Important

If you forward JSON logs to the default Elasticsearch instance managed by OpenShift Logging, it generates new indices based on your configuration. To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.

Structure types

You can use the following structure types in the 

ClusterLogForwarder
CR to construct index names for the Elasticsearch log store: 

  • structuredTypeKey
    (string, optional) is the name of a message field. The value of that field, if present, is used to construct the index name. 

    • kubernetes.labels.<key>
      is the Kubernetes pod label whose value is used to construct the index name. 
    • openshift.labels.<key>
      is the 
      pipeline.label.<key>
      element in the 
      ClusterLogForwarder
      CR whose value is used to construct the index name. 
    • kubernetes.container_name
      uses the container name to construct the index name. 
  • structuredTypeName
    : (string, optional) If 
    structuredTypeKey
    is not set or its key is not present, OpenShift Logging uses the value of 
    structuredTypeName
    as the structured type. When you use both 
    structuredTypeKey
    and 
    structuredTypeName
    together, 
    structuredTypeName
    provides a fallback index name if the key in 
    structuredTypeKey
    is missing from the JSON log data.
Note

Although you can set the value of 

structuredTypeKey
to any field shown in the "Log Record Fields" topic, the most useful fields are shown in the preceding list of structure types.

A structuredTypeKey: kubernetes.labels.<key> example

Suppose the following:

  • Your cluster is running application pods that produce JSON logs in two different formats, "apache" and "google".
  • The user labels these application pods with 
    logFormat=apache
    and 
    logFormat=google
    .
  • You use the following snippet in your 
    ClusterLogForwarder
    CR YAML file. 
outputDefaults:
 elasticsearch:
    structuredTypeKey: kubernetes.labels.logFormat
1 

    structuredTypeName: nologformat
pipelines:
- inputRefs: <application>
  outputRefs: default
  parse: json
2
1
Uses the value of the key-value pair that is formed by the Kubernetes logFormat label.
2
Enables parsing JSON logs.

In that case, the following structured log record goes to the 

app-apache-write
index: 

{
  "structured":{"name":"fred","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "apache", ...}}
}

And the following structured log record goes to the 

app-google-write
index: 

{
  "structured":{"name":"wilma","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "google", ...}}
}

A structuredTypeKey: openshift.labels.<key> example

Suppose that you use the following snippet in your 

ClusterLogForwarder
CR YAML file. 

outputDefaults:
 elasticsearch:
    structuredTypeKey: openshift.labels.myLabel
1 

    structuredTypeName: nologformat
pipelines:
 - name: application-logs
   inputRefs:
   - application
   - audit
   outputRefs:
   - elasticsearch-secure
   - default
   parse: json
   labels:
     myLabel: myValue
2
1
Uses the value of the key-value pair that is formed by the OpenShift myLabel label.
2
The myLabel element gives its string value, myValue, to the structured log record.

In that case, the following structured log record goes to the 

app-myValue-write
index: 

{
  "structured":{"name":"fred","home":"bedrock"},
  "openshift":{"labels":{"myLabel": "myValue", ...}}
}

Additional considerations

  • The Elasticsearch index for structured records is formed by prepending "app-" to the structured type and appending "-write".
  • Unstructured records are not sent to the structured index. They are indexed as usual in the application, infrastructure, or audit indices.
  • If there is no non-empty structured type, forward an unstructured record with no 
    structured
    field.

It is important not to overload Elasticsearch with too many indices. Only use distinct structured types for distinct log formats, not for each application or namespace. For example, most Apache applications use the same JSON log format and structured type, such as 

LogApache
.

8.3. Forwarding JSON logs to the Elasticsearch log store
Copiar enlace

For an Elasticsearch log store, if your JSON log entries follow different schemas, configure the 

ClusterLogForwarder
custom resource (CR) to group each JSON schema into a single output definition. This way, Elasticsearch uses a separate index for each schema.

Important

Because forwarding different schemas to the same index can cause type conflicts and cardinality problems, you must perform this configuration before you forward data to the Elasticsearch store.

To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.

Procedure

  1. Add the following snippet to your 

    ClusterLogForwarder
    CR YAML file. 

    outputDefaults:
 elasticsearch:
    structuredTypeKey: <log record field>
    structuredTypeName: <name>
pipelines:
- inputRefs:
  - application
  outputRefs: default
  parse: json
  2. Optional: Use 
    structuredTypeKey
    to specify one of the log record fields, as described in the preceding topic, Configuring JSON log data for Elasticsearch. Otherwise, remove this line.

  3. Optional: Use 

    structuredTypeName
    to specify a 
    <name>
    , as described in the preceding topic, Configuring JSON log data for Elasticsearch. Otherwise, remove this line.

    Important

    To parse JSON logs, you must set either 

    structuredTypeKey
    or 
    structuredTypeName
    , or both 
    structuredTypeKey
    and 
    structuredTypeName
    .

  4. For 
    inputRefs
    , specify which log types to forward by using that pipeline, such as 
    application,
     
    infrastructure
    , or 
    audit
    .
  5. Add the 
    parse: json
    element to pipelines.

  6. Create the CR object: 

    $ oc create -f <file-name>.yaml

    The Red Hat OpenShift Logging Operator redeploys the Fluentd pods. However, if they do not redeploy, delete the Fluentd pods to force them to redeploy. 

    $ oc delete pod --selector logging-infra=collector
Red Hat logoGithubredditYoutubeTwitter

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Ayudamos a los usuarios de Red Hat a innovar y alcanzar sus objetivos con nuestros productos y servicios con contenido en el que pueden confiar. Explore nuestras recientes actualizaciones.

Hacer que el código abierto sea más inclusivo

Red Hat se compromete a reemplazar el lenguaje problemático en nuestro código, documentación y propiedades web. Para más detalles, consulte el Blog de Red Hat.

Acerca de Red Hat

Ofrecemos soluciones reforzadas que facilitan a las empresas trabajar en plataformas y entornos, desde el centro de datos central hasta el perímetro de la red.

Theme

© 2026 Red HatVolver arriba