Chapter 10. Configuring logging for Kafka components

Procedure

1. Switch to the kafka user:

   su - kafka

2. List all the broker loggers for a broker by using the kafka-configs.sh tool:

   /opt/kafka/bin/kafka-configs.sh --bootstrap-server <broker_address> --describe --entity-type broker-loggers --entity-name BROKER-ID

   For example, for broker 0:

   /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type broker-loggers --entity-name 0

   This returns the logging level for each logger: TRACE, DEBUG, INFO, WARN, ERROR, or FATAL.

   For example:

   #...
   kafka.controller.ControllerChannelManager=INFO sensitive=false synonyms={}
   kafka.log.TimeIndex=INFO sensitive=false synonyms={}

3. Change the logging level for one or more broker loggers. Use the --alter and --add-config options and specify each logger and its level as a comma-separated list in double quotes.

   /opt/kafka/bin/kafka-configs.sh --bootstrap-server <broker_address> --alter --add-config "LOGGER-ONE=NEW-LEVEL,LOGGER-TWO=NEW-LEVEL" --entity-type broker-loggers --entity-name BROKER-ID

   For example, for broker 0:

   /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "kafka.controller.ControllerChannelManager=WARN,kafka.log.TimeIndex=WARN" --entity-type broker-loggers --entity-name 0

   If successful this returns:

   Completed updating config for broker: 0.

Procedure

1. Switch to the kafka user:

   su - kafka

2. Check the current logging level for the loggers configured in the connect-log4j.properties file:

   $ cat /opt/kafka/config/connect-log4j.properties
   
   # ...
   log4j.rootLogger=INFO, stdout, connectAppender
   # ...
   log4j.logger.org.apache.zookeeper=ERROR
   log4j.logger.org.reflections=ERROR

   Use a curl command to check the logging levels from the admin/loggers endpoint of the Kafka Connect API:

   curl -s http://localhost:8083/admin/loggers/ | jq
   
   {
     "org.apache.zookeeper": {
       "level": "ERROR"
     },
     "org.reflections": {
       "level": "ERROR"
     },
     "root": {
       "level": "INFO"
     }
   }

   jq prints the output in JSON format. The list shows standard org and root level loggers, plus any specific loggers with modified logging levels.

   If you configure TLS (Transport Layer Security) authentication for the admin.listeners configuration in Kafka Connect, then the address of the loggers endpoint is the value specified for admin.listeners with the protocol as https, such as https://localhost:8083.

   You can also get the log level of a specific logger:

   curl -s http://localhost:8083/admin/loggers/org.apache.kafka.connect.mirror.MirrorCheckpointConnector | jq
   
   {
     "level": "INFO"
   }

3. Use a PUT method to change the log level for a logger:

   curl -Ss -X PUT -H 'Content-Type: application/json' -d '{"level": "TRACE"}' http://localhost:8083/admin/loggers/root
   
   {
     # ...
   
     "org.reflections": {
       "level": "TRACE"
     },
     "org.reflections.Reflections": {
       "level": "TRACE"
     },
     "root": {
       "level": "TRACE"
     }
   }

   If you change the root logger, the logging level for loggers that used the root logging level by default are also changed.