Search

Chapter 3. Editing kubelet log level verbosity and gathering logs

download PDF

To troubleshoot some issues with nodes, establish the kubelet’s log level verbosity depending on the issue to be tracked.

3.1. Modifying the kubelet as a one-time scenario

To modify the kubelet in a one-time scenario without rebooting the node due to the change of machine-config(spec":{"paused":false}}), allowing you to modify the kubelet without affecting the service, follow this procedure.

Procedure

  1. Connect to the node in debug mode:

    $ oc debug node/<node>
    $ chroot /host

    Alternatively, it is possible to SSH to the node and become root.

  2. After access is established, check the default log level:

    $ systemctl cat kubelet

    Example output

    # /etc/systemd/system/kubelet.service.d/20-logging.conf
    [Service]
    Environment="KUBELET_LOG_LEVEL=2"

  3. Define the new verbosity required in a new /etc/systemd/system/kubelet.service.d/30-logging.conf file, which overrides /etc/systemd/system/kubelet.service.d/20-logging.conf. In this example, the verbosity is changed from 2 to 8:

    $ echo -e "[Service]\nEnvironment=\"KUBELET_LOG_LEVEL=8\"" > /etc/systemd/system/kubelet.service.d/30-logging.conf
  4. Reload systemd and restart the service:

    $ systemctl daemon-reload
    $ systemctl restart kubelet
  5. Gather the logs, and then revert the log level increase:

    $ rm -f /etc/systemd/system/kubelet.service.d/30-logging.conf
    $ systemctl daemon-reload
    $ systemctl restart kubelet

3.2. Persistent kubelet log level configuration

Procedure

  • Use the following MachineConfig object for persistent kubelet log level configuration:

     apiVersion: machineconfiguration.openshift.io/v1
     kind: MachineConfig
     metadata:
       labels:
         machineconfiguration.openshift.io/role: master
       name: 99-master-kubelet-loglevel
     spec:
       config:
         ignition:
           version: 3.2.0
         systemd:
           units:
             - name: kubelet.service
               enabled: true
               dropins:
                 - name: 30-logging.conf
                   contents: |
                     [Service]
                     Environment="KUBELET_LOG_LEVEL=2"

    Generally, it is recommended to apply 0-4 as debug-level logs and 5-8 as trace-level logs.

3.3. Log verbosity descriptions

Log verbosityDescription

--v=0

Always visible to an Operator.

--v=1

A reasonable default log level if you do not want verbosity.

--v=2

Useful steady state information about the service and important log messages that might correlate to significant changes in the system. This is the recommended default log level.

--v=3

Extended information about changes.

--v=4

Debug level verbosity.

--v=6

Display requested resources.

--v=7

Display HTTP request headers.

--v=8

Display HTTP request contents.

3.4. Gathering kubelet logs

Procedure

  • After the kubelet’s log level verbosity is configured properly, you can gather logs by running the following commands:

    $ oc adm node-logs --role master -u kubelet
    $ oc adm node-logs --role worker -u kubelet

    Alternatively, inside the node, run the following command:

    $ journalctl -b -f -u kubelet.service
  • To collect master container logs, run the following command:

    $ sudo tail -f /var/log/containers/*
  • To directly gather the logs of all nodes, run the following command:

    - for n in $(oc get node --no-headers | awk '{print $1}'); do oc adm node-logs $n | gzip > $n.log.gz; done
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.