Chapter 2. Configuring logging

Procedure

1. To activate the Ceph debugging output, dout(), at runtime:

   ceph tell TYPE.ID injectargs --debug-SUBSYSTEM VALUE [--NAME VALUE]

   Copy to Clipboard Toggle word wrap

2. Replace:

   • TYPE with the type of Ceph daemons (osd, mon, or mds)
   • ID with a specific ID of the Ceph daemon. Alternatively, use * to apply the runtime setting to all daemons of a particular type.
   • SUBSYSTEM with a specific subsystem.

   • VALUE with a number from 1 to 20, where 1 is terse and 20 is verbose.

     For example, to set the log level for the OSD subsystem on the OSD named osd.0 to 0 and the memory level to 5:

     # ceph tell osd.0 injectargs --debug-osd 0/5

     Copy to Clipboard Toggle word wrap

1. Log in to the host with a running Ceph daemon, for example, ceph-osd or ceph-mon.

2. Display the configuration:

   Syntax

   ceph daemon NAME config show | less

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph daemon osd.0 config show | less

   Copy to Clipboard Toggle word wrap

Procedure

1. To activate Ceph debugging output, dout() at boot time, add the debugging settings to the Ceph configuration file.

   1. For subsystems common to each daemon, add the settings under the [global] section.

   2. For subsystems for particular daemons, add the settings under a daemon section, such as [mon], [osd], or [mds].

      Example

      [global]
              debug_ms = 1/5
      
      [mon]
              debug_mon = 20
              debug_paxos = 1/5
              debug_auth = 2
      
      [osd]
              debug_osd = 1/5
              debug_monc = 5/20
      
      [mds]
              debug_mds = 1

      Copy to Clipboard Toggle word wrap

Procedure

1. Add the size setting after the rotation frequency to the log rotation file:

   rotate 7
   weekly
   size SIZE
   compress
   sharedscripts

   Copy to Clipboard Toggle word wrap

   For example, to rotate a log file when it reaches 500 MB:

   rotate 7
   weekly
   size 500 MB
   compress
   sharedscripts
   size 500M

   Copy to Clipboard Toggle word wrap
   Note

   The size value can be expressed as '500 MB' or '500M'.

2. Open the crontab editor:

   [root@mon ~]# crontab -e

   Copy to Clipboard Toggle word wrap

3. Add an entry to check the /etc/logrotate.d/ceph-<fsid> file. For example, to instruct Cron to check /etc/logrotate.d/ceph-<fsid> every 30 minutes:

   30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph-d3bb5396-c404-11ee-9e65-002590fc2a2e >/dev/null 2>&1

   Copy to Clipboard Toggle word wrap

Procedure

1. Find where the logs are located:

   Syntax

   logrotate -f

   Copy to Clipboard Toggle word wrap

   Example

   [root@host01 ~]# logrotate -f
   /etc/logrotate.d/ceph-12ab345c-1a2b-11ed-b736-fa163e4f6220

   Copy to Clipboard Toggle word wrap

2. List the logs within the specified location:

   Syntax

   ll LOG_LOCATION

   Copy to Clipboard Toggle word wrap

   Example

   [root@host01 ~]# ll /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220
    -rw-r--r--. 1 ceph ceph    412 Sep 28 09:26 opslog.log.1.gz

   Copy to Clipboard Toggle word wrap

3. List the current buckets:

   Example

   [root@host01 ~]# /usr/local/bin/s3cmd ls

   Copy to Clipboard Toggle word wrap

4. Create a bucket:

   Syntax

   /usr/local/bin/s3cmd mb s3://NEW_BUCKET_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@host01 ~]# /usr/local/bin/s3cmd mb s3://bucket1
   Bucket `s3://bucket1` created

   Copy to Clipboard Toggle word wrap

5. List the current logs:

   Syntax

   ll LOG_LOCATION

   Copy to Clipboard Toggle word wrap

   Example

   [root@host01 ~]# ll /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220
    total 852
    ...
    -rw-r--r--. 1 ceph ceph    920 Jun 29 02:17 opslog.log
    -rw-r--r--. 1 ceph ceph    412 Jun 28 09:26 opslog.log.1.gz

   Copy to Clipboard Toggle word wrap

6. Collect the logs:

   Syntax

   tail -f LOG_LOCATION/opslog.log

   Copy to Clipboard Toggle word wrap

   Example

   [root@host01 ~]# tail -f /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220/opslog.log
   
   {"bucket":"","time":"2022-09-29T06:17:03.133488Z","time_local":"2022-09-
   29T06:17:03.133488+0000","remote_addr":"10.0.211.66","user":"test1",
   "operation":"list_buckets","uri":"GET /
   HTTP/1.1","http_status":"200","error_code":"","bytes_sent":232,
   "bytes_received":0,"object_size":0,"total_time":9,"user_agent":"","referrer":
   "","trans_id":"tx00000c80881a9acd2952a-006335385f-175e5-primary",
   "authentication_type":"Local","access_key_id":"1234","temp_url":false}
   
   {"bucket":"cn1","time":"2022-09-29T06:17:10.521156Z","time_local":"2022-09-
   29T06:17:10.521156+0000","remote_addr":"10.0.211.66","user":"test1",
   "operation":"create_bucket","uri":"PUT /cn1/
   HTTP/1.1","http_status":"200","error_code":"","bytes_sent":0,
   "bytes_received":0,"object_size":0,"total_time":106,"user_agent":"",
   "referrer":"","trans_id":"tx0000058d60c593632c017-0063353866-175e5-primary",
   "authentication_type":"Local","access_key_id":"1234","temp_url":false}

   Copy to Clipboard Toggle word wrap