Chapter 2. Configuring logging

Procedure

1. To activate the Ceph debugging output, dout(), at runtime:

   ceph tell TYPE.ID injectargs --debug-SUBSYSTEM VALUE [--NAME VALUE]

2. Replace:

   • TYPE with the type of Ceph daemons (osd, mon, or mds)
   • ID with a specific ID of the Ceph daemon. Alternatively, use * to apply the runtime setting to all daemons of a particular type.
   • SUBSYSTEM with a specific subsystem.

   • VALUE with a number from 1 to 20, where 1 is terse and 20 is verbose.

     For example, to set the log level for the OSD subsystem on the OSD named osd.0 to 0 and the memory level to 5:

     # ceph tell osd.0 injectargs --debug-osd 0/5

1. Log in to the host with a running Ceph daemon, for example, ceph-osd or ceph-mon.

2. Display the configuration:

   Syntax

   ceph daemon NAME config show | less

   Example

   [ceph: root@host01 /]# ceph daemon osd.0 config show | less

Procedure

1. To activate Ceph debugging output, dout() at boot time, add the debugging settings to the Ceph configuration file.

   1. For subsystems common to each daemon, add the settings under the [global] section.

   2. For subsystems for particular daemons, add the settings under a daemon section, such as [mon], [osd], or [mds].

      Example

      [global]
              debug_ms = 1/5
      
      [mon]
              debug_mon = 20
              debug_paxos = 1/5
              debug_auth = 2
      
      [osd]
              debug_osd = 1/5
              debug_monc = 5/20
      
      [mds]
              debug_mds = 1

Procedure

1. Add the size setting after the rotation frequency to the log rotation file:

   rotate 7
   weekly
   size SIZE
   compress
   sharedscripts

   For example, to rotate a log file when it reaches 500 MB:

   rotate 7
   weekly
   size 500 MB
   compress
   sharedscripts
   size 500M

   Note

   The size value can be expressed as '500 MB' or '500M'.

2. Open the crontab editor:

   [root@mon ~]# crontab -e

3. Add an entry to check the /etc/logrotate.d/ceph-<fsid> file. For example, to instruct Cron to check /etc/logrotate.d/ceph-<fsid> every 30 minutes:

   30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph-d3bb5396-c404-11ee-9e65-002590fc2a2e >/dev/null 2>&1

Procedure

1. Find where the logs are located:

   Syntax

   logrotate -f

   Example

   [root@host01 ~]# logrotate -f
   /etc/logrotate.d/ceph-12ab345c-1a2b-11ed-b736-fa163e4f6220

2. List the logs within the specified location:

   Syntax

   ll LOG_LOCATION

   Example

   [root@host01 ~]# ll /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220
    -rw-r--r--. 1 ceph ceph    412 Sep 28 09:26 opslog.log.1.gz

3. List the current buckets:

   Example

   [root@host01 ~]# /usr/local/bin/s3cmd ls

4. Create a bucket:

   Syntax

   /usr/local/bin/s3cmd mb s3://NEW_BUCKET_NAME

   Example

   [root@host01 ~]# /usr/local/bin/s3cmd mb s3://bucket1
   Bucket `s3://bucket1` created

5. List the current logs:

   Syntax

   ll LOG_LOCATION

   Example

   [root@host01 ~]# ll /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220
    total 852
    ...
    -rw-r--r--. 1 ceph ceph    920 Jun 29 02:17 opslog.log
    -rw-r--r--. 1 ceph ceph    412 Jun 28 09:26 opslog.log.1.gz

6. Collect the logs:

   Syntax

   tail -f LOG_LOCATION/opslog.log

   Example

   [root@host01 ~]# tail -f /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220/opslog.log
   
   {"bucket":"","time":"2022-09-29T06:17:03.133488Z","time_local":"2022-09-
   29T06:17:03.133488+0000","remote_addr":"10.0.211.66","user":"test1",
   "operation":"list_buckets","uri":"GET /
   HTTP/1.1","http_status":"200","error_code":"","bytes_sent":232,
   "bytes_received":0,"object_size":0,"total_time":9,"user_agent":"","referrer":
   "","trans_id":"tx00000c80881a9acd2952a-006335385f-175e5-primary",
   "authentication_type":"Local","access_key_id":"1234","temp_url":false}
   
   {"bucket":"cn1","time":"2022-09-29T06:17:10.521156Z","time_local":"2022-09-
   29T06:17:10.521156+0000","remote_addr":"10.0.211.66","user":"test1",
   "operation":"create_bucket","uri":"PUT /cn1/
   HTTP/1.1","http_status":"200","error_code":"","bytes_sent":0,
   "bytes_received":0,"object_size":0,"total_time":106,"user_agent":"",
   "referrer":"","trans_id":"tx0000058d60c593632c017-0063353866-175e5-primary",
   "authentication_type":"Local","access_key_id":"1234","temp_url":false}