Red Hat Summit이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 17. Monitoring Red Hat Gluster Storage Gluster Workload
Monitoring storage volumes is helpful when conducting a capacity planning or performance tuning activity on a Red Hat Gluster Storage volume. You can monitor the Red Hat Gluster Storage volumes with different parameters and use those system outputs to identify and troubleshoot issues.
You can use the
volume top and volume profile commands to view vital performance information and identify bottlenecks on each brick of a volume.
You can also perform a statedump of the brick processes and NFS server process of a volume, and also view volume status and volume information.
Note
If you restart the server process, the existing
profile and top information will be reset.
17.1. Profiling volumes
링크 복사링크가 클립보드에 복사되었습니다!
17.1.1. Server-side volume profiling using volume profile
링크 복사링크가 클립보드에 복사되었습니다!
The
volume profile command provides an interface to get the per-brick or NFS server I/O information for each file operation of a volume. This information helps in identifying the bottlenecks in the storage system.
This section describes how to use the
volume profile command.
17.1.1.1. Start Profiling
링크 복사링크가 클립보드에 복사되었습니다!
To view the file operation information of each brick, start the profiling command:
# gluster volume profile VOLNAME start
For example, to start profiling on test-volume:
# gluster volume profile test-volume start
Starting volume profile on test has been successfulImportant
Running
profile command can affect system performance while the profile information is being collected. Red Hat recommends that profiling should only be used for debugging.
When profiling is started on the volume, the following additional options are displayed when using the
volume info command:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on17.1.1.2. Displaying the I/O Information
링크 복사링크가 클립보드에 복사되었습니다!
To view the I/O information of the bricks on a volume, use the following command:
# gluster volume profile VOLNAME info
For example, to view the I/O information of test-volume:
# gluster v profile glustervol info
Brick: rhsqaci-vm33.lab.eng.blr.redhat.com:/bricks/brick0/1
-----------------------------------------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 11 RELEASE
0.00 0.00 us 0.00 us 0.00 us 238 RELEASEDIR
0.35 380.05 us 380.05 us 380.05 us 1 SETXATTR
0.40 107.73 us 5.50 us 413.31 us 4 OPENDIR
0.62 167.65 us 91.33 us 339.28 us 4 STATFS
0.86 187.42 us 28.50 us 534.96 us 5 GETXATTR
2.16 106.54 us 32.16 us 383.58 us 22 ENTRYLK
2.17 106.97 us 39.01 us 251.65 us 22 FLUSH
2.92 263.57 us 189.06 us 495.05 us 12 SETATTR
3.22 124.60 us 43.08 us 311.69 us 28 INODELK
3.41 616.76 us 319.27 us 1028.72 us 6 READDIR
10.11 997.03 us 413.73 us 3507.02 us 11 CREATE
73.79 256.58 us 50.02 us 924.61 us 312 LOOKUP
Duration: 46537 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 1 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 11 RELEASE
0.00 0.00 us 0.00 us 0.00 us 4 RELEASEDIR
0.35 380.05 us 380.05 us 380.05 us 1 SETXATTR
0.40 107.73 us 5.50 us 413.31 us 4 OPENDIR
0.62 167.65 us 91.33 us 339.28 us 4 STATFS
0.86 187.42 us 28.50 us 534.96 us 5 GETXATTR
2.16 106.54 us 32.16 us 383.58 us 22 ENTRYLK
2.17 106.97 us 39.01 us 251.65 us 22 FLUSH
2.92 263.57 us 189.06 us 495.05 us 12 SETATTR
3.22 124.60 us 43.08 us 311.69 us 28 INODELK
3.41 616.76 us 319.27 us 1028.72 us 6 READDIR
10.11 997.03 us 413.73 us 3507.02 us 11 CREATE
73.79 256.58 us 50.02 us 924.61 us 312 LOOKUP
Duration: 347 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: rhsqaci-vm33.lab.eng.blr.redhat.com:/bricks/brick1/1
-----------------------------------------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 12 RELEASE
0.00 0.00 us 0.00 us 0.00 us 238 RELEASEDIR
0.14 146.24 us 146.24 us 146.24 us 1 OPEN
0.26 266.64 us 266.64 us 266.64 us 1 SETXATTR
0.26 67.88 us 2.50 us 243.52 us 4 OPENDIR
0.42 108.83 us 81.87 us 139.11 us 4 STATFS
0.98 201.26 us 82.36 us 306.38 us 5 GETXATTR
2.49 116.34 us 23.53 us 304.10 us 22 ENTRYLK
2.75 236.13 us 124.73 us 358.80 us 12 SETATTR
3.12 114.82 us 44.34 us 550.01 us 28 INODELK
3.17 142.00 us 23.16 us 388.56 us 23 FLUSH
4.37 748.73 us 324.70 us 1115.96 us 6 READDIR
6.57 614.44 us 364.94 us 807.17 us 11 CREATE
75.46 248.88 us 66.43 us 599.31 us 312 LOOKUP
Duration: 46537 seconds
Data Read: 0 bytes
Data Written: 0 bytes
To view the I/O information of the NFS server on a specified volume, use the following command:
# gluster volume profile VOLNAME info nfs
For example, to view the I/O information of the NFS server on test-volume:
# gluster volume profile test-volume info nfs
NFS Server : localhost
----------------------
Cumulative Stats:
Block Size: 32768b+ 65536b+
No. of Reads: 0 0
No. of Writes: 1000 1000
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.01 410.33 us 194.00 us 641.00 us 3 STATFS
0.60 465.44 us 346.00 us 867.00 us 147 FSTAT
1.63 187.21 us 67.00 us 6081.00 us 1000 SETATTR
1.94 221.40 us 58.00 us 55399.00 us 1002 ACCESS
2.55 301.39 us 52.00 us 75922.00 us 968 STAT
2.85 326.18 us 88.00 us 66184.00 us 1000 TRUNCATE
4.47 511.89 us 60.00 us 101282.00 us 1000 FLUSH
5.02 3907.40 us 1723.00 us 19508.00 us 147 READDIRP
25.42 2876.37 us 101.00 us 843209.00 us 1012 LOOKUP
55.52 3179.16 us 124.00 us 121158.00 us 2000 WRITE
Duration: 7074 seconds
Data Read: 0 bytes
Data Written: 102400000 bytes
Interval 1 Stats:
Block Size: 32768b+ 65536b+
No. of Reads: 0 0
No. of Writes: 1000 1000
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.01 410.33 us 194.00 us 641.00 us 3 STATFS
0.60 465.44 us 346.00 us 867.00 us 147 FSTAT
1.63 187.21 us 67.00 us 6081.00 us 1000 SETATTR
1.94 221.40 us 58.00 us 55399.00 us 1002 ACCESS
2.55 301.39 us 52.00 us 75922.00 us 968 STAT
2.85 326.18 us 88.00 us 66184.00 us 1000 TRUNCATE
4.47 511.89 us 60.00 us 101282.00 us 1000 FLUSH
5.02 3907.40 us 1723.00 us 19508.00 us 147 READDIRP
25.41 2878.07 us 101.00 us 843209.00 us 1011 LOOKUP
55.53 3179.16 us 124.00 us 121158.00 us 2000 WRITE
Duration: 330 seconds
Data Read: 0 bytes
Data Written: 102400000 bytes17.1.1.3. Stop Profiling
링크 복사링크가 클립보드에 복사되었습니다!
To stop profiling on a volume, use the following command:
# gluster volume profile VOLNAME stop
For example, to stop profiling on test-volume:
# gluster volume profile test-volume stop
Stopping volume profile on test has been successful17.1.2. Client-side volume profiling (FUSE only)
링크 복사링크가 클립보드에 복사되었습니다!
Red Hat Gluster Storage lets you profile how your mount point is being accessed, so that you can investigate latency issues even when you cannot instrument the application accessing your storage.
The io-stats translator records statistics of all file system activity on a Red Hat Gluster Storage volume that travels through a FUSE mount point. It collects information on files opened from the FUSE mount path, the read and write throughput for these files, the number of blocks read and written, and the latency observed for different file operations.
Run the following command to output all recorded statistics for the specified mount point to the specified output file.
# setfattr -n trusted.io-stats-dump -v output_file_id mount_point
This generates a number of files in the
/var/run/gluster directory. The output_file_id is not the whole file name, but is used as part of the name of the generated files.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}