Edition 2
Copyright © 2008 Red Hat, Inc
1801 Varsity Drive
Raleigh, NC 27606-2072 USA
Phone: +1 919 754 3700
Phone: 888 733 4281
Fax: +1 919 754 3701
/etc/fstabMono-spaced Bold
To see the contents of the filemy_next_bestselling_novelin your current working directory, enter thecat my_next_bestselling_novelcommand at the shell prompt and press Enter to execute the command.
Press Enter to execute the command.Press Ctrl+Alt+F2 to switch to the first virtual terminal. Press Ctrl+Alt+F1 to return to your X-Windows session.
mono-spaced bold. For example:
File-related classes includefilesystemfor file systems,filefor files, anddirfor directories. Each class has its own associated set of permissions.
Choose → → from the main menu bar to launch Mouse Preferences. In the Buttons tab, click the Left-handed mouse check box and click to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).To insert a special character into a gedit file, choose → → from the main menu bar. Next, choose → from the Character Map menu bar, type the name of the character in the Search field and click . The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the button. Now switch back to your document and choose → from the gedit menu bar.
Mono-spaced Bold ItalicProportional Bold Italic
To connect to a remote machine using ssh, typesshat a shell prompt. If the remote machine isusername@domain.nameexample.comand your username on that machine is john, typessh john@example.com.Themount -o remountcommand remounts the named file system. For example, to remount thefile-system/homefile system, the command ismount -o remount /home.To see the version of a currently installed package, use therpm -qcommand. It will return a result as follows:package.package-version-release
Publican is a DocBook publishing system.
mono-spaced roman and presented thus:
books Desktop documentation drafts mss photos stuff svn books_tests Desktop1 downloads images notes scripts svgs
mono-spaced roman but add syntax highlighting as follows:
package org.jboss.book.jca.ex1; import javax.naming.InitialContext; public class ExClient { public static void main(String args[]) throws Exception { InitialContext iniCtx = new InitialContext(); Object ref = iniCtx.lookup("EchoBean"); EchoHome home = (EchoHome) ref; Echo echo = home.create(); System.out.println("Created Echo"); System.out.println("Echo.echo('Hello') = " + echo.echo("Hello")); } }
rhel-isa.
System Downtime Scheduled for Friday Night
Starting this Friday at 6pm (midnight for our associates in Berlin), all financial applications will be unavailable for a period of approximately four hours.During this time, changes to both the hardware and software on the Finance database server will be performed. These changes should greatly reduce the time required to run the Accounts Payable and Accounts Receivable applications, and the weekly Balance Sheet report.Other than the change in runtime, most people should notice no other change. However, those of you that have written your own SQL queries should be aware that the layout of some indices will change. This is documented on the company intranet website, on the Finance page.Should you have any questions, comments, or concerns, please contact System Administration at extension 4321.
System Downtime Scheduled for Tonight
Reminder: The system downtime announced this past Monday will take place as scheduled tonight at 6pm (midnight for the Berlin office). You can find the original announcement on the company intranet website, on the System Administration page.Several people have asked whether they should stop working early tonight to make sure their work is backed up prior to the downtime. This will not be necessary, as the work being done tonight will not impact any work done on your personal workstations.Remember, those of you that have written your own SQL queries should be aware that the layout of some indices will change. This is documented on the company intranet website, on the Finance page.
System Downtime Complete
The system downtime scheduled for Friday night (refer to the System Administration page on the company intranet website) has been completed. Unfortunately, hardware issues prevented one of the tasks from being completed. Due to this, the remaining tasks took longer than the originally-scheduled four hours. Instead, all systems were back in production by midnight (6am Saturday for the Berlin office).Because of the remaining hardware issues, performance of the AP, AR, and the Balance Sheet report will be slightly improved, but not to the extent originally planned. A second downtime will be announced and scheduled as soon as the issues that prevented completion of the task have been resolved.Please note that the downtime did change some database indices; people that have written their own SQL queries should consult the Finance page on the company intranet website.Please contact System Administration at extension 4321 with any questions.
cron and at commands are most commonly used in these roles.
cron can schedule the execution of commands or scripts for recurring intervals ranging in length from minutes to months. The crontab command is used to manipulate the files controlling the cron daemon that actually schedules each cron job for execution.
at command (and the closely-related command batch) are more appropriate for scheduling the execution of one-time scripts or commands. These commands implement a rudimentary batch subsystem consisting of multiple queues with varying scheduling priorities. The priorities are known as niceness levels (due to the name of the command -- nice). Both at and batch are perfect for tasks that must start at a given time but are not time-critical in terms of finishing.
bash command shell
perl scripting language
python scripting language
bash shell tend to make more extensive use of the many small utility programs (for example, to perform character string manipulation), while perl scripts perform more of these types of operations using features built into the language itself. A script written using python can fully exploit the language's object-oriented capabilities, making complex scripts more easily extensible.
grep and sed) that are part of Red Hat Enterprise Linux. Learning perl (and python), on the other hand, tends to be a more "self-contained" process. However, many perl language constructs are based on the syntax of various traditional UNIX utility programs, and as such are familiar to those Red Hat Enterprise Linux system administrators with shell scripting experience.
Vim text editor
vim and Emacs are primarily text-based in nature.
bob to write and group finance to read the file.
syslogd, which can log system information locally (normally to files in the /var/log/ directory) or to a remote system (which acts as a dedicated log server for multiple computers.)
crontab(1) and crontab(5) man pages -- Learn how to schedule commands and scripts for automatic execution at regular intervals.
at(1) man page -- Learn how to schedule commands and scripts for execution at a later time.
bash(1) man page -- Learn more about the default shell and shell script writing.
perl(1) man page -- Review pointers to the many man pages that make up perl's online documentation.
python(1) man page -- Learn more about options, files, and environment variables controlling the Python interpreter.
gedit(1) man page and menu entry -- Learn how to edit text files with this graphical text editor.
emacs(1) man page -- Learn more about this highly-flexible text editor, including how to run its online tutorial.
vim(1) man page -- Learn how to use this powerful text editor.
evolution(1) man page and menu entry -- Learn how to manage your email with this graphical email client.
mutt(1) man page and files in /usr/share/doc/mutt-<version> -- Learn how to manage your email with this text-based email client.
pam(8) man page and files in /usr/share/doc/pam-<version> -- Learn how authentication takes place under Red Hat Enterprise Linux.
[1] Be sure to send this message out as soon as the work is done, before you leave for home. Once you have left the office, it is much too easy to forget, leaving your users in the dark as to whether they can use the system or not.
[2] And of course a system administrator that expects the unexpected would naturally use RAID (or related technologies) to lessen the impact of a critical disk drive failing during production.
[3] Again, system administrators that think ahead configure their systems to make it as easy as possible to quickly add a new disk drive to the system.
free
top (and GNOME System Monitor, a more graphically oriented version of top)
vmstat
freefree command displays system memory utilization. Here is an example of its output:
total used free shared buffers cached Mem: 255508 240268 15240 0 7592 86188 -/+ buffers/cache: 146488 109020 Swap: 530136 26268 503868 Mem: row displays physical memory utilization, while the Swap: row displays the utilization of the system swap space, and the -/+ buffers/cache: row displays the amount of physical memory currently devoted to system buffers.
free by default only displays memory utilization information once, it is only useful for very short-term monitoring, or quickly determining if a memory-related problem is currently in progress. Although free has the ability to repetitively display memory utilization figures via its -s option, the output scrolls, making it difficult to easily detect changes in memory utilization.
free -s would be to run free using the watch command. For example, to display memory utilization every two seconds (the default display interval for watch), use this command:
watch freewatch command issues the free command every two seconds, updating by clearing the screen and writing the new output to the same screen location. This makes it much easier to determine how memory utilization changes over time, since watch creates a single updated view with no scrolling. You can control the delay between updates by using the -n option, and can cause any changes between updates to be highlighted by using the -d option, as in the following command:
watch -n 1 -d freewatch man page.
watch command runs until interrupted with Ctrl+C. The watch command is something to keep in mind; it can come in handy in many situations.
topfree displays only memory-related information, the top command does a little bit of everything. CPU utilization, process statistics, memory utilization -- top monitors it all. In addition, unlike the free command, top's default behavior is to run continuously; there is no need to use the watch command. Here is a sample display:
14:06:32 up 4 days, 21:20, 4 users, load average: 0.00, 0.00, 0.00 77 processes: 76 sleeping, 1 running, 0 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 19.6% 0.0% 0.0% 0.0% 0.0% 0.0% 180.2% cpu00 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0% cpu01 19.6% 0.0% 0.0% 0.0% 0.0% 0.0% 80.3% Mem: 1028548k av, 716604k used, 311944k free, 0k shrd, 131056k buff 324996k actv, 108692k in_d, 13988k in_c Swap: 1020116k av, 5276k used, 1014840k free 382228k cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 17578 root 15 0 13456 13M 9020 S 18.5 1.3 26:35 1 rhn-applet-gu 19154 root 20 0 1176 1176 892 R 0.9 0.1 0:00 1 top 1 root 15 0 168 160 108 S 0.0 0.0 0:09 0 init 2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0 3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1 4 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd 5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0 6 root 35 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1 9 root 15 0 0 0 0 SW 0.0 0.0 0:07 1 bdflush 7 root 15 0 0 0 0 SW 0.0 0.0 1:19 0 kswapd 8 root 15 0 0 0 0 SW 0.0 0.0 0:14 1 kscand 10 root 15 0 0 0 0 SW 0.0 0.0 0:03 1 kupdated 11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd top is running. For example, top by default displays both idle and non-idle processes. To display only non-idle processes, press i; a second press returns to the default display mode.
top appears like a simple display-only program, this is not the case. That is because top uses single character commands to perform various operations. For example, if you are logged in as root, it is possible to change the priority and even kill any process on your system. Therefore, until you have reviewed top's help screen (type ? to display it), it is safest to only type q (which exits top).
toptop, the GNOME System Monitor displays information related to overall system status, process counts, memory and swap utilization, and process-level statistics.
vmstatvmstat. With vmstat, it is possible to get an overview of process, memory, swap, I/O, system, and CPU activity in one line of numbers:
procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 5276 315000 130744 380184 1 1 2 24 14 50 1 1 47 0 r -- The number of runnable processes waiting for access to the CPU
b -- The number of processes in an uninterruptible sleep state
swpd -- The amount of virtual memory used
free -- The amount of free memory
buff -- The amount of memory used for buffers
cache -- The amount of memory used as page cache
si -- The amount of memory swapped in from disk
so -- The amount of memory swapped out to disk
bi -- Blocks sent to a block device
bo -- Blocks received from a block device
in -- The number of interrupts per second
cs -- The number of context switches per second
us -- The percentage of the time the CPU ran user-level code
sy -- The percentage of the time the CPU ran system-level code
id -- The percentage of the time the CPU was idle
wa -- I/O wait
vmstat is run without any options, only one line is displayed. This line contains averages, calculated from the time the system was last booted.
vmstat's ability to repetitively display resource utilization data at set intervals. For example, the command vmstat 1 displays one new line of utilization data every second, while the command vmstat 1 10 displays one new line per second, but only for the next ten seconds.
vmstat can be used to quickly determine resource utilization and performance issues. But to gain more insight into those issues, a different kind of tool is required -- a tool capable of more in-depth data collection and analysis.
iostatmpstatsadcsadc collects system resource utilization information and writes it to a file.
sarsadc, sar reports can be generated interactively or written to a file for more intensive analysis.
iostat commandiostat command at its most basic provides an overview of CPU and disk I/O statistics:
Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com) 07/11/2003 avg-cpu: %user %nice %sys %idle 6.11 2.56 2.15 89.18 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 1.68 15.69 22.42 31175836 44543290 iostat displays an overview of the system's average CPU utilization since the last reboot. The CPU utilization report includes the following percentages:
nice(2))
dev<major-number>-sequence-number, where <major-number><sequence-number>iostat. For more information, refer to the iostat(1) man page.
mpstat commandmpstat command produces the following output:
Linux 2.6.11-1.1369_FC4 (example.redhat.com) 02/07/2006 01:22:23 PM CPU %user %nice %system %iowait %irq %soft %idle intr/s 01:22:23 PM all 0.02 0.00 0.02 0.02 0.02 0.00 99.92 1011.86 mpstat allows the utilization for each CPU to be displayed individually, making it possible to determine how effectively each CPU is being used.
sadc commandsadc command collects system utilization data and writes it to a file for later analysis. By default, the data is written to files in the /var/log/sa/ directory. The files are named sa<dd>, where <dd>sadc is normally run by the sa1 script. This script is periodically invoked by cron via the file sysstat, which is located in /etc/cron.d/. The sa1 script invokes sadc for a single one-second measuring interval. By default, cron runs sa1 every 10 minutes, adding the data collected during each interval to the current /var/log/sa/sa<dd> file.
sar commandsar command produces system utilization reports based on the data collected by sadc. As configured in Red Hat Enterprise Linux, sar is automatically run to process the files automatically collected by sadc. The report files are written to /var/log/sa/ and are named sar<dd>, where <dd>sar is normally run by the sa2 script. This script is periodically invoked by cron via the file sysstat, which is located in /etc/cron.d/. By default, cron runs sa2 once a day at 23:53, allowing it to produce a report for the entire day's data.
sar Reportssar report produced by the default Red Hat Enterprise Linux configuration consists of multiple sections, with each section containing a specific type of data, ordered by the time of day that the data was collected. Since sadc is configured to perform a one-second measurement interval every ten minutes, the default sar reports contain data in ten-minute increments, from 00:00 to 23:50[7].
sar report, with the data from 00:30 through 23:40 removed to save space:
00:00:01 CPU %user %nice %system %idle 00:10:00 all 6.39 1.96 0.66 90.98 00:20:01 all 1.61 3.16 1.09 94.14 … 23:50:01 all 44.07 0.02 0.77 55.14 Average: all 5.80 4.99 2.87 86.34 iostat.
00:00:01 CPU %user %nice %system %idle 00:10:00 0 4.19 1.75 0.70 93.37 00:10:00 1 8.59 2.18 0.63 88.60 00:20:01 0 1.87 3.21 1.14 93.78 00:20:01 1 1.35 3.12 1.04 94.49 … 23:50:01 0 42.84 0.03 0.80 56.33 23:50:01 1 45.29 0.01 0.74 53.95 Average: 0 6.00 5.01 2.74 86.25 Average: 1 5.61 4.97 2.99 86.43 sar configuration; some are explored in upcoming chapters. For more information about the data contained in each section, refer to the sar(1) man page.
opcontrol command supports the --list-events option, which displays the event types available for the currently-installed processor, along with suggested minimum counter values for each.
oprofile.o kernel module, and the oprofiled daemon.
op_timeoprofppgprof-style output
op_to_sourceop_visualiseopcontrol command.
opcontrol to configure the type of data to be collected with the following command:
opcontrol \ --vmlinux=/boot/vmlinux-`uname -r` \ --ctr0-event=CPU_CLK_UNHALTED \ --ctr0-count=6000opcontrol to:
--vmlinux=/boot/vmlinux-`uname -r`)
--ctr0-event=CPU_CLK_UNHALTED)
--ctr0-count=6000)
oprofile kernel module is loaded by using the lsmod command:
Module Size Used by Not tainted oprofile 75616 1 … /dev/oprofile/) is mounted with the ls /dev/oprofile/ command:
0 buffer buffer_watershed cpu_type enable stats 1 buffer_size cpu_buffer_size dump kernel_only /root/.oprofile/daemonrc file contains the settings required by the data collection software:
CTR_EVENT[0]=CPU_CLK_UNHALTED CTR_COUNT[0]=6000 CTR_KERNEL[0]=1 CTR_USER[0]=1 CTR_UM[0]=0 CTR_EVENT_VAL[0]=121 CTR_EVENT[1]= CTR_COUNT[1]= CTR_KERNEL[1]=1 CTR_USER[1]=1 CTR_UM[1]=0 CTR_EVENT_VAL[1]= one_enabled=1 SEPARATE_LIB_SAMPLES=0 SEPARATE_KERNEL_SAMPLES=0 VMLINUX=/boot/vmlinux-2.4.21-1.1931.2.349.2.2.entsmp opcontrol to actually start data collection with the opcontrol --start command:
Using log file /var/lib/oprofile/oprofiled.log Daemon started. Profiler running. oprofiled daemon is running with the command ps x | grep -i oprofiled:
32019 ? S 0:00 /usr/bin/oprofiled --separate-lib-samples=0 … 32021 pts/0 S 0:00 grep -i oprofiled oprofiled command line displayed by ps is much longer; however, it has been truncated here for formatting purposes.)
/var/lib/oprofile/samples/ directory. The files in this directory follow a somewhat unusual naming convention. Here is an example:
}usr}bin}less#0 /) characters replaced by right curly brackets (}), and ending with a pound sign (#) followed by a number (in this case, 0.) Therefore, the file used in this example represents data collected while /usr/bin/less was running.
opcontrol --dump command to force the samples to disk.
op_time is used to display (in reverse order -- from highest number of samples to lowest) the samples that have been collected:
3321080 48.8021 0.0000 /boot/vmlinux-2.4.21-1.1931.2.349.2.2.entsmp 761776 11.1940 0.0000 /usr/bin/oprofiled 368933 5.4213 0.0000 /lib/tls/libc-2.3.2.so 293570 4.3139 0.0000 /usr/lib/libgobject-2.0.so.0.200.2 205231 3.0158 0.0000 /usr/lib/libgdk-x11-2.0.so.0.200.2 167575 2.4625 0.0000 /usr/lib/libglib-2.0.so.0.200.2 123095 1.8088 0.0000 /lib/libcrypto.so.0.9.7a 105677 1.5529 0.0000 /usr/X11R6/bin/XFree86 … less is a good idea when producing a report interactively, as the reports can be hundreds of lines long. The example given here has been truncated for that reason.
<sample-count><sample-percent><unused-field><executable-name>
<sample-count><sample-percent><unused-field><executable-name>XFree86. It is worth noting that for the system running this sample session, the counter value of 6000 used represents the minimum value recommended by opcontrol --list-events. This means that -- at least for this particular system -- OProfile overhead at its highest consumes roughly 11% of the CPU.
free(1) man page -- Learn how to display free and used memory statistics.
top(1) man page -- Learn how to display CPU utilization and process-level statistics.
watch(1) man page -- Learn how to periodically execute a user-specified program, displaying fullscreen output.
vmstat(8) man page -- Learn how to display a concise overview of process, memory, swap, I/O, system, and CPU utilization.
iostat(1) man page -- Learn how to display CPU and I/O statistics.
mpstat(1) man page -- Learn how to display individual CPU statistics on multiprocessor systems.
sadc(8) man page -- Learn how to collects system utilization data.
sa1(8) man page -- Learn about a script that runs sadc periodically.
sar(1) man page -- Learn how to produce system resource utilization reports.
sa2(8) man page -- Learn how to produce daily system resource utilization report files.
nice(1) man page -- Learn how to change process scheduling priority.
oprofile(1) man page -- Learn how to profile system performance.
op_visualise(1) man page -- Learn how to graphically display OProfile data.
[4] Assuming a single-processor computer system.
[5] More information on buses, datapaths, and bandwidth is available in Chapter 3, Bandwidth and Processing Power.
[6]
Device major numbers can be found by using ls -l to display the desired device file in /dev/. The major number appears after the device's group specification.
[7] Due to changing system loads, the actual time at which the data was collected may vary by a second or two.
[8] OProfile can also use a fallback mechanism (known as TIMER_INT) for those system architectures that lack performance monitoring hardware.
vmstat, it is possible to determine if overall device activity is excessive by examining the bi and bo fields; in addition, taking note of the si and so fields give you a bit more insight into how much disk activity is due to swap-related I/O:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 248088 158636 480804 0 0 2 6 120 120 10 3 87 0 bi field shows two blocks/second written to block devices (primarily disk drives), while the bo field shows six blocks/second read from block devices. We can determine that none of this activity was due to swapping, as the si and so fields both show a swap-related I/O rate of zero kilobytes/second.
iostat, it is possible to gain a bit more insight into disk-related activity:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003 avg-cpu: %user %nice %sys %idle 5.34 4.60 2.83 87.24 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev8-0 1.10 6.21 25.08 961342 3881610 dev8-1 0.00 0.00 0.00 16 0 /dev/sda, the first SCSI disk) averaged slightly more than one I/O operation per second (the tsp field). Most of the I/O activity for this device were writes (the Blk_wrtn field), with slightly more than 25 blocks written each second (the Blk_wrtn/s field).
iostat's -x option:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003 avg-cpu: %user %nice %sys %idle 5.37 4.54 2.81 87.27 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz /dev/sda 13.57 2.86 0.36 0.77 32.20 29.05 16.10 14.53 54.52 /dev/sda1 0.17 0.00 0.00 0.00 0.34 0.00 0.17 0.00 133.40 /dev/sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11.56 /dev/sda3 0.31 2.11 0.29 0.62 4.74 21.80 2.37 10.90 29.42 /dev/sda4 0.09 0.75 0.04 0.15 1.06 7.24 0.53 3.62 43.01 iostat output is now displaying statistics on a per-partition level. By using df to associate mount points with device names, it is possible to use this report to determine if, for example, the partition containing /home/ is experiencing an excessive workload.
iostat -x is longer and contains more information than this; here is the remainder of each line (with the device column added for easier reading):
Device: avgqu-sz await svctm %util /dev/sda 0.24 20.86 3.80 0.43 /dev/sda1 0.00 141.18 122.73 0.03 /dev/sda2 0.00 6.00 6.00 0.00 /dev/sda3 0.12 12.84 2.68 0.24 /dev/sda4 0.11 57.47 8.94 0.17 /dev/sda2 is the system swap partition; it is obvious from the many fields reading 0.00 for this partition that swapping is not a problem on this system.
/dev/sda1. The statistics for this partition are unusual; the overall activity seems low, but why are the average I/O request size (the avgrq-sz field), average wait time (the await field), and the average service time (the svctm field) so much larger than the other partitions? The answer is that this partition contains the /boot/ directory, which is where the kernel and initial ramdisk are stored. When the system boots, the read I/Os (notice that only the rsec/s and rkB/s fields are non-zero; no writing is done here on a regular basis) used during the boot process are for large numbers of blocks, resulting in the relatively long wait and service times iostat displays.
sar for a longer-term overview of I/O statistics; for example, sar -b displays a general I/O report:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003 12:00:00 AM tps rtps wtps bread/s bwrtn/s 12:10:00 AM 0.51 0.01 0.50 0.25 14.32 12:20:01 AM 0.48 0.00 0.48 0.00 13.32 … 06:00:02 PM 1.24 0.00 1.24 0.01 36.23 Average: 1.11 0.31 0.80 68.14 34.79 iostat's initial display, the statistics are grouped for all block devices.
sar -d:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003 12:00:00 AM DEV tps sect/s 12:10:00 AM dev8-0 0.51 14.57 12:10:00 AM dev8-1 0.00 0.00 12:20:01 AM dev8-0 0.48 13.32 12:20:01 AM dev8-1 0.00 0.00 … 06:00:02 PM dev8-0 1.24 36.25 06:00:02 PM dev8-1 0.00 0.00 Average: dev8-0 1.11 102.93 Average: dev8-1 0.00 0.00 sar, it is possible to accurately determine how much CPU power is being consumed and by what.
top is the first resource monitoring tool discussed in Chapter 2, Resource Monitoring to provide a more in-depth representation of CPU utilization. Here is a top report from a dual-processor workstation:
9:44pm up 2 days, 2 min, 1 user, load average: 0.14, 0.12, 0.09 90 processes: 82 sleeping, 1 running, 7 zombie, 0 stopped CPU0 states: 0.4% user, 1.1% system, 0.0% nice, 97.4% idle CPU1 states: 0.5% user, 1.3% system, 0.0% nice, 97.1% idle Mem: 1288720K av, 1056260K used, 232460K free, 0K shrd, 145644K buff Swap: 522104K av, 0K used, 522104K free 469764K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 30997 ed 16 0 1100 1100 840 R 1.7 0.0 0:00 top 1120 root 5 -10 249M 174M 71508 S 0.9 13.8 254:59 X 1260 ed 15 0 54408 53M 6864 S 0.7 4.2 12:09 gnome-terminal 888 root 15 0 2428 2428 1796 S 0.1 0.1 0:06 sendmail 1264 ed 15 0 16336 15M 9480 S 0.1 1.2 1:58 rhn-applet-gui 1 root 15 0 476 476 424 S 0.0 0.0 0:05 init 2 root 0K 0 0 0 0 SW 0.0 0.0 0:00 migration_CPU0 3 root 0K 0 0 0 0 SW 0.0 0.0 0:00 migration_CPU1 4 root 15 0 0 0 0 SW 0.0 0.0 0:01 keventd 5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU0 6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU1 7 root 15 0 0 0 0 SW 0.0 0.0 0:05 kswapd 8 root 15 0 0 0 0 SW 0.0 0.0 0:00 bdflush 9 root 15 0 0 0 0 SW 0.0 0.0 0:01 kupdated 10 root 25 0 0 0 0 SW 0.0 0.0 0:00 mdrecoveryd top does), which represent the load average for the past 1, 5, and 15 minutes, indicating that the system in this example was not very busy.
top itself; in other words, the one runnable process on this otherwise-idle system was top taking a "picture" of itself.
vmstat, we obtain a slightly different understanding of our example system:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 233276 146636 469808 0 0 7 7 14 27 10 3 87 0 0 0 0 233276 146636 469808 0 0 0 0 523 138 3 0 96 0 0 0 0 233276 146636 469808 0 0 0 0 557 385 2 1 97 0 0 0 0 233276 146636 469808 0 0 0 0 544 343 2 0 97 0 0 0 0 233276 146636 469808 0 0 0 0 517 89 2 0 98 0 0 0 0 233276 146636 469808 0 0 0 32 518 102 2 0 98 0 0 0 0 233276 146636 469808 0 0 0 0 516 91 2 1 98 0 0 0 0 233276 146636 469808 0 0 0 0 516 72 2 0 98 0 0 0 0 233276 146636 469808 0 0 0 0 516 88 2 0 97 0 0 0 0 233276 146636 469808 0 0 0 0 516 81 2 0 97 0 vmstat 1 10 to sample the system every second for ten times. At first, the CPU-related statistics (the us, sy, and id fields) seem similar to what top displayed, and maybe even appear a bit less detailed. However, unlike top, we can also gain a bit of insight into how the CPU is being used.
system fields, we notice that the CPU is handling about 500 interrupts per second on average and is switching between processes anywhere from 80 to nearly 400 times a second. If you think this seems like a lot of activity, think again, because the user-level processing (the us field) is only averaging 2%, while system-level processing (the sy field) is usually under 1%. Again, this is an idle system.
iostat and mpstat provide little additional information over what we have already experienced with top and vmstat. However, sar produces a number of reports that can come in handy when monitoring CPU utilization.
sar -q, which displays the run queue length, total number of processes, and the load averages for the past one and five minutes. Here is a sample:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003 12:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 12:10:00 AM 3 122 0.07 0.28 12:20:01 AM 5 123 0.00 0.03 … 09:50:00 AM 5 124 0.67 0.65 Average: 4 123 0.26 0.26 sar report is produced by the command sar -u:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003 12:00:01 AM CPU %user %nice %system %idle 12:10:00 AM all 3.69 20.10 1.06 75.15 12:20:01 AM all 1.73 0.22 0.80 97.25 … 10:00:00 AM all 35.17 0.83 1.06 62.93 Average: all 7.47 4.85 3.87 83.81 sar makes the data available on an ongoing basis and is therefore more useful for obtaining long-term averages, or for the production of CPU utilization graphs.
sar -U command can produce statistics for an individual processor or for all processors. Here is an example of output from sar -U ALL:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003 12:00:01 AM CPU %user %nice %system %idle 12:10:00 AM 0 3.46 21.47 1.09 73.98 12:10:00 AM 1 3.91 18.73 1.03 76.33 12:20:01 AM 0 1.63 0.25 0.78 97.34 12:20:01 AM 1 1.82 0.20 0.81 97.17 … 10:00:00 AM 0 39.12 0.75 1.04 59.09 10:00:00 AM 1 31.22 0.92 1.09 66.77 Average: 0 7.61 4.91 3.86 83.61 Average: 1 7.33 4.78 3.88 84.02 sar -w command reports on the number of context switches per second, making it possible to gain additional insight in where CPU cycles are being spent:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003 12:00:01 AM cswch/s 12:10:00 AM 537.97 12:20:01 AM 339.43 … 10:10:00 AM 319.42 Average: 1158.25 sar reports on interrupt activity. The first, (produced using the sar -I SUM command) displays a single "interrupts per second" statistic:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003 12:00:01 AM INTR intr/s 12:10:00 AM sum 539.15 12:20:01 AM sum 539.49 … 10:40:01 AM sum 539.10 Average: sum 541.00 sar -I PROC, it is possible to break down interrupt activity by processor (on multiprocessor systems) and by interrupt level (from 0 to 15):
Linux 2.4.21-1.1931.2.349.2.2.entsmp (pigdog.example.com) 07/21/2003 12:00:00 AM CPU i000/s i001/s i002/s i008/s i009/s i011/s i012/s 12:10:01 AM 0 512.01 0.00 0.00 0.00 3.44 0.00 0.00 12:10:01 AM CPU i000/s i001/s i002/s i008/s i009/s i011/s i012/s 12:20:01 AM 0 512.00 0.00 0.00 0.00 3.73 0.00 0.00 … 10:30:01 AM CPU i000/s i001/s i002/s i003/s i008/s i009/s i010/s 10:40:02 AM 0 512.00 1.67 0.00 0.00 0.00 15.08 0.00 Average: 0 512.00 0.42 0.00 N/A 0.00 6.03 N/A i002/s field illustrating the rate for interrupt level 2). If this were a multiprocessor system, there would be one line per sample period for each CPU.
sar adds or removes specific interrupt fields if no data is collected for that field. The example report above provides an example of this, the end of the report includes interrupt levels (3 and 10) that were not present at the start of the sampling period.
sar reports -- sar -I ALL and sar -I XALL. However, the default configuration for the sadc data collection utility does not collect the information necessary for these reports. This can be changed by editing the file /etc/cron.d/sysstat, and changing this line:
*/10 * * * * root /usr/lib/sa/sa1 1 1
*/10 * * * * root /usr/lib/sa/sa1 -I 1 1 sadc, and results in larger data file sizes. Therefore, make sure your system configuration can support the additional space consumption.
vmstat(8) man page -- Learn how to display a concise overview of process, memory, swap, I/O, system, and CPU utilization.
iostat(1) man page -- Learn how to display CPU and I/O statistics.
sar(1) man page -- Learn how to produce system resource utilization reports.
sadc(8) man page -- Learn how to collect system utilization data.
sa1(8) man page -- Learn about a script that runs sadc periodically.
top(1) man page -- Learn how to display CPU utilization and process-level statistics.
N is accessed by the CPU, it is highly likely that address N+1 will be accessed next. This makes sense, as most programs consist of large sections of instructions that execute -- in order -- one after the other.
X is accessed, it is likely that other addresses surrounding X will also be accessed in the future.
free, it is possible to get a concise (if somewhat simplistic) overview of memory and swap utilization. Here is an example:
total used free shared buffers cached Mem: 1288720 361448 927272 0 27844 187632 -/+ buffers/cache: 145972 1142748 Swap: 522104 0 522104
total used free shared buffers cached Mem: 255088 246604 8484 0 6492 111320 -/+ buffers/cache: 128792 126296 Swap: 530136 111308 418828 free, vmstat has the benefit of displaying more than memory utilization statistics. Here is the output from vmstat 1 10:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 111304 9728 7036 107204 0 0 6 10 120 24 10 2 89 0 2 0 111304 9728 7036 107204 0 0 0 0 526 1653 96 4 0 0 1 0 111304 9616 7036 107204 0 0 0 0 552 2219 94 5 1 0 1 0 111304 9616 7036 107204 0 0 0 0 624 699 98 2 0 0 2 0 111304 9616 7052 107204 0 0 0 48 603 1466 95 5 0 0 3 0 111304 9620 7052 107204 0 0 0 0 768 932 90 4 6 0 3 0 111304 9440 7076 107360 92 0 244 0 820 1230 85 9 6 0 2 0 111304 9276 7076 107368 0 0 0 0 832 1060 87 6 7 0 3 0 111304 9624 7092 107372 0 0 16 0 813 1655 93 5 2 0 2 0 111304 9624 7108 107372 0 0 0 972 1189 1165 68 9 23 0 free field) varies somewhat, and there is a bit of swap-related I/O (the si and so fields), but overall this system is running well. It is doubtful, however, how much additional workload it could handle, given the current memory utilization.
sar, it is possible to examine this aspect of system performance in much more detail.
sar -r report, we can examine memory and swap utilization more closely:
Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com) 07/22/2003 12:00:01 AM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached 12:10:00 AM 240468 1048252 81.34 0 133724 485772 12:20:00 AM 240508 1048212 81.34 0 134172 485600 … 08:40:00 PM 934132 354588 27.51 0 26080 185364 Average: 324346 964374 74.83 0 96072 467559 kbmemfree and kbmemused fields show the typical free and used memory statistics, with the percentage of memory used displayed in the %memused field. The kbbuffers and kbcached fields show how many kilobytes of memory are allocated to buffers and the system-wide data cache.
kbmemshrd field is always zero for systems (such as Red Hat Enterprise Linux) using the 2.4 Linux kernel.
12:00:01 AM kbswpfree kbswpused %swpused 12:10:00 AM 522104 0 0.00 12:20:00 AM 522104 0 0.00 … 08:40:00 PM 522104 0 0.00 Average: 522104 0 0.00 kbswpfree and kbswpused fields show the amount of free and used swap space, in kilobytes, with the %swpused field showing the swap space used as a percentage.
sar -W report. Here is an example:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003 12:00:01 AM pswpin/s pswpout/s 12:10:01 AM 0.15 2.56 12:20:00 AM 0.00 0.00 … 03:30:01 PM 0.42 2.56 Average: 0.11 0.37 pswpin/s) as there were going out to swap (pswpout/s).
sar -B report:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003 12:00:01 AM pgpgin/s pgpgout/s activepg inadtypg inaclnpg inatarpg 12:10:00 AM 0.03 8.61 195393 20654 30352 49279 12:20:00 AM 0.01 7.51 195385 20655 30336 49275 … 08:40:00 PM 0.00 7.79 71236 1371 6760 15873 Average: 201.54 201.54 169367 18999 35146 44702 pgpgin/s) and paged out to disk (pgpgout/s). These statistics serve as a barometer of overall virtual memory activity.
activepg field) averages approximately 660MB[13].
inadtypg field shows how many inactive pages are dirty (modified) and may need to be written to disk. The inaclnpg field, on the other hand, shows how many inactive pages are clean (unmodified) and do not need to be written to disk.
inatarpg field represents the desired size of the inactive list. This value is calculated by the Linux kernel and is sized such that the inactive list remains large enough to act as a pool for page replacement purposes.
sar -R report. Here is a sample report:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003 12:00:01 AM frmpg/s shmpg/s bufpg/s campg/s 12:10:00 AM -0.10 0.00 0.12 -0.07 12:20:00 AM 0.02 0.00 0.19 -0.07 … 08:50:01 PM -3.19 0.00 0.46 0.81 Average: 0.01 0.00 -0.00 -0.00 sar report are unique, in that they may be positive, negative, or zero. When positive, the value indicates the rate at which pages of this type are increasing. When negative, the value indicates the rate at which pages of this type are decreasing. A value of zero indicates that pages of this type are neither increasing or decreasing.
frmpg/s field) and nearly 1 page per second added to the page cache (the campg/s field). The list of pages used as buffers (the bufpg/s field) gained approximately one page every two seconds, while the shared memory page list (the shmpg/s field) neither gained nor lost any pages.
free(1) man page -- Learn how to display free and used memory statistics.
vmstat(8) man page -- Learn how to display a concise overview of process, memory, swap, I/O, system, and CPU utilization.
sar(1) man page -- Learn how to produce system resource utilization reports.
sa2(8) man page -- Learn how to produce daily system resource utilization report files.
[11] While "RAM" is an acronym for "Random Access Memory," and a term that could easily apply to any storage technology allowing the non-sequential access of stored data, when system administrators talk about RAM they invariably mean main system memory.
[12] A reasonably active system always experiences some level of page fault activity, due to page faults incurred as newly-launched applications are brought into memory.
[13] The page size under Red Hat Enterprise Linux on the x86 system used in this example is 4096 bytes. Systems based on other architectures may have different page sizes.
/etc/fstab| Cylinder | Head | Sector |
|---|---|---|
| 1014 |
X
|
X
|
| Cylinder | Head | Sector |
|---|---|---|
| 1014 | 2 |
X
|
| Cylinder | Head | Sector |
|---|---|---|
| 1014 | 2 | 12 |
| Adjacent Cylinder | Full-Stroke |
|---|---|
| 0.6 | 8.2 |
accounting, people that work in engineering would have their directories under engineering, and so on.
engineering directory, it would be a straightforward process to:
engineering directory
engineering directory on the original storage to something like engineering-archive (before deleting it entirely after running smoothly with the new configuration for a month)
N-drive RAID 0 array results in the removal of 1/Nth of all the data, rendering the array useless
n disk drives in it, every nth chunk is dedicated to parity.
x is calculated by mathematically combining the data from each chunk x stored on all the other drives in the array. If the data in a chunk is updated, the corresponding parity chunk must be recalculated and updated as well.
n drives, only 1/nth of the total available storage is dedicated to redundancy
/dev/ directory. The format for each file name depends on several aspects of the actual hardware and how it has been configured. The important points are as follows:
sd -- The device is SCSI-based
hd -- The device is ATA-based
hda or sda.
sda through sdz, the next 26 would be named sdaa through sdaz, and so on.
/dev/hda1 -- The first partition on the first ATA drive
/dev/sdb12 -- The twelfth partition on the second SCSI drive
/dev/sdad4 -- The fourth partition on the thirtieth SCSI drive
/dev/hdc -- The entire third ATA device
/dev/sdb -- The entire second SCSI device
/dev/sda) are not changed in any way
/dev/sda now has a new name, because the first SCSI drive on the new controller is now /dev/sda
devlabeldevlabel software attempts to address the device naming issue in a different manner than file system labels. The devlabel software is run by Red Hat Enterprise Linux whenever the system reboots (and whenever hotpluggable devices are inserted or removed).
devlabel runs, it reads its configuration file (/etc/sysconfig/devlabel) to obtain the list of devices for which it is responsible. For each device on the list, there is a symbolic link (chosen by the system administrator) and the device's UUID (Universal Unique IDentifier).
devlabel command makes sure the symbolic link always refers to the originally-specified device -- even if that device's name has changed. In this way, a system administrator can configure a system to refer to /dev/projdisk instead of /dev/sda12, for example.
devlabel must only search the system for the matching UUID and update the symbolic link appropriately.
devlabel, refer to the System Administrators Guide.
devlabel-managed symbolic link
foo in its root directory; the full path to the directory would be /foo/. Next, assume that this system has a partition that is to be mounted, and that the partition's mount point is to be /foo/. If that partition had a file by the name of bar.txt in its top-level directory, after the partition was mounted you could access the file with the following full file specification:
/foo/bar.txt/foo/ directory will be read from or written to that partition.
/home/ -- that is because the login directories for all user accounts are normally located under /home/. If /home/ is used as a mount point, all users' files are written to a dedicated partition and will not fill up the operating system's file system.
/etc/mtab
/proc/mounts
df command
/etc/mtab/etc/mtab is a normal file that is updated by the mount program whenever file systems are mounted or unmounted. Here is a sample /etc/mtab:
/dev/sda3 / ext3 rw 0 0 none /proc proc rw 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/sda1 /boot ext3 rw 0 0 none /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/sda4 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 /etc/mtab file is meant to be used to display the status of currently-mounted file systems only. It should not be manually modified.
ro) or read-write (rw), along with any other mount options
/etc/fstab[24])
/proc/mounts/proc/mounts file is part of the proc virtual file system. As with the other files under /proc/, the mounts "file" does not exist on any disk drive in your Red Hat Enterprise Linux system.
cat /proc/mounts, we can view the status of all mounted file systems:
rootfs / rootfs rw 0 0 /dev/root / ext3 rw 0 0 /proc /proc proc rw 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/sda1 /boot ext3 rw 0 0 none /dev/pts devpts rw 0 0 /dev/sda4 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 /proc/mounts is very similar to that of /etc/mtab. There are a number of file systems mounted that have nothing to do with disk drives. Among these are the /proc/ file system itself (along with two other file systems mounted under /proc/), pseudo-ttys, and shared memory.
/proc/mounts is the best way to be 100% sure of seeing what is mounted on your Red Hat Enterprise Linux system, as the kernel is providing this information. Other methods can, under rare circumstances, be inaccurate.
df Command/etc/mtab or /proc/mounts lets you know what file systems are currently mounted, it does little beyond that. Most of the time you are more interested in one particular aspect of the file systems that are currently mounted -- the amount of free space on them.
df command. Here is some sample output from df:
Filesystem 1k-blocks Used Available Use% Mounted on /dev/sda3 8428196 4280980 3719084 54% / /dev/sda1 124427 18815 99188 16% /boot /dev/sda4 8428196 4094232 3905832 52% /home none 644600 0 644600 0% /dev/shm /etc/mtab and /proc/mount are immediately obvious:
df it is very easy to see where the problem lies.
/etc/exports. For more information, see the exports(5) man page and the System Administrators Guide.
/etc/fstab/etc/fstab file. This file is used to control what file systems are mounted when the system boots, as well as to supply default values for other file systems that may be mounted manually from time to time. Here is a sample /etc/fstab file:
LABEL=/ / ext3 defaults 1 1 /dev/sda1 /boot ext3 defaults 1 2 /dev/cdrom /mnt/cdrom iso9660 noauto,owner,kudzu,ro 0 0 /dev/homedisk /home ext3 defaults 1 2 /dev/sda2 swap swap defaults 0 0 /dev/sda1), a file system label specification (LABEL=/), or a devlabel-managed symbolic link (/dev/homedisk)
/boot)
auto may be specified to select automatic detection of the file system to be mounted, which is handy for removable media units such as diskette drives)
mount's behavior (noauto,owner,kudzu)
dump backup utility is used, the number in this field controls dump's handling of the specified file system
fsck checks the integrity of the file systems
/etc/fstab
fdisk utility program
parted, another command-line utility program
fdisk are included:
fdisk, this is done by including the device name when you start fdisk:
fdisk /dev/hda fdisk displays the partition table by using the p command:
Command (m for help): p Disk /dev/hda: 255 heads, 63 sectors, 1244 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 17 136521 83 Linux /dev/hda2 18 83 530145 82 Linux swap /dev/hda3 84 475 3148740 83 Linux /dev/hda4 476 1244 6176992+ 83 Linux d command in fdisk:
Command (m for help): d Partition number (1-4): 1fdisk, this is a two-step process -- first, creating the partition (using the n command):
Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-767): 1 Last cylinder or +size or +sizeM or +sizeK: +512M t command):
Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): 82fdisk by using the w command:
Command (m for help): wmkfs utility program. However, mkfs does not actually do the work of writing the file-system-specific information onto a disk drive; instead it passes control to one of several other programs that actually create the file system.
mkfs.<fstype> man page for the file system you have selected. For example, look at the mkfs.ext3 man page to see the options available to you when creating a new ext3 file system. In general, the mkfs.<fstype> programs provide reasonable defaults for most configurations; however here are some of the options that system administrators most commonly change:
/etc/fstab
/etc/fstab/etc/fstab”, you must add the necessary line(s) to /etc/fstab to ensure that the new file system(s) are mounted whenever the system reboots. Once you have updated /etc/fstab, test your work by issuing an "incomplete" mount, specifying only the device or mount point. Something similar to one of the following commands is sufficient:
mount /home mount /dev/hda3 /home or /dev/hda3 with the mount point or device for your specific situation.)
/etc/fstab entry is correct, mount obtains the missing information from it and completes the mount operation.
/etc/fstab is configured properly to automatically mount the new storage every time the system boots (although if you can afford a quick reboot, it would not hurt to do so -- just to be sure).
/etc/fstab
/etc/fstab/etc/fstab file. You can identify the proper lines by one of the following methods:
/etc/fstab
/etc/fstab
/etc/fstab that identify swap partitions on the disk drive to be removed; they can be easily overlooked.
umountumount command. If a swap partition exists on the disk drive, it must be either be deactivated with the swapoff command, or the system should be rebooted.
umount command requires you to specify either the device file name, or the partition's mount point:
umount /dev/hda2 umount /home /etc/fstab entry.
swapoff to disable swapping to a partition, you must specify the device file name representing the swap partition:
swapoff /dev/hda4 swapoff, boot into rescue mode and remove the partition's /etc/fstab entry.
badblocks -ws <device-name><device-name>/dev/hdb for the second ATA hard drive.
badblocks runs:
Writing pattern 0xaaaaaaaa: done Reading and comparing: done Writing pattern 0x55555555: done Reading and comparing: done Writing pattern 0xffffffff: done Reading and comparing: done Writing pattern 0x00000000: done Reading and comparing: done badblocks is actually writing four different data patterns to every block on the disk drive. For large disk drives, this process can take a long time -- quite often several hours.
rm command. That is because when you delete a file using rm it only marks the file as deleted -- it does not erase the contents of the file.
/home/ directory was on its own file system, disk quotas could be enabled there, enforcing equitable disk usage by all users. However the root file system could be left without disk quotas, eliminating the complexity of maintaining disk quotas on a file system where only the operating system itself resides.
/etc/fstab
quotacheck
/etc/fstab file controls the mounting of file systems under Red Hat Enterprise Linux. Because disk quotas are implemented on a per-file-system basis, there are two options -- usrquota and grpquota -- that must be added to that file to enable disk quotas.
usrquota option enables user-based disk quotas, while the grpquota option enables group-based quotas. One or both of these options may be enabled by placing them in the options field for the desired file system.
quotacheck command is used to create the disk quota files and to collect the current usage information from already existing files. The disk quota files (named aquota.user and aquota.group for user- and group-based quotas) contain the necessary quota-related information and reside in the file system's root directory.
edquota command is used.
edquota command. Here is an example:
Disk quotas for user matt (uid 500): Filesystem blocks soft hard inodes soft hard /dev/md3 6618000 0 0 17397 0 0
Disk quotas for user matt (uid 500): Filesystem blocks soft hard inodes soft hard /dev/md3 6618000 6900000 7000000 17397 0 0 edquota program can also be used to set the per-file-system grace period by using the -t option.
repquota utility program. Using the command repquota /home produces this output:
*** Report for user quotas on device /dev/md3 Block grace time: 7days; Inode grace time: 7days Block limits File limits User used soft hard grace used soft hard grace ---------------------------------------------------------------------- root -- 32836 0 0 4 0 0 matt -- 6618000 6900000 7000000 17397 0 0 repquota can be found in the System Administrators Guide, in the chapter on disk quotas.
quotacheck. However, many system administrators recommend running quotacheck on a regular basis, even if the system has not crashed.
quotacheck when enabling disk quotas.
quotacheck command:
quotacheck -avugquotacheck on a regular basis is to use cron. Most system administrators run quotacheck once a week, though there may be valid reasons to pick a longer or shorter interval, depending on your specific conditions.
mdadm program (refer to man mdadm for more information).
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hd[bc]1 mdadm --detail --scan > /dev/mdadm.conf
/dev/md0 is now ready to be formatted and mounted. The process at this point is no different than for formatting and mounting a single disk drive.
/proc/mdstat/proc/mdstat is the easiest way to check on the status of all RAID arrays on a particular system. Here is a sample mdstat (view with the command cat /proc/mdstat):
Personalities : [raid1] read_ahead 1024 sectors md1 : active raid1 hda3[0] hdc3[1] 522048 blocks [2/2] [UU] md0 : active raid1 hda2[0] hdc2[1] 4192896 blocks [2/2] [UU] md2 : active raid1 hda1[0] hdc1[1] 128384 blocks [2/2] [UU] unused devices: <none> /proc/mdstat and contains the following information:
/dev/ part)
U meaning the device is OK, and _ indicating that the device has failed)
/proc/mdstat show that a problem exists with one of the RAID arrays, you can rebuild it by performing the following steps:
mdadm --manage /dev/md0 -r /dev/sdc3
fdisk, replace the removed disk and re-format the replacement disk.
mdadm --manage /dev/md0 -a /dev/sdc3
mdadm --manage --set-faulty /dev/md0 /dev/sdc3
watch -n 1 cat /proc/mdstat
mdadm --manage /dev/md0 -r /dev/sdc3
mdadm --manage /dev/md0 -a /dev/sdc3
mdadm --detail /dev/md0
exports(5) man page -- Learn about the NFS configuration file format.
fstab(5) man page -- Learn about the file system information configuration file format.
swapoff(8) man page -- Learn how to disable swap partitions.
df(1) man page -- Learn how to display disk space usage on mounted file systems.
fdisk(8) man page -- Learn about this partition table maintenance utility program.
mkfs(8), mke2fs(8) man pages -- Learn about these file system creation utility programs.
badblocks(8) man page -- Learn how to test a device for bad blocks.
quotacheck(8) man page -- Learn how to verify block and inode usage for users and groups and optionally creates disk quota files.
edquota(8) man page -- Learn about this disk quota maintenance utility program.
repquota(8) man page -- Learn about this disk quota reporting utility program.
raidtab(5) man page -- Learn about the software RAID configuration file format.
mdadm(8) man page -- Learn about this software RAID array management utility program.
lvm(8) man page -- Learn about Logical Volume Management.
devlabel(8) man page -- Learn about persistent storage device access.
devlabel, partitioning, disk quotas, NFS, and Samba.
[14] Some optical devices -- notably CD-ROM drives -- use somewhat different approaches to data storage; these differences are pointed out at the appropriate points within the chapter.
[15] In some optical devices (such as CD-ROM drives), the access arm is continually moving, causing the head assembly to describe a spiral path over the surface of the platter. This is a fundamental difference in how the storage medium is used and reflects the CD-ROM's origins as a medium for music storage, where continuous data retrieval is a more common operation than searching for a specific data point.
[16] While early mass storage devices used the same number of sectors for every track, later devices divided the range of cylinders into different "zones," with each zone having a different number of sectors per track. The reason for this is to take advantage of the additional space between sectors in the outer cylinders, where there is more unused space between sectors.
[17] Some storage hardware (usually those that incorporate removable drive "carriers") is designed so that the act of plugging a module into place automatically sets the SCSI ID to an appropriate value.
[18] Fast-80 is not technically a change in bus speed; instead the 40MHz bus was retained, but data was clocked at both the rising and falling of each clock pulse, effectively doubling the throughput.
[19] Actually, this is not entirely true. All hard drives include some amount of on-board cache memory that is used to improve read performance. However, any I/O request to read data must eventually be satisfied by physically reading the data from the storage medium. This means that, while cache may alleviate read I/O performance problems, it can never totally eliminate the time required to physically read the data from the storage medium.
[20] Some optical disk drives exhibit this behavior, due to the physical constraints of the technologies used to implement optical data storage.
[21] When early RAID research began, the acronym stood for Redundant Array of Inexpensive Disks, but over time the "standalone" disks that RAID was intended to supplant became cheaper and cheaper, rendering the price comparison meaningless.
[22] I/O performance is reduced while operating with one drive unavailable, due to the overhead involved in reconstructing the missing data.
[23] There is also an impact from the parity calculations required for each write. However, depending on the specific RAID 5 implementation (specifically, where in the system the parity calculations are performed), this impact can range from sizable to nearly nonexistent.
[24]
Refer to Section 5.9.5, “Mounting File Systems Automatically with /etc/fstab” for more information.
| Password Length | Potential Passwords |
|---|---|
| 1 | 26 |
| 2 | 676 |
| 3 | 17,576 |
| 4 | 456,976 |
| 5 | 11,881,376 |
| 6 | 308,915,776 |
accounts, this information can then be placed in a shared directory (owned by the accounts group) with group read and write permissions on the directory.
finance, and make all finance personnel members of that group. If the financial information is too sensitive for the company at large, but vital for senior officials within the organization, then grant the senior officials group-level permission to access the directories and data used by the finance department by adding all senior officials to the finance group.
r — Indicates that a given category of user can read a file.
w — Indicates that a given category of user can write to a file.
x — Indicates that a given category of user can execute the contents of a file.
-) indicates that no access is permitted.
ls -l. For example, if the user juan creates an executable file named foo, the output of the command ls -l foo would appear like this:
-rwxrwxr-x 1 juan juan 0 Sep 26 12:25 foo rwx. This first set of symbols define owner access — in this example, the owner juan has full access, and may read, write, and execute the file. The next set of rwx symbols define group access (again, with full access), while the last set of symbols define the types of access permitted for all other users. Here, all other users may read and execute the file, but may not modify it in any way.
juan launches an application, the application runs using user juan's context. However, in some cases the application may need a more privileged level of access in order to accomplish a task. Such applications include those that edit system settings or log in users. For this reason, special permissions have been created.
s in the place of the x in the owner category. If the owner of the file does not have execute permissions, a capital S reflects this fact.
s in place of the x in the group category. If the group owning the file or directory does not have execute permissions, a capital S reflects this fact.
t in place of the x in the everyone category. If the everyone category does not have execute permissions, the T is capitalized to reflect this fact.
/tmp/ directory for exactly this reason.
/etc/passwd and /etc/group files on a file server and a user's workstation differ in the UIDs or GIDs they contain, improper application of permissions can lead to security issues.
juan has a UID of 500 on a desktop computer, files juan creates on a file server will be created with owner UID 500. However, if user bob logs in locally to the file server (or even some other computer), and bob's account also has a UID of 500, bob will have full access to juan's files, and vice versa.
root user, and are treated specially by Red Hat Enterprise Linux — all access is automatically granted.
/etc/ directory. When a system administrator creates new user accounts, these files must either be edited manually or applications must be used to make the necessary changes.
/etc/ directory that store user and group information under Red Hat Enterprise Linux.
/etc/passwd/etc/passwd file is world-readable and contains a list of users, each on a separate line. On each line is a colon delimited list containing the following information:
x if shadow passwords are being used — more on this later).
finger access this field to provide additional user information.
/home/juan/.
/bin/bash. If this field is left blank, /bin/sh is used. If it is set to a non-existent file, then the user will be unable to log into the system.
/etc/passwd entry:
root:x:0:0:root:/root:/bin/bash root user has a shadow password, as well as a UID and GID of 0. The root user has /root/ as a home directory, and uses /bin/bash for a shell.
/etc/passwd, see the passwd(5) man page.
/etc/shadow/etc/passwd file must be world-readable (the main reason being that this file is used to perform the translation from UID to username), there is a risk involved in storing everyone's password in /etc/passwd. True, the passwords are encrypted. However, it is possible to perform attacks against passwords if the encrypted password is available.
/etc/passwd can be obtained by an attacker, an attack that can be carried out in secret becomes possible. Instead of risking detection by having to attempt an actual login with every potential password generated by password-cracker, an attacker can use a password cracker in the following manner:
/etc/passwd
/etc/shadow file is readable only by the root user and contains password (and optional password aging information) for each user. As in the /etc/passwd file, each user's information is on a separate line. Each of these lines is a colon delimited list including the following information:
crypt(3) library function or the md5 hash algorithm. In this field, values other than a validly-formatted encrypted or hashed password are used to control user logins and to show the password status. For example, if the value is ! or *, the account is locked and the user is not allowed to log in. If the value is !! a password has never been set before (and the user, not having set a password, will not be able to log in).
/etc/shadow:
juan:$1$.QKDPc5E$SWlkjRWexrXYgc98F.:12825:0:90:5:30:13096: juan:
/etc/shadow file, see the shadow(5) man page.
/etc/group/etc/group file is world-readable and contains a list of groups, each on a separate line. Each line is a four field, colon delimited list including the following information:
newgrp command and typing the password stored here. If a lower case x is in this field, then shadow group passwords are being used.
/etc/group:
general:x:502:juan,shelley,bob general group is using shadow passwords, has a GID of 502, and that juan, shelley, and bob are members.
/etc/group, see the group(5) man page.
/etc/gshadow/etc/gshadow file is readable only by the root user and contains an encrypted password for each group, as well as group membership and administrator information. Just as in the /etc/group file, each group's information is on a separate line. Each of these lines is a colon delimited list including the following information:
newgrp command. If the value of this field is !, then no user is allowed to access the group using the newgrp command. A value of !! is treated the same as a value of ! — however, it also indicates that a password has never been set before. If the value is null, only group members can log into the group.
gpasswd command.
/etc/gshadow:
general:!!:shelley:juan,bob general group has no password and does not allow non-members to join using the newgrp command. In addition, shelley is a group administrator, and juan and bob are regular, non-administrative members.
| Application | Function |
|---|---|
/usr/sbin/useradd
| Adds user accounts. This tool is also used to specify primary and secondary group membership. |
/usr/sbin/userdel
| Deletes user accounts. |
/usr/sbin/usermod
|
Edits account attributes including some functions related to password aging. For more fine-grained control, use the passwd command. usermod is also used to specify primary and secondary group membership.
|
passwd
| Sets passwords. Although primarily used to change a user's password, it also controls all aspects of password aging. |
/usr/sbin/chpasswd
| Reads in a file consisting of username and password pairs, and updates each users' password accordingly. |
chage
|
Changes the user's password aging policies. The passwd command can also be used for this purpose.
|
chfn
| Changes the user's GECOS information. |
chsh
| Changes the user's default shell. |
| Application | Function |
|---|---|
/usr/sbin/groupadd
|
Adds groups, but does not assign users to those groups. The useradd and usermod programs should then be used to assign users to a given group.
|
/usr/sbin/groupdel
| Deletes groups. |
/usr/sbin/groupmod
|
Modifies group names or GIDs, but does not change group membership. The useradd and usermod programs should be used to assign users to a given group.
|
gpasswd
| Changes group membership and sets passwords to allow non-group members who know the group password to join the group. It is also used to specify group administrators. |
/usr/sbin/grpck
|
Checks the integrity of the /etc/group and /etc/gshadow files.
|
| Application | Function |
|---|---|
chgrp
| Changes which group owns a given file. |
chmod
| Changes access permissions for a given file. It is also capable of assigning special permissions. |
chown
| Changes a file's ownership (and can also change group). |
passwd(5) man page — Learn more about the file format information for the /etc/passwd file.
group(5) man page — Learn more about the file format information for the /etc/group file.
shadow(5) man page — Learn more about the file format information for the /etc/shadow file.
useradd(8) man page — Learn how to create or update user accounts.
userdel(8) man page — Learn how to delete user accounts.
usermod(8) man page — Learn how to modify user accounts.
passwd(1) man page — Learn how to update a user's password.
chpasswd(8) man page — Learn how to update many users' passwords at one time.
chage(1) man page — Learn how to change user password aging information.
chfn(1) man page — Learn how to change a user's GECOS (finger) information.
chsh(1) man page — Learn how to change a user's login shell.
groupadd(8) man page — Learn how to create a new group.
groupdel(8) man page — Learn how to delete a group.
groupmod(8) man page — Learn how to modify a group.
gpasswd(1) man page — Learn how to administer the /etc/group and /etc/gshadow files.
grpck(1) man page — Learn how to verify the integrity of the /etc/group and /etc/gshadow files.
chgrp(1) man page — Learn how to change group-level ownership.
chmod(1) man page — Learn how to change file access permissions.
chown(1) man page — Learn how to change file owner and group.
[25] GECOS stands for General Electric Comprehensive Operating Supervisor. This field was used at Bell Labs, in the original UNIX implementation. The lab had many different computers, including one running GECOS. This field was used to store information when the UNIX system sent batch and print jobs to the GECOS system.
system-config-printer. This command automatically determines whether to run the graphical or text-based version depending on whether the command is executed in the graphical desktop environment or from a text-based console.
system-config-printer-tui from a shell prompt.
/etc/printcap file or the files in the /etc/cups/ directory. Each time the printer daemon (cups) is started or restarted, new configuration files are dynamically created. The files are dynamically created when changes are applied with the Printer Configuration Tool as well.
lpr(1) man page -- Learn how to print selected files on the printer of your choice.
lprm(1) man page -- Learn how to remove print jobs from a printer queue.
cupsd(8) man page -- Learn about the CUPS printer daemon.
cupsd.conf(5) man page -- Learn about the file format for the CUPS printer daemon configuration file.
classes.conf(5) man page -- Learn about the file format for the CUPS class configuration file.
/usr/share/doc/cups-<version> -- Learn more about the CUPS printing system.
tartar utility is well known among UNIX system administrators. It is the archiving method of choice for sharing ad-hoc bits of source code and files between systems. The tar implementation included with Red Hat Enterprise Linux is GNU tar, one of the more feature-rich tar implementations.
tar, backing up the contents of a directory can be as simple as issuing a command similar to the following:
tar cf /mnt/backup/home-backup.tar /home/ home-backup.tar in /mnt/backup/. The archive contains the contents of the /home/ directory.
tar czf /mnt/backup/home-backup.tar.gz /home/ tar; to learn more about them, read the tar(1) man page.
cpiocpio utility is another traditional UNIX program. It is an excellent general-purpose program for moving data from one place to another and, as such, can serve well as a backup program.
cpio is a bit different from tar. Unlike tar, cpio reads the names of the files it is to process via standard input. A common method of generating a list of files for cpio is to use programs such as find whose output is then piped to cpio:
find /home/ | cpio -o > /mnt/backup/home-backup.cpio cpio archive file (containing the everything in /home/) called home-backup.cpio and residing in the /mnt/backup/ directory.
find has a rich set of file selection tests, sophisticated backups can easily be created. For example, the following command performs a backup of only those files that have not been accessed within the past year:
find /home/ -atime +365 | cpio -o > /mnt/backup/home-backup.cpio cpio (and find); to learn more about them read the cpio(1) and find(1) man pages.
dump/restore: Not Recommended for Mounted File Systems!dump and restore programs are Linux equivalents to the UNIX programs of the same name. As such, many system administrators with UNIX experience may feel that dump and restore are viable candidates for a good backup program under Red Hat Enterprise Linux. However, one method of using dump can cause problems. Here is Linus Torvald's comment on the subject:
From: Linus Torvalds To: Neil Conway Subject: Re: [PATCH] SMP race in ext2 - metadata corruption. Date: Fri, 27 Apr 2001 09:59:46 -0700 (PDT) Cc: Kernel Mailing List <linux-kernel At vger Dot kernel Dot org> [ linux-kernel added back as a cc ] On Fri, 27 Apr 2001, Neil Conway wrote: > > I'm surprised that dump is deprecated (by you at least ;-)). What to > use instead for backups on machines that can't umount disks regularly? Note that dump simply won't work reliably at all even in 2.4.x: the buffer cache and the page cache (where all the actual data is) are not coherent. This is only going to get even worse in 2.5.x, when the directories are moved into the page cache as well. So anybody who depends on "dump" getting backups right is already playing Russian roulette with their backups. It's not at all guaranteed to get the right results - you may end up having stale data in the buffer cache that ends up being "backed up". Dump was a stupid program in the first place. Leave it behind. > I've always thought "tar" was a bit undesirable (updates atimes or > ctimes for example). Right now, the cpio/tar/xxx solutions are definitely the best ones, and will work on multiple filesystems (another limitation of "dump"). Whatever problems they have, they are still better than the _guaranteed_(*) data corruptions of "dump". However, it may be that in the long run it would be advantageous to have a "filesystem maintenance interface" for doing things like backups and defragmentation.. Linus (*) Dump may work fine for you a thousand times. But it _will_ fail under the right circumstances. And there is nothing you can do about it. dump/restore on mounted file systems is strongly discouraged. However, dump was originally designed to backup unmounted file systems; therefore, in situations where it is possible to take a file system offline with umount, dump remains a viable backup technology.
tar or dump to do the actual backups (although under Red Hat Enterprise Linux using tar is preferable, due to the issues with dump raised in Section 8.4.2.3, “dump/restore: Not Recommended for Mounted File Systems!”). As such, AMANDA backups do not require AMANDA in order to restore files -- a decided plus.
amanda(8) man page.
tar(1) man page -- Learn how to archive data.
dump(8) man page -- Learn how to dump file system contents.
restore(8) man page -- Learn how to retrieve file system contents saved by dump.
cpio(1) man page -- Learn how to copy files to and from archives.
find(1) man page -- Learn how to search for files.
amanda(8) man page -- Learn more about the commands that are part of the AMANDA backup system.
/usr/share/doc/amanda-server-<version>/ -- Learn more about AMANDA by reviewing these various documents and example files.
[26] And this would likely be considered a best-case response time, as technicians usually are responsible for territories that extend away from their office in all directions. If you are at one end of their territory and the only available technician is at the other end, the response time will be even longer.
[27] UPS technology is discussed in more detail in Section 8.1.3.2.3.2, “Providing Power For the Next Few Minutes”.
[28] If the operators at your organization do not have a set of operating procedures, work with them, your management, and your users to get them created. Without them, a data center is out of control and likely to experience severe problems in the course of day-to-day operations.
[29] We are using the term data in this section to describe anything that is processed via backup software. This includes operating system software, application software, as well as actual data. No matter what it is, as far as backup software is concerned, it is all data.
[30]
The .gz extension is traditionally used to signify that the file has been compressed with gzip. Sometimes .tar.gz is shortened to .tgz to keep file names reasonably sized.
| Revision History | |||
|---|---|---|---|
| Revision 1.0 | Tue Sep 23 2008 | ||
| |||