Chapter 11. Disk I/O: MonitoringThis chapter provides an overview of the AIX-specific tools (sar, nmon, and topas) available to monitor disk I/O activity. These commands let you quickly troubleshoot a performance problem and capture data for historical trending and analysis. Don't expect to see iostat here. That Unix utility lets you quickly determine whether there is an imbalanced I/O load between your physical disks and adapters. But unless you decide to write your own scripting tools using iostat, it won't help you with long-term trending and capturing data. 11.1. sarThe sar command, whose syntax is given in Chapter 8, is one of those older, generic Unix tools that have been improved over the years. Although I generally prefer to use more specific AIX tools, such as nmon and topas, sar provides strong information with respect to disk I/O. Let's run a typical sar command to examine I/O activity: # sar -d 1 2 Here's a breakdown of the column headings:
You want to be wary of any disk that approaches 100 percent utilization or shows a large number of queue requests waiting for disk. Although the sample output shows some activity, we have no real I/O problems because no waiting for I/O is occurring. We should continue to monitor this system to ensure other disks in addition to hdisk0 are being used. Where sar differs from iostat is in its ability to capture data for long-term analysis and trending using its system activity data collector (sadc) utility. Usually turned off in cron, the sadc utility lets you capture data for historical trending and analysis. Here's how this works. As delivered by default on AIX systems, two shell scripts, /usr/lib/sa/sa1 and /usr/lib/sa/sa2, which are normally commented out provide daily reports on the activity of the system. The sar command actually calls the sadc routine to access system data. The following example shows how the shell scripts are usually kicked off from cron: # crontab -l | grep sa1 11.2. topasWhat about something a little more user-friendly? Did you say topas? The topas command is a nice performance-monitoring tool that you can use for a number of purposes, including monitoring your disk subsystem. Let's take a look at the topas output from a disk perspective:
Code View:
Scroll
/
Show All
Topas output for host - Testhost Events/Queues FILE/TTY No I/O activity at all is going on here. Besides the physical disk, pay close attention to the "Wait" information (in the CPU section up top), which also helps you determine whether the system is I/O-bound. If you see high numbers here, you can then use other tools, such as filemon, fileplace, lslv, or lsof, to help you figure out which processes, adapters, or file systems are causing your bottlenecks. The topas command is useful for quickly troubleshooting an issue when you want a little more than iostat can provide. In a sense, topas is a graphical mix of iostat and vmstat, although recent improvements now provide the ability to capture data for historical analysis. These improvements, introduced in AIX 5.3, no doubt were made because of the popularity of nmon. While nmon provides a front end similar to topas, it is much more useful in terms of long-term trending and analysis. Further, as you learned in Chapter 5, nmon gives system administrators the ability to output data to an Excel spreadsheet for presentation in graphical charts (tailor-made for senior management and functional teams) that clearly illustrate bottlenecks. The nmon analyzer tool provides the hooks into nmon. (Figure 5.1 in Chapter 5 shows some sample output from the nmon analyzer.) With respect to disk I/O, nmon reports the following data: disk I/O rates, data transfers, read/write ratios, and disk adapter statistics. Here is one small example of where nmon really shines. Let's say you want to know which processes are hogging most of the disk I/O, and you want to be able to correlate that activity with the actual disk to clearly illustrate I/O per process. nmon usage helps you here more than any other tool. To perform this task with nmon, use the –t option; set your timing and then sort by I/O channel. How do you use nmon to capture data and import it into the analyzer? Use the open-source sudo command and run nmon for three hours, taking a snapshot every 30 seconds: # sudo nmon -f -t -r test1 -s 30 -c 180 Next, sort the created output file: # sort -A testsystem_yymmdd.nmon > testsystem_yymmdd.csv Then FTP the .csv file to your PC, start the nmon analyzer spreadsheet (enabling macros), and click on Analyze nmon data. The nmon command also helps track the configuration of asynchronous I/O servers. 11.3. Logical Volume MonitoringSay that a ticket has just been opened up with the service desk that relates to slow performance on some database server. You suspect there might be an I/O issue, so you start with iostat. iostat, the equivalent of using vmstat for virtual memory, is arguably the most effective way to get a first glance at what is happening with your I/O subsystem. Let's run iostat, in this case once a second:
Code View:
Scroll
/
Show All
# iostat 1 The command reports the following information:
You need to watch % tm_act very carefully because if this utilization exceeds roughly 60 to 70 percent, that usually indicates that processes are starting to wait for I/O. This might be your first clue of impending I/O problems. Moving data to less busy drives can obviously help ease this burden. Generally speaking, the more drives your data hits, the better. Just like anything else, too much of a good thing can also be bad, and you also have to make sure you don't have too many drives hitting any one adapter. One way to determine whether an adapter is saturated is to sum the Kbps amounts for all disks attached to one adapter. The total should be below the disk adapter's throughput rating, usually less than 70 percent. Using the –a flag with iostat helps you drill down further to examine adapter utilization. In the following output, there clearly are no bottlenecks: # iostat -a 11.4. AIX LVM CommandsWe examined disk placement earlier, and I stressed the importance of architecting your systems correctly from the beginning. Unfortunately, you don't always have that option. As system administrators, we sometimes inherit systems that must be fixed. Let's look at a sample layout of the logical volumes on disks to determine whether we need to change definitions or rearrange data. We'll examine a volume group and find the logical volumes that are a part of it. The lsvg command provides volume group information: # lsvg -l data2 Now, let's use lslv, which provides information about logical volumes:
Code View:
Scroll
/
Show All
# lslv data2lv This view provides a detailed description of the logical volume attributes. What do we have here? The intra-policy is at the center, which normally is the best policy for I/O-intensive logical volumes. As you recall from an earlier discussion, there are exceptions to this rule. Unfortunately, you've just hit one of them. Because Mirror Write Consistency Check (MWCC) is on, the volume would have been better served if it were placed on the edge. Let's look at its inter-policy. The inter-policy is minimum, which is usually the best policy if availability matters more than performance. Further, there are twice as many physical partitions as logical partitions, which signifies that you are mirroring your systems. In this case, let's assume you were told that raw performance was the most important objective, so the logical volume wasn't configured to reflect the reality of how the volume is being used. Further, if you are mirroring the system and using an external storage array, the situation would even be worse, because you're already providing mirroring at the hardware layer, which is actually more effective than using AIX mirroring. The lslv command's –l (lowercase L) flag lists all the physical volumes associated with the logical volumes and shows the distribution for each logical volume: # lslv -l data2lv With this detail, you can determine that 100 percent of the physical partitions on the disk are allocated to this logical volume. The distribution section of the output shows the actual number of physical partitions within each physical volume. From here, you can detail the volume's intra-disk policy. Let's drill down even further, using the -p flag: # lspv -p hdisk2 The preceding view shows you what is free on the physical volume, what has been used, and which partitions are used where. The order of the fields is as follows: edge, middle, center, inner-middle, inner-edge. The sample report shows that most of the data is in the middle and some is at the center. This is a nice view. You can do a lot with lsvg and lslv; run a man on these commands to find out more about them. One of the best tools for looking at LVM use is lvmstat. Because the lvmstat view is not enabled by default, you need to enable it before running the tool: # lvmstat -v data2vg -e The following command takes a snapshot of Logical Volume Manager information every second for 10 intervals: # lvmstat -v data2vg 1 10 The resulting output shows the most utilized logical volumes on your system since you started the data collection tool: # lvmstat -v data2vg This detail is very helpful when drilling down to the logical volume layer in tuning your systems:
11.5. filemon and fileplaceThis section introduces two important I/O tools, filemon and fileplace, and discusses how you can use them during systems administration each day. 11.6. filemonfilemon [-d] [-i Trace_File -n Gennames_File] [-o File] [-O Levels]The filemon command uses a trace facility to report on the I/O activity of physical and logical storage, including your actual files. The I/O activity monitored is based on the time interval specified when running the trace. The command reports on all layers of file system utilization, including the LVM, virtual memory, and physical disk layers. Run without any flags, filemon executes in the background while application programs or system commands are being run and monitored. The trace starts automatically and runs until it is stopped. At that time, the command generates an I/O activity report and exits. It can also process a trace file that has been recorded by the trace facility. You can then generate reports from this file. Because reports generated to standard output usually scroll past your screen, I advise using the –o option to write the output to a file: # filemon -o dbmon.out -O all When we check out the file, here is what we see:
Code View:
Scroll
/
Show All
Sun Aug 19 17:50:45 2007Look for long seek times because they can result in decreased application performance. By examining the read and write sequence counts in detail, you can further determine whether the access is sequential or random. This information helps you when it is time to do I/O tuning. The sample output clearly illustrates that there is no I/O bottleneck to speak of in this case. The filemon command provides a tremendous amount of detail; to be honest, I've found it gives too much information at times. Further, using filemon can impose a large performance hit. I don't typically like to recommend performance tools that impose such a substantial overhead, so I'll reiterate that although filemon certainly has a purpose, you need to be very careful when using it. 11.7. fileplacefileplace [ {-l|-p} [-i] [-v] ] File | [-m LogicalVolumeName]
The fileplace
command reports the placement of a file's blocks within a file system.
The command is commonly used to examine and assess the efficiency of a
file's placement on disk. For what purposes do you use it? One reason
would be to help determine whether some of your heavily used files are
substantially fragmented.
The fileplace command can also help you identify the physical volume with the highest utilization and determine whether the drive or I/O adapter is causing the bottleneck. Let's look at an example of a frequently accessed file:
Code View:
Scroll
/
Show All
# fileplace -pv dbfileYou should be interested in space efficiency and sequentiality here. Higher space efficiency means files are less fragmented and provide better sequential file access. A higher sequentiality tells you that the files are more contiguously allocated, which is also better for sequential file access. In the example, space efficiency could be better, while sequentiality is quite high. If space and sequentiality are too low, you might want to consider file system reorganization. You would do this with the reorgvg command, which can improve logical volume utilization and efficiency. |
|
|
