I/O問題一直是一個比較難定位的問題，今天線上環(huán)境遇到了I/O 引起的CPU負載問題，看到了如下這篇比較好的文章，完飯后的我還在和西紅柿和黃瓜在減肥的路上抗?fàn)帲脤⒃姆g成中文，供廣大同胞品鑒

 
Linux has many tools available for troubleshooting some are easy to use, some are more advanced.

Linux 有許多可用來查找問題的簡單工具，也有許多是更高級的

I/O Wait is an issue that requires use of some of the more advanced tools as well as an advanced usage of some of the basic tools. The reason I/O Wait is difficult to troubleshoot is due to the fact that by default there are plenty of tools to tell you that your system is I/O bound, but not as many that can narrow the problem to a specific process or processes.

I/O Wait 就是一個需要使用高級的工具來debug的問題，當(dāng)然也有許多基本工具的高級用法。I/O wait的問題難以定位的原因是因為我們有很多工具可以告訴你說I/O 受限了，但是并沒有告訴你具體是那個進程引起的（哪些進程們）

Answering whether or not I/O is causing system slowness

確認是否是I/O問題導(dǎo)致系統(tǒng)緩慢

To identify whether I/O is causing system slowness you can use several commands but the easiest is the unix command top.

確認是否是I/O導(dǎo)致的系統(tǒng)緩慢我們可以使用多個命令，但是，最簡單的是unix的命令 top

 # top
 top - 14:31:20 up 35 min, 4 users, load average: 2.25, 1.74, 1.68
 Tasks: 71 total, 1 running, 70 sleeping, 0 stopped, 0 zombie
 Cpu(s): 2.3%us, 1.7%sy, 0.0%ni, 0.0%id, 96.0%wa, 0.0%hi, 0.0%si, 0.0%st
 Mem: 245440k total, 241004k used, 4436k free, 496k buffers
 Swap: 409596k total, 5436k used, 404160k free, 182812k cached
 
From the CPU(s) line you can see the current percentage of CPU in I/O Wait; The higher the number the more cpu resources are waiting for I/O access.

從Cpu一行我們可以看到浪費在I/O Wait上的CPU百分比；這個數(shù)字越高說明越多的CPU資源在等待I/O權(quán)限

wa -- iowait
 Amount of time the CPU has been waiting for I/O to complete.
 
Finding which disk is being written to

查找那塊磁盤正在被寫入

The above top command shows I/O Wait from the system as a whole but it does not tell you what disk is being affected; for this we will use the iostatcommand.

上邊的top命令從一個整體上說明了I/O wait，但是并沒有說明是哪塊磁盤影響的，想知道是哪塊磁盤引發(fā)的問題，我們用到了另外一個命令 iostat 命令

 $ iostat -x 2 5
 avg-cpu: %user %nice %system %iowait %steal %idle
  3.66 0.00 47.64 48.69 0.00 0.00
 
 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 44.50 39.27 117.28 29.32 11220.94 13126.70 332.17 65.77 462.79 9.80 2274.71 7.60 111.41
 dm-0 0.00 0.00 83.25 9.95 10515.18 4295.29 317.84 57.01 648.54 16.73 5935.79 11.48 107.02
 dm-1 0.00 0.00 57.07 40.84 228.27 163.35 8.00 93.84 979.61 13.94 2329.08 10.93 107.02
 
The iostat command in the example will print a report every 2 seconds for 5 intervals; the -x tells iostat to print out an extended report.

上邊的例子中，iostat 會每2秒更新一次，一共打印5次信息， -x 的選項是打印出擴展信息

The 1st report from iostat will print statistics based on the last time the system was booted; for this reason in most circumstances the first report from iostat should be ignored. Every sub-sequential report printed will be based on the time since the previous interval. For example in our command we will print a report 5 times, the 2nd report are disk statistics gathered since the 1st run of the report, the 3rd is based from the 2nd and so on.

第一個iostat 報告會打印出系統(tǒng)最后一次啟動后的統(tǒng)計信息，這也就是說，在多數(shù)情況下，第一個打印出來的信息應(yīng)該被忽略，剩下的報告，都是基于上一次間隔的時間。舉例子來說，這個命令會打印5次，第二次的報告是從第一次報告出來一個后的統(tǒng)計信息，第三次是基于第二次 ，依次類推

In the above example the %utilized for sda is 111.41% this is a good indicator that our problem lies with processes writing to sda. While the test system in my example only has 1 disk this type of information is extremely helpful when the server has multiple disks as this can narrow down the search for which process is utilizing I/O.

在上面的例子中，sda的%utilized 是111.41%，這個很好的說明了有進程正在寫入到sda磁盤中。因為例子中的測試系統(tǒng)只有一塊磁盤，當(dāng)一個服務(wù)器中有多快磁盤的時候，這個命令可以很好的縮小我們需要查找的進程的范圍

Aside from %utilized there is a wealth of information in the output of iostat; items such as read and write requests per millisecond(rrqm/s & wrqm/s), reads and writes per second (r/s & w/s) and plenty more. In our example our program seems to be read and write heavy this information will be helpful when trying to identify the offending process.

除了%utilized 外，我們可以得到更改豐富的資源從iostat，例如每毫秒讀寫請求（rrqm/s & wrqm/s)），每秒讀寫的（(r/s & w/s)，當(dāng)然還有更多。在上邊的例子中，我們的項目看起來正在讀寫非常多的信息。這個對我們查找相應(yīng)的進程非常有用。

Finding the processes that are causing high I/O

查找引起高I/O wait 對應(yīng)的進程

iotop

 # iotop
 Total DISK READ: 8.00 M/s | Total DISK WRITE: 20.36 M/s
  TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
 15758 be/4 root 7.99 M/s 8.01 M/s 0.00 % 61.97 % bonnie++ -n 0 -u 0 -r 239 -s 478 -f -b -d /tmp
 
The simplest method of finding which process is utilizing storage the most is to use the command iotop. After looking at the statistics it is easy to identify bonnie++ as the process causing the most I/O utilization on this machine.

最簡單的方式來發(fā)現(xiàn)罪魁禍首是使用命令iotop，通過查看iotop的統(tǒng)計信息，我們可以很容易的指導(dǎo)bonnie++就是罪魁禍首

While iotop is a great command and easy to use, it is not installed on all (or the main) Linux distributions by default; and I personally prefer not to rely on commands that are not installed by default. A systems administrator may find themselves on a system where they simply cannot install the non-defualt packages until a scheduled time which may be far too late depending on the issue.

雖然iotop是一個非常強大的工具，并且使用簡單，但是它并不是默認安裝在所有的linux操作系統(tǒng)中。并且我個人傾向不要太依賴那些默認沒有安裝的命令。一個系統(tǒng)管理員可能會發(fā)現(xiàn)他無法立即安裝額外的除默認程序之外的軟件，除非等到后邊的維護的時間。

If iotop is not available the below steps will also allow you to narrow down the offending process/processes.

如果iotop并沒有安裝，下面的步驟會教會你如何縮小目標進程的范圍

Process list “state”

進程的狀態(tài)

The ps command has statistics for memory and cpu but it does not have a statistic for disk I/O. While it may not have a statistic for I/O it does show the processes state which can be used to indicate whether or not a process is waiting for I/O.

ps 命令對內(nèi)存和CPU有一個統(tǒng)計，但是他沒有對磁盤I/O的統(tǒng)計，雖然他沒有顯示磁盤I/O，但是它顯示進行的狀態(tài)，我們可以用來知道一個進程是否正在等待I/O

The ps state field provides the processes current state; below is a list of states from the man page.

ps state狀態(tài)來表示了process現(xiàn)在的狀態(tài)，下面是各個狀態(tài)的幫助文檔

PROCESS STATE CODES
 D uninterruptible sleep (usually IO)
 R running or runnable (on run queue)
 S interruptible sleep (waiting for an event to complete)
 T stopped, either by a job control signal or because it is being traced.
 W paging (not valid since the 2.6.xx kernel)
 X dead (should never be seen)
 Z defunct ("zombie") process, terminated but not reaped by its parent.
 
Processes that are waiting for I/O are commonly in an “uninterruptible sleep” state or “D”; given this information we can simply find the processes that are constantly in a wait state.

那些等待I/O的進程的狀態(tài)一般是“uninterruptible sleep”，或者“D”，我們可以很容易的查找到正在等待I/O的進程

Example:

 # for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done
 D 248 [jbd2/dm-0-8]
 D 16528 bonnie++ -n 0 -u 0 -r 239 -s 478 -f -b -d /tmp
 ----
 D 22 [kswapd0]
 D 16528 bonnie++ -n 0 -u 0 -r 239 -s 478 -f -b -d /tmp
 ----
 D 22 [kswapd0]
 D 16528 bonnie++ -n 0 -u 0 -r 239 -s 478 -f -b -d /tmp
 ----
 D 22 [kswapd0]
 D 16528 bonnie++ -n 0 -u 0 -r 239 -s 478 -f -b -d /tmp
 ----
 D 16528 bonnie++ -n 0 -u 0 -r 239 -s 478 -f -b -d /tmp
 ----
 
The above for loop will print the processes in a “D” state every 5 seconds for 10 intervals.

上邊的例子會循環(huán)的輸出狀態(tài)是D的進程，每5秒一次，一共10次

From the output above the bonnie++ process with a pid of 16528 is waiting for I/O more often than any other process. At this point the bonnie++ seems likely to be causing the I/O Wait, but just because the process is in an uninterruptible sleep state does not necessarily prove that it is the cause of I/O wait.

從輸出我們可以知道 bonnie++ 的pid是16528 ，它比其他的進程更可疑，它老師在waiting，這個時候，bonnie++貌似就是我們想找到的進程，但是，單純從它的狀態(tài)，我們沒有辦法證明是bonnie++引起的I/O問題

To help confirm our suspicions we can use the /proc file system. Within each processes directory there is a file called “io” which holds the same I/O statistics that iotop is utilizing.

為了確認我們的懷疑，我們可以使用 /proc文件系統(tǒng)，每個進程目錄下都有一個叫io的文件，里邊保存這和iotop類似的信息

 # cat /proc/16528/io
 rchar: 48752567
 wchar: 549961789
 syscr: 5967
 syscw: 67138
 read_bytes: 49020928
 write_bytes: 549961728
 cancelled_write_bytes: 0

 
The read_bytes and write_bytes are the number of bytes that this specific process has written and read from the storage layer. In this case the bonnie++ process has read 46 MB and written 524 MB to disk. While for some processes this may not be a lot, in our example this is enough write and reads to cause the high i/o wait that this system is seeing.

read_bytes和write_bytes是這個進程從磁盤讀寫的字節(jié)，在這個例子中，bonnie++進程讀取了46M的數(shù)據(jù)并且寫入了524的數(shù)據(jù)到磁盤上。這樣的數(shù)據(jù)對于其他的進程可能并不是很多，但是在我們例子中，這足夠引發(fā)系統(tǒng)的問題。

Finding what files are being written too heavily

查找那個文件引起的I/Owait

The lsof command will show you all of the files open by a specific process or all processes depending on the options provided. From this list one can make an educated guess as to what files are likely being written to often based on the size of the file and the amounts present in the “io” file within /proc.

lsof 命令可以展示一個進程打開的所有文件，或者打開一個文件的所有進程。從這個列表中，我們可以找到具體是什么文件被寫入，根據(jù)文件的大小和/proc中io文件的具體數(shù)據(jù)

（這段翻譯的有點繞，翻譯好的直接留言好了http://www.）

To narrow down the output we will use the -p <pid> options to print only files open by the specific process id.

我們可以使用-p <pid>的方式來減少輸出，pid是具體的進程

 # lsof -p 16528
 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
 bonnie++ 16528 root cwd DIR 252,0 4096 130597 /tmp
 <truncated>
 bonnie++ 16528 root 8u REG 252,0 501219328 131869 /tmp/Bonnie.16528
 bonnie++ 16528 root 9u REG 252,0 501219328 131869 /tmp/Bonnie.16528
 bonnie++ 16528 root 10u REG 252,0 501219328 131869 /tmp/Bonnie.16528
 bonnie++ 16528 root 11u REG 252,0 501219328 131869 /tmp/Bonnie.16528
 bonnie++ 16528 root 12u REG 252,0 501219328 131869 <strong>/tmp/Bonnie.16528</strong>
 
To even further confirm that these files are being written to the heavily we can see if the /tmp filesystem is part of sda.

為了更深入的確認這些文件被頻繁的讀寫，我們可以通過如下命令來查看

 # df /tmp
 Filesystem 1K-blocks Used Available Use% Mounted on
 /dev/mapper/workstation-root 7667140 2628608 4653920 37% /
 
From the output of df we can determine that /tmp is part of the root logical volume in the workstation volume group.

從上面的命令結(jié)果來看，我們可以確定/tmp 是我們環(huán)境的邏輯磁盤的根目錄

 # pvdisplay
  --- Physical volume ---
  PV Name /dev/sda5
  VG Name workstation
  PV Size 7.76 GiB / not usable 2.00 MiB
  Allocatable yes
  PE Size 4.00 MiB
  Total PE 1986
  Free PE 8
  Allocated PE 1978
  PV UUID CLbABb-GcLB-l5z3-TCj3-IOK3-SQ2p-RDPW5S
 
Using pvdisplay we can see that the /dev/sda5 partition part of the sda disk is the partition that the workstation volume group is using and in turn is where/tmp exists. Given this information it is safe to say that the large files listed in the lsof above are likely the files being read & written to frequently.

通過pvdisplay我們能看到/dev/sda5其實就是我們用來創(chuàng)建邏輯磁盤的具體磁盤。通過以上的信息我們可以放心的說lsof的結(jié)果就是我們要查找的文件