How can I find out what is generating iowait on my server?

A question I’m asked daily is “How can I find out what is generating iowait on my server?” Sure, you can dig through pages of lsof output, restart services, or run strace, but it can be a frustrating process. I saw a process on this blog post, and I changed the regexes to fit Red Hat and CentOS systems a bit better:

# /etc/init.d/syslog stop
# echo 1 > /proc/sys/vm/block_dump
# dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)’ | sort | uniq -c | sort -rn | head
1526 mysqld
819 httpd
429 kjournald
35 qmail
27 in
7 imapd
6 irqbalance
5 pop
4 pdflush
3 spamc

In my specific situation, it looks like MySQL is the biggest abuser of my disk, followed by Apache and the filesystem journaling. As expected, qmail is a large contender, too.

Don’t forget to set things back to their normal state when you’re done!

# echo 0 > /proc/sys/vm/block_dump
# /etc/init.d/syslog start