Very slow file operations
Posted: 2015/02/18 17:55:28
I recently rebuild a machine with Centos 5.8 with a brand new disk. This machine is old and has been running 5.8 for a long period with no issues so the only thing which changed recently is the new much larger disk. However all the file operations are extremely slow since the machine was rebuilt with the new disk. My knowledge about OS admin stuff is very limited and I will appreciate if someone can point me to where should I be looking at for finding the root cause. What commands or utilities I could use?
There is a java based back up program running on this machine which is constantly reading from the disk but the same program was running on this machine before the rebuild too and machine has been behaving fine for almost 2 years while this program ran at all times. But now the machine is not very usable and even the back up program is crawling.
This is the output of the iostat command:
avg-cpu: %user %nice %system %iowait %steal %idle
0.67 0.41 1.70 52.66 0.00 44.56
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 24.87 197.21 53.98 238224706 65205070
sda1 0.00 0.00 0.00 2266 14
sda2 24.87 197.21 53.98 238222160 65205056
sdb 91.76 138.66 903.42 167496875 1091305280
sdb1 91.76 138.66 903.42 167496635 1091305280
dm-0 157.98 335.87 957.40 405716746 1156511560
dm-1 0.00 0.00 0.00 1376 216
And here is the output from the iotop command:
8723 be/4 root 0.00 B/s 0.00 B/s 0.00 % 98.49 % [pdflush]
588 be/3 root 0.00 B/s 0.00 B/s 0.00 % 75.71 % [kjournald]
8 be/3 root 0.00 B/s 0.00 B/s 0.00 % 5.19 % [events/0]
13161 be/4 root 0.00 B/s 0.00 B/s 0.00 % 1.25 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~ang:./skin com.backup42.desktop.CPDesktop
10668 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.63 % artsd -F 10 -S 4096 -s 60 -m artsmessage -l 3 -f
16681 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.09 % sshd: root [priv]
3792 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.09 % python -E /usr/bin/sealert -s
13060 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.09 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~hplan/lang com.backup42.service.CPService
13141 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~ang:./skin com.backup42.desktop.CPDesktop
10672 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % artsmessage -i Sound server informational~l continue, using the null output device.
13049 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~hplan/lang com.backup42.service.CPService
13162 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~ang:./skin com.backup42.desktop.CPDeskt
I dont know what to look for here..
Can someone please help?
Thanks
--Abhishek
There is a java based back up program running on this machine which is constantly reading from the disk but the same program was running on this machine before the rebuild too and machine has been behaving fine for almost 2 years while this program ran at all times. But now the machine is not very usable and even the back up program is crawling.
This is the output of the iostat command:
avg-cpu: %user %nice %system %iowait %steal %idle
0.67 0.41 1.70 52.66 0.00 44.56
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 24.87 197.21 53.98 238224706 65205070
sda1 0.00 0.00 0.00 2266 14
sda2 24.87 197.21 53.98 238222160 65205056
sdb 91.76 138.66 903.42 167496875 1091305280
sdb1 91.76 138.66 903.42 167496635 1091305280
dm-0 157.98 335.87 957.40 405716746 1156511560
dm-1 0.00 0.00 0.00 1376 216
And here is the output from the iotop command:
8723 be/4 root 0.00 B/s 0.00 B/s 0.00 % 98.49 % [pdflush]
588 be/3 root 0.00 B/s 0.00 B/s 0.00 % 75.71 % [kjournald]
8 be/3 root 0.00 B/s 0.00 B/s 0.00 % 5.19 % [events/0]
13161 be/4 root 0.00 B/s 0.00 B/s 0.00 % 1.25 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~ang:./skin com.backup42.desktop.CPDesktop
10668 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.63 % artsd -F 10 -S 4096 -s 60 -m artsmessage -l 3 -f
16681 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.09 % sshd: root [priv]
3792 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.09 % python -E /usr/bin/sealert -s
13060 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.09 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~hplan/lang com.backup42.service.CPService
13141 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~ang:./skin com.backup42.desktop.CPDesktop
10672 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % artsmessage -i Sound server informational~l continue, using the null output device.
13049 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~hplan/lang com.backup42.service.CPService
13162 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dfile.encoding=UTF-8 -Dapp=CrashPla~ang:./skin com.backup42.desktop.CPDeskt
I dont know what to look for here..
Can someone please help?
Thanks
--Abhishek