Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

[KellyLesperance]

We have a cluster of 23 HP DL380p Gen8 hosts running Kafka. Basic specs:

2x E5-2650
128 GB RAM
12 x 4 TB 7200 RPM SATA drives connected to an HP H220 HBA
Dual port 10 GB NIC

The drives are configured as one large RAID-10 volume with mdadm, filesystem is XFS. The OS is not installed on the drive - we PXE boot a CentOS image we've built with minimal packages installed, and do the OS configuration via puppet. Originally, the hosts were running CentOS 6.5, with Kafka 0.8.1, without issue. We recently upgraded to CentOS 7.2 and Kafka 0.9, and that's when the trouble started.

What we're seeing is that when the weekly raid-check script executes, performance nose dives, and I/O wait skyrockets. The raid check starts out fairly fast (20000K/sec - the limit that's been set), but then quickly drops down to about 4000K/Sec. dev.raid.speed sysctls are at the defaults:

Code: Select all

dev.raid.speed_limit_max = 200000
dev.raid.speed_limit_min = 1000

Here's 10 seconds of iostat output, which illustrates the issue:

Code: Select all

[root@r1k1log] # iostat 1 10
Linux 3.10.0-327.18.2.el7.x86_64 (r1k1) 	05/24/16 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.80    0.06    1.89   14.79    0.00   74.46

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              52.59      2033.16     10682.78 1210398902 6359779847
sdb              52.46      2031.25     10682.78 1209265338 6359779847
sdc              52.40      2033.21     10683.53 1210433924 6360229587
sdd              52.22      2031.16     10683.53 1209212513 6360229587
sdf              52.20      2031.17     10682.06 1209216701 6359354331
sdg              52.62      2033.22     10684.17 1210437080 6360606756
sdh              52.57      2031.21     10684.17 1209242746 6360606756
sde              51.67      2033.17     10682.06 1210408935 6359354331
sdj              51.90      2031.13     10684.48 1209191501 6360795559
sdi              52.47      2033.16     10684.48 1210399262 6360795559
sdk              52.09      2033.15     10684.36 1210396915 6360724971
sdl              51.95      2031.20     10684.36 1209235241 6360724971
md127           138.20        74.49     64101.35   44348810 38161468777

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.57    0.09    1.33   26.19    0.00   63.81

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              28.00       512.00      8416.00        512       8416
sdb              28.00       512.00      8416.00        512       8416
sdc              25.00       448.00      8876.00        448       8876
sdd              24.00       448.00      8364.00        448       8364
sdf              23.00       448.00      8192.00        448       8192
sdg              24.00       512.00      7680.00        512       7680
sdh              24.00       512.00      7680.00        512       7680
sde              23.00       448.00      8192.00        448       8192
sdj              23.00       512.00      7680.00        512       7680
sdi              23.00       512.00      7680.00        512       7680
sdk              23.00       512.00      7680.00        512       7680
sdl              23.00       512.00      7680.00        512       7680
md127           101.00         0.00     48012.00          0      48012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.50    0.00    1.04   24.27    0.00   68.19

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              26.00       512.00      9216.00        512       9216
sdb              26.00       512.00      9216.00        512       9216
sdc              27.00       576.00      9204.00        576       9204
sdd              28.00       576.00      9716.00        576       9716
sdf              31.00       768.00      9728.00        768       9728
sdg              28.00       512.00     10240.00        512      10240
sdh              28.00       512.00     10240.00        512      10240
sde              31.00       768.00      9728.00        768       9728
sdj              28.00       512.00      9744.00        512       9744
sdi              28.00       512.00      9744.00        512       9744
sdk              27.00       512.00      9728.00        512       9728
sdl              27.00       512.00      9728.00        512       9728
md127           114.00         0.00     57860.00          0      57860

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.24    0.00    1.32   20.02    0.00   69.42

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              50.00       512.00     20408.00        512      20408
sdb              50.00       512.00     20408.00        512      20408
sdc              48.00       512.00     19984.00        512      19984
sdd              48.00       512.00     19984.00        512      19984
sdf              50.00       704.00     19968.00        704      19968
sdg              47.00       512.00     19968.00        512      19968
sdh              47.00       512.00     19968.00        512      19968
sde              50.00       704.00     19968.00        704      19968
sdj              48.00       512.00     19972.00        512      19972
sdi              48.00       512.00     19972.00        512      19972
sdk              48.00       512.00     19980.00        512      19980
sdl              48.00       512.00     19980.00        512      19980
md127           241.00         0.00    120280.00          0     120280

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.98    0.00    0.98   18.42    0.00   72.63

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              39.00       640.00     14076.00        640      14076
sdb              39.00       640.00     14076.00        640      14076
sdc              36.00       512.00     14324.00        512      14324
sdd              36.00       512.00     14324.00        512      14324
sdf              36.00       576.00     13824.00        576      13824
sdg              43.00      1024.00     13824.00       1024      13824
sdh              43.00      1024.00     13824.00       1024      13824
sde              36.00       576.00     13824.00        576      13824
sdj              44.00      1024.00     14104.00       1024      14104
sdi              44.00      1024.00     14104.00       1024      14104
sdk              45.00      1024.00     14336.00       1024      14336
sdl              45.00      1024.00     14336.00       1024      14336
md127           168.00         0.00     84488.00          0      84488

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.39    0.00    1.01   19.48    0.00   72.13

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              22.00       896.00      4096.00        896       4096
sdb              22.00       896.00      4096.00        896       4096
sdc              19.00       640.00      4344.00        640       4344
sdd              19.00       640.00      4344.00        640       4344
sdf              18.00       512.00      5120.00        512       5120
sdg              18.00       512.00      5120.00        512       5120
sdh              18.00       512.00      5120.00        512       5120
sde              18.00       512.00      5120.00        512       5120
sdj              18.00       512.00      4624.00        512       4624
sdi              18.00       512.00      4624.00        512       4624
sdk              18.00       512.00      4608.00        512       4608
sdl              18.00       512.00      4608.00        512       4608
md127            57.00         0.00     27912.00          0      27912

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.92    0.00    1.58   21.84    0.00   65.66

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              23.00       576.00      7168.00        576       7168
sdb              23.00       576.00      7168.00        576       7168
sdc              29.00       896.00      7680.00        896       7680
sdd              29.00       896.00      7680.00        896       7680
sdf              31.00      1024.00      7680.00       1024       7680
sdg              31.00      1024.00      7680.00       1024       7680
sdh              31.00      1024.00      7680.00       1024       7680
sde              31.00      1024.00      7680.00       1024       7680
sdj              30.00      1024.00      7168.00       1024       7168
sdi              31.00      1024.00      7680.00       1024       7680
sdk              32.00      1024.00      7424.00       1024       7424
sdl              32.00      1024.00      7424.00       1024       7424
md127            89.00         0.00     44800.00          0      44800

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.89    0.03    2.63   21.54    0.00   61.91

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              30.00       960.00      7680.00        960       7680
sdb              30.00       960.00      7680.00        960       7680
sdc              32.00      1024.00      7684.00       1024       7684
sdd              32.00      1024.00      7684.00       1024       7684
sdf              31.00      1024.00      7680.00       1024       7680
sdg              31.00      1024.00      7680.00       1024       7680
sdh              31.00      1024.00      7680.00       1024       7680
sde              31.00      1024.00      7680.00       1024       7680
sdj              32.00      1024.00      8192.00       1024       8192
sdi              31.00      1024.00      7680.00       1024       7680
sdk              26.00       704.00      7680.00        704       7680
sdl              26.00       704.00      7680.00        704       7680
md127            92.00         0.00     46596.00          0      46596

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.24    0.00    2.22   19.89    0.00   63.65

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              33.00      1024.00      7244.00       1024       7244
sdb              33.00      1024.00      7244.00       1024       7244
sdc              31.00      1024.00      7668.00       1024       7668
sdd              31.00      1024.00      7668.00       1024       7668
sdf              31.00      1024.00      7680.00       1024       7680
sdg              26.00       768.00      6672.00        768       6672
sdh              26.00       768.00      6672.00        768       6672
sde              31.00      1024.00      7680.00       1024       7680
sdj              21.00       512.00      6656.00        512       6656
sdi              21.00       512.00      6656.00        512       6656
sdk              27.00       832.00      7168.00        832       7168
sdl              27.00       832.00      7168.00        832       7168
md127            88.00         0.00     43088.00          0      43088

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.02    0.13    1.42   23.90    0.00   66.53

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              30.00      1024.00      7168.00       1024       7168
sdb              30.00      1024.00      7168.00       1024       7168
sdc              29.00       960.00      7168.00        960       7168
sdd              29.00       960.00      7168.00        960       7168
sdf              23.00       512.00      7668.00        512       7668
sdg              28.00       768.00      7680.00        768       7680
sdh              28.00       768.00      7680.00        768       7680
sde              23.00       512.00      7668.00        512       7668
sdj              30.00      1024.00      6672.00       1024       6672
sdi              30.00      1024.00      6672.00       1024       6672
sdk              30.00      1024.00      7168.00       1024       7168
sdl              30.00      1024.00      7168.00       1024       7168
md127            87.00         0.00     43524.00          0      43524

Details of the array:

Code: Select all

[root@r1k1] # cat /proc/mdstat 
Personalities : [raid10] 
md127 : active raid10 sdf[5] sdi[8] sdh[7] sdk[10] sdb[1] sdj[9] sdc[2] sdd[3] sdl[11] sde[13] sdg[12] sda[0]
      23441323008 blocks super 1.2 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
      [======>..............]  check = 30.8% (7237496960/23441323008) finish=62944.5min speed=4290K/sec
      
unused devices: <none>
[root@r1k1] # mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Thu Sep 18 09:57:57 2014
     Raid Level : raid10
     Array Size : 23441323008 (22355.39 GiB 24003.91 GB)
  Used Dev Size : 3906887168 (3725.90 GiB 4000.65 GB)
   Raid Devices : 12
  Total Devices : 12
    Persistence : Superblock is persistent

    Update Time : Tue May 24 15:32:56 2016
          State : active, checking 
 Active Devices : 12
Working Devices : 12
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

   Check Status : 30% complete

           Name : localhost:kafka
           UUID : b6b98e3e:65ee06c3:3599d781:98908041
         Events : 2459193

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync set-A   /dev/sda
       1       8       16        1      active sync set-B   /dev/sdb
       2       8       32        2      active sync set-A   /dev/sdc
       3       8       48        3      active sync set-B   /dev/sdd
      13       8       64        4      active sync set-A   /dev/sde
       5       8       80        5      active sync set-B   /dev/sdf
      12       8       96        6      active sync set-A   /dev/sdg
       7       8      112        7      active sync set-B   /dev/sdh
       8       8      128        8      active sync set-A   /dev/sdi
       9       8      144        9      active sync set-B   /dev/sdj
      10       8      160       10      active sync set-A   /dev/sdk
      11       8      176       11      active sync set-B   /dev/sdl

We've tried changing the I/O scheduler, queue_depth, queue_type, read-ahead, etc, but nothing has helped. We've also upgraded all of the firmware, and installed HP's mpt2sas driver.

We have 4 other Kafka clusters, however they're HP DL180 G6 servers. We completed the same CentOS 6.5 -> 7.2/Kafka 0.8 -> 0.9 upgrade on those clusters, and there has been no impact to their performance.

We've been banging our heads against the wall for a few weeks now, really hoping someone from the community can point us in the right direction.

[aks]

Sounds like the issue is with the raid-check script - is it a custom script?

[KellyLesperance]

No, it's the CentOS-provided /usr/sbin/raid-check script. By default, it's scheduled to run every Sunday at 1:00.

[aks]

From memory that's just a bash script, so isolating the where the error is could just be a question of commenting out parts running and checking. One thing that does come to mind could be the MAXCONCURRENT variable. If you set this to 1, RAID devices would be locked sequentially - perhaps the bus is being flooded?
But like I say, this is from memory.

[KellyLesperance]

We've only 1 array on the system, so MAXCONCURRENT isn't going to change anything, unfortunately.

Essentially, the script just does "echo check > /sys/block/md###/md/sync_action", and the md subsystem does it's thing.

[aks]

I think check will read all blocks but doesn't trigger any writes unless a read error is detected (source: https://raid.wiki.kernel.org/index.php/ ... OMMANDS.3F)

So it looks like check is triggering writes (based on the writes stats in your iostat post above). Are you 100% sure that all 12 of your disks are okay?

[KellyLesperance]

I am confident that the disks are good - we're talking 23 systems all showing the same symptoms.

thanks,

Kelly

[aks]

Okay, how many GB/s are read and how many are written? If we have large (well any positive integer really) then the check "thinks" there's something wrong.
Also, (I can't remember if we checked this), are the RAID arrays good BEFORE the check (i.e.: they're not sync'ing or something like that)?

[KellyLesperance]

So, for the most part, we're doing writes - reads typically get pulled from cache, rather than hitting the disk.
The array on each host is in a good state prior to the check kicking off.

Here's an iostat sample with the check running:

Code: Select all

[root@r1k1 ~] # iostat -xdmc 1 10
Linux 3.10.0-327.18.2.el7.x86_64 (r1k1) 	05/31/16 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.66    0.06    2.22   12.07    0.00   73.99

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               1.87     0.20   29.67   23.05     1.98    10.41   481.21     2.46   46.75    2.09  104.23   4.45  23.47
sdk               2.17     0.23   29.37   23.15     1.98    10.41   483.08     2.45   46.65    2.26  102.98   4.49  23.59
sdc               1.93     0.17   29.61   22.98     1.98    10.41   482.40     2.48   47.25    2.14  105.36   4.50  23.65
sdd               1.96     0.17   29.55   22.98     1.98    10.41   482.93     2.60   49.49    2.22  110.28   4.59  24.13
sda               1.87     0.18   29.67   23.14     1.98    10.41   480.30     2.44   46.25    2.04  102.96   4.45  23.50
sdj               2.28     0.20   29.23   23.05     1.98    10.41   485.27     2.54   48.58    2.38  107.17   4.62  24.17
sdl               2.25     0.23   29.26   23.15     1.98    10.41   484.16     2.49   47.50    2.33  104.59   4.54  23.78
sdh               1.91     0.23   29.60   23.34     1.98    10.41   479.23     2.49   46.97    2.14  103.81   4.45  23.53
sde               2.12     0.16   29.43   22.72     1.98    10.41   486.37     2.65   50.74    2.47  113.24   4.72  24.61
sdf               1.89     0.16   29.63   22.72     1.98    10.41   484.54     2.42   46.27    2.06  103.92   4.45  23.28
sdg               1.94     0.23   29.60   23.34     1.98    10.41   479.16     2.57   48.60    2.18  107.47   4.54  24.02
sdb               1.88     0.18   29.63   23.14     1.98    10.41   480.70     2.51   47.50    2.08  105.66   4.51  23.81
md127             0.00     0.00    0.48  138.93     0.10    62.46   919.03     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.93    0.00    1.30   14.86    0.00   73.92

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    7.00   10.00     0.44     5.00   655.06     0.19   10.94    0.43   18.30   8.35  14.20
sdk               0.00     0.00    8.00   10.00     0.50     5.00   625.78     0.21   11.67    0.25   20.80   8.44  15.20
sdc               0.00     0.00    8.00   14.00     0.50     6.06   610.91     0.32   14.68    0.50   22.79   9.82  21.60
sdd               0.00     0.00    8.00   14.00     0.50     6.06   610.91     0.33   14.86    0.50   23.07  10.45  23.00
sda               0.00     0.00    8.00   12.00     0.50     6.00   665.60     0.22   11.35    0.25   18.75   8.90  17.80
sdj               0.00     0.00    7.00   10.00     0.44     5.00   655.06     0.20   12.00    0.43   20.10   8.53  14.50
sdl               0.00     0.00    8.00   10.00     0.50     5.00   625.78     0.22   12.11    0.25   21.60   8.72  15.70
sdh               0.00     0.00    0.00   11.00     0.00     5.02   933.82     0.24   20.45    0.00   20.45  16.18  17.80
sde               0.00     0.00    1.00   12.00     0.06     5.53   881.23     0.25   17.69    0.00   19.17  13.69  17.80
sdf               0.00     0.00    1.00   12.00     0.06     5.53   881.23     0.27   19.31    0.00   20.92  14.62  19.00
sdg               0.00     0.00    0.00   11.00     0.00     5.02   933.82     0.24   20.18    0.00   20.18  16.18  17.80
sdb               0.00     0.00    8.00   12.00     0.50     6.00   665.60     0.23   11.65    0.25   19.25   8.50  17.00
md127             0.00     0.00    0.00   69.00     0.00    32.61   967.88     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.80    0.03    1.01   17.95    0.00   71.21

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    8.00   16.00     0.50     7.54   685.67     0.29   11.46    0.25   17.06  10.42  25.00
sdk               0.00     0.00    8.00   15.00     0.50     7.50   712.35     0.23    9.83    0.38   14.87   8.39  19.30
sdc               0.00     0.00    1.00   15.00     0.06     7.50   968.00     0.27   16.75    0.00   17.87  14.31  22.90
sdd               0.00     0.00    1.00   15.00     0.06     7.50   968.00     0.27   16.62    0.00   17.73  14.37  23.00
sda               0.00     0.00    8.00   15.00     0.50     7.50   712.35     0.22    9.70    0.38   14.67   9.22  21.20
sdj               0.00     0.00    8.00   16.00     0.50     7.54   685.67     0.29   11.62    0.25   17.31  10.83  26.00
sdl               0.00     0.00    8.00   15.00     0.50     7.50   712.35     0.23   10.09    0.50   15.20   8.74  20.10
sdh               0.00     0.00    8.00   17.00     0.50     8.50   737.28     0.28   11.68    0.25   17.06   9.40  23.50
sde               0.00     0.00    7.00   16.00     0.44     7.97   748.52     0.37   16.09    8.43   19.44  12.61  29.00
sdf               0.00     0.00    7.00   17.00     0.44     8.47   760.00     0.36   15.58    8.14   18.65  12.00  28.80
sdg               0.00     0.00    8.00   17.00     0.50     8.50   737.28     0.29   12.20    0.25   17.82  10.00  25.00
sdb               0.00     0.00    8.00   15.00     0.50     7.50   712.35     0.21    9.26    0.38   14.00   8.65  19.90
md127             0.00     0.00    0.00   94.00     0.00    46.50  1013.19     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.78    0.03    0.75   18.65    0.00   75.79

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    8.00   12.00     0.50     6.00   665.60     0.26   13.90    0.25   23.00  13.20  26.40
sdk               0.00     0.00    3.00   12.00     0.19     6.00   844.80     0.18   12.13    0.67   15.00  12.13  18.20
sdc               0.00     0.00    7.00   12.00     0.44     5.97   690.53     0.18    9.42    0.29   14.75   8.95  17.00
sdd               0.00     0.00    7.00   12.00     0.44     5.97   690.53     0.20   10.63    0.29   16.67  10.26  19.50
sda               0.00     0.00    0.00   12.00     0.00     6.00  1024.00     0.19   15.67    0.00   15.67  15.67  18.80
sdj               0.00     0.00    8.00   12.00     0.50     6.00   665.60     0.19   10.35    0.25   17.08   9.65  19.30
sdl               0.00     0.00    3.00   12.00     0.19     6.00   844.80     0.18   12.13    0.67   15.00  12.13  18.20
sdh               0.00     0.00    8.00   11.00     0.50     5.50   646.74     0.17    8.95    0.38   15.18   8.42  16.00
sde               0.00     0.00    8.00   12.00     0.50     5.54   618.00     0.17    9.35    0.50   15.25   8.25  16.50
sdf               0.00     0.00    8.00   11.00     0.50     5.04   596.63     0.17    9.05    0.62   15.18   8.42  16.00
sdg               0.00     0.00    8.00   11.00     0.50     5.50   646.74     0.15    8.00    0.50   13.45   7.79  14.80
sdb               0.00     0.00    0.00   12.00     0.00     6.00  1024.00     0.39   32.25    0.00   32.25  32.25  38.70
md127             0.00     0.00    0.00   71.00     0.00    35.00  1009.69     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.31    0.09    2.21   16.13    0.00   69.25

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    8.00   16.00     0.50     8.00   725.33     0.27   11.12    0.38   16.50  10.62  25.50
sdk               0.00     0.00    7.00   16.00     0.44     8.00   751.30     0.26   11.48    0.00   16.50  10.04  23.10
sdc               0.00     0.00    8.00   17.00     0.50     8.04   699.20     0.28   11.28    0.38   16.41  10.20  25.50
sdd               0.00     0.00    8.00   17.00     0.50     8.04   699.20     0.26   10.44    0.38   15.18   9.96  24.90
sda               0.00     0.00    8.00   16.00     0.50     8.00   725.33     0.26   11.00    0.62   16.19  10.17  24.40
sdj               0.00     0.00    8.00   16.00     0.50     8.00   725.33     0.25   10.21    0.38   15.12   9.67  23.20
sdl               0.00     0.00    7.00   16.00     0.44     8.00   751.30     0.25   10.91    0.14   15.62  10.00  23.00
sdh               0.00     0.00    8.00   16.00     0.50     7.97   723.00     0.25   10.50    0.25   15.62   9.92  23.80
sde               0.00     0.00    8.00   15.00     0.50     7.50   712.35     0.25   10.30    0.50   15.53  10.65  24.50
sdf               0.00     0.00    8.00   15.00     0.50     7.50   712.35     0.24    9.70    0.50   14.60  10.00  23.00
sdg               0.00     0.00    8.00   16.00     0.50     7.97   723.00     0.23    9.62    0.25   14.31   9.12  21.90
sdb               0.00     0.00    8.00   16.00     0.50     8.00   725.33     0.26   10.96    0.62   16.12  10.33  24.80
md127             0.00     0.00    0.00   96.00     0.00    47.51  1013.50     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.49    0.03    3.01   15.63    0.00   63.84

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00   15.00     0.00     7.50  1024.00     0.35   23.53    0.00   23.53  10.73  16.10
sdk               0.00     0.00    6.00   16.00     0.38     7.50   733.09     0.38   17.32    0.67   23.56   9.36  20.60
sdc               0.00     0.00    8.00   13.00     0.50     6.50   682.67     0.22   10.43    0.38   16.62   6.05  12.70
sdd               0.00     0.00    8.00   13.00     0.50     6.50   682.67     0.28   13.14    0.38   21.00   6.90  14.50
sda               0.00     0.00    8.00   13.00     0.50     6.50   682.67     0.28   11.38    0.25   18.23   7.62  16.00
sdj               0.00     0.00    0.00   15.00     0.00     7.50  1024.00     1.15   76.73    0.00   76.73  18.87  28.30
sdl               0.00     0.00    6.00   16.00     0.38     7.50   733.09     0.31   14.18    0.67   19.25   8.27  18.20
sdh               0.00     0.00    0.00   15.00     0.00     7.50  1024.00     0.29   19.20    0.00   19.20  11.33  17.00
sde               0.00     0.00    6.00   15.00     0.38     7.50   768.00     0.34   16.76    0.33   23.33   9.19  19.30
sdf               0.00     0.00    6.00   15.00     0.38     7.50   768.00     0.29   14.67    0.33   20.40   8.10  17.00
sdg               0.00     0.00    0.00   15.00     0.00     7.50  1024.00     0.30   19.87    0.00   19.87  11.40  17.10
sdb               0.00     0.00    8.00   14.00     0.50     7.00   698.18     0.27   11.55    0.25   18.00   7.27  16.00
md127             0.00     0.00    0.00   87.00     0.00    43.00  1012.23     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.90    0.00    2.05   24.34    0.00   58.70

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    8.00   14.00     0.50     6.52   653.09     0.22    9.86    0.62   15.14   8.82  19.40
sdk               0.00     0.00    8.00   14.00     0.50     6.97   695.64     0.22   10.05    0.50   15.50   9.59  21.10
sdc               0.00     0.00    4.00   15.00     0.25     7.03   784.84     0.22   11.37    0.00   14.40   9.79  18.60
sdd               0.00     0.00    4.00   15.00     0.25     7.03   784.84     0.23   12.26    0.00   15.53  10.00  19.00
sda               0.00     0.00    8.00   15.00     0.50     7.50   712.35     0.22   11.09    0.50   16.73   8.78  20.20
sdj               0.00     0.00    8.00   14.00     0.50     6.52   653.09     0.31   14.18    0.62   21.93  13.73  30.20
sdl               0.00     0.00    8.00   14.00     0.50     6.97   695.64     0.21    9.59    0.50   14.79   8.95  19.70
sdh               0.00     0.00    8.00   14.00     0.50     7.00   698.18     0.21    9.50    0.00   14.93   8.36  18.40
sde               0.00     0.00    2.00   16.00     0.12     7.06   817.78     0.23   12.72    0.50   14.25  11.44  20.60
sdf               0.00     0.00    2.00   16.00     0.12     7.06   817.78     0.25   13.89    0.50   15.56  12.33  22.20
sdg               0.00     0.00    8.00   13.00     0.50     6.50   682.67     0.28    9.38    0.00   15.15  12.33  25.90
sdb               0.00     0.00    8.00   14.00     0.50     7.00   698.18     0.17    8.68    0.50   13.36   7.32  16.10
md127             0.00     0.00    0.00   86.00     0.00    41.08   978.33     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.55    0.00    0.75   33.29    0.00   57.40

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    8.00   12.00     0.50     5.50   614.40     0.25    9.90    0.38   16.25   9.25  18.50
sdk               0.00     0.00    8.00   11.00     0.50     5.50   646.74     0.24   12.58    0.12   21.64   9.53  18.10
sdc               0.00     0.00    4.00   12.00     0.25     6.00   800.00     0.23   14.12    0.50   18.67  11.00  17.60
sdd               0.00     0.00    4.00   12.00     0.25     6.00   800.00     0.19   11.69    0.50   15.42   9.25  14.80
sda               0.00     0.00    1.00   12.00     0.06     6.00   955.08     0.20   15.23    0.00   16.50  12.15  15.80
sdj               0.00     0.00    8.00   13.00     0.50     6.00   633.90     0.25   10.67    0.38   17.00   9.10  19.10
sdl               0.00     0.00    8.00   11.00     0.50     5.50   646.74     0.19    9.95    0.12   17.09   8.00  15.20
sdh               0.00     0.00    8.00   13.00     0.50     6.03   636.95     0.22   10.48    0.25   16.77   8.00  16.80
sde               0.00     0.00    8.00   13.00     0.50     6.40   672.76     0.30   14.48    7.75   18.62  11.48  24.10
sdf               0.00     0.00    8.00   13.00     0.50     6.40   672.76     0.30   14.19    7.50   18.31  11.48  24.10
sdg               0.00     0.00    8.00   14.00     0.50     6.53   654.55     0.27   15.91    0.25   24.86   9.45  20.80
sdb               0.00     0.00    1.00   12.00     0.06     6.00   955.08     0.20   15.23    0.00   16.50  12.62  16.40
md127             0.00     0.00    0.00   75.00     0.00    36.43   994.77     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.53    0.03    0.38   41.01    0.00   51.05

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    1.00   12.00     0.06     5.61   892.92     0.44   38.00    1.00   41.08  26.54  34.50
sdk               0.00     0.00    0.00   11.00     0.00     5.50  1024.00     0.21   19.18    0.00   19.18  17.45  19.20
sdc               0.00     0.00    8.00   11.00     0.50     5.50   646.74     0.22   11.53    0.25   19.73  10.95  20.80
sdd               0.00     0.00    8.00   11.00     0.50     5.50   646.74     0.17    8.95    0.25   15.27   8.47  16.10
sda               0.00     0.00    7.00   11.00     0.44     5.47   672.44     0.17    9.28    0.57   14.82   8.72  15.70
sdj               0.00     0.00    1.00   11.00     0.06     5.11   882.00     0.14   14.17    1.00   15.36  12.08  14.50
sdl               0.00     0.00    0.00   11.00     0.00     5.50  1024.00     0.19   17.09    0.00   17.09  15.91  17.50
sdh               0.00     0.00    8.00   10.00     0.50     5.00   625.78     0.20   10.94    0.25   19.50  10.00  18.00
sde               0.00     0.00    8.00   10.00     0.50     5.00   625.78     0.24   13.06    0.25   23.30  12.44  22.40
sdf               0.00     0.00    8.00   10.00     0.50     5.00   625.78     0.17    9.39    0.25   16.70   8.94  16.10
sdg               0.00     0.00    8.00   10.00     0.50     5.00   625.78     0.19   10.67    0.25   19.00   9.83  17.70
sdb               0.00     0.00    7.00   11.00     0.44     5.47   672.44     0.22   11.78    0.57   18.91  10.83  19.50
md127             0.00     0.00    0.00   65.00     0.00    32.08  1010.71     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.71    0.03    0.82   42.61    0.00   49.83

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00    19.00    7.00   56.00     0.44    15.06   503.75     2.51   39.81    0.14   44.77   5.56  35.00
sdk               0.00    10.00   10.00   64.00     0.52    14.80   423.89     2.34   31.66    1.60   36.36   5.68  42.00
sdc               0.00     3.00    7.00   59.00     0.44    14.21   454.55     2.28   34.03    0.29   38.03   6.92  45.70
sdd               0.00     3.00    7.00   60.00     0.44    14.71   463.04     2.23   33.33    0.29   37.18   6.01  40.30
sda               0.00     4.00    8.00   57.00     0.50    15.18   494.06     1.67   25.78    0.38   29.35   6.74  43.80
sdj               0.00    19.00    7.00   56.00     0.44    15.06   503.75     2.72   43.21    0.14   48.59   6.10  38.40
sdl               0.00    10.00    8.00   64.00     0.50    14.80   435.22     2.54   35.35    0.62   39.69   5.43  39.10
sdh               0.00     6.00    0.00   61.00     0.00    15.24   511.75     1.69   27.64    0.00   27.64   5.92  36.10
sde               0.00     1.00    0.00   46.00     0.00    14.92   664.17     3.16   68.87    0.00   68.87  11.02  50.70
sdf               0.00     1.00    0.00   46.00     0.00    14.92   664.17     1.56   33.93    0.00   33.93   8.00  36.80
sdg               0.00     6.00    3.00   61.00     0.02    15.24   488.52     2.92   45.64    9.33   47.43   7.14  45.70
sdb               0.00     4.00    8.00   57.00     0.50    15.18   494.06     1.90   29.23    0.38   33.28   7.22  46.90
md127             0.00     0.00    5.00  385.00     0.04    88.91   467.11     0.00    0.00    0.00    0.00   0.00   0.00

And here's iostat without:

Code: Select all

[root@r1k1 ~] # iostat -xdmc 1 10
Linux 3.10.0-327.18.2.el7.x86_64 (r1k1) 	05/31/16 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.66    0.06    2.22   12.07    0.00   73.99

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               1.87     0.20   29.66   23.06     1.98    10.42   481.40     2.47   46.86    2.09  104.45   4.45  23.48
sdk               2.17     0.23   29.37   23.16     1.98    10.42   483.27     2.46   46.77    2.26  103.20   4.49  23.60
sdc               1.93     0.17   29.60   22.99     1.98    10.42   482.59     2.49   47.39    2.14  105.64   4.50  23.65
sdd               1.96     0.17   29.54   22.99     1.98    10.42   483.13     2.61   49.62    2.22  110.53   4.59  24.14
sda               1.87     0.18   29.67   23.15     1.98    10.42   480.49     2.45   46.36    2.04  103.15   4.45  23.50
sdj               2.28     0.20   29.22   23.06     1.98    10.42   485.46     2.55   48.74    2.38  107.47   4.62  24.18
sdl               2.25     0.23   29.25   23.16     1.98    10.42   484.35     2.50   47.61    2.33  104.79   4.54  23.79
sdh               1.91     0.23   29.59   23.36     1.98    10.42   479.42     2.49   47.11    2.14  104.09   4.45  23.54
sde               2.12     0.16   29.42   22.74     1.98    10.42   486.57     2.65   50.90    2.47  113.56   4.72  24.61
sdf               1.89     0.16   29.62   22.74     1.98    10.42   484.73     2.43   46.38    2.06  104.12   4.45  23.28
sdg               1.94     0.23   29.60   23.36     1.98    10.42   479.34     2.58   48.77    2.18  107.82   4.54  24.03
sdb               1.88     0.18   29.63   23.15     1.98    10.42   480.89     2.51   47.61    2.08  105.88   4.51  23.82
md127             0.00     0.00    0.48  139.02     0.10    62.51   919.08     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.09    0.00    2.57    0.10    0.00   85.25

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00    4.00     0.00     0.04    18.50     0.05   13.25    0.00   13.25  13.25   5.30
sdk               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   14.00    0.00   14.00  14.00   2.80
sdc               0.00     0.00    0.00    3.00     0.00     0.39   264.67     0.04   13.33    0.00   13.33  13.33   4.00
sdd               0.00     0.00    0.00    3.00     0.00     0.39   264.67     0.05   18.00    0.00   18.00  18.00   5.40
sda               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.02    8.50    0.00    8.50   8.50   1.70
sdj               0.00     0.00    0.00    4.00     0.00     0.04    18.50     0.06   15.75    0.00   15.75  15.75   6.30
sdl               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.02   11.50    0.00   11.50  11.50   2.30
sdh               0.00     0.00    0.00    4.00     0.00     0.49   248.50     0.05   12.00    0.00   12.00  12.00   4.80
sde               0.00     0.00    0.00    3.00     0.00     0.50   342.00     0.05   15.33    0.00   15.33  15.33   4.60
sdf               0.00     0.00    0.00    3.00     0.00     0.50   342.00     0.04   11.67    0.00   11.67  11.67   3.50
sdg               0.00     0.00    0.00    4.00     0.00     0.49   248.50     0.06   14.50    0.00   14.50  14.50   5.80
sdb               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   13.00    0.00   13.00  13.00   2.60
md127             0.00     0.00    0.00    7.00     0.00     1.41   411.43     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.23    0.03    2.98    0.00    0.00   79.75

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdk               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdj               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdl               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdh               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sde               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdf               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdg               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md127             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.63    0.00    3.27    0.00    0.00   80.10

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00    3.00     0.00     0.03    22.00     0.04   13.67    0.00   13.67  13.67   4.10
sdk               0.00     0.00    0.00    3.00     0.00     0.02    11.33     0.04   12.33    0.00   12.33  12.33   3.70
sdc               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.02    9.50    0.00    9.50   9.50   1.90
sdd               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   12.50    0.00   12.50  12.50   2.50
sda               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   15.00    0.00   15.00  15.00   3.00
sdj               0.00     0.00    0.00    3.00     0.00     0.03    22.00     0.06   18.67    0.00   18.67  18.67   5.60
sdl               0.00     0.00    0.00    3.00     0.00     0.02    11.33     0.03    8.33    0.00    8.33   8.33   2.50
sdh               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   13.50    0.00   13.50  13.50   2.70
sde               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   13.50    0.00   13.50  13.50   2.70
sdf               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   13.00    0.00   13.00  13.00   2.60
sdg               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.02   12.00    0.00   12.00  12.00   2.40
sdb               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.02   11.00    0.00   11.00  11.00   2.20
md127             0.00     0.00    0.00    2.00     0.00     0.05    48.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.39    0.03    3.07    0.73    0.00   80.78

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     1.00    0.00   41.00     0.00    17.91   894.85     2.91   39.46    0.00   39.46   5.46  22.40
sdk               0.00     0.00    0.00   42.00     0.00    20.50   999.64     2.76   39.67    0.00   39.67   4.02  16.90
sdc               0.00     2.00    0.00   40.00     0.00    18.61   953.02     2.80   39.25    0.00   39.25   4.42  17.70
sdd               0.00     2.00    0.00   37.00     0.00    17.11   947.27     3.03   46.03    0.00   46.03   5.00  18.50
sda               0.00     0.00    0.00   42.00     0.00    19.94   972.21     3.78   62.50    0.00   62.50   6.81  28.60
sdj               0.00     1.00    0.00   41.00     0.00    17.91   894.85     2.92   39.73    0.00   39.73   5.46  22.40
sdl               0.00     0.00    0.00   42.00     0.00    20.50   999.64     2.75   39.38    0.00   39.38   4.07  17.10
sdh               0.00     0.00    0.00   42.00     0.00    18.96   924.60     3.07   43.90    0.00   43.90   4.90  20.60
sde               0.00     0.00    0.00   40.00     0.00    19.50   998.23     2.98   42.80    0.00   42.80   4.47  17.90
sdf               0.00     0.00    0.00   36.00     0.00    17.50   995.36     3.13   48.00    0.00   48.00   4.92  17.70
sdg               0.00     0.00    0.00   41.00     0.00    18.46   922.17     3.01   42.73    0.00   42.73   4.90  20.10
sdb               0.00     0.00    0.00   45.00     0.00    21.44   975.67     2.69   36.29    0.00   36.29   4.18  18.80
md127             0.00     0.00    0.00  484.00     0.00   232.36   983.21     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.37    0.13    2.98    0.00    0.00   81.53

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00   43.00     0.00    21.00  1000.21     4.58  136.56    0.00  136.56   5.49  23.60
sdk               0.00     0.00    0.00   37.00     0.00    18.00   996.35     4.40  148.46    0.00  148.46   8.95  33.10
sdc               0.00     0.00    0.00   41.00     0.00    20.00   999.05     4.39  137.17    0.00  137.17   6.02  24.70
sdd               0.00     0.00    0.00   44.00     0.00    21.50  1000.75     5.32  151.09    0.00  151.09   5.30  23.30
sda               0.00     0.00    0.00   39.00     0.00    18.75   984.85     3.81  127.41    0.00  127.41   5.74  22.40
sdj               0.00     0.00    0.00   43.00     0.00    21.00  1000.21     4.90  144.07    0.00  144.07   5.56  23.90
sdl               0.00     0.00    0.00   37.00     0.00    18.00   996.35     3.75  130.84    0.00  130.84   9.43  34.90
sdh               0.00     0.00    0.00   41.00     0.00    20.00   999.05     6.01  176.54    0.00  176.54   6.90  28.30
sde               0.00     0.00    0.00   43.00     0.00    19.18   913.70     6.13  172.12    0.00  172.12   6.35  27.30
sdf               0.00     0.00    0.00   47.00     0.00    21.18   923.09     5.18  140.11    0.00  140.11   5.09  23.90
sdg               0.00     0.00    0.00   42.00     0.00    20.50   999.64     6.05  174.10    0.00  174.10   6.69  28.10
sdb               0.00     0.00    0.00   36.00     0.00    17.25   981.58     3.41  124.14    0.00  124.14   6.06  21.80
md127             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.26    0.03    4.51    0.35    0.00   76.85

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00   34.00     0.00    14.85   894.41     1.65   48.65    0.00   48.65   4.03  13.70
sdk               0.00     1.00    0.00   31.00     0.00    14.50   958.00     1.80   58.13    0.00   58.13   3.90  12.10
sdc               0.00     0.00    0.00   31.00     0.00    14.42   952.84     1.49   48.03    0.00   48.03   3.58  11.10
sdd               0.00     0.00    0.00   31.00     0.00    14.42   952.84     1.63   52.61    0.00   52.61   6.16  19.10
sda               0.00     0.00    0.00   30.00     0.00    14.00   955.80     1.60   53.30    0.00   53.30   4.27  12.80
sdj               0.00     0.00    0.00   34.00     0.00    14.85   894.41     1.71   50.44    0.00   50.44   4.62  15.70
sdl               0.00     1.00    0.00   31.00     0.00    14.50   958.00     1.79   57.84    0.00   57.84   4.00  12.40
sdh               0.00     0.00    0.00   31.00     0.00    14.50   958.00     1.80   58.19    0.00   58.19   3.97  12.30
sde               0.00     0.00    0.00   32.00     0.00    14.50   928.31     1.83   57.16    0.00   57.16   4.59  14.70
sdf               0.00     0.00    0.00   32.00     0.00    14.50   928.31     1.69   52.91    0.00   52.91   5.16  16.50
sdg               0.00     0.00    0.00   31.00     0.00    14.50   958.00     1.74   56.26    0.00   56.26   6.23  19.30
sdb               0.00     0.00    0.00   30.00     0.00    14.00   955.80     1.55   51.83    0.00   51.83   4.17  12.50
md127             0.00     0.00    0.00  179.00     0.00    86.77   992.80     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.20    0.00    3.59    0.00    0.00   80.22

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdk               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdj               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdl               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdh               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sde               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdf               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdg               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md127             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.65    0.03    2.15    0.28    0.00   85.88

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     1.00    0.00   34.00     0.00    15.03   905.50     2.19   58.12    0.00   58.12   5.15  17.50
sdk               0.00     0.00    0.00   32.00     0.00    14.02   897.09     2.24   59.97    0.00   59.97   4.25  13.60
sdc               0.00     0.00    0.00   31.00     0.00    14.00   925.00     2.20   57.35    0.00   57.35   4.52  14.00
sdd               0.00     0.00    0.00   34.00     0.00    15.50   933.74     2.01   56.03    0.00   56.03   4.38  14.90
sda               0.00     0.00    0.00   33.00     0.00    14.94   927.36     1.95   52.73    0.00   52.73   4.64  15.30
sdj               0.00     1.00    0.00   35.00     0.00    15.53   908.89     2.14   58.03    0.00   58.03   4.60  16.10
sdl               0.00     0.00    0.00   35.00     0.00    15.52   907.97     1.98   56.54    0.00   56.54   4.37  15.30
sdh               0.00     0.00    0.00   35.00     0.00    16.00   936.31     1.94   55.46    0.00   55.46   4.69  16.40
sde               0.00     0.00    0.00   35.00     0.00    16.00   936.31     2.12   57.46    0.00   57.46   4.51  15.80
sdf               0.00     0.00    0.00   35.00     0.00    16.00   936.31     2.01   54.34    0.00   54.34   4.29  15.00
sdg               0.00     0.00    0.00   32.00     0.00    14.50   928.09     2.19   58.34    0.00   58.34   4.53  14.50
sdb               0.00     0.00    0.00   35.00     0.00    15.94   932.89     1.96   55.86    0.00   55.86   4.60  16.10
md127             0.00     0.00    0.00  195.00     0.00    95.77  1005.83     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.83    0.00    1.65    0.06    0.00   89.46

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00    6.00     0.00     1.06   363.00     0.05   44.17    0.00   44.17   8.67   5.20
sdk               0.00     0.00    0.00    4.00     0.00     1.50   768.25     0.04   87.25    0.00   87.25   9.25   3.70
sdc               0.00     0.00    0.00    5.00     0.00     2.00   819.40     0.05   91.80    0.00   91.80   6.60   3.30
sdd               0.00     0.00    0.00    3.00     0.00     0.50   342.00     0.03   43.67    0.00   43.67   8.33   2.50
sda               0.00     0.00    0.00    4.00     0.00     1.00   512.50     0.04   61.75    0.00   61.75   8.25   3.30
sdj               0.00     0.00    0.00    5.00     0.00     0.56   230.80     0.04   30.00    0.00   30.00   8.80   4.40
sdl               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.02    8.00    0.00    8.00   8.00   1.60
sdh               0.00     0.00    0.00    3.00     0.00     0.00     3.33     0.04   14.33    0.00   14.33  14.33   4.30
sde               0.00     0.00    0.00    3.00     0.00     0.28   192.67     0.03   45.00    0.00   45.00   9.67   2.90
sdf               0.00     0.00    0.00    3.00     0.00     0.28   192.67     0.03   45.33    0.00   45.33  10.33   3.10
sdg               0.00     0.00    0.00    5.00     0.00     1.50   616.20     0.05   70.60    0.00   70.60   8.20   4.10
sdb               0.00     0.00    0.00    2.00     0.00     0.00     1.00     0.03   14.00    0.00   14.00  14.00   2.80
md127             0.00     0.00    0.00    4.00     0.00     0.07    34.00     0.00    0.00    0.00    0.00   0.00   0.00

Writes per second for each device doubled when I idled the raid check. Reads per second went from an average of 7 down to 0, so obviously the RAID check is the only thing actually reading data from the devices.

Thanks!

Kelly

[aks]

reads typically get pulled from cache

Only if:
1. The read is in the cache.
2. The chace is larger than the amount of data you're reading (which would kind of defeat the point of having disks).
3. You're not issuing a "bypass chace and go right to disk" call.

The average size (in sectors) of the requests that were issued to the device goes (persumably to the max) and stays there when check is running and goes up and down when it's not.

Anyway, further analysis is left up to you.

I'm going to try and attach pictures of the RAID stats with and with the check to show the constant max and the spikes.

I'll also attach a libre office spreadsheet with all the stats, grouped by devices and charted (you just change the data ranges per chart for the device you want to look at).

Okay none of that worked. I have a tarball with all the data, where can I upload it? Alternatively, you can do it yourself....