RAID 5 - unable to mount xfs volume

[elkingsparx]

8TB x 12 drive RAID on Dell R510

Noticed the following error on boot.

Code: Select all

[Thu May  4 11:08:58 2023] megaraid_sas 0000:02:00.0: 155182 (736502926s/0x0001/FATAL) - VD bad block table on VD 05/c is full; unable to log block dcf73d10 (on PD 04(e0x20/s4) at dcf73d10)
[Thu May  4 11:09:02 2023]  sdf: sdf1
[Thu May  4 11:09:02 2023] sd 0:2:5:0: [sdf] Attached SCSI disk

Code: Select all

[Thu May  4 11:10:09 2023] megaraid_sas 0000:02:00.0: 155184 (736502998s/0x0001/FATAL) - VD bad block table on VD 05/c is full; unable to log block dde08958 (on PD 04(e0x20/s4) at dde08958)
[Thu May  4 11:19:47 2023] megaraid_sas 0000:02:00.0: 155186 (736503575s/0x0001/FATAL) - VD bad block table on VD 05/c is full; unable to log block eacc3658 (on PD 04(e0x20/s4) at eacc3658)
[Thu May  4 11:20:19 2023] megaraid_sas 0000:02:00.0: 155188 (736503608s/0x0001/FATAL) - VD bad block table on VD 05/c is full; unable to log block eafc3718 (on PD 04(e0x20/s4) at eafc3718)

RAID Status

Code: Select all

/dev/md0:
           Version : 1.2
     Creation Time : Mon Dec 18 12:31:05 2017
        Raid Level : raid5
        Array Size : 85946640384 (80.04 TiB 88.01 TB)
     Used Dev Size : 7813330944 (7.28 TiB 8.00 TB)
      Raid Devices : 12
     Total Devices : 11
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu May  4 11:47:35 2023
             State : clean, degraded 
    Active Devices : 11
   Working Devices : 11
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : localhost.localdomain:0
              UUID : 13ab13f0:d4b757b3:8b17660b:fdd87b30
            Events : 124736

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1
       -       0        0        4      removed
      13       8       97        5      active sync   /dev/sdg1
       6       8      113        6      active sync   /dev/sdh1
       7       8      129        7      active sync   /dev/sdi1
       8       8      145        8      active sync   /dev/sdj1
       9       8      161        9      active sync   /dev/sdk1
      10       8      177       10      active sync   /dev/sdl1
      12       8      193       11      active sync   /dev/sdm1

Error on trying to mount the volume

Code: Select all

[Thu May  4 11:52:02 2023] XFS (md0): Mounting V5 Filesystem
[Thu May  4 11:52:02 2023] XFS (md0): totally zeroed log
[Thu May  4 11:52:02 2023] XFS (md0): Corruption warning: Metadata has LSN (110:3200) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
[Thu May  4 11:52:02 2023] XFS (md0): log mount/recovery failed: error -22
[Thu May  4 11:52:02 2023] XFS (md0): log mount failed

[BShT]

Please unmount and run xfs_repair

[elkingsparx]

Currently running, not yet complete but this is the log so far.

Code: Select all

# xfs_repair -nv /dev/md0
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
        - block cache size set to 1750272 entries
Phase 2 - using internal log
        - zero log...
totally zeroed log
zero_log: head block 0 tail block 0
        - scan filesystem freespace and inode maps...
xfs_repair: read failed: Input/output error
bad magic # 0 in btbno block 64/19995675
expected level 1 got 0 in btbno block 64/19995675
bad btree nrecs (0, min=168, max=336) in btbno block 64/19995675
agf_btreeblks 24, counted 12 in ag 64
xfs_repair: read failed: Input/output error
bad magic # 0 in btbno block 54/11389217
bad btree nrecs (0, min=252, max=505) in btbno block 54/11389217
invalid start block 0 in record 0 of bno btree block 54/11389217
invalid start block 0 in record 1 of bno btree block 54/11389217
invalid start block 0 in record 2 of bno btree block 54/11389217
invalid start block 0 in record 3 of bno btree block 54/11389217
invalid start block 0 in record 4 of bno btree block 54/11389217
invalid start block 0 in record 5 of bno btree block 54/11389217
invalid start block 0 in record 6 of bno btree block 54/11389217
invalid start block 0 in record 7 of bno btree block 54/11389217
invalid start block 0 in record 8 of bno btree block 54/11389217
invalid start block 0 in record 9 of bno btree block 54/11389217
invalid start block 0 in record 10 of bno btree block 54/11389217
invalid start block 0 in record 11 of bno btree block 54/11389217
invalid start block 0 in record 12 of bno btree block 54/11389217
invalid start block 0 in record 13 of bno btree block 54/11389217
invalid start block 0 in record 14 of bno btree block 54/11389217
invalid start block 0 in record 15 of bno btree block 54/11389217
invalid start block 0 in record 16 of bno btree block 54/11389217
invalid start block 0 in record 17 of bno btree block 54/11389217
invalid start block 0 in record 18 of bno btree block 54/11389217
invalid start block 0 in record 19 of bno btree block 54/11389217
invalid start block 0 in record 20 of bno btree block 54/11389217
invalid start block 0 in record 21 of bno btree block 54/11389217
invalid start block 0 in record 22 of bno btree block 54/11389217
invalid start block 0 in record 23 of bno btree block 54/11389217
invalid start block 0 in record 24 of bno btree block 54/11389217
invalid start block 0 in record 25 of bno btree block 54/11389217
invalid start block 0 in record 26 of bno btree block 54/11389217
invalid start block 0 in record 27 of bno btree block 54/11389217
invalid start block 0 in record 28 of bno btree block 54/11389217
invalid start block 0 in record 29 of bno btree block 54/11389217
invalid start block 0 in record 30 of bno btree block 54/11389217
invalid start block 0 in record 31 of bno btree block 54/11389217
invalid start block 0 in record 32 of bno btree block 54/11389217
invalid start block 0 in record 33 of bno btree block 54/11389217
invalid start block 0 in record 34 of bno btree block 54/11389217
invalid start block 0 in record 35 of bno btree block 54/11389217
invalid start block 0 in record 36 of bno btree block 54/11389217
invalid start block 0 in record 37 of bno btree block 54/11389217
invalid start block 0 in record 38 of bno btree block 54/11389217
invalid start block 0 in record 39 of bno btree block 54/11389217
invalid start block 0 in record 40 of bno btree block 54/11389217
invalid start block 0 in record 41 of bno btree block 54/11389217
invalid start block 0 in record 42 of bno btree block 54/11389217
invalid start block 0 in record 43 of bno btree block 54/11389217
invalid start block 0 in record 44 of bno btree block 54/11389217
invalid start block 0 in record 45 of bno btree block 54/11389217
invalid start block 0 in record 46 of bno btree block 54/11389217
invalid start block 0 in record 47 of bno btree block 54/11389217
invalid start block 0 in record 48 of bno btree block 54/11389217
invalid start block 0 in record 49 of bno btree block 54/11389217
invalid start block 0 in record 50 of bno btree block 54/11389217
invalid start block 0 in record 51 of bno btree block 54/11389217
invalid start block 0 in record 52 of bno btree block 54/11389217
invalid start block 0 in record 53 of bno btree block 54/11389217
invalid start block 0 in record 54 of bno btree block 54/11389217
invalid start block 0 in record 55 of bno btree block 54/11389217
invalid start block 0 in record 56 of bno btree block 54/11389217
invalid start block 0 in record 57 of bno btree block 54/11389217
invalid start block 0 in record 58 of bno btree block 54/11389217
invalid start block 0 in record 59 of bno btree block 54/11389217
invalid start block 0 in record 60 of bno btree block 54/11389217
invalid start block 0 in record 61 of bno btree block 54/11389217
invalid start block 0 in record 62 of bno btree block 54/11389217
invalid start block 0 in record 63 of bno btree block 54/11389217
invalid start block 0 in record 64 of bno btree block 54/11389217
invalid start block 0 in record 65 of bno btree block 54/11389217
invalid start block 0 in record 66 of bno btree block 54/11389217
invalid start block 0 in record 67 of bno btree block 54/11389217
invalid start block 0 in record 68 of bno btree block 54/11389217
invalid start block 0 in record 69 of bno btree block 54/11389217
invalid start block 0 in record 70 of bno btree block 54/11389217
invalid start block 0 in record 71 of bno btree block 54/11389217
invalid start block 0 in record 72 of bno btree block 54/11389217
invalid start block 0 in record 73 of bno btree block 54/11389217
invalid start block 0 in record 74 of bno btree block 54/11389217
invalid start block 0 in record 75 of bno btree block 54/11389217
invalid start block 0 in record 76 of bno btree block 54/11389217
invalid start block 0 in record 77 of bno btree block 54/11389217
invalid start block 0 in record 78 of bno btree block 54/11389217
invalid start block 0 in record 79 of bno btree block 54/11389217
invalid start block 0 in record 80 of bno btree block 54/11389217
invalid start block 0 in record 81 of bno btree block 54/11389217
invalid start block 0 in record 82 of bno btree block 54/11389217
invalid start block 0 in record 83 of bno btree block 54/11389217
invalid start block 0 in record 84 of bno btree block 54/11389217
invalid start block 0 in record 85 of bno btree block 54/11389217
invalid start block 0 in record 86 of bno btree block 54/11389217
invalid start block 0 in record 87 of bno btree block 54/11389217
invalid start block 0 in record 88 of bno btree block 54/11389217
invalid start block 0 in record 89 of bno btree block 54/11389217
invalid start block 0 in record 90 of bno btree block 54/11389217
invalid start block 0 in record 91 of bno btree block 54/11389217
invalid start block 0 in record 92 of bno btree block 54/11389217
invalid start block 0 in record 93 of bno btree block 54/11389217
invalid start block 0 in record 94 of bno btree block 54/11389217
invalid start block 0 in record 95 of bno btree block 54/11389217
invalid start block 0 in record 96 of bno btree block 54/11389217
invalid start block 0 in record 97 of bno btree block 54/11389217
invalid start block 0 in record 98 of bno btree block 54/11389217
invalid start block 0 in record 99 of bno btree block 54/11389217
invalid start block 0 in record 100 of bno btree block 54/11389217
invalid start block 0 in record 101 of bno btree block 54/11389217
invalid start block 0 in record 102 of bno btree block 54/11389217
invalid start block 0 in record 103 of bno btree block 54/11389217
invalid start block 0 in record 104 of bno btree block 54/11389217
invalid start block 0 in record 105 of bno btree block 54/11389217
invalid start block 0 in record 106 of bno btree block 54/11389217
invalid start block 0 in record 107 of bno btree block 54/11389217
invalid start block 0 in record 108 of bno btree block 54/11389217
invalid start block 0 in record 109 of bno btree block 54/11389217
invalid start block 0 in record 110 of bno btree block 54/11389217
invalid start block 0 in record 111 of bno btree block 54/11389217
invalid start block 0 in record 112 of bno btree block 54/11389217
invalid start block 0 in record 113 of bno btree block 54/11389217
invalid start block 0 in record 114 of bno btree block 54/11389217
invalid start block 0 in record 115 of bno btree block 54/11389217
invalid start block 0 in record 116 of bno btree block 54/11389217
invalid start block 0 in record 117 of bno btree block 54/11389217
invalid start block 0 in record 118 of bno btree block 54/11389217
invalid start block 0 in record 119 of bno btree block 54/11389217
invalid start block 0 in record 120 of bno btree block 54/11389217
invalid start block 0 in record 121 of bno btree block 54/11389217
invalid start block 0 in record 122 of bno btree block 54/11389217
invalid start block 0 in record 123 of bno btree block 54/11389217
invalid start block 0 in record 124 of bno btree block 54/11389217
invalid start block 0 in record 125 of bno btree block 54/11389217
invalid start block 0 in record 126 of bno btree block 54/11389217
invalid start block 0 in record 127 of bno btree block 54/11389217
invalid start block 0 in record 128 of bno btree block 54/11389217
invalid start block 0 in record 129 of bno btree block 54/11389217
invalid start block 0 in record 130 of bno btree block 54/11389217
invalid start block 0 in record 131 of bno btree block 54/11389217
invalid start block 0 in record 132 of bno btree block 54/11389217
invalid start block 0 in record 133 of bno btree block 54/11389217
invalid start block 0 in record 134 of bno btree block 54/11389217
invalid start block 0 in record 135 of bno btree block 54/11389217
invalid start block 0 in record 136 of bno btree block 54/11389217
invalid start block 0 in record 137 of bno btree block 54/11389217
invalid start block 0 in record 138 of bno btree block 54/11389217
invalid start block 0 in record 139 of bno btree block 54/11389217
invalid start block 0 in record 140 of bno btree block 54/11389217
invalid start block 0 in record 141 of bno btree block 54/11389217
invalid start block 0 in record 142 of bno btree block 54/11389217
invalid start block 0 in record 143 of bno btree block 54/11389217
invalid start block 0 in record 144 of bno btree block 54/11389217
invalid start block 0 in record 145 of bno btree block 54/11389217
invalid start block 0 in record 146 of bno btree block 54/11389217
invalid start block 0 in record 147 of bno btree block 54/11389217
invalid start block 0 in record 148 of bno btree block 54/11389217
invalid start block 0 in record 149 of bno btree block 54/11389217
invalid start block 0 in record 150 of bno btree block 54/11389217
invalid start block 0 in record 151 of bno btree block 54/11389217
invalid start block 0 in record 152 of bno btree block 54/11389217
invalid start block 0 in record 153 of bno btree block 54/11389217
invalid start block 0 in record 154 of bno btree block 54/11389217
invalid start block 0 in record 155 of bno btree block 54/11389217
invalid start block 0 in record 156 of bno btree block 54/11389217
invalid start block 0 in record 157 of bno btree block 54/11389217
invalid start block 0 in record 158 of bno btree block 54/11389217
invalid start block 0 in record 159 of bno btree block 54/11389217
invalid start block 0 in record 160 of bno btree block 54/11389217
invalid start block 0 in record 161 of bno btree block 54/11389217
invalid start block 0 in record 162 of bno btree block 54/11389217
invalid start block 0 in record 163 of bno btree block 54/11389217
invalid start block 0 in record 164 of bno btree block 54/11389217
invalid start block 0 in record 165 of bno btree block 54/11389217
invalid start block 0 in record 166 of bno btree block 54/11389217
invalid start block 0 in record 167 of bno btree block 54/11389217
invalid start block 0 in record 168 of bno btree block 54/11389217
invalid start block 0 in record 169 of bno btree block 54/11389217
invalid start block 0 in record 170 of bno btree block 54/11389217
invalid start block 0 in record 171 of bno btree block 54/11389217
invalid start block 0 in record 172 of bno btree block 54/11389217
invalid start block 0 in record 173 of bno btree block 54/11389217
invalid start block 0 in record 174 of bno btree block 54/11389217
invalid start block 0 in record 175 of bno btree block 54/11389217
invalid start block 0 in record 176 of bno btree block 54/11389217
invalid start block 0 in record 177 of bno btree block 54/11389217
invalid start block 0 in record 178 of bno btree block 54/11389217
invalid start block 0 in record 179 of bno btree block 54/11389217
invalid start block 0 in record 180 of bno btree block 54/11389217
invalid start block 0 in record 181 of bno btree block 54/11389217
invalid start block 0 in record 182 of bno btree block 54/11389217
invalid start block 0 in record 183 of bno btree block 54/11389217
invalid start block 0 in record 184 of bno btree block 54/11389217
invalid start block 0 in record 185 of bno btree block 54/11389217
invalid start block 0 in record 186 of bno btree block 54/11389217
invalid start block 0 in record 187 of bno btree block 54/11389217
invalid start block 0 in record 188 of bno btree block 54/11389217
invalid start block 0 in record 189 of bno btree block 54/11389217
invalid start block 0 in record 190 of bno btree block 54/11389217
invalid start block 0 in record 191 of bno btree block 54/11389217
invalid start block 0 in record 192 of bno btree block 54/11389217
invalid start block 0 in record 193 of bno btree block 54/11389217
invalid start block 0 in record 194 of bno btree block 54/11389217
invalid start block 0 in record 195 of bno btree block 54/11389217
invalid start block 0 in record 196 of bno btree block 54/11389217
invalid start block 0 in record 197 of bno btree block 54/11389217
invalid start block 0 in record 198 of bno btree block 54/11389217
invalid start block 0 in record 199 of bno btree block 54/11389217
invalid start block 0 in record 200 of bno btree block 54/11389217
invalid start block 0 in record 201 of bno btree block 54/11389217
invalid start block 0 in record 202 of bno btree block 54/11389217
invalid start block 0 in record 203 of bno btree block 54/11389217
invalid start block 0 in record 204 of bno btree block 54/11389217
invalid start block 0 in record 205 of bno btree block 54/11389217
invalid start block 0 in record 206 of bno btree block 54/11389217
invalid start block 0 in record 207 of bno btree block 54/11389217
invalid start block 0 in record 208 of bno btree block 54/11389217
invalid start block 0 in record 209 of bno btree block 54/11389217
invalid start block 0 in record 210 of bno btree block 54/11389217
invalid start block 0 in record 211 of bno btree block 54/11389217
invalid start block 0 in record 212 of bno btree block 54/11389217
invalid start block 0 in record 213 of bno btree block 54/11389217
invalid start block 0 in record 214 of bno btree block 54/11389217
invalid start block 0 in record 215 of bno btree block 54/11389217
invalid start block 0 in record 216 of bno btree block 54/11389217
invalid start block 0 in record 217 of bno btree block 54/11389217
invalid start block 0 in record 218 of bno btree block 54/11389217
invalid start block 0 in record 219 of bno btree block 54/11389217
invalid start block 0 in record 220 of bno btree block 54/11389217
invalid start block 0 in record 221 of bno btree block 54/11389217
invalid start block 0 in record 222 of bno btree block 54/11389217
invalid start block 0 in record 223 of bno btree block 54/11389217
invalid start block 0 in record 224 of bno btree block 54/11389217
invalid start block 0 in record 225 of bno btree block 54/11389217
invalid start block 0 in record 226 of bno btree block 54/11389217
invalid start block 0 in record 227 of bno btree block 54/11389217
invalid start block 0 in record 228 of bno btree block 54/11389217
invalid start block 0 in record 229 of bno btree block 54/11389217
invalid start block 0 in record 230 of bno btree block 54/11389217
invalid start block 0 in record 231 of bno btree block 54/11389217
invalid start block 0 in record 232 of bno btree block 54/11389217
invalid start block 0 in record 233 of bno btree block 54/11389217
invalid start block 0 in record 234 of bno btree block 54/11389217
invalid start block 0 in record 235 of bno btree block 54/11389217
invalid start block 0 in record 236 of bno btree block 54/11389217
invalid start block 0 in record 237 of bno btree block 54/11389217
invalid start block 0 in record 238 of bno btree block 54/11389217
invalid start block 0 in record 239 of bno btree block 54/11389217
invalid start block 0 in record 240 of bno btree block 54/11389217
invalid start block 0 in record 241 of bno btree block 54/11389217
invalid start block 0 in record 242 of bno btree block 54/11389217
invalid start block 0 in record 243 of bno btree block 54/11389217
invalid start block 0 in record 244 of bno btree block 54/11389217
invalid start block 0 in record 245 of bno btree block 54/11389217
invalid start block 0 in record 246 of bno btree block 54/11389217
invalid start block 0 in record 247 of bno btree block 54/11389217
invalid start block 0 in record 248 of bno btree block 54/11389217
invalid start block 0 in record 249 of bno btree block 54/11389217
invalid start block 0 in record 250 of bno btree block 54/11389217
invalid start block 0 in record 251 of bno btree block 54/11389217
sb_fdblocks 37419319, counted 37419307
        - 11:56:30: scanning filesystem freespace - 81 of 81 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - 11:56:30: scanning agi unlinked lists - 81 of 81 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 30
        - agno = 0
        - agno = 15
        - agno = 45
        - agno = 60
        - agno = 75
        - agno = 46
        - 12:11:15: process known inodes and inode discovery - 369088 of 7834304 inodes done
        - 12:11:15: Phase 3: elapsed time 14 minutes, 45 seconds - processed 25022 inodes per minute
        - 12:11:15: Phase 3: 4% done - estimated remaining time 4 hours, 58 minutes, 20 seconds
        - agno = 76
        - agno = 47
        - agno = 31
        - 12:26:15: process known inodes and inode discovery - 742528 of 7834304 inodes done
        - 12:26:15: Phase 3: elapsed time 29 minutes, 45 seconds - processed 24958 inodes per minute
        - 12:26:15: Phase 3: 9% done - estimated remaining time 4 hours, 44 minutes, 8 seconds
        - agno = 48
        - agno = 61
        - agno = 1
        - agno = 16
        - agno = 32
        - agno = 77
        - agno = 49
        - 12:41:15: process known inodes and inode discovery - 1151808 of 7834304 inodes done
        - 12:41:15: Phase 3: elapsed time 44 minutes, 45 seconds - processed 25738 inodes per minute
        - 12:41:15: Phase 3: 14% done - estimated remaining time 4 hours, 19 minutes, 37 seconds
        - agno = 2
        - agno = 33
        - agno = 17
        - agno = 50
        - 12:56:15: process known inodes and inode discovery - 1515584 of 7834304 inodes done
        - 12:56:15: Phase 3: elapsed time 59 minutes, 45 seconds - processed 25365 inodes per minute
        - 12:56:15: Phase 3: 19% done - estimated remaining time 4 hours, 9 minutes, 6 seconds
        - agno = 51
        - agno = 3
        - agno = 18
        - agno = 34
        - agno = 62
        - 13:11:15: process known inodes and inode discovery - 1926464 of 7834304 inodes done
        - 13:11:15: Phase 3: elapsed time 1 hour, 14 minutes, 45 seconds - processed 25772 inodes per minute
        - 13:11:15: Phase 3: 24% done - estimated remaining time 3 hours, 49 minutes, 14 seconds
        - agno = 35
        - agno = 19
        - agno = 52
xfs_repair: read failed: Input/output error
xfs_repair: read failed: Input/output error
bad CRC for inode 111692349440
bad magic number 0x0 on inode 111692349440
bad version number 0x0 on inode 111692349440
inode identifier 0 mismatch on inode 111692349440
bad CRC for inode 111692349441
bad magic number 0x0 on inode 111692349441
bad version number 0x0 on inode 111692349441
inode identifier 0 mismatch on inode 111692349441
bad CRC for inode 111692349442
bad magic number 0x0 on inode 111692349442
bad version number 0x0 on inode 111692349442
inode identifier 0 mismatch on inode 111692349442
bad CRC for inode 111692349443
bad magic number 0x0 on inode 111692349443
bad version number 0x0 on inode 111692349443
inode identifier 0 mismatch on inode 111692349443
bad CRC for inode 111692349444
bad magic number 0x0 on inode 111692349444
bad version number 0x0 on inode 111692349444
inode identifier 0 mismatch on inode 111692349444
bad CRC for inode 111692349445
bad magic number 0x0 on inode 111692349445
bad version number 0x0 on inode 111692349445
inode identifier 0 mismatch on inode 111692349445
bad CRC for inode 111692349446
bad magic number 0x0 on inode 111692349446
bad version number 0x0 on inode 111692349446
inode identifier 0 mismatch on inode 111692349446
bad CRC for inode 111692349447
bad magic number 0x0 on inode 111692349447
bad version number 0x0 on inode 111692349447
inode identifier 0 mismatch on inode 111692349447
bad CRC for inode 111692349448
bad magic number 0x0 on inode 111692349448
bad version number 0x0 on inode 111692349448
inode identifier 0 mismatch on inode 111692349448
bad CRC for inode 111692349449
bad magic number 0x0 on inode 111692349449
bad version number 0x0 on inode 111692349449
inode identifier 0 mismatch on inode 111692349449
bad CRC for inode 111692349450
bad magic number 0x0 on inode 111692349450
bad version number 0x0 on inode 111692349450
inode identifier 0 mismatch on inode 111692349450
bad CRC for inode 111692349451
bad magic number 0x0 on inode 111692349451
bad version number 0x0 on inode 111692349451
inode identifier 0 mismatch on inode 111692349451
bad CRC for inode 111692349452
bad magic number 0x0 on inode 111692349452
bad version number 0x0 on inode 111692349452
inode identifier 0 mismatch on inode 111692349452
bad CRC for inode 111692349453
bad magic number 0x0 on inode 111692349453
bad version number 0x0 on inode 111692349453
inode identifier 0 mismatch on inode 111692349453
bad CRC for inode 111692349454
bad magic number 0x0 on inode 111692349454
bad version number 0x0 on inode 111692349454
inode identifier 0 mismatch on inode 111692349454
bad CRC for inode 111692349455
bad magic number 0x0 on inode 111692349455
bad version number 0x0 on inode 111692349455
inode identifier 0 mismatch on inode 111692349455
bad CRC for inode 111692349456
bad magic number 0x0 on inode 111692349456
bad version number 0x0 on inode 111692349456
inode identifier 0 mismatch on inode 111692349456
bad CRC for inode 111692349457
bad magic number 0x0 on inode 111692349457
bad version number 0x0 on inode 111692349457
inode identifier 0 mismatch on inode 111692349457
bad CRC for inode 111692349458
bad magic number 0x0 on inode 111692349458
bad version number 0x0 on inode 111692349458
inode identifier 0 mismatch on inode 111692349458
bad CRC for inode 111692349459
bad magic number 0x0 on inode 111692349459
bad version number 0x0 on inode 111692349459
inode identifier 0 mismatch on inode 111692349459
bad CRC for inode 111692349460
bad magic number 0x0 on inode 111692349460
bad version number 0x0 on inode 111692349460
inode identifier 0 mismatch on inode 111692349460
bad CRC for inode 111692349461
bad magic number 0x0 on inode 111692349461
bad version number 0x0 on inode 111692349461
inode identifier 0 mismatch on inode 111692349461
bad CRC for inode 111692349462
bad magic number 0x0 on inode 111692349462
bad version number 0x0 on inode 111692349462
inode identifier 0 mismatch on inode 111692349462
bad CRC for inode 111692349463
bad magic number 0x0 on inode 111692349463
bad version number 0x0 on inode 111692349463
inode identifier 0 mismatch on inode 111692349463
bad CRC for inode 111692349464
bad magic number 0x0 on inode 111692349464
bad version number 0x0 on inode 111692349464
inode identifier 0 mismatch on inode 111692349464
bad CRC for inode 111692349465
bad magic number 0x0 on inode 111692349465
bad version number 0x0 on inode 111692349465
inode identifier 0 mismatch on inode 111692349465
bad CRC for inode 111692349466
bad magic number 0x0 on inode 111692349466
bad version number 0x0 on inode 111692349466
inode identifier 0 mismatch on inode 111692349466
bad CRC for inode 111692349467
bad magic number 0x0 on inode 111692349467
bad version number 0x0 on inode 111692349467
inode identifier 0 mismatch on inode 111692349467
bad CRC for inode 111692349468
bad magic number 0x0 on inode 111692349468
bad version number 0x0 on inode 111692349468
inode identifier 0 mismatch on inode 111692349468
bad CRC for inode 111692349469
bad magic number 0x0 on inode 111692349469
bad version number 0x0 on inode 111692349469
inode identifier 0 mismatch on inode 111692349469
bad CRC for inode 111692349470
bad magic number 0x0 on inode 111692349470
bad version number 0x0 on inode 111692349470
inode identifier 0 mismatch on inode 111692349470
bad CRC for inode 111692349471
bad magic number 0x0 on inode 111692349471
bad version number 0x0 on inode 111692349471
inode identifier 0 mismatch on inode 111692349471
bad CRC for inode 111692349440, would rewrite
bad magic number 0x0 on inode 111692349440, would reset magic number
bad version number 0x0 on inode 111692349440, would reset version number
inode identifier 0 mismatch on inode 111692349440
would have cleared inode 111692349440
bad CRC for inode 111692349441, would rewrite
bad magic number 0x0 on inode 111692349441, would reset magic number
bad version number 0x0 on inode 111692349441, would reset version number
inode identifier 0 mismatch on inode 111692349441
would have cleared inode 111692349441
bad CRC for inode 111692349442, would rewrite
bad magic number 0x0 on inode 111692349442, would reset magic number
bad version number 0x0 on inode 111692349442, would reset version number
inode identifier 0 mismatch on inode 111692349442
would have cleared inode 111692349442
bad CRC for inode 111692349443, would rewrite
bad magic number 0x0 on inode 111692349443, would reset magic number
bad version number 0x0 on inode 111692349443, would reset version number
inode identifier 0 mismatch on inode 111692349443
would have cleared inode 111692349443
bad CRC for inode 111692349444, would rewrite
bad magic number 0x0 on inode 111692349444, would reset magic number
bad version number 0x0 on inode 111692349444, would reset version number
inode identifier 0 mismatch on inode 111692349444
would have cleared inode 111692349444
bad CRC for inode 111692349445, would rewrite
bad magic number 0x0 on inode 111692349445, would reset magic number
bad version number 0x0 on inode 111692349445, would reset version number
inode identifier 0 mismatch on inode 111692349445
would have cleared inode 111692349445
bad CRC for inode 111692349446, would rewrite
bad magic number 0x0 on inode 111692349446, would reset magic number
bad version number 0x0 on inode 111692349446, would reset version number
inode identifier 0 mismatch on inode 111692349446
would have cleared inode 111692349446
bad CRC for inode 111692349447, would rewrite
bad magic number 0x0 on inode 111692349447, would reset magic number
bad version number 0x0 on inode 111692349447, would reset version number
inode identifier 0 mismatch on inode 111692349447
would have cleared inode 111692349447
bad CRC for inode 111692349448, would rewrite
bad magic number 0x0 on inode 111692349448, would reset magic number
bad version number 0x0 on inode 111692349448, would reset version number
inode identifier 0 mismatch on inode 111692349448
would have cleared inode 111692349448
bad CRC for inode 111692349449, would rewrite
bad magic number 0x0 on inode 111692349449, would reset magic number
bad version number 0x0 on inode 111692349449, would reset version number
inode identifier 0 mismatch on inode 111692349449
would have cleared inode 111692349449
bad CRC for inode 111692349450, would rewrite
bad magic number 0x0 on inode 111692349450, would reset magic number
bad version number 0x0 on inode 111692349450, would reset version number
inode identifier 0 mismatch on inode 111692349450
would have cleared inode 111692349450
bad CRC for inode 111692349451, would rewrite
bad magic number 0x0 on inode 111692349451, would reset magic number
bad version number 0x0 on inode 111692349451, would reset version number
inode identifier 0 mismatch on inode 111692349451
would have cleared inode 111692349451
bad CRC for inode 111692349452, would rewrite
bad magic number 0x0 on inode 111692349452, would reset magic number
bad version number 0x0 on inode 111692349452, would reset version number
inode identifier 0 mismatch on inode 111692349452
would have cleared inode 111692349452
bad CRC for inode 111692349453, would rewrite
bad magic number 0x0 on inode 111692349453, would reset magic number
bad version number 0x0 on inode 111692349453, would reset version number
inode identifier 0 mismatch on inode 111692349453
would have cleared inode 111692349453
bad CRC for inode 111692349454, would rewrite
bad magic number 0x0 on inode 111692349454, would reset magic number
bad version number 0x0 on inode 111692349454, would reset version number
inode identifier 0 mismatch on inode 111692349454
would have cleared inode 111692349454
bad CRC for inode 111692349455, would rewrite
bad magic number 0x0 on inode 111692349455, would reset magic number
bad version number 0x0 on inode 111692349455, would reset version number
inode identifier 0 mismatch on inode 111692349455
would have cleared inode 111692349455
bad CRC for inode 111692349456, would rewrite
bad magic number 0x0 on inode 111692349456, would reset magic number
bad version number 0x0 on inode 111692349456, would reset version number
inode identifier 0 mismatch on inode 111692349456
would have cleared inode 111692349456
bad CRC for inode 111692349457, would rewrite
bad magic number 0x0 on inode 111692349457, would reset magic number
bad version number 0x0 on inode 111692349457, would reset version number
inode identifier 0 mismatch on inode 111692349457
would have cleared inode 111692349457
bad CRC for inode 111692349458, would rewrite
bad magic number 0x0 on inode 111692349458, would reset magic number
bad version number 0x0 on inode 111692349458, would reset version number
inode identifier 0 mismatch on inode 111692349458
would have cleared inode 111692349458
bad CRC for inode 111692349459, would rewrite
bad magic number 0x0 on inode 111692349459, would reset magic number
bad version number 0x0 on inode 111692349459, would reset version number
inode identifier 0 mismatch on inode 111692349459
would have cleared inode 111692349459
bad CRC for inode 111692349460, would rewrite
bad magic number 0x0 on inode 111692349460, would reset magic number
bad version number 0x0 on inode 111692349460, would reset version number
inode identifier 0 mismatch on inode 111692349460
would have cleared inode 111692349460
bad CRC for inode 111692349461, would rewrite
bad magic number 0x0 on inode 111692349461, would reset magic number
bad version number 0x0 on inode 111692349461, would reset version number
inode identifier 0 mismatch on inode 111692349461
would have cleared inode 111692349461
bad CRC for inode 111692349462, would rewrite
bad magic number 0x0 on inode 111692349462, would reset magic number
bad version number 0x0 on inode 111692349462, would reset version number
inode identifier 0 mismatch on inode 111692349462
would have cleared inode 111692349462
bad CRC for inode 111692349463, would rewrite
bad magic number 0x0 on inode 111692349463, would reset magic number
bad version number 0x0 on inode 111692349463, would reset version number
inode identifier 0 mismatch on inode 111692349463
would have cleared inode 111692349463
bad CRC for inode 111692349464, would rewrite
bad magic number 0x0 on inode 111692349464, would reset magic number
bad version number 0x0 on inode 111692349464, would reset version number
inode identifier 0 mismatch on inode 111692349464
would have cleared inode 111692349464
bad CRC for inode 111692349465, would rewrite
bad magic number 0x0 on inode 111692349465, would reset magic number
bad version number 0x0 on inode 111692349465, would reset version number
inode identifier 0 mismatch on inode 111692349465
would have cleared inode 111692349465
bad CRC for inode 111692349466, would rewrite
bad magic number 0x0 on inode 111692349466, would reset magic number
bad version number 0x0 on inode 111692349466, would reset version number
inode identifier 0 mismatch on inode 111692349466
would have cleared inode 111692349466
bad CRC for inode 111692349467, would rewrite
bad magic number 0x0 on inode 111692349467, would reset magic number
bad version number 0x0 on inode 111692349467, would reset version number
inode identifier 0 mismatch on inode 111692349467
would have cleared inode 111692349467
bad CRC for inode 111692349468, would rewrite
bad magic number 0x0 on inode 111692349468, would reset magic number
bad version number 0x0 on inode 111692349468, would reset version number
inode identifier 0 mismatch on inode 111692349468
would have cleared inode 111692349468
bad CRC for inode 111692349469, would rewrite
bad magic number 0x0 on inode 111692349469, would reset magic number
bad version number 0x0 on inode 111692349469, would reset version number
inode identifier 0 mismatch on inode 111692349469
would have cleared inode 111692349469
bad CRC for inode 111692349470, would rewrite
bad magic number 0x0 on inode 111692349470, would reset magic number
bad version number 0x0 on inode 111692349470, would reset version number
inode identifier 0 mismatch on inode 111692349470
would have cleared inode 111692349470
bad CRC for inode 111692349471, would rewrite
bad magic number 0x0 on inode 111692349471, would reset magic number
bad version number 0x0 on inode 111692349471, would reset version number
inode identifier 0 mismatch on inode 111692349471
would have cleared inode 111692349471
        - 13:26:15: process known inodes and inode discovery - 2267392 of 7834304 inodes done
        - 13:26:15: Phase 3: elapsed time 1 hour, 29 minutes, 45 seconds - processed 25263 inodes per minute
        - 13:26:15: Phase 3: 28% done - estimated remaining time 3 hours, 40 minutes, 21 seconds
        - agno = 4
        - agno = 53
        - 13:41:15: process known inodes and inode discovery - 2597312 of 7834304 inodes done
        - 13:41:15: Phase 3: elapsed time 1 hour, 44 minutes, 45 seconds - processed 24795 inodes per minute
        - 13:41:15: Phase 3: 33% done - estimated remaining time 3 hours, 31 minutes, 12 seconds
        - agno = 36
        - agno = 20
        - agno = 78
        - 13:56:15: process known inodes and inode discovery - 2899200 of 7834304 inodes done
        - 13:56:15: Phase 3: elapsed time 1 hour, 59 minutes, 45 seconds - processed 24210 inodes per minute
        - 13:56:15: Phase 3: 37% done - estimated remaining time 3 hours, 23 minutes, 50 seconds
        - agno = 37
        - agno = 5
        - 14:11:15: process known inodes and inode discovery - 3244736 of 7834304 inodes done
        - 14:11:15: Phase 3: elapsed time 2 hours, 14 minutes, 45 seconds - processed 24079 inodes per minute
        - 14:11:15: Phase 3: 41% done - estimated remaining time 3 hours, 10 minutes, 35 seconds
        - agno = 21
        - agno = 63
        - 14:26:15: process known inodes and inode discovery - 3579008 of 7834304 inodes done
        - 14:26:15: Phase 3: elapsed time 2 hours, 29 minutes, 45 seconds - processed 23899 inodes per minute
        - 14:26:15: Phase 3: 45% done - estimated remaining time 2 hours, 58 minutes, 2 seconds
        - agno = 38
        - agno = 6
        - agno = 22
        - agno = 79
        - agno = 54
        - 14:41:15: process known inodes and inode discovery - 3916864 of 7834304 inodes done
        - 14:41:15: Phase 3: elapsed time 2 hours, 44 minutes, 45 seconds - processed 23774 inodes per minute
        - 14:41:15: Phase 3: 49% done - estimated remaining time 2 hours, 44 minutes, 46 seconds
xfs_repair: read failed: Input/output error
xfs_repair: read failed: Input/output error
bad CRC for inode 116054477664
bad magic number 0x0 on inode 116054477664
bad version number 0x0 on inode 116054477664
inode identifier 0 mismatch on inode 116054477664
bad CRC for inode 116054477665

[TrevorH]

Those errors are from your RAID controller so the problem is at a lower level than the filesystem. It looks like one of the disks in the array is going to die soon as the message is telling you that the internal bad block list for the disk is full up. I amm not sure why xfs_repair is listing all those problems but I think the real error is not something it can fix, it needs hardware looked at. I'd check "on PD 04" that disk.

[tunk]

You may be able to run smartctl on the RAID disks, something like this:
fdisk -l
smartctl --scan
smartctl -a -d megaraid,YY /dev/sdX

E.g.:
# fdisk -l
....
Disk /dev/sdb: ....
Disk model: PERC Hxxx
....
# smartctl --scan
....
/dev/sdb -d scsi # /dev/sdb, SCSI device
....
/dev/bus/5 -d megaraid,22 # /dev/bus/5 [megaraid_disk_22], SCSI device
....
# smartctl -a -d megaraid,22 /dev/sdb

[elkingsparx]

Thanks for the reply. If I replace the failing drive the raid should be able to rebuild?

[tunk]

Never had to do it, but in theory, I think the answer is yes.
In practice you will put all disk at high stress, running them
continuously for (a) day(s) increasing the probability of the
others failing. Usually I use RAID 6 for arrays with this
many disks.

[TrevorH]

Depends on your RAID level but, yes, if you replace the disk and it's not RAID 0 then it should rebuild. There is a chance that another disk may crap out during the rebuild because it will have to read all of the contents of all of them in order to rebuild the missing one.

[tunk]

If you have other storage it may be safer to first copy the data there:
- you can divide the copy in sections letting the disks cool down and
rest in the pauses
- you only copy your data, whereas a RAID rebuild will read/write the
whole disks. I.e. if it's ~50% full you will reduce the stress with ~50%

Edit:
Come to think about it, I think there's been discussions about what
is worse, high temperature or heating/cooling cycles. If the latter,
then the first argument isn't valid.

[elkingsparx]

Was unable to mount the volume so I'm unable to access the data, that's why I was running xfs_repair.

Code: Select all

megaraid_sas 0000:02:00.0: 155184 (736502998s/0x0001/FATAL) - VD bad block table on VD 05/c is full; unable to log block dde08958 (on PD 04(e0x20/s4) at dde08958)
megaraid_sas 0000:02:00.0: 155186 (736503575s/0x0001/FATAL) - VD bad block table on VD 05/c is full; unable to log block eacc3658 (on PD 04(e0x20/s4) at eacc3658)
megaraid_sas 0000:02:00.0: 155188 (736503608s/0x0001/FATAL) - VD bad block table on VD 05/c is full; unable to log block eafc3718 (on PD 04(e0x20/s4) at eafc3718)
md: md0 stopped.
md: kicking non-fresh sdf1 from array!
async_tx: api initialized (async)
xor: measuring software checksum speed
   prefetch64-sse:  6796.000 MB/sec
   generic_sse:  5956.000 MB/sec
xor: using function: prefetch64-sse (6796.000 MB/sec)
raid6: sse2x1   gen()  3910 MB/s
raid6: sse2x2   gen()  4605 MB/s
raid6: sse2x4   gen()  5093 MB/s
raid6: using algorithm sse2x4 gen() (5093 MB/s)
raid6: using ssse3x2 recovery algorithm
md/raid:md0: device sdb1 operational as raid disk 0
md/raid:md0: device sdm1 operational as raid disk 11
md/raid:md0: device sdl1 operational as raid disk 10
md/raid:md0: device sdk1 operational as raid disk 9
md/raid:md0: device sdj1 operational as raid disk 8
md/raid:md0: device sdi1 operational as raid disk 7
md/raid:md0: device sdh1 operational as raid disk 6
md/raid:md0: device sdg1 operational as raid disk 5
md/raid:md0: device sde1 operational as raid disk 3
md/raid:md0: device sdd1 operational as raid disk 2
md/raid:md0: device sdc1 operational as raid disk 1
md/raid:md0: raid level 5 active with 11 out of 12 devices, algorithm 2
md0: detected capacity change from 0 to 88009359753216
md: data-check of RAID array md0
md: md0: data-check done.
md: data-check of RAID array md0
md: md0: data-check done.
md: requested-resync of RAID array md0
md: md0: requested-resync done.
XFS (md0): Mounting V5 Filesystem
XFS (md0): totally zeroed log
XFS (md0): Corruption warning: Metadata has LSN (110:3200) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
XFS (md0): log mount/recovery failed: error -22
XFS (md0): log mount failed