Hardware Question - S.M.A.R.T. reports disk failure imminent

Issues related to applications and software problems and general support
Post Reply
mjz
Posts: 115
Joined: 2020/02/13 22:05:02

Hardware Question - S.M.A.R.T. reports disk failure imminent

Post by mjz » 2021/07/28 17:32:04

I use 'gsmartcontrol' to monitor my disks. One drive is reporting near failure (see output below).

I backed up the drive, reformatted it and then put it back in service curious how long it will last.
The drive reported 2,071 reallocated sectors which is roughly a megabyte in size.

After a month, the drive is still running fine - I have even used it as a running drive (data duplicated) with lots of data and no issue.

Given only a megabyte of drive space out of 2 TB is bad, perhaps S.M.A.R.T is too conservative?

Of course I won't rely on this drive, just curious if others have had drives report as "failed", but continue to work just fine.

I decided to wipe the drive by writing to all sectors to see what happens (i.e. dd if=/dev/zero of=/dev/sdb oflag=seek_bytes seek=400000 status=progress &) by forcing it to read/write
to all sectors. It's running now with no issues so far.


=== START OF INFORMATION SECTION ===
Model Family: Western Digital Re
Device Model: WDC WD2000FYYZ-01UL1B0
Serial Number: WD-WMC1P0155372
LU WWN Device Id: 5 0014ee 0039b2d16
Firmware Version: 01.01K01
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 28 08:08:05 2021 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM level is: 128 (minimum power consumption without standby)
Rd look-ahead is: Enabled
Write cache is: Disabled
ATA Security is: Disabled, NOT FROZEN [SEC1]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
See vendor-specific Attribute list for failed Attributes.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Hardware Question - S.M.A.R.T. reports disk failure imminent

Post by TrevorH » 2021/07/28 22:05:39

UUse smartctl -a /dev/sdb
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

mjz
Posts: 115
Joined: 2020/02/13 22:05:02

Re: Hardware Question - S.M.A.R.T. reports disk failure imminent

Post by mjz » 2021/07/28 22:49:26

Code: Select all

smartctl -a /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-5.4.0-80-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Re
Device Model:     WDC WD2000FYYZ-01UL1B0
Serial Number:    WD-WMC1P0155372
LU WWN Device Id: 5 0014ee 0039b2d16
Firmware Version: 01.01K01
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jul 28 17:48:06 2021 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
See vendor-specific Attribute list for failed Attributes.

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(24540) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 267) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x70bd)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   123   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   238   168   021    Pre-fail  Always       -       3058
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       701
  5 Reallocated_Sector_Ct   0x0033   133   133   140    Pre-fail  Always   FAILING_NOW 2071
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   095   095   000    Old_age   Always       -       4327
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       169
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       48
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       652
194 Temperature_Celsius     0x0022   114   104   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   169   169   000    Old_age   Always       -       31
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       2
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   199   000    Old_age   Offline      -       2

SMART Error Log Version: 1
ATA Error Count: 6 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 6 occurred at disk power-on lifetime: 3987 hours (166 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 c0 c8 75 e7  Error: UNC 8 sectors at LBA = 0x0775c8c0 = 125159616

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 c0 c8 75 e7 0a      01:24:43.607  READ DMA
  c8 00 18 c8 52 20 e0 0a      01:24:43.607  READ DMA
  ec 00 00 00 00 00 a0 0a      01:24:43.604  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      01:24:43.604  SET FEATURES [Set transfer mode]

Error 5 occurred at disk power-on lifetime: 3987 hours (166 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 c0 c8 75 e7  Error: UNC 8 sectors at LBA = 0x0775c8c0 = 125159616

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 c0 c8 75 e7 0a      01:24:41.709  READ DMA
  c8 00 10 00 4f 20 e0 0a      01:24:41.709  READ DMA
  ec 00 00 00 00 00 a0 0a      01:24:41.706  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      01:24:41.706  SET FEATURES [Set transfer mode]

Error 4 occurred at disk power-on lifetime: 3987 hours (166 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 c0 c8 75 e7  Error: UNC 8 sectors at LBA = 0x0775c8c0 = 125159616

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 c0 c8 75 e7 0a      01:24:39.811  READ DMA
  c8 00 08 f8 4e 20 e0 0a      01:24:39.811  READ DMA
  ec 00 00 00 00 00 a0 0a      01:24:39.808  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      01:24:39.808  SET FEATURES [Set transfer mode]

Error 3 occurred at disk power-on lifetime: 3987 hours (166 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 c0 c8 75 e7  Error: UNC 8 sectors at LBA = 0x0775c8c0 = 125159616

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 c0 c8 75 e7 0a      01:24:37.975  READ DMA
  c8 00 08 50 4e 20 e0 0a      01:24:37.975  READ DMA
  c8 00 08 b8 c8 75 e7 0a      01:24:37.975  READ DMA
  c8 00 08 70 18 20 e0 0a      01:24:37.971  READ DMA
  c8 00 08 b0 c8 75 e7 0a      01:24:37.971  READ DMA

Error 2 occurred at disk power-on lifetime: 3987 hours (166 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 80 c0 c8 75 e7  Error: UNC 128 sectors at LBA = 0x0775c8c0 = 125159616

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 80 a8 c8 75 e7 0a      01:24:36.036  READ DMA
  c8 00 08 c0 4e 20 e0 0a      01:24:36.036  READ DMA
  c8 00 20 88 c8 75 e7 0a      01:24:36.035  READ DMA
  c8 00 08 80 c8 75 e7 0a      01:24:36.035  READ DMA
  c8 00 08 18 2e 20 e0 0a      01:24:36.016  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      4259         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Hardware Question - S.M.A.R.T. reports disk failure imminent

Post by TrevorH » 2021/07/28 23:32:12

There's a good chance that's still under warranty if it's only been powered on for 4327 hours.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

gerry666uk
Posts: 98
Joined: 2020/02/10 19:06:06

Re: Hardware Question - S.M.A.R.T. reports disk failure imminent

Post by gerry666uk » 2021/07/29 22:10:00

It sounds like the 'Reallocated_Sector_Ct' has exceeded some threshold.
I'm not sure why the 'value' is different to the 'raw value'.
2071 re-mapped sectors doesn't sound very much on a big drive, but it could mean that a sector op has failed over two thousand times over the life of the drive, and also that the re-map area might be getting filled up.
I don't think you can test this with 'dd' because the re-mapping will be at a lower level and 'dd' will just skip over it.
The recommendation is to replace the drive.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Hardware Question - S.M.A.R.T. reports disk failure imminent

Post by jlehtone » 2021/07/30 07:14:22

gerry666uk wrote:
2021/07/29 22:10:00
I don't think you can test this with 'dd' because the re-mapping will be at a lower level and 'dd' will just skip over it.
Sector reallocation occurs only when writes fail. The dd writes.
The SMART counters before and after that dd would have been interesting; how many reallocs did the dd invoke?

There are already Current_Pending_Sector: 2; some attempts to reallocate have not yet succeeded.

Yes, replace the drive.


Low hours do not always mean new drive.
When did WD Re came to market? About 2013 or earlier? When did they discontinue? After 2016?
They are EOL: https://www.hdstorageworks.com/Western-Digital-EOL.asp
The date of assembly is on the sticker on top of the drive.

That said, WD had a support page. You could check if they still offer RMA for your drive.

jabuzzard
Posts: 30
Joined: 2008/09/18 12:35:09

Re: Hardware Question - S.M.A.R.T. reports disk failure imminent

Post by jabuzzard » 2021/08/04 22:11:12

If you have 200 reallocated sectors you want to be copying the data off ASAP. If you have 2000 reallocated sectors power down the drive NOW till you are in a position to copy the data off. Unless it's in a RAID array in which case get a replacement drive because failure is imminent in my ~20 years experience doing storage professionally.

Post Reply