one-disk RAID1 gets an hot spare when second disk added, tries and fail to become two-disks RAID1

Options

Hi.

I had a NAS326 where one of the two disks was showing bad stats in SMART. So I bought a new one, pulled out the defective one and replaced it.

Initially the NAS was doing a lot of beeping and I could not even access the web interface. I pulled out the new disk and was able to access with just the original one.

Then I added the new disk and things seemed to work. However, it is reporting that the Disk Group 1 is degraded. Looking at it, it says that the volume is made by just disk2. As you see in the screenshot,

disk1 is still in Disk Group 1, but as an hot spare rather than in the RAID1, which I think is what should have happened.

In fact when the NAS restarts it tries to synchronise disk1 and to make Disk Group 1 a RAID1 with two disks, if I understood correctly what the interface is telling me.

However, when the rebuild reaches 100%, nothing changes and disk1 is still an hot spare.

Note that the disks are identical (WD Red 4Tb).

I don't understand what is happening, any idea about what is going wrong and how I can solve this?

Thanks in advance

Accepted Solution

  • frangio68
    frangio68 Posts: 5
    First Comment
    Answer ✓
    Options

    It did work!

    I ran

    dd if=/dev/zero of=/dev/sdb count=1 seek=7812092248
    

    and it worked flawlessly. Well, at the beginning it did show everything as red rather than orange, but then I rebooted the NAS, which automatically started the rebuilding of the RAID. I was hopeful since the log did no longer show the line

    end_request: I/O error, dev sdb, sector 7812092248

    and in fact after a night's work the rebuilding ended. I now have both disks in the RAID1 and everything is reported as green.

    Thank you very much Mijzelf, you really are a deity of NASes :-) The incident is closed.

All Replies

  • Mijzelf
    Mijzelf Posts: 2,642  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    Options

    Normally when a disk is added as spare, it is because it couldn't be added as member. In case of raid5 this means that another member went out of business while re-syncing, and leaves you with one or more healthy members, one dropped one, and one spare one. And an array which is down.

    For raid1 I would expect the same behavior, the remaining member dropped from the array, and a spare one added. But maybe I'm mistaken.

    Can you enable the ssh server, login over ssh, and post the output of

    su
    mdadm —examine /dev/sd[ab]3
    

  • frangio68
    frangio68 Posts: 5
    First Comment
    Options

    Hi.

    Thanks for taking interest in the case.

    The command "mdadm —examine /dev/sd[ab]3" does not work, but it does with "—examine"; I'm saying this only for the sake of future readers of the thread.

    The output is below. Hope this tells you something useful. Thank you in advance for any hint.

    /dev/sda3:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x0
    Array UUID : 8bfc4464:7d84b135:8fda12d9:d4194d32
    Name : NAS326:2 (local to host NAS326)
    Creation Time : Thu Mar 3 04:05:06 2016
    Raid Level : raid1
    Raid Devices : 2

    Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
    Array Size : 3902886912 (3722.08 GiB 3996.56 GB)
    Data Offset : 262144 sectors
    Super Offset : 8 sectors
    State : clean
    Device UUID : dbc7fa48:5b9a47ae:0f8bd22d:69efa45d

    Update Time : Thu Jun 29 15:33:50 2023
    Checksum : 94881cc4 - correct
    Events : 98742

    Device Role : spare
    Array State : A. ('A' == active, '.' == missing)
    /dev/sdb3:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x0
    Array UUID : 8bfc4464:7d84b135:8fda12d9:d4194d32
    Name : NAS326:2 (local to host NAS326)
    Creation Time : Thu Mar 3 04:05:06 2016
    Raid Level : raid1
    Raid Devices : 2

    Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
    Array Size : 3902886912 (3722.08 GiB 3996.56 GB)
    Data Offset : 262144 sectors
    Super Offset : 8 sectors
    State : clean
    Device UUID : 557f100f:32f8c4a4:c2640ed5:1aa179f7

    Update Time : Thu Jun 29 15:33:50 2023
    Checksum : 922bbd78 - correct
    Events : 98742

    Device Role : Active device 0
    Array State : A. ('A' == active, '.' == missing)

  • Mijzelf
    Mijzelf Posts: 2,642  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    Options

    I don't see any anomalies (except that the forum software b0rked up your output. It has added some code tags and removed some lines in return.)

    I think that you have 'Current Pending Sectors'. Smartctl should be able to tell:

    su
    smartctl -a /dev/sdb
    

    That isn't necessarily a problem. Sometimes a bit flips. When you know that a bit on a modern harddisk is only a few square nanometers in size, and there are billions of them, it is easy to understand that after times some bits are flipped, or no longer readable. Especially bits which have not been written (= refreshed) for a long time. For that reason each sector has some parity bits to restore unreadable bits. But if too many bits are unreadable, and/or flipped, the recovery fails. That doesn't mean there is something wrong with that sector. It could be perfectly fine, yet unreadable. If you would write new data to it, it could store it.

    When adding a new disk to a raid1 array, the whole surface is copied, no matter if it is in use or not. The raid array is at a lower level than the filesystem, and so it doesn't know about files, just about sectors. When it finds a 'Current Pending Sector' it doesn't know if that sector is important or not. It just stops resyncing. In case of a raid5 array the unreadable disk is dropped, and the new disk is made spare. Maybe your version of the raid manager keeps the only remaining disk in case of raid1.

    To get the array healthy again, you'll have to get rid of that unreadable sector. One way to do so is to make a bit-by-bit copy to another disk. The unreadable sector will be ignored, and the 'copy' will be all zero's, or something like that. After you have booted the system with the copy disk (convincing the system that it is the 'actual array') you can add the orignal disk. The raid manager will copy back all sectors. The unreadable sector is overwritten with fresh data, and everybody is happy.

    I have written in this thread about creating bit-by-bit copies using ddrescue. Read it, and let me know if there are questions.

  • frangio68
    frangio68 Posts: 5
    First Comment
    Options

    Hi again.

    This is the output:

    ~ # smartctl -a /dev/sdb
    smartctl 6.3 2014-07-26 r3976 [armv7l-linux-3.10.39] (local build)
    Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

    === START OF INFORMATION SECTION ===
    Model Family: Western Digital Red (AF)
    Device Model: WDC WD40EFRX-68N32N0
    Serial Number: WD-WCC7K4CURTJE
    LU WWN Device Id: 5 0014ee 2bc69cb3c
    Firmware Version: 82.00A82
    User Capacity: 4,000,787,030,016 bytes [4.00 TB]
    Sector Sizes: 512 bytes logical, 4096 bytes physical
    Rotation Rate: 5400 rpm
    Form Factor: 3.5 inches
    Device is: In smartctl database [for details use: -P show]
    ATA Version is: ACS-3 (unknown minor revision code: 0x006d)
    SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is: Sat Jul 1 15:59:12 2023 CEST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled

    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED

    General SMART Values:
    Offline data collection status: (0x00) Offline data collection activity
    was never started.
    Auto Offline Data Collection: Disabled.
    Self-test execution status: ( 0) The previous self-test routine completed
    without error or no self-test has ever
    been run.
    Total time to complete Offline
    data collection: (43080) seconds.
    Offline data collection
    capabilities: (0x7b) SMART execute Offline immediate.
    Auto Offline data collection on/off support.
    Suspend Offline collection upon new
    command.
    Offline surface scan supported.
    Self-test supported.
    Conveyance Self-test supported.
    Selective Self-test supported.
    SMART capabilities: (0x0003) Saves SMART data before entering
    power-saving mode.
    Supports SMART auto save timer.
    Error logging capability: (0x01) Error logging supported.
    General Purpose Logging supported.
    Short self-test routine
    recommended polling time: ( 2) minutes.
    Extended self-test routine
    recommended polling time: ( 457) minutes.
    Conveyance self-test routine
    recommended polling time: ( 5) minutes.
    SCT capabilities: (0x303d) SCT Status supported.
    SCT Error Recovery Control supported.
    SCT Feature Control supported.
    SCT Data Table supported.

    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
    1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 4
    3 Spin_Up_Time 0x0027 169 168 021 Pre-fail Always - 6525
    4 Start_Stop_Count 0x0032 074 074 000 Old_age Always - 26657
    5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
    7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
    9 Power_On_Hours 0x0032 061 061 000 Old_age Always - 28708
    10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
    11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
    12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 126
    192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 88
    193 Load_Cycle_Count 0x0032 192 192 000 Old_age Always - 26909
    194 Temperature_Celsius 0x0022 116 097 000 Old_age Always - 34
    196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
    197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
    198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
    199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
    200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0

    SMART Error Log Version: 1
    No Errors Logged

    SMART Self-test log structure revision number 1
    Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

    1 Short offline Completed without error 00% 0 -2 Short offline Completed without error 00% 0 -

    SMART Selective self-test log data structure revision number 1
    SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
    1 0 0 Not_testing
    2 0 0 Not_testing
    3 0 0 Not_testing
    4 0 0 Not_testing
    5 0 0 Not_testing
    Selective self-test flags (0x0):
    After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.

    It seems some of the SMART stuff is on the bad side. Does it mean I should replace the disk?

    Thanks

  • Mijzelf
    Mijzelf Posts: 2,642  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    Options

    It seems some of the SMART stuff is on the bad side. Does it mean I should replace the disk?

    Which stuff do you mean? I'm certainly not an expert, but this disk looks healthy to me.

    197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0

    There are no Current_Pending_Sectors. So that is not the cause of your troubles. The only remaining way I know to find out what is the problem is looking at the kernel log:

    dmesg | less
    

    This shows the log, (dmesg does), and you can scroll down and up using the cursor keys (less does). And I'm almost sure the forum software will screw up my code. The pipe sign in between is on my keyboard shift-\.

    The kernel log doesn't survive a reboot. So if you rebooted after the last attempt to resync the array it's not in the log. And it's cyclic. So old entries can be overwritten by new ones. Don't know how noisy a 326 is. When the first or second line is '[timestamp] Linux Version …' you have the complete log.

  • frangio68
    frangio68 Posts: 5
    First Comment
    Options

    Hi and thanks again.

    Which stuff do you mean? I'm certainly not an expert, but this disk looks healthy to me.

    Well, seeing the three "pre-fail" entries 1, 3 and 5 is not exactly reassuring, but let's move on.

    197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0

    Thera are no Current_Pending_Sectors. So that is not the cause of your
    troubles. The only remaining way I know to find out what is the problem
    is looking at the kernel log:

    dmesg | less
    

    The log is indeed rather verbose. Towards the end, however, some stuff that could be relevant shows up. It starts with

    EXT4-fs (dm-1): warning: mounting fs with errors, running e2fsck is recommended
    EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: usrquota,data=ordered,barrier=1
    EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)

    Then there are at least three instances of

    EXT4-fs (dm-1): error count: 3
    EXT4-fs (dm-1): initial error at 1683748757: ext4_mb_generate_buddy:755
    EXT4-fs (dm-1): last error at 1683748760: ext4_mb_generate_buddy:755
    ata2.00: exception Emask 0x0 SAct 0xf SErr 0x0 action 0x0
    ata2.00: irq_stat 0x40000008
    ata2.00: failed command: READ FPDMA QUEUED
    ata2.00: cmd 60/00:00:00:10:a3/04:00:d1:01:00/40 tag 0 ncq 524288 in
    res 41/40:00:58:11:a3/00:00:d1:01:00/00 Emask 0x409 (media error)
    ata2.00: status: { DRDY ERR }
    ata2.00: error: { UNC }
    ata2.00: configured for UDMA/133
    sd 1:0:0:0: [sdb] Unhandled sense code
    sd 1:0:0:0: [sdb]Result: hostbyte=0x00 driverbyte=0x08
    sd 1:0:0:0: [sdb]Sense Key : 0x3 [current] [descriptor]
    Descriptor sense data with sense descriptors (in hex):
    72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 01
    d1 a3 11 58
    sd 1:0:0:0: [sdb]ASC=0x11 ASCQ=0x4
    sd 1:0:0:0: [sdb] CDB:
    cdb[0]=0x88: 88 00 00 00 00 01 d1 a3 10 00 00 00 04 00 00 00
    end_request: I/O error, dev sdb, sector 7812092248
    ata2: EH complete

    which seem to culminate with

    RAID1 conf printout:
    --- wd:1 rd:2
    disk 0, wo:0, o:1, dev:sdb3
    disk 1, wo:1, o:1, dev:sda3
    RAID1 conf printout:
    --- wd:1 rd:2
    disk 0, wo:0, o:1, dev:sdb3
    EXT4-fs (dm-1): error count: 3
    EXT4-fs (dm-1): initial error at 1683748757: ext4_mb_generate_buddy:755
    EXT4-fs (dm-1): last error at 1683748760: ext4_mb_generate_buddy:755
    EXT4-fs (dm-1): error count: 3
    EXT4-fs (dm-1): initial error at 1683748757: ext4_mb_generate_buddy:755
    EXT4-fs (dm-1): last error at 1683748760: ext4_mb_generate_buddy:755

    Taking a clue from the first lines I also tried

    ~ # e2fsck /dev/sdb

    but I got

    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdb is in use.
    e2fsck: Cannot continue, aborting.

    If any of this rings a bell …

    Thanks

  • Mijzelf
    Mijzelf Posts: 2,642  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    Options

    Well, seeing the three "pre-fail" entries 1, 3 and 5 is not exactly reassuring, but let's move on.

    'Pre-fail' is not a verdict, it's a category. There are 'Pre-fail' and Old_age' values, the first category means the value says something about a nearby fail, the old age values just keeps track of operating hours and such.

    It's the value which counts. A Spin_Up_Time of 6525 (msec I think) can be good or bad, hard to say. But the vendor knows, and he gives it a 169 out of 200, and tells you should be worried when it falls below 21.

    end_request: I/O error, dev sdb, sector 7812092248

    This is definitely a problem. Raid rebuilding will stop here. And as the disk has about 7814037168 sectors, that is less than a % from the end, so the rebuild will go on to 100%, and then fail.

    Strange that this doesn't show up in SMART as Current_Pending_Sector or Reallocated_Sector_Ct. Although it could be one of the 4 Raw_Read_Error_Rate.

    This can be solved in several ways.

    • You can shrink to raid array+embedded logical volume+embedded filesystem by 2‰, putting the sector outside the array.
    • You can create a bit-by-bit copy of that disk, as described before.
    • You can manually overwrite that sector. If it's a soft-error the sector will be fine after that, if it's a hard error the sector will be replaced by a spare one, and Reallocated_Sector_Ct will go up to 1.
    dd if=/dev/zero of=/dev/sdb count=1 seek=7812092248
    

    There are 2 problems, however. I don't know if the busybox dd on the box can handle that big numbers (it has to be a 64 bits value, 32 bits goes to 4000000000), and I don't know if the sector in dmesg is 0 or 1 based. But you can first try the reverse, copy the sector from disk:

    dd if=/dev/sdb of=/dev/null count=1 seek=7812092248 
    

    That should give an I/O error, it should show up in dmesg, and probably the Raw_Read_Error_Rate is incremented. If it doesn't, try

    dd if=/dev/sdb of=/dev/null count=1 seek=7812092247
    

    or …49

    If both don't work, busybox can't handle this. If you have found the right sector number you know how to overwrite it.

    EXT4-fs (dm-1): error count: 3

    While an unreadable sector can cause filesystem errors, the filesystem error cannot cause raid rebuilding problems. The raid array is below the filesystem.

    ~ # e2fsck /dev/sdb

    The filesystem is not on the raw disk surface, but on the logical volume (probably /dev/dm-1) on the raid array (/dev/md2) on 3th partition (/dev/sdb3). So the command should be

    e2fsck /dev/dm-1
    

    which will also fail, because the filesystem is mounted. Elsewhere in this forum I have written about how to run e2fsck on the data volume of a ZyXEL NAS.

  • Mijzelf
    Mijzelf Posts: 2,642  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    Options

    On second thought, the command to read a sector is

    dd if=/dev/sdb of=/dev/null count=1 skip=7812092247
    

    man dd

  • frangio68
    frangio68 Posts: 5
    First Comment
    Answer ✓
    Options

    It did work!

    I ran

    dd if=/dev/zero of=/dev/sdb count=1 seek=7812092248
    

    and it worked flawlessly. Well, at the beginning it did show everything as red rather than orange, but then I rebooted the NAS, which automatically started the rebuilding of the RAID. I was hopeful since the log did no longer show the line

    end_request: I/O error, dev sdb, sector 7812092248

    and in fact after a night's work the rebuilding ended. I now have both disks in the RAID1 and everything is reported as green.

    Thank you very much Mijzelf, you really are a deity of NASes :-) The incident is closed.

Consumer Product Help Center