one-disk RAID1 gets an hot spare when second disk added, tries and fail to become two-disks RAID1

frangio68

Hi.

I had a NAS326 where one of the two disks was showing bad stats in SMART. So I bought a new one, pulled out the defective one and replaced it.

Initially the NAS was doing a lot of beeping and I could not even access the web interface. I pulled out the new disk and was able to access with just the original one.

Then I added the new disk and things seemed to work. However, it is reporting that the Disk Group 1 is degraded. Looking at it, it says that the volume is made by just disk2. As you see in the screenshot,

disk1 is still in Disk Group 1, but as an hot spare rather than in the RAID1, which I think is what should have happened.

In fact when the NAS restarts it tries to synchronise disk1 and to make Disk Group 1 a RAID1 with two disks, if I understood correctly what the interface is telling me.

However, when the rebuild reaches 100%, nothing changes and disk1 is still an hot spare.

Note that the disks are identical (WD Red 4Tb).

I don't understand what is happening, any idea about what is going wrong and how I can solve this?

Thanks in advance

frangio68

It did work!

I ran

dd if=/dev/zero of=/dev/sdb count=1 seek=7812092248

and it worked flawlessly. Well, at the beginning it did show everything as red rather than orange, but then I rebooted the NAS, which automatically started the rebuilding of the RAID. I was hopeful since the log did no longer show the line

end_request: I/O error, dev sdb, sector 7812092248

and in fact after a night's work the rebuilding ended. I now have both disks in the RAID1 and everything is reported as green.

Thank you very much Mijzelf, you really are a deity of NASes :-) The incident is closed.

Mijzelf

Normally when a disk is added as spare, it is because it couldn't be added as member. In case of raid5 this means that another member went out of business while re-syncing, and leaves you with one or more healthy members, one dropped one, and one spare one. And an array which is down.

For raid1 I would expect the same behavior, the remaining member dropped from the array, and a spare one added. But maybe I'm mistaken.

Can you enable the ssh server, login over ssh, and post the output of

su
mdadm —examine /dev/sd[ab]3

frangio68

Hi.

Thanks for taking interest in the case.

The command "mdadm —examine /dev/sd[ab]3" does not work, but it does with "—examine"; I'm saying this only for the sake of future readers of the thread.

The output is below. Hope this tells you something useful. Thank you in advance for any hint.

/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 8bfc4464:7d84b135:8fda12d9:d4194d32
Name : NAS326:2 (local to host NAS326)
Creation Time : Thu Mar 3 04:05:06 2016
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
Array Size : 3902886912 (3722.08 GiB 3996.56 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : dbc7fa48:5b9a47ae:0f8bd22d:69efa45d

Update Time : Thu Jun 29 15:33:50 2023

   Checksum : 94881cc4 - correct

     Events : 98742

Device Role : spare
Array State : A. ('A' == active, '.' == missing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 8bfc4464:7d84b135:8fda12d9:d4194d32
Name : NAS326:2 (local to host NAS326)
Creation Time : Thu Mar 3 04:05:06 2016
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
Array Size : 3902886912 (3722.08 GiB 3996.56 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 557f100f:32f8c4a4:c2640ed5:1aa179f7

Update Time : Thu Jun 29 15:33:50 2023

   Checksum : 922bbd78 - correct

     Events : 98742

Device Role : Active device 0
Array State : A. ('A' == active, '.' == missing)

Mijzelf

I don't see any anomalies (except that the forum software b0rked up your output. It has added some code tags and removed some lines in return.)

I think that you have 'Current Pending Sectors'. Smartctl should be able to tell:

su
smartctl -a /dev/sdb

That isn't necessarily a problem. Sometimes a bit flips. When you know that a bit on a modern harddisk is only a few square nanometers in size, and there are billions of them, it is easy to understand that after times some bits are flipped, or no longer readable. Especially bits which have not been written (= refreshed) for a long time. For that reason each sector has some parity bits to restore unreadable bits. But if too many bits are unreadable, and/or flipped, the recovery fails. That doesn't mean there is something wrong with that sector. It could be perfectly fine, yet unreadable. If you would write new data to it, it could store it.

When adding a new disk to a raid1 array, the whole surface is copied, no matter if it is in use or not. The raid array is at a lower level than the filesystem, and so it doesn't know about files, just about sectors. When it finds a 'Current Pending Sector' it doesn't know if that sector is important or not. It just stops resyncing. In case of a raid5 array the unreadable disk is dropped, and the new disk is made spare. Maybe your version of the raid manager keeps the only remaining disk in case of raid1.

To get the array healthy again, you'll have to get rid of that unreadable sector. One way to do so is to make a bit-by-bit copy to another disk. The unreadable sector will be ignored, and the 'copy' will be all zero's, or something like that. After you have booted the system with the copy disk (convincing the system that it is the 'actual array') you can add the orignal disk. The raid manager will copy back all sectors. The unreadable sector is overwritten with fresh data, and everybody is happy.

I have written in this thread about creating bit-by-bit copies using ddrescue. Read it, and let me know if there are questions.

frangio68

Hi again.

This is the output:

~ # smartctl -a /dev/sdb
smartctl 6.3 2014-07-26 r3976 [armv7l-linux-3.10.39] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (AF)
Device Model: WDC WD40EFRX-68N32N0
Serial Number: WD-WCC7K4CURTJE
LU WWN Device Id: 5 0014ee 2bc69cb3c
Firmware Version: 82.00A82
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 (unknown minor revision code: 0x006d)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jul 1 15:59:12 2023 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (43080) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 457) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 4
3 Spin_Up_Time 0x0027 169 168 021 Pre-fail Always - 6525
4 Start_Stop_Count 0x0032 074 074 000 Old_age Always - 26657
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 061 061 000 Old_age Always - 28708
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 126
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 88
193 Load_Cycle_Count 0x0032 192 192 000 Old_age Always - 26909
194 Temperature_Celsius 0x0022 116 097 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 0 -2 Short offline Completed without error 00% 0 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

It seems some of the SMART stuff is on the bad side. Does it mean I should replace the disk?

Thanks

Mijzelf

It seems some of the SMART stuff is on the bad side. Does it mean I should replace the disk?

Which stuff do you mean? I'm certainly not an expert, but this disk looks healthy to me.

197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0

There are no Current_Pending_Sectors. So that is not the cause of your troubles. The only remaining way I know to find out what is the problem is looking at the kernel log:

dmesg | less

This shows the log, (dmesg does), and you can scroll down and up using the cursor keys (less does). And I'm almost sure the forum software will screw up my code. The pipe sign in between is on my keyboard shift-\.

The kernel log doesn't survive a reboot. So if you rebooted after the last attempt to resync the array it's not in the log. And it's cyclic. So old entries can be overwritten by new ones. Don't know how noisy a 326 is. When the first or second line is '[timestamp] Linux Version …' you have the complete log.

frangio68

Hi and thanks again.

Which stuff do you mean? I'm certainly not an expert, but this disk looks healthy to me.

Well, seeing the three "pre-fail" entries 1, 3 and 5 is not exactly reassuring, but let's move on.

197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0

Thera are no Current_Pending_Sectors. So that is not the cause of your
troubles. The only remaining way I know to find out what is the problem
is looking at the kernel log:

dmesg | less

The log is indeed rather verbose. Towards the end, however, some stuff that could be relevant shows up. It starts with

EXT4-fs (dm-1): warning: mounting fs with errors, running e2fsck is recommended
EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: usrquota,data=ordered,barrier=1
EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)

Then there are at least three instances of

EXT4-fs (dm-1): error count: 3
EXT4-fs (dm-1): initial error at 1683748757: ext4_mb_generate_buddy:755
EXT4-fs (dm-1): last error at 1683748760: ext4_mb_generate_buddy:755
ata2.00: exception Emask 0x0 SAct 0xf SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000008
ata2.00: failed command: READ FPDMA QUEUED
ata2.00: cmd 60/00:00:00:10:a3/04:00:d1:01:00/40 tag 0 ncq 524288 in
res 41/40:00:58:11:a3/00:00:d1:01:00/00 Emask 0x409 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
sd 1:0:0:0: [sdb] Unhandled sense code
sd 1:0:0:0: [sdb]Result: hostbyte=0x00 driverbyte=0x08
sd 1:0:0:0: [sdb]Sense Key : 0x3 [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 01
d1 a3 11 58
sd 1:0:0:0: [sdb]ASC=0x11 ASCQ=0x4
sd 1:0:0:0: [sdb] CDB:
cdb[0]=0x88: 88 00 00 00 00 01 d1 a3 10 00 00 00 04 00 00 00
end_request: I/O error, dev sdb, sector 7812092248
ata2: EH complete

which seem to culminate with

RAID1 conf printout:
--- wd:1 rd:2
disk 0, wo:0, o:1, dev:sdb3
disk 1, wo:1, o:1, dev:sda3
RAID1 conf printout:
--- wd:1 rd:2
disk 0, wo:0, o:1, dev:sdb3
EXT4-fs (dm-1): error count: 3
EXT4-fs (dm-1): initial error at 1683748757: ext4_mb_generate_buddy:755
EXT4-fs (dm-1): last error at 1683748760: ext4_mb_generate_buddy:755
EXT4-fs (dm-1): error count: 3
EXT4-fs (dm-1): initial error at 1683748757: ext4_mb_generate_buddy:755
EXT4-fs (dm-1): last error at 1683748760: ext4_mb_generate_buddy:755

Taking a clue from the first lines I also tried

~ # e2fsck /dev/sdb

but I got

e2fsck 1.42.12 (29-Aug-2014)
/dev/sdb is in use.
e2fsck: Cannot continue, aborting.

If any of this rings a bell …

Thanks

Mijzelf

Well, seeing the three "pre-fail" entries 1, 3 and 5 is not exactly reassuring, but let's move on.

'Pre-fail' is not a verdict, it's a category. There are 'Pre-fail' and Old_age' values, the first category means the value says something about a nearby fail, the old age values just keeps track of operating hours and such.

It's the value which counts. A Spin_Up_Time of 6525 (msec I think) can be good or bad, hard to say. But the vendor knows, and he gives it a 169 out of 200, and tells you should be worried when it falls below 21.

end_request: I/O error, dev sdb, sector 7812092248

This is definitely a problem. Raid rebuilding will stop here. And as the disk has about 7814037168 sectors, that is less than a % from the end, so the rebuild will go on to 100%, and then fail.

Strange that this doesn't show up in SMART as Current_Pending_Sector or Reallocated_Sector_Ct. Although it could be one of the 4 Raw_Read_Error_Rate.

This can be solved in several ways.

• You can shrink to raid array+embedded logical volume+embedded filesystem by 2‰, putting the sector outside the array.
• You can create a bit-by-bit copy of that disk, as described before.
• You can manually overwrite that sector. If it's a soft-error the sector will be fine after that, if it's a hard error the sector will be replaced by a spare one, and Reallocated_Sector_Ct will go up to 1.

dd if=/dev/zero of=/dev/sdb count=1 seek=7812092248

There are 2 problems, however. I don't know if the busybox dd on the box can handle that big numbers (it has to be a 64 bits value, 32 bits goes to 4000000000), and I don't know if the sector in dmesg is 0 or 1 based. But you can first try the reverse, copy the sector from disk:

dd if=/dev/sdb of=/dev/null count=1 seek=7812092248 

That should give an I/O error, it should show up in dmesg, and probably the Raw_Read_Error_Rate is incremented. If it doesn't, try

dd if=/dev/sdb of=/dev/null count=1 seek=7812092247

or …49

If both don't work, busybox can't handle this. If you have found the right sector number you know how to overwrite it.

EXT4-fs (dm-1): error count: 3

While an unreadable sector can cause filesystem errors, the filesystem error cannot cause raid rebuilding problems. The raid array is below the filesystem.

~ # e2fsck /dev/sdb

The filesystem is not on the raw disk surface, but on the logical volume (probably /dev/dm-1) on the raid array (/dev/md2) on 3th partition (/dev/sdb3). So the command should be

e2fsck /dev/dm-1

which will also fail, because the filesystem is mounted. Elsewhere in this forum I have written about how to run e2fsck on the data volume of a ZyXEL NAS.

Mijzelf

On second thought, the command to read a sector is

dd if=/dev/sdb of=/dev/null count=1 skip=7812092247

man dd

frangio68

It did work!

I ran

dd if=/dev/zero of=/dev/sdb count=1 seek=7812092248

and it worked flawlessly. Well, at the beginning it did show everything as red rather than orange, but then I rebooted the NAS, which automatically started the rebuilding of the RAID. I was hopeful since the log did no longer show the line

end_request: I/O error, dev sdb, sector 7812092248

and in fact after a night's work the rebuilding ended. I now have both disks in the RAID1 and everything is reported as green.

Thank you very much Mijzelf, you really are a deity of NASes :-) The incident is closed.