NAS542 lost RAID5 after repair

Mijzelf

This forum software is amazing. It has cut 5 lines for each disk from your listing. (Here you can see a complete listing), the lines 'Update Time' to 'Chunk Size' are missing for each disk, and exchange by a grey block. Beats me.

Fortunately is doesn't matter for diagnosis, although the 'Update Time' can give some extra information for investigation.

The 2nd disk is still the new one, I suppose? This is a summary of the array state:

/dev/sda3:

Device Role : Active device 0

Array State : AAAA ('A' == active, '.' == missing)

/dev/sdb3:

Device Role : spare

Array State : ..AA ('A' == active, '.' == missing)

/dev/sdc3:

Device Role : Active device 2

Array State : ..AA ('A' == active, '.' == missing)

/dev/sdd3:

Device Role : Active device 3

Array State : ..AA ('A' == active, '.' == missing)

A healthy 4 disk raid5 array has 4 members 'Active device 0…3', and each member has a local array status of 'AAAA'. IIn your case disk 1 has state 'AAAA', while the others are '..AA'. This means disk 1 is dropped while the other 3 disks were still available. Then you exchanged disk 2, which couldn't be added to the array, as only 2 members were left. For that reason it's spare. Disks 3 and 4 are still valid.

It would be nice if disk 1 could be inserted to the array again, unfortunately I don't know how. But it is possible to create a new array while keeping the current content. To do so you have to execute next commands:

su
mdadm ++stop /dev/md2
mdadm ++create ++assume-clean ++level=5 ++raid-devices=4 ++metadata=1.2 ++chunk=64K ++layout=left-symmetric /dev/md2 /dev/sda3 missing /dev/sdc3 /dev/sdd3

Every '+' should be exchange by a 'minus'. This will create a degraded array with member 2 missing. The 2nd command (stop) could error out, that is no problem.

indiana1938

Thanks a lot for your kind support.

Just to be save: shall I leave defective HDD #2 in the NAS or shall I remove it physically from NAS housing before executing mentioned commands?

Mijzelf

Leave it. If you remove it the device node will shift, and another create command is needed.

Defective? This is not the new disk? (Not that it matters, mdadm will ignore it when creating the degraded array)

indiana1938

Ok, got following message:

mdadm: array /dev/md2 started.

Now I started web interface and got following message:

After klicking "OK", following window appears:

Here is the window with volume information:

And the last one with information about Disk Group:

Shall I replace the defective HDD by the new one or what shall I do?

Sorry, but I'm a bloody noob on this story and don't want to do anything wrong.

Mijzelf

The original volume is gone, as the firmware recognizes the volume by the UUID of the raid array. And that is changed.

I think the firmware will show the new volume after a reboot. And yes, you can exchange the defective disk.

indiana1938

Ok, here is the current status:

After reboot the NAS told me, that there is a degraded RAID and offered to repair it.

I accepted and after the repair process was finished, the RAID was …. gone again.
No RAID, no volume.

Therefore I made a reboot again and the RAID was still gone.

I made the analysis again via SSH and here is what the NAS returned:

mdadm --examine /dev/sd[abcd]3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1009ded:cd1c154e:a47066f1:401f45dd
Name : NAS542:2 (local to host NAS542)
Creation Time : Tue Mar 14 15:04:05 2023
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)
Array Size : 5848150464 (5577.23 GiB 5988.51 GB)
Used Dev Size : 3898766976 (1859.08 GiB 1996.17 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3ee637ac:f9bd7b55:62bbd5d1:1ce143d0

Update Time : Tue Mar 14 16:05:44 2023

   Checksum : 578faca9 - correct

     Events : 146


     Layout : left-symmetric

 Chunk Size : 64K

Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1009ded:cd1c154e:a47066f1:401f45dd
Name : NAS542:2 (local to host NAS542)
Creation Time : Tue Mar 14 15:04:05 2023
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)
Array Size : 5848151040 (5577.23 GiB 5988.51 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 97c9e64f:9de1443b:47673189:cd2f3fb5

Update Time : Tue Mar 14 16:13:47 2023

   Checksum : 7d60b1aa - correct

     Events : 158


     Layout : left-symmetric

 Chunk Size : 64K

Device Role : spare
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1009ded:cd1c154e:a47066f1:401f45dd
Name : NAS542:2 (local to host NAS542)
Creation Time : Tue Mar 14 15:04:05 2023
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)
Array Size : 5848151040 (5577.23 GiB 5988.51 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 796d50b5:0a226a8b:cce6d797:449fd317

Update Time : Tue Mar 14 16:13:47 2023

   Checksum : a42a84f3 - correct

     Events : 158


     Layout : left-symmetric

 Chunk Size : 64K

Device Role : Active device 2
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1009ded:cd1c154e:a47066f1:401f45dd
Name : NAS542:2 (local to host NAS542)
Creation Time : Tue Mar 14 15:04:05 2023
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)
Array Size : 5848151040 (5577.23 GiB 5988.51 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c9565907:5578146d:9d6dfac8:5fc606de

Update Time : Tue Mar 14 16:13:47 2023

   Checksum : cf33727b - correct

     Events : 158


     Layout : left-symmetric

 Chunk Size : 64K

Device Role : Active device 3
Array State : ..AA ('A' == active, '.' == missing)

Any further idea? I saved the information frm NAS also in a text file for your reference.

NAS542_Analysis.txt

Mijzelf

The first disk is dropped again. I suppose there is some sector unreadable, and while resyncing disk 2 it has to be read. That fails, and the disk is dropped.

You have 4 new disk on the shelf, don't you? I suggest you to create a bit-by-bit copy from disk 1 to an empty, new one. Then re-create the array degraded with the new disk 1, like you did before.

To do so, remove the other 3 disks (and label them), and insert a new one in slot 2. Execute

cat /proc/partitions

to find the disk nodes. The command shows all block devices, and you should see the old disk with three partitions, and the new disk without. for instance sda, sda1, sda2, sda3, and sdb without partitions. I expect the old disk to be sda, and the new one sdb.

Then I send you a download link to a tool ddrescue. Put that in /tmp using WinSCP. Then execute

chmod a+x /tmp/ddrescue.armv7l
su
/tmp/ddrescue.armv7l /dev/sda /dev/sdb

This will copy all data from disk sda to disk sdb. Doublecheck that sdb is the new disk, and that the command line is OK. Ddrescue will happily overwrite anything you offer as target.

I think the copying will be done at 80~100MB/sec, so you can calculate how long it will take. You'll need to keep the ssh client open during that time.

indiana1938

Here is the result of cat /proc/partitions:

major minor #blocks name

7 0 146432 loop0
31 0 256 mtdblock0
31 1 512 mtdblock1
31 2 256 mtdblock2
31 3 10240 mtdblock3
31 4 10240 mtdblock4
31 5 112640 mtdblock5
31 6 10240 mtdblock6
31 7 112640 mtdblock7
31 8 6144 mtdblock8
8 0 1953514584 sda
8 1 1998848 sda1
8 2 1999872 sda2
8 3 1949514752 sda3
8 16 1953514584 sdb
31 9 102424 mtdblock9
9 0 1997760 md0
9 1 1998784 md1
31 10 4464 mtdblock10

So is command " /tmp/ddrescue.armv7l /dev/sda /dev/sdb " correct?

And thanks for the link. :-)

Mijzelf

Correct. sda has 3 partitions, sdb has none. So sda is the old disk, sdb the new one.

indiana1938

Just tried to execute " chmod a+x /tmp/ddrescue.armv7l " and got following message:

chmod: /tmp/ddrescue.armv7l: No such file or directory

What's wrong?

I copied the file to /tmp directory by using WinSCP.