NAS540 - Can any data be recovered?

Mijzelf

The physical sequence does not really matter. What matters is the role the disks have in the array. A four disk raid5 array has four different roles, numbered from 0 to 3. When you create such an array by firmware, the physical sequence is used.

Normally the disks are from left to right called sda, sdb, sdc and sdd. When you have an SD card inserted a boot (or maybe an USB thumb disk) it is shifted, and in your case it's sdb-sde.

The third partition on each disk is used for the data volume, so that is sdb3-sde3.

We recreated the array, as it's (AFAIK) not possible to force-reinsert a member to an existing array, while your array was down due to too less members left. So one or more members had to be force-reinserted.

The command to create the array is

mdadm --create <options> mdX role0 role1 role2 role3

where mdX is the device name of the array to be created, and role0-role3 are the device names of the members with that role. One of these may be 'missing', to create a degraded array.

We know the roles of sdc3 and sdd3 (role1 and 2), as they were listed in the 'mdadm --examine' in the beginning of this thread. sdb3 didn't have a valid raid header, and sde3 was not listed because an sda was injected, but I assumed it to be role3.

Now things get complicated. The initial created array didn't give a valid filesystem, and I assumed that was because sdb3 was actually empty. But maybe it was actually sde3 which was empty, and the non existing raid header of sdb3 was a result of the crash.

So we tried both 'missing /dev/sdc3 /dev/sdd3 /dev/sde3' and '/dev/sdb3 /dev/sdc3 /dev/sdd3 missing'. But if you swapped the disks, that should have been 'missing /dev/sdc3 /dev/sdd3 /dev/sdb3' and and '/dev/sde3 /dev/sdc3 /dev/sdd3 missing'.

If you now swapped them back again, the original sequence works again after re-creating the array. You can swap the disks of a created array without problems, as the raid manager reads the headers. (Of course it becomes more complicated to repair things, if you don't know the logical sequence of the disks. So it's better to keep the logical and physical sequence the same.)

When you pull the left disk, all other disks shift after a reboot. sdc becomes sdb. So that is also something to keep in mind. And if you pull your SD card, everything also shifts.

The command 'cat /proc/mdstat'  shows the active raid arrays. On a NAS5xx there are normally at least 3 of them, one swap array, one firmware array (md0 and md1, don't know which is what) and one or more data arrays, md2 and higher. So you can ignore md0 and md1. (Although /proc/mdstat in this case tells they have 4 members, meaning all disks at least partly work, which is nice to know)

You can see in /proc/mdstat that md0 is also a raid5 array. It is absolutely possible that their roles are identical to the roles in md2. That could be a way to find the original sequence of the disks:

mdadm --examine /dev/sd[bcde]1

Problem here is that it is possible to think of scenarios which make their sequence different. For instance exchanging the disks (md0 is still fine, but the physical sequence is different from the logical) and then create a new data volume. Boom! Different logical sequence.

I don't know what 'resource is busy'  means here. Assuming you are trying to create the array, I think that either the array device, or one of the members is busy. For the array device you can do a

mdadm --stop /dev/md2

for the members I don't know. It depends on how they are busy. For the raid device you can also use /dev/md3 or higher, as these are certainly not busy.

I hope I made clear how things work, that should make it easier to understand what we (and mainly you) are doing.

octhrope

This is great information and thank you for writing all of it for the community at large. 

For me i think im out of options, i guess the lesson is learned.  Unless you have any other suggestions?

Thank you for all your time and energy on this, I really appreciate all the information and assistance. Wouldn't have made it this far without you.