NSA325 degraded RAID1 after replacing HDD "impossible" to repair.

Paolo DeF
Paolo DeF Posts: 11  Freshman Member
First Comment Friend Collector Third Anniversary

Hi everyone.

I bougth the NSA325 NAS some 10+ years ago. At first I used 2 HDD, 2TB each, in RAID1. Let's call them VeryOld1 (VO1) and VeryOld2(VO2), model ST2000VN000. 3 years ago, VO2 stopped working, so I replaced it with a new HDD, called Old1 (O1), 4TB, model ST4000VN008. I just put it in the bay, turned on the NAS, the system recognized a degraded RAID1, repaired it, and I got myself a fully functional 2TB RAID1 system again.

Few weeks ago, I bought a new HDD, 4TB, model ST4000NE00 - New1 (N1). The idea was to remove VO1, put in N1 with O1, rebuild the RAID1 and expand the volume to 4TB. Please note both VO1 and O1 are perfectly fine, according to SMART.

First (1: O1+N1) attempt: it failed. RAID was not even degraded, but DOWN. No access to the content from exported shares. After few tries, I turned off the NAS, removed N1, put back in VO1, and turned on (2: VO1+O1). The system recognized a DEGRADED RAID1, but failed to repair it. I had access to the content via NFS export.
I checked online but was not able to repair it. So I made another attempt: keep VO1, remove O1, put in N1. (3: VO1+N1).
System found a degraded RAID, and repaired! VO1 and N1 working.So I thought I had it. Turned it off, removed VO1, put in O1 instead. (4: O1+N1): RAID DOWN. Same result as (1).

I sweared, removed O1, put back in VO1. Same configuration as (3), that was working fine: RAID DEGRADED. Repair failed.Sweared again.

Now I'm with O1+V1. The system sees only disk2 in RAID. I'm attaching a few outputs.

Can anyone help me? I guess there is something wrong with the partitions (or with my head…). I thought of formatting O1, try to rebuild the RAID with VO1+O1 and, if it works, format N1, rebuild the RAID with VO1+N1, and finally try to put together O1+N1. But I really hope there is a smarter way…

Cheers,

Paolo

mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Jan 10 14:46:48 2014
     Raid Level : raid1
     Array Size : 1952996792 (1862.52 GiB 1999.87 GB)
  Used Dev Size : 1952996792 (1862.52 GiB 1999.87 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Mon Mar 18 17:47:50 2024
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : NSA325:0  (local to host NSA325)
           UUID : c37b646d:3f15bc83:6dbb0bd8:bc6152d8
         Events : 11894640

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       8       18        1      active sync   /dev/sdb2

fdisk -l

Disk /dev/sda: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xxxxxxx9

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          64      514048+   8  AIX
/dev/sda2              65      243201  1952997952+  20  Unknown

Disk /dev/sdb: 4000.7 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xxxxxxxc

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1          64      514048+  83  Linux
/dev/sdb2              65      267349  2146966762+  20  Unknown
mdadm --examine /dev/sd[ab]2
/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c37b646d:3f15bc83:6dbb0bd8:bc6152d8
           Name : NSA325:0  (local to host NSA325)
  Creation Time : Fri Jan 10 14:46:48 2014
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1952996928 (1862.52 GiB 1999.87 GB)
     Array Size : 1952996792 (1862.52 GiB 1999.87 GB)
  Used Dev Size : 1952996792 (1862.52 GiB 1999.87 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : d4e91f24:03a4c98f:7e4ad766:d3ce6ad0

    Update Time : Sun Mar 17 16:37:53 2024
       Checksum : 246c25ce - correct
         Events : 11890526


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c37b646d:3f15bc83:6dbb0bd8:bc6152d8
           Name : NSA325:0  (local to host NSA325)
  Creation Time : Fri Jan 10 14:46:48 2014
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 2146965738 (2047.51 GiB 2198.49 GB)
     Array Size : 1952996792 (1862.52 GiB 1999.87 GB)
  Used Dev Size : 1952996792 (1862.52 GiB 1999.87 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f488f869:d666e828:7fc1e4bb:9a493b90

    Update Time : Mon Mar 18 17:48:15 2024
       Checksum : 2f6360ef - correct
         Events : 11894642


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)

Accepted Solution

  • Mijzelf
    Mijzelf Posts: 2,790  Guru Member
    250 Answers 2500 Comments Friend Collector Seventh Anniversary
    Answer ✓

    That's weird. I see no real reason why this wouldn't work. I wouldn't expect the 'Avail Dev Size' to be different on both disks. The new one is supposed to be a clone of the old one, including partition table. But AFAIK this shouldn't be a problem for the raid array.

    You won't be able to enlarge this array to 4TB, when you manage to get 2 4TB disks in sync. The partition table is MBR, which means it can't define partitions >2TB. When you start over, (and wipe the current partition table?) it should create a GPT partition table, which allows bigger partitions.

All Replies

  • Mijzelf
    Mijzelf Posts: 2,790  Guru Member
    250 Answers 2500 Comments Friend Collector Seventh Anniversary
    Answer ✓

    That's weird. I see no real reason why this wouldn't work. I wouldn't expect the 'Avail Dev Size' to be different on both disks. The new one is supposed to be a clone of the old one, including partition table. But AFAIK this shouldn't be a problem for the raid array.

    You won't be able to enlarge this array to 4TB, when you manage to get 2 4TB disks in sync. The partition table is MBR, which means it can't define partitions >2TB. When you start over, (and wipe the current partition table?) it should create a GPT partition table, which allows bigger partitions.

  • Paolo DeF
    Paolo DeF Posts: 11  Freshman Member
    First Comment Friend Collector Third Anniversary

    Hi Mijzef, thanks a lot. I'm puzzled as well. I'm taking the proper set of actions is:

    1. Leave only N1 (new 4TB) in the NAS. Delete volume/reset HDD (Would it be better to do it from terminal? which command do you recommend? Or it would even better to recreate manually the proper partions? How?)
    2. Put again VO1 (very old 2TB) in the NAS with the "emptied" N1
    3. Rebuild RAID (hopefully it works… at this point, how can I check the partition has been recreated as GPT in N1?)
    4. Remove both, insert only O1 (old 4TB) . Delete volume/reset HDD (same as point #1)
    5. Insert N1 with O1
    6. Keep fingers crossed

    If this doesn't work… Move VO1 in the PC, mount it, put O1 and N1 in the NAS, delete volume, create a "clean" 4TB RAID1, copy from VO1. Maybe it could be even faster like this, but I'm worried I could loose all the groups, shares, exported and so on…

    What do you think?

    Cheers, Paolo

  • Mijzelf
    Mijzelf Posts: 2,790  Guru Member
    250 Answers 2500 Comments Friend Collector Seventh Anniversary

    AFAIK the firmware is supposed to clone the partition table of the remaining raid member. So when doing your 1,2 strategy disk N1 will not get a GPT.

    I think the best approach is to put only N1 into the box, wipe the partition table

    dd if=/dev/zero of=/dev/sda bs=1M count=1
    

    reboot, to get the non-existing table active, then create a single disk raid1 array. You can see if it's GPT as the volume is >2TB.

    Then insert VO1, with some luck it will be recognized and mounted as degraded array. Copy your files over to the array on N1. Better use the file browser or command line, using Samba will down- and upload everthing.

    And finally insert O1 and add it to the single disk raid1 array, making it redundant.

    There might be a problem, if you are using ssh now, you will loose it as soon as you pull the current volume. The ssh server is an installable package, and is installed on that volume. So you'll have to use the telnet backdoor instead.

  • Paolo DeF
    Paolo DeF Posts: 11  Freshman Member
    First Comment Friend Collector Third Anniversary
    edited March 19

    Hi, thank you again. There is just 1 thing that worries me. VO1 is the only disk I'm able to use to retrieve data from (in the NAS). If i put only O1, N1 or both, I'm not able to retrieve data from interface or export. No folders or anything. RAID is DOWN, Disk Usage = N/A. There are no active packages and such.

    I'm worried that, if I insert VO1 within the newly rebuilt N1, I could loose the content of VO1. I Backed up most of the important stuff, but don't have room enough for everything. "with some luck" sounds ominous to me 😁

    Also, I'm guessing I'd loose groups, shares, NFS exports… is that correct?

    In the meantime, I started what you suggested:

    dd if=/dev/zero of=/dev/sda bs=1M count=1
    1+0 records in
    1+0 records out
    1048576 bytes (1.0MB) copied, 0.005526 seconds, 181.0MB/s

    Restarted, and there are no volumes. But I can't create a single disk RAID1 array, only JBOD is available

    I created a JBOD volume just to be sure: 3,58 TB.

    No packages available (and retrieve from internet is not working; but it was to be expected, I found your old posts about this).

    Not sure how to proceed further.

    Sorry for the flooding of question and doubts and having you guide me step by step.

  • Mijzelf
    Mijzelf Posts: 2,790  Guru Member
    250 Answers 2500 Comments Friend Collector Seventh Anniversary

    There is just 1 thing that worries me. VO1 is the only disk I'm able to use to retrieve data from (in the NAS). If i put only O1, N1 or both, I'm not able to retrieve data from interface or export. No folders or anything. RAID is DOWN, Disk Usage = N/A.

    I suppose somehow the raid manager chokes in O1 and N1. If you plugin O1 next to N1, you can try to assemble and mount the array manually.

    Assuming O1 is sdb, the commands to do so are:

    mdadm -A /dev/md1 /dev/sdb2 —run
    mkdir /tmp/mountpoint
    mount /dev/md1 /tmp/mountpoint
    

    Now you should be able to see your data in /tmp/mountpoint

    ls /tmp/mountpoint/
    

    It is possible that the firmware dives in and moves the mountpoint. In that case you should be able to see it in the filebrowser and/or sambe shares.

    You can check if the disk is sdb with

    cat /proc/partitions
    

    if sdb2 is around 2TB and sda2 around 4TB, sdb is O1. Else it's N1

    There are no active packages and such.

    The packages are installed in the data volume. So if the array is down, there are no packages.

    "with some luck" sounds ominous to me 😁

    Yeah. That was about the automatic assemble&mount. Manually should always be possible.

    Also, I'm guessing I'd loose groups, shares, NFS exports… is that correct?

    Correct. Although the shares are on certain level just directories, and that directories can be copied. Then you can re-activate them as share. But owner, access rights and such are gone.

    About groups I'm not sure. The user config is stored on a flash partition, so that might survive a volume change. But maybe it invalidates if the volume changes.

    Don't know exactly what JBOD is. Have a look with mdadm, I think it's just a single disk raid1

    mdadm -E /dev/md0
    

  • Paolo DeF
    Paolo DeF Posts: 11  Freshman Member
    First Comment Friend Collector Third Anniversary

    Hi Mijzef, thanks a lot again, and sorry for the long delay.

    JBOD = Just a Bunch Of Disks, kind of similar to RAID 0.

    In the meantime, I reverted to VO1 and O1, with a 2TB "healthy" RAID1. Reason is, I have a very limited amount of time available as per now. I'm almost sure I can create a "blank" 4TB RAID1 and then copy the data, thanks to your help, but I'd also need to config the metarepository, install packages, create shares, users, groups, exports… I can't afford it as per now 😅

    Unless: there is a way to export most of the config files? E.g. the NFS export are in /etc/exports and such.

  • Mijzelf
    Mijzelf Posts: 2,790  Guru Member
    250 Answers 2500 Comments Friend Collector Seventh Anniversary

    As far as I remember there is an option in the webinterface to ex- and import user settings. But maybe that is only on the firmware 5 devices.

Consumer Product Help Center