Rebuilding RAID 5 structure ME -> PANIC

I had problem with my raid disk 3 on NAS542.
Decided to update the firmware and to replace the failed HDD.
Updated successfully the FW to the latest version.
Then replaced the failed drive #3 with a new one.
Started rebuilding the RAID 5 structure. At 1.8% the device crashed and I had to reboot.
After the reboot the error was that HDD 1 is not part of the RAID volume (out of sync).
I think drives 1, 2 and 4 work properly. The error is aftermath of the reboot.
Someone help me recover the data. I'm lost. And noob with RAID too.

All Replies

  • ~ # cat /proc/mdstat
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
    md3 : inactive sda3[5](S)
          1949513728 blocks super 1.2

    md2 : inactive sdb3[1](S) sdd3[3](S)
          3898767360 blocks super 1.2

    md1 : active raid1 sda2[0] sdd2[3] sdb2[1]
          1998784 blocks super 1.2 [4/3] [UU_U]

    md0 : active raid1 sdc1[4](F) sda1[0] sdd1[3] sdb1[1]
          1997760 blocks super 1.2 [4/3] [UU_U]

    unused devices: <none>
  • ~ # cat /proc/partitions
    major minor  #blocks  name

       7        0     147456 loop0
      31        0        256 mtdblock0
      31        1        512 mtdblock1
      31        2        256 mtdblock2
      31        3      10240 mtdblock3
      31        4      10240 mtdblock4
      31        5     112640 mtdblock5
      31        6      10240 mtdblock6
      31        7     112640 mtdblock7
      31        8       6144 mtdblock8
       8        0 1953514584 sda
       8        1    1998848 sda1
       8        2    1999872 sda2
       8        3 1949514752 sda3
       8       16 1953514584 sdb
       8       17    1998848 sdb1
       8       18    1999872 sdb2
       8       19 1949514752 sdb3
       8       32 1953514584 sdc
       8       33    1998848 sdc1
       8       34    1999872 sdc2
       8       35 1949514752 sdc3
       8       48 1953514584 sdd
       8       49    1998848 sdd1
       8       50    1999872 sdd2
       8       51 1949514752 sdd3
      31        9     102424 mtdblock9
       9        0    1997760 md0
       9        1    1998784 md1
      31       10       4464 mtdblock10

  • ~ # cat /proc/mounts
    rootfs / rootfs rw 0 0
    /proc /proc proc rw,relatime 0 0
    /sys /sys sysfs rw,relatime 0 0
    none /proc/bus/usb usbfs rw,relatime 0 0
    devpts /dev/pts devpts rw,relatime,mode=600 0 0
    ubi5:ubi_rootfs1 /firmware/mnt/nand ubifs ro,relatime 0 0
    /dev/md0 /firmware/mnt/sysdisk ext4 ro,relatime,user_xattr,barrier=1,data=ordered 0 0
    /dev/loop0 /ram_bin ext2 ro,relatime,user_xattr,barrier=1 0 0
    /dev/loop0 /usr ext2 ro,relatime,user_xattr,barrier=1 0 0
    /dev/loop0 /lib/security ext2 ro,relatime,user_xattr,barrier=1 0 0
    /dev/loop0 /lib/modules ext2 ro,relatime,user_xattr,barrier=1 0 0
    /dev/loop0 /lib/locale ext2 ro,relatime,user_xattr,barrier=1 0 0
    /dev/ram0 /tmp/tmpfs tmpfs rw,relatime,size=5120k 0 0
    /dev/ram0 /usr/local/etc tmpfs rw,relatime,size=5120k 0 0
    ubi3:ubi_config /etc/zyxel ubifs rw,relatime 0 0
    configfs /sys/kernel/config configfs rw,relatime 0 0

  • mdadm --examine /dev/sd[abcdef]3
    /dev/sda3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 4e742b3f:b0b2794c:83062d84:2833deb8
               Name : NAS542:1
      Creation Time : Wed Apr 26 23:28:17 2017
         Raid Level : raid1
       Raid Devices : 4

     Avail Dev Size : 3899027456 (1859.20 GiB 1996.30 GB)
         Array Size : 1998784 (1952.27 MiB 2046.75 MB)
      Used Dev Size : 3997568 (1952.27 MiB 2046.75 MB)
        Data Offset : 2048 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 37b12f05:983e4e69:c0f59ed2:35fba7f2

        Update Time : Fri Jan 22 04:34:36 2021
           Checksum : ada3c810 - correct
             Events : 411


       Device Role : spare
       Array State : AAAA ('A' == active, '.' == missing)
    /dev/sdb3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 1ff44e8b:f8340d1c:23c7ab5d:43d85bc6
               Name : NAS542:2
      Creation Time : Wed Apr 26 23:28:19 2017
         Raid Level : raid5
       Raid Devices : 4

     Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)
         Array Size : 5848151040 (5577.23 GiB 5988.51 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 3de071e5:9b79e2a9:67c0e9a2:e37e8f44

        Update Time : Fri Jan 22 02:51:33 2021
           Checksum : dc8683f7 - correct
             Events : 15479

             Layout : left-symmetric
         Chunk Size : 64K

       Device Role : Active device 1
       Array State : .A.A ('A' == active, '.' == missing)
    mdadm: No md superblock detected on /dev/sdc3.
    /dev/sdd3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 1ff44e8b:f8340d1c:23c7ab5d:43d85bc6
               Name : NAS542:2
      Creation Time : Wed Apr 26 23:28:19 2017
         Raid Level : raid5
       Raid Devices : 4

     Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)
         Array Size : 5848151040 (5577.23 GiB 5988.51 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : b24b3047:f2263e4e:9dc80f40:afb75e4d

        Update Time : Fri Jan 22 02:51:33 2021
           Checksum : 8895ddc6 - correct
             Events : 15479

             Layout : left-symmetric
         Chunk Size : 64K

       Device Role : Active device 3
       Array State : .A.A ('A' == active, '.' == missing)
    mdadm: cannot open /dev/sde3: No such device or address
    mdadm: cannot open /dev/sdf3: No such device or address

  • NiceMan
    NiceMan Posts: 7
    edited February 2021
    OK
    I did it!
    All data recovered!
    Recreated the RAID5
    Remounted the volume
    Learned a lot tonight.
    All done... :)

  • Hello,
    I have raid degraded, I replace disk 2, follow the steps to repair it, after a few minutes of loading took me back to the repair page. I check and this is my status: 

    ~ # mdadm --examine /dev/sd[abcd]3
    /dev/sda3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : ba107b29:c0145772:da954bd1:e8ed408c
               Name : NasProductie:2  (local to host NasProductie)
      Creation Time : Wed May  2 12:23:26 2018
         Raid Level : raid5
       Raid Devices : 4

     Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)
         Array Size : 17569172928 (16755.27 GiB 17990.83 GB)
      Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : active
        Device UUID : 25519dae:6ffb4334:78ac2f19:040a1bc1

        Update Time : Fri Jan 28 20:47:48 2022
           Checksum : bbe1e0e2 - correct
             Events : 20244

             Layout : left-symmetric
         Chunk Size : 64K

       Device Role : Active device 0
       Array State : AAAA ('A' == active, '.' == missing)
    /dev/sdb3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : ba107b29:c0145772:da954bd1:e8ed408c
               Name : NasProductie:2  (local to host NasProductie)
      Creation Time : Wed May  2 12:23:26 2018
         Raid Level : raid5
       Raid Devices : 4

     Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)
         Array Size : 17569172928 (16755.27 GiB 17990.83 GB)
      Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 98cda793:3772a83f:396cc157:68bfdcbb

        Update Time : Sat Jan 29 11:05:35 2022
           Checksum : e5a7135e - correct
             Events : 20517

             Layout : left-symmetric
         Chunk Size : 64K

       Device Role : spare
       Array State : ..AA ('A' == active, '.' == missing)
    /dev/sdc3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : ba107b29:c0145772:da954bd1:e8ed408c
               Name : NasProductie:2  (local to host NasProductie)
      Creation Time : Wed May  2 12:23:26 2018
         Raid Level : raid5
       Raid Devices : 4

     Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)
         Array Size : 17569172928 (16755.27 GiB 17990.83 GB)
      Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : c9f527ed:fe6bbd25:c086f5a3:c58af25c

        Update Time : Sat Jan 29 11:05:35 2022
           Checksum : 12858769 - correct
             Events : 20517

             Layout : left-symmetric
         Chunk Size : 64K

       Device Role : Active device 2
       Array State : ..AA ('A' == active, '.' == missing)
    /dev/sdd3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : ba107b29:c0145772:da954bd1:e8ed408c
               Name : NasProductie:2  (local to host NasProductie)
      Creation Time : Wed May  2 12:23:26 2018
         Raid Level : raid5
       Raid Devices : 4

     Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)
         Array Size : 17569172928 (16755.27 GiB 17990.83 GB)
      Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 882a0bb0:cc7ccefa:29f17a31:15af39f6

        Update Time : Sat Jan 29 11:05:35 2022
           Checksum : d1465bb0 - correct
             Events : 20517

             Layout : left-symmetric
         Chunk Size : 64K

       Device Role : Active device 3
       Array State : ..AA ('A' == active, '.' == missing)
    ~ #

    Some advice please ?

Consumer Product Help Center