Rebuilding RAID 5 structure ME -> PANIC

NiceMan

I had problem with my raid disk 3 on NAS542.
Decided to update the firmware and to replace the failed HDD.
Updated successfully the FW to the latest version.
Then replaced the failed drive #3 with a new one.
Started rebuilding the RAID 5 structure. At 1.8% the device crashed and I had to reboot.
After the reboot the error was that HDD 1 is not part of the RAID volume (out of sync).
I think drives 1, 2 and 4 work properly. The error is aftermath of the reboot.
Someone help me recover the data. I'm lost. And noob with RAID too.

NiceMan

~ # cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]

md3 : inactive sda3[5](S)

      1949513728 blocks super 1.2

md2 : inactive sdb3[1](S) sdd3[3](S)

      3898767360 blocks super 1.2

md1 : active raid1 sda2[0] sdd2[3] sdb2[1]

      1998784 blocks super 1.2 [4/3] [UU_U]

md0 : active raid1 sdc1[4](F) sda1[0] sdd1[3] sdb1[1]

      1997760 blocks super 1.2 [4/3] [UU_U]

unused devices: <none>

NiceMan

~ # cat /proc/partitions

major minor  #blocks  name

   7        0     147456 loop0

  31        0        256 mtdblock0

  31        1        512 mtdblock1

  31        2        256 mtdblock2

  31        3      10240 mtdblock3

  31        4      10240 mtdblock4

  31        5     112640 mtdblock5

  31        6      10240 mtdblock6

  31        7     112640 mtdblock7

  31        8       6144 mtdblock8

   8        0 1953514584 sda

   8        1    1998848 sda1

   8        2    1999872 sda2

   8        3 1949514752 sda3

   8       16 1953514584 sdb

   8       17    1998848 sdb1

   8       18    1999872 sdb2

   8       19 1949514752 sdb3

   8       32 1953514584 sdc

   8       33    1998848 sdc1

   8       34    1999872 sdc2

   8       35 1949514752 sdc3

   8       48 1953514584 sdd

   8       49    1998848 sdd1

   8       50    1999872 sdd2

   8       51 1949514752 sdd3

  31        9     102424 mtdblock9

   9        0    1997760 md0

   9        1    1998784 md1

  31       10       4464 mtdblock10

NiceMan

~ # cat /proc/mounts

rootfs / rootfs rw 0 0

/proc /proc proc rw,relatime 0 0

/sys /sys sysfs rw,relatime 0 0

none /proc/bus/usb usbfs rw,relatime 0 0

devpts /dev/pts devpts rw,relatime,mode=600 0 0

ubi5:ubi_rootfs1 /firmware/mnt/nand ubifs ro,relatime 0 0

/dev/md0 /firmware/mnt/sysdisk ext4 ro,relatime,user_xattr,barrier=1,data=ordered 0 0

/dev/loop0 /ram_bin ext2 ro,relatime,user_xattr,barrier=1 0 0

/dev/loop0 /usr ext2 ro,relatime,user_xattr,barrier=1 0 0

/dev/loop0 /lib/security ext2 ro,relatime,user_xattr,barrier=1 0 0

/dev/loop0 /lib/modules ext2 ro,relatime,user_xattr,barrier=1 0 0

/dev/loop0 /lib/locale ext2 ro,relatime,user_xattr,barrier=1 0 0

/dev/ram0 /tmp/tmpfs tmpfs rw,relatime,size=5120k 0 0

/dev/ram0 /usr/local/etc tmpfs rw,relatime,size=5120k 0 0

ubi3:ubi_config /etc/zyxel ubifs rw,relatime 0 0

configfs /sys/kernel/config configfs rw,relatime 0 0

NiceMan

mdadm --examine /dev/sd[abcdef]3

/dev/sda3:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : 4e742b3f:b0b2794c:83062d84:2833deb8

           Name : NAS542:1

  Creation Time : Wed Apr 26 23:28:17 2017

     Raid Level : raid1

   Raid Devices : 4

 Avail Dev Size : 3899027456 (1859.20 GiB 1996.30 GB)

     Array Size : 1998784 (1952.27 MiB 2046.75 MB)

  Used Dev Size : 3997568 (1952.27 MiB 2046.75 MB)

    Data Offset : 2048 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : 37b12f05:983e4e69:c0f59ed2:35fba7f2

    Update Time : Fri Jan 22 04:34:36 2021

       Checksum : ada3c810 - correct

         Events : 411

   Device Role : spare

   Array State : AAAA ('A' == active, '.' == missing)

/dev/sdb3:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : 1ff44e8b:f8340d1c:23c7ab5d:43d85bc6

           Name : NAS542:2

  Creation Time : Wed Apr 26 23:28:19 2017

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)

     Array Size : 5848151040 (5577.23 GiB 5988.51 GB)

    Data Offset : 262144 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : 3de071e5:9b79e2a9:67c0e9a2:e37e8f44

    Update Time : Fri Jan 22 02:51:33 2021

       Checksum : dc8683f7 - correct

         Events : 15479

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : Active device 1

   Array State : .A.A ('A' == active, '.' == missing)

mdadm: No md superblock detected on /dev/sdc3.

/dev/sdd3:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : 1ff44e8b:f8340d1c:23c7ab5d:43d85bc6

           Name : NAS542:2

  Creation Time : Wed Apr 26 23:28:19 2017

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 3898767360 (1859.08 GiB 1996.17 GB)

     Array Size : 5848151040 (5577.23 GiB 5988.51 GB)

    Data Offset : 262144 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : b24b3047:f2263e4e:9dc80f40:afb75e4d

    Update Time : Fri Jan 22 02:51:33 2021

       Checksum : 8895ddc6 - correct

         Events : 15479

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : Active device 3

   Array State : .A.A ('A' == active, '.' == missing)

mdadm: cannot open /dev/sde3: No such device or address

mdadm: cannot open /dev/sdf3: No such device or address

NiceMan

OK
I did it!
All data recovered!
Recreated the RAID5
Remounted the volume
Learned a lot tonight.
All done...

ValentinIstoc

Hello,
I have raid degraded, I replace disk 2, follow the steps to repair it, after a few minutes of loading took me back to the repair page. I check and this is my status: 

~ # mdadm --examine /dev/sd[abcd]3

/dev/sda3:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : ba107b29:c0145772:da954bd1:e8ed408c

           Name : NasProductie:2  (local to host NasProductie)

  Creation Time : Wed May  2 12:23:26 2018

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)

     Array Size : 17569172928 (16755.27 GiB 17990.83 GB)

  Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)

    Data Offset : 262144 sectors

   Super Offset : 8 sectors

          State : active

    Device UUID : 25519dae:6ffb4334:78ac2f19:040a1bc1

    Update Time : Fri Jan 28 20:47:48 2022

       Checksum : bbe1e0e2 - correct

         Events : 20244

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : Active device 0

   Array State : AAAA ('A' == active, '.' == missing)

/dev/sdb3:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : ba107b29:c0145772:da954bd1:e8ed408c

           Name : NasProductie:2  (local to host NasProductie)

  Creation Time : Wed May  2 12:23:26 2018

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)

     Array Size : 17569172928 (16755.27 GiB 17990.83 GB)

  Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)

    Data Offset : 262144 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : 98cda793:3772a83f:396cc157:68bfdcbb

    Update Time : Sat Jan 29 11:05:35 2022

       Checksum : e5a7135e - correct

         Events : 20517

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : spare

   Array State : ..AA ('A' == active, '.' == missing)

/dev/sdc3:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : ba107b29:c0145772:da954bd1:e8ed408c

           Name : NasProductie:2  (local to host NasProductie)

  Creation Time : Wed May  2 12:23:26 2018

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)

     Array Size : 17569172928 (16755.27 GiB 17990.83 GB)

  Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)

    Data Offset : 262144 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : c9f527ed:fe6bbd25:c086f5a3:c58af25c

    Update Time : Sat Jan 29 11:05:35 2022

       Checksum : 12858769 - correct

         Events : 20517

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : Active device 2

   Array State : ..AA ('A' == active, '.' == missing)

/dev/sdd3:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : ba107b29:c0145772:da954bd1:e8ed408c

           Name : NasProductie:2  (local to host NasProductie)

  Creation Time : Wed May  2 12:23:26 2018

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 11712782336 (5585.09 GiB 5996.94 GB)

     Array Size : 17569172928 (16755.27 GiB 17990.83 GB)

  Used Dev Size : 11712781952 (5585.09 GiB 5996.94 GB)

    Data Offset : 262144 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : 882a0bb0:cc7ccefa:29f17a31:15af39f6

    Update Time : Sat Jan 29 11:05:35 2022

       Checksum : d1465bb0 - correct

         Events : 20517

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : Active device 3

   Array State : ..AA ('A' == active, '.' == missing)

~ #

Some advice please ?