NAS542 inaccessible due to 16TB limit

alex_c

Hi,

I have a RAID-5 (single volume on disk group) setup with 3x6TB HDDs to which I just added a 4th one. All was fine until I tried expanding the volume from the web UI to the maximum available capacity (16.3TB). Now I cannot access the share or nor the web UI. I no longer hear the boot successful beep either. My NAS has an older firmware (5.21ABAG.2) and it could be because of a bug that was fixed in the latest version, but unfortunately the harm is already done and I ended up with a volume >16TB that most likely makes the NAS choke up.

How can I get my RAID array and NAS back in a good state without loosing the data?

I managed to get inside with ssh and found the relevant LVM volume of 16.3TB, but the file system is still reported at ~10.9TB (the size it was before adding the 4th HDD). I'm guessing the NAS managed to extend the volume but not the filesystem?!. Would I be able to use lvreduce or something similar to shrink the volume to under 16TB and then expand the file system?

#NAS_July_2020

Snow

How about press reset to default setting? it should not affect the data on disks.

Mijzelf

The limit is 16TiB (4096 * 2^32), not 16TB (16 * 10^12). 16TiB ~ 17.6TB, so I don't see where the number 16.3 comes from.

Anyway, how long did the conversion take? When you convert a 3 disk raid5 system to a 4 disk raid5 system, I'd expect it to take hours, maybe days. If it didn't take that long, I don't think the filesystem is 'reshuffled' or whatever the name may be for re-arranging the data blocks.

Have you tried to mount the 10.9TB filesystem?

alex_c

@Mijzelf , you are technically correct, I meant TiB. You'll have to excuse me, old habits die hard. 16.3TiB is the maximum size possible for the raid array (each hard-drive has a usable capacity of around ~5.45TiB, so 3*5.45 is approx. 16.3TiB for data, and another 5.45 is left for parity)

Syncing upon adding the 4th disk took around 3 days, after which I added the unallocated space to the sole existing disk group and then proceeded to expand the logical volume to fill the available space. The latter took just a couple of minutes, but the volume, which was initially 10.9TiB (2*5.45), was not increased and then, after restarting the NAS, I got the behavior described in the original post.

Some inspection after logging in with `ssh` revealed that the LV was expanded to 16.3TiB, but the filesystem was still 10.9TiB, as reported by `df -h` so I'm assuming the web UI choked up trying to resize the filesystem. I didn't have to manually mount it as it was already mounted and accessible and I could `ls` the files inside.

So I took a deep breath and managed to do a `lvresize` with -L 11T and then restarted the NAS and now I can get into the web UI and access the shares. I also tried resizing the filesystem with `resize2fs` to fill in the volume, but it doesn't let me while it's mounted and online resize doesn't work either for some reason. It's saying permission denied, even though I logged in with root. I guess I can live with the ~100GiB wasted for now on that volume.

Next step is to try to create a second volume for the remaining unallocated ~5TiB on the group, using the GUI. I'm currently waiting for the array to finish resyncing because I had previously booted the NAS with only 3 drives plugged in and couldn't just `mdadm --re-add` the 4th one because of a mismatch in the number of events.

Mijzelf

Ah. I think resize2fs failed on the 16.3TiB volume because the limit is 16TiB. (The kernel is just too old to support >2^32 blocks on ext4). A bit strange the firmware allowed you to resize the volume to that size. Or did you do it manually, using mdadm?

Indeed it's not possible to resize the filesystem while mounted. There is a way around, it is possible to 'inject' a telnet daemon the shutdown script, and block that script directly after unmounting everything.

https://homeforum.zyxel.com/discussion/comment/5867/#Comment_5867

https://homeforum.zyxel.com/discussion/comment/8443/#Comment_8443

alex_c

Yes, the web UI allowed me to resize that volume, but according to the firmware changelog this is a bug that was fixed in v5.21(ABAG.6)C0 on June 12:

Fix if total size (unallocated capacity + current volume size) is larger than 16 TB (16,383 GB), Editing volume size by clicking "MAX" button will make volume size larger than 16 TB. After applying this setting, user can not login web GUI.

So, had I updated the firmware to the latest version before resizing, this wouldn't have happened .