NSA325v2: Resizing to 16 TB (RAID 1) fails

cschiefner

Dear all -

I recently bought my second NSA325v2 case and - after having taken Verified Hard Disk List for NAS Series into account - put in two 16 TB Toshiba MG08ACA16TE disks in RAID 1 mode: that worked like a charm.

Now, I wanted to upgrade my first NSA325v2 with a pair of the very same 16 TB disks: first old disk out, first new one in: resync - fine. Second old disk out, second new one in: resync again - fine, too.

But then resizing the healthy volume from the old 6 TB to the new 16 TB fails: it simply does not happen - without any particular error message.

Both devices run under firmware V4.81(AALS.1).

I have read up pretty much everthing in:

https://community.zyxel.com/en/search?query=e2fsck&scope=site&source=community

and:

https://community.zyxel.com/en/search?query=resize&scope=site&source=community

, but have not yet been able to nail it.

Observation #1:

The bulit-in e2fsck (1.41.14 (22-Dec-2010)) used to show four or five stages of checking - that is not the case any longer, it simply terminates with:

e2fsck 1.41.14 (22-Dec-2010)
/dev/md0: clean 254897/366256128 files 1464795842/1465005312 blocks
[old device, upgraded from 6 to 16 TB]

and:

e2fsck 1.41.14 (22-Dec-2010)
/dev/md0: clean 459721/976592896 files 1707650418/3906344414 blocks
[new device, set up with 16 TB right away]

, respectively.

Observation #2:

What has caught my eye is @Mijzelf's notion of "e2fsck 1.45.5 (07-Jan-2020)" in his answer to this question:

https://community.zyxel.com/en/discussion/13364/zyxel-nsa325-v2-shows-errors-when-running-e2fsck

- could this be the issue that the built-in e2fsck and/or resize are too old to handle the 16 TB?

Any hints appreciated!

Thanks & best,

-C.

Mijzelf

- could this be the issue that the built-in e2fsck and/or resize are too old to handle the 16 TB?

e2fsck won't be the problem, as

/dev/md0: clean 254897/366256128 files 1464795842/1465005312 blocks

the volume is clean. Don't know if resize2fs is able to handle up to 16TB, but I know it can't handle a big resize without swap space. The box has 2 swap files, one on the first (system) partition, and one on the data partition. The seconds can't be used while resizing, and I have no idea if the remaining swap would be enough.

In your case I'd try to resize manually, by injecting a telnet daemon in the shutdown script, as I wrote in some other posts in this forum. If resize2fs fails with a “resize2fs: Memory allocation failed while trying to resize /dev/md0” you can add extra swap space on an USB disk, or something like thet.

Mijzelf

- could this be the issue that the built-in e2fsck and/or resize are too old to handle the 16 TB?

e2fsck won't be the problem, as

/dev/md0: clean 254897/366256128 files 1464795842/1465005312 blocks

the volume is clean. Don't know if resize2fs is able to handle up to 16TB, but I know it can't handle a big resize without swap space. The box has 2 swap files, one on the first (system) partition, and one on the data partition. The seconds can't be used while resizing, and I have no idea if the remaining swap would be enough.

In your case I'd try to resize manually, by injecting a telnet daemon in the shutdown script, as I wrote in some other posts in this forum. If resize2fs fails with a “resize2fs: Memory allocation failed while trying to resize /dev/md0” you can add extra swap space on an USB disk, or something like thet.

cschiefner

Hi @Mijzelf,

thanks a bunch for your comprehensive answer and hints - that I am able to digest only now, as other pressing technical issues have come up in the meantime.

Wrt. “Resizing manually” I believe you are referring to:

https://community.zyxel.com/en/discussion/7766/expanding-raid-volume-on-nas326-is-not-working

, right?

And in case telnetd got never deactivated - I initially activated it around the box's FFP installation -, putting /bin/sh in the shutdown script should be enough - correct?

And I take it that "resize2fs /dev/md0” is meant to do its trick without any further parameters?

An last but not least: I have a 120 GB SSD with a SATA/USB adapter at hand for the external swap space, as you have suggested. Howver, I fail to locate to locate the to-be-moved swap space.

What I can find is:

root@NASigoreng:~# cat /etc/fstab
/dev/ram0 / ext2 defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
none /dev/pts devpts defaults 0 0

root@NASigoreng:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mtdblock8 0.0K 0.0K 0.0K 91% /zyxel/mnt/nand
/dev/sda1 473T 466T 6.8T 99% /zyxel/mnt/sysdisk
/dev/loop0 136T 121T 16T 89% /usr
/dev/ram0 1.3T 1.1G 1.3T 1% /tmp/tmpfs
/dev/mtdblock4 2.6T 376G 2.2T 15% /etc/zyxel
/dev/sdc1 8.6T 64G 8.5T 1% /e-data/547B-87F8
/dev/sdc2 477T 125T 328T 28% /e-data/8b365ae2-2ffc-44d0-8480-9923d744ff08
/dev/md0 1.4E 1.4E 205T 100% /i-data/87bc26aa
root@NASigoreng:~# ls -latr /zyxel/mnt/sysdisk
total 475988
drwxr-xr-x 7 root root 0 Jun 23 2017 ..
drwxr-xr-x 2 root root 12288 Feb 2 13:52 lost+found
-rw-r--r-- 1 root root 146800640 Feb 2 13:53 sysdisk.img
drwxr-xr-x 3 root root 1024 Feb 2 13:54 .
-rwxrwxrwx 1 root root 338690048 Feb 2 13:54 swap_ul6545p
root@NASigoreng:~#

/zyxel/mnt/sysdisk/swap_ul6545p looks a candidate to me, with a size of merely 323 MB. But how to move a swap file - and not a file system - to an external USB storage?

Again, any (further) hints appreciated - thanks in advance and all the best,

-C.

cschiefner

Addition to my previous post as I have just found out by looking at /etc/init.d/rc.shutdown :

/sbin/swapoff`cat /proc/swaps|awk '{print $1}'|grep -v Filename`

terminates the two swap spaces, I believe.

The swap space on the external USB could then be prepared according to e.g. https://askubuntu.com/questions/173676/how-to-make-a-usb-stick-swap-disk - but how to eventually activate it?

/etc/fstab does not seem to be right place as it does not bear any notion of a swap space.

Mijzelf

And I take it that "resize2fs /dev/md0” is meant to do its trick without any further parameters?

Yes.

But how to move a swap file - and not a file system - to an external USB storage?

You do not move it, but just create a new one. Assuming the USB storage is auto mounted, you can find it in /e-data/. Then create a swapfile

dd if=/dev/zero of=/e-data/*/swapfile bs=1M count=1000
mkswap /e-data/*/swapfile

The * will only work if there is only one directory in /e-data/. Else you need to be more specific. This will create a file of 1GB.

Then enable the swap

swapon /e-data/*/swapfile

You can check it with

cat /proc/swaps

cschiefner

Hi @Mijzelf,

after having deactivated the two standard swap spaces and activated the new 1 GB one on the external USB SSD, I invoked e2fsck and resize2fs in a manner a bit more chatty:

• /usr/sbin/e2fsck.new -fv /dev/md0

and:

• nohup /usr/sbin/resize2fs -p /dev/md0 2>&1 > /tmp/cschiefner.txt &

And:

• tail -f /tmp/cschiefner.txt

eventually revealed it:

• Extending the inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/usr/sbin/resize2fs: Memory allocation failed while trying to resize /dev/md0

At least I now know why resizing fails.

But I wonder what to do next: would resizing in two steps 6 TB → 11 TB and then 11 TB → 16 TB possibly work as extending the inode table would happen in smaller chunks?

Any suggestions appreciated!

Thanks and best,

-C.

Mijzelf

I'd expect that either the end size or the number of files in the filesystem would determine the max of memory usage. In that case it wouldn't help to do it in multiple steps.

How about adding more swap? The rule of thumb is that swap should be twice the memory size, but you are not interested in efficient swap usage, only in getting the job done. I'm not sure, but think it could be useful to enlarge the swap to 3.5GiB (so swap+ram is 4GiB). Don't know if a 32 bits Arm system can handle more more, due to lack of address space.

BTW, -p adds a 'percentage bar'. Don't know how often that is updated, but each update will write a new line to /tmp/cschiefner.txt, and as /tmp/ is also in ram, that file shouldn't get too big.

cschiefner

Hi @Mijzelf & all -

I got bold and blew the swap file up to 10 GB by:

dd if=/dev/zero of=/e-data/[…]/swapfile bs=1M count=10000

for the next attempt.

Et voilà:

root@NASigoreng:~# tail -f /tmp/cschiefner.txt

resize2fs 1.41.14 (22-Dec-2010)

Resizing the filesystem on /dev/md0 to 3906344448 (4k) blocks.

Begin pass 1 (max = 74504)

Extending the inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The filesystem on /dev/md0 is now 3906344448 blocks long.

And another e2fsck confirmed it's all fine.

😊

P.S. #1: A dd if=/dev/zero of=/e-data/[…]/swapfile bs=1M count=3584 to enlarge the swap space to only 3.5 GB as @Mijzelf has suggested would probably have been enough already. But when I eventually read this I had given the above a trial already.

P.S. #2: I shall post a complete write-up shortly.

cschiefner

And now for the promised write-up:

1. Have an external USB storage of at least 4 GB in size formatted as plain, off-the-shelf NTFS
2. Log into the system by either telnet or ssh
3. Check /e-data/ for content with:
   ls -latr /e-data/
4. Plug the external USB storage into the front-side USB socket: it should be auto-mounted by the system to /e-data/
5. Check /e-data/ again for content with:
   ls -latr /e-data/
   The new entry 'ABCD1234' (CAUTION: just a placeholder for further referencing it! In my case it's been '16b9121d7cdde35b484594f1c610f0f4'. One needs to replace it by the actual directory name!) is the external USB storage
6. Create a 3.5 GB ('count=3584') file on the external USB storage:
   dd if=/dev/zero of=/e-data/ABCD1234/swapfile bs=1M count=3584
7. Make it a new swap space:
   mkswap /e-data/ABCD1234/swapfile
8. Enable the new swap space:
   /sbin/swapon /e-data/ABCD1234/swapfile
9. Check success with:
   cat /proc/swaps
   The new swap space should be listed
10. Make a backup of the shutdown script:
    cp -pir /etc/init.d/rc.shutdown /etc/init.d/rc.shutdown.ORIG
11. Change the shutdown script:
    vi /etc/init.d/rc.shutdown
12. Within vi, while executing the editor:
    /# swapoff<ENTER>
A<ENTER>
/sbin/telnetd<ENTER>
/bin/sh<ESC>
:wq<ENTER>
13. Start the shutdown process:
    poweroff
14. Log into the system again by telnet
15. Disbale all swap spaces:
    /sbin/swapoff `cat /proc/swaps|awk '{print $1}'|grep -v Filename`
16. Enable the new swap space again:
    /sbin/swapon /e-data/ABCD1234/swapfile
17. Check success with:
    cat /proc/swaps
    The new swap space should be listed again
18. Unmount as specified in the shutdown script:
    /bin/umount `cat /proc/mounts|grep /dev/md|awk '{print $2}'`
/bin/umount /etc/zyxel
19. Check the RAID 1 file system:
    /usr/sbin/e2fsck.new -fv /dev/md0
20. In case it's clean, start the resizing process:
    nohup /usr/sbin/resize2fs -p /dev/md0 2>&1 > /tmp/cschiefner.txt &
    The temporary log file /tmp/cschiefner.txt will stay well below 1K in size
21. Follow the progess:
    tail -f /tmp/cschiefner.txt
    It should read along the lines of:
    resize2fs 1.41.14 (22-Dec-2010)
Resizing the filesystem on /dev/md0 to 3906344448 (4k) blocks.
Begin pass 1 (max = 74504)
Extending the inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/md0 is now 3906344448 blocks long.
22. Check the RAID 1 file system again:
    /usr/sbin/e2fsck.new -fv /dev/md0
23. In case it's still clean, shut it off:
    mdadm -Ss
24. Complete the the shutdown process:
    killall -9 /bin/sh
25. In case the system wouldn't simply switch off, press the power-on button until the system beeps a second time.

That's it, you're done! ☺️