ZyXEL NSA325-v2 shows errors when running e2fsck

Frik

Hi all,

Recently I upgraded my NAS-325 v2 with 2x4TB hard discs, I got everything working with the help of @Mijzelf (thanks a lot).

I did a scan of /dev/md0 with e2fsck, just to make sure the file system is ok :

root@NSA325-v2:~# e2fsck.new -n /dev/md0

e2fsck 1.41.14 (22-Dec-2010)

Warning!  /dev/md0 is mounted.

Warning: skipping journal recovery because doing a read-only filesystem check.

/dev/md0 contains a file system with errors, check forced.

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

Free blocks count wrong (427235202, counted=427235211).

Fix? no

Free inodes count wrong (244079835, counted=244079836).

Fix? no

/dev/md0: ********** WARNING: Filesystem still has errors **********

So I followed the instructions in another post on this forum (https://community.zyxel.com/en/discussion/9871/nsa-325-v2-problems-with-disk1-in-two-different-systems) to unmount the discs and to do a scan with the -f option, hoping that it would repair the errors for me :

root@NSA325-v2:~# e2fsck.new -f /dev/md0

output : 

e2fsck 1.41.14 (22-Dec-2010)

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

/dev/md0: 82725/244162560 files (11.6% non-contiguous), 549394012/976629214 blocks

So what's strange, is that it's not showing any errors when checking with -f. I rebooted the box and tried e2fsck -n again, but it's still telling me there are errors on the file system. Any ideas?

Mijzelf

Fascinating. Did you see your error was actually the same as the one of myZyxel? One free inode too much, and some free blocks.

Anyway, you were running the check on a mounted filesystem, which was in use. As e2fsck cannot take a snapshot, it is not avoidable that the filesystems changes while it is running. I just tried it on my disk, run an e2fsck -nf, which gave a lot of errors:

e2fsck -nf /dev/sda2
e2fsck 1.45.5 (07-Jan-2020)
Warning!  /dev/sda2 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Inode 8268542 extent tree (at level 1) could be shorter.  Optimize? no

Inode 8526873 extent tree (at level 1) could be narrower.  Optimize? no

Deleted inode 9971134 has zero dtime.  Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(4626432--4627455) -(4634624--4634794) -(57286914--57286915) +(57772824--57772825)
Fix? no

Free blocks count wrong for group #1748 (25331, counted=25329).
Fix? no

Free blocks count wrong for group #1763 (30267, counted=30269).
Fix? no

Free blocks count wrong (31170141, counted=31170116).
Fix? no

Inode bitmap differences:  -9971134
Fix? no

Free inodes count wrong (14181379, counted=14181376).
Fix? no

Block bitmap differences: Group 1748 block bitmap does not match checksum.
IGNORED.
Group 1763 block bitmap does not match checksum.
IGNORED.

/dev/sda2: ********** WARNING: Filesystem still has errors **********

/dev/sda2: 1063933/15245312 files (0.2% non-contiguous), 29801555/60971696 blocks

Alarming, isn't it? I compiled some stuff and run it again. Guess what? Similar but different errors. So don't put too much value on the result of a file system check on a mounted system.

Mijzelf

Fascinating. Did you see your error was actually the same as the one of myZyxel? One free inode too much, and some free blocks.

Anyway, you were running the check on a mounted filesystem, which was in use. As e2fsck cannot take a snapshot, it is not avoidable that the filesystems changes while it is running. I just tried it on my disk, run an e2fsck -nf, which gave a lot of errors:

e2fsck -nf /dev/sda2
e2fsck 1.45.5 (07-Jan-2020)
Warning!  /dev/sda2 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Inode 8268542 extent tree (at level 1) could be shorter.  Optimize? no

Inode 8526873 extent tree (at level 1) could be narrower.  Optimize? no

Deleted inode 9971134 has zero dtime.  Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(4626432--4627455) -(4634624--4634794) -(57286914--57286915) +(57772824--57772825)
Fix? no

Free blocks count wrong for group #1748 (25331, counted=25329).
Fix? no

Free blocks count wrong for group #1763 (30267, counted=30269).
Fix? no

Free blocks count wrong (31170141, counted=31170116).
Fix? no

Inode bitmap differences:  -9971134
Fix? no

Free inodes count wrong (14181379, counted=14181376).
Fix? no

Block bitmap differences: Group 1748 block bitmap does not match checksum.
IGNORED.
Group 1763 block bitmap does not match checksum.
IGNORED.

/dev/sda2: ********** WARNING: Filesystem still has errors **********

/dev/sda2: 1063933/15245312 files (0.2% non-contiguous), 29801555/60971696 blocks

Alarming, isn't it? I compiled some stuff and run it again. Guess what? Similar but different errors. So don't put too much value on the result of a file system check on a mounted system.

Frik

Ok, thanks for that, it makes sense. I should have read the man page for e2fsck first, there it also states : 

Note that in general it is not safe to run e2fsck on mounted filesystems. The only exception is if the -n option is specified, and -c, -l, or -L options are not specified. However, even if it is safe to do so, the results printed by e2fsck are not valid if the filesystem is mounted. If e2fsck asks whether or not you should check a filesystem which is mounted, the only correct answer is ''no''. 

Albeit that there's no man page for e2fsck on the box....I guess it has something to do with the Linux on there (Busybox is too small to also add man pages maybe ?)  

But I guess the moral of the story is that you can trust e2fsck when it's run on an unmounted file system. So the fact that the scan that I did on the unmounted system did not return any error should be reassuring, right ?

Mijzelf

I guess it has something to do with the Linux on there (Busybox is too small to also add man pages maybe ?) 

Who would ever consult a manpage on an embedded system? Even on my desktop I rather just google for 'man whatever' than using my local copy. The layout and search possibilities of man are just no on par. Anyway, on embedded systems it's not common practice to install the man pages, just to save space. Actually the 325 has some man pages, but not the viewer. Someone forgot to strip them, or maybe the license of that package mandates the pages are distributed as well.

So the fact that the scan that I did on the unmounted system did not return any error should be reassuring, right ?

Right.

Frik

Something strange happened... I tried to run e2fsck again on the unmounted file system by rebooting but injecting telnetd in rc.shutdown (as before) but this time /dev/md0 refused to be unmounted.

So I inserted the following lines in /etc/init.d/rc.shutdown, just below the line # swapoff

# swapoff

# swapoff

/sbin/telnetd

/bin/sleep 999999

So this is after I issued poweroff and logged in again via telnet, then tried unmounting again. This time it did not work as smoothly....

root@NSA325-v2:~# umount /dev/md0

umount: /i-data/a11fd9e7: device is busy.

        (In some cases useful info about processes that use

         the device is found by lsof(8) or fuser(1))

So I tried :

root@NSA325-v2:~# cat /proc/swaps

Filename                                Type            Size    Used    Priority

/dev/sdb1                               partition       498680  0       -1

/i-data/a11fd9e7/.system/swap_ul6545p   file            524280  0       -2

root@NSA325-v2:~# swapoff /i-data/a11fd9e7/.system/swap_ul6545p

And now I could unmount :

root@NSA325-v2:~# umount -v /dev/md0

/dev/md0 umounted

So I managed to unmount it, but I had to first deactivate the swap on /i-data. I guess this is OK, right? 

Mijzelf

As long as there are open files on a filesystem, it cannot be unmounted. That's right. Don't know if it's OK that on that place in the rc.shutdown the volume was not yet unmounted, nor unmountable. If below your telnetd line there is code that could fix that, it's OK. When the box powers off with the filesystem still mounted, that's not OK. The only way to be sure there are no filesystem changes left in memory is by unmounting it.