NAS326 recovery after ransomware infection

Adamyno

I successfully got a ransomware infection. It was probably my fault that I got to this point. The last firmware update was about fixing a serious vulnerability (CVSS 9.8). The latest FW was on the device, but it is possible that the ransomware was already on the NAS before that. It is also possible that an application in an external repo was vulnerable. Maybe the Syncthing. I didn't do too much to be safe. I have been putting off the task for a long time. All I did was put the web interface on a non-default, high port number.

Most of my smaller files have the 0XXX extension. E.g. jpg, pdf, docx, etc. Videos and larger files are not affected (I think due to lack of resources). Presumably this is why I notice the infection late because the movies worked on the TV. Even when I noticed the infection, I only opened the shared folders to move a small file. that's when I saw that .TXT file in the root folder.

I wanted to try several applications that might be able to decrypt the encrypted files, but I can't do it because I would have to put the 3TB HDDs in a desktop machine in order to mount the RAID array. Unfortunately, such a large disc does not work properly via USB and I do not have a desktop computer. Another option is to copy the more important infected files somewhere and then try the tools (I don't believe in miracles), but as long as the NAS is running, it will encrypt continuously in its current state. As a root user, all that is visible in the "top" is that the cpu load is high, but it is not visible what is taking up the resources.

I was thinking how about doing a hard reset? I understand that then the system rewrites the configuration files to the disks and then even if the malware is there, it will not run. But that's just a guess. If, on the other hand, you have written yourself into the flash, then the reset is useless. How do I flash the latest firmware again? Does it even make sense? Any other ideas? Data is no longer important. The important thing is that the device can be used reliably again.

Mijzelf

I hardly dare to ask, did you already reboot the box?

If you did, I'm aware of only 4 ways to get software running on boot. (In decreasing difficulty)

1. Change the root filesystem
2. Change the loop mounted file on /usr
   
3. Use the package start mechanism
4. Use /i-data/.system/zy-pkgs/USRPKG_DEPS_START

1) The rootfile system is embedded in the kernel as initramfs. While it is possible to change it, it involves compiling & flashing a new kernel. Which automatically means the malware has to have different versions for different hardware.

2) The major part of the firmware is loopmounted on /usr/. The content is in one big file sysdisk.img containing an ext2 filesystem, and the file is located in /dev/md0 which is mounted on /firmware/mnt/sysdisk/. The md5sum of this file is calculated on boot, and compared to an md5sum stored in flash. If it doesn't match a fresh copy is extracted from a compressed blob in flash. (This is part of the update mechanism)

sysdisk.img is mounted read-only. It is possible to remount it rw, but any change will be reverted on reboot, due to that md5sum. There is a way around, if some file exists (firmware/mnt/sysdisk/mount.sda1.rw.flag), sysdisk.img will be mounted rw, and the md5sum will not be calculated.

So if /usr/ is writable, try to remove that flag file and reboot. If /usr/ is then ro, you probably removed your malware.

3) The script which starts all packages (/etc/init.d/zypkg_controller.sh) contains next snippet:

    # - start to startup packages not in ${ZYPKG_DEPS} and ${USRPKG_DEPS_START}
    cat /etc/zyxel/pkg_conf/status | grep Installed-Rule | awk '{print $2}' |  while read zypkg; do
        PKGName=`echo ${zypkg} | awk -F "/" '{print $5}'`
        cat /i-data/.system/zy-pkgs/ZYPKG_DEPS | grep $PKGName > /dev/null 2>&1
        if [ "$?" != "0" ]; then
            echo ${Processed_Packages} | grep ${PKGName} > /dev/null 2>&1
            if [ "$?" != "0" ]; then
                EXEINIFILE=${zypkg}/etc/init.d/$PKGName
                if [ ! -x ${EXEINIFILE} ]; then
                    write_log "---> Error: start-up program \"${zypkg}\" is not existed or not excutable"
                else
                    $EXEINIFILE startup
                    if [ "$?" == "0" ]; then
                        write_log "---> start \"$PKGName\" successfully."
                    else
                        write_log "---> start \"$PKGName\" failed."
                    fi
                fi
            fi
        fi
    done

So all lines in /etc/zyxel/pkg_conf/status containing 'Installed-Rule' will be read, and their second block is treated a directory. If that directory contains an executable script/binary /etc/init.d/<last-part-of-directoryname>, it is executed. My /etc/zyxel/pkg_conf/status contains a line 'Installed-Rule: /i-data/38aa86ff/.PKG/Tweaks/', which means /i-data/38aa86ff/.PKG/Tweaks/etc/init.d/Tweaks will be executed. (If it exists and is executable)

Check your /etc/zyxel/pkg_conf/status for strange lines.

4) Any executable script/binary in /i-data/.system/zy-pkgs/, which name is listed in /i-data/.system/zy-pkgs/USRPKG_DEPS_START, will be executed on boot. By default this file doesn't exist.

It's certainly possible that there are more ways to get an executable started which I'm not aware of.

AFAIK flashing the same firmware again can be done by executing 'echo 1 >/firmware/mnt/info/revision' which make the the updater think revision 1 is flashed. (In reality it's something like 52000)

Adamyno

The device has been rebooted (several times).

I did a factory reset from the menu because it didn't work with the physical button.

I tried reinstalling the current firmware (I don't know if it was successful because the web interface was not clear).

After that, the transmission setting (9091) was still there under the upnp, which I couldn't even turn off, even though the transmission was theoretically no longer installed due to the factory reset.

After that, I deleted the volume group via the web interface and emptied all the HDDs. I re-added the volumes to RAID1. After that I had an empty drive and I was able to turn everything off even under upnp. After a while, however, the anomalies came. I can't sync the device to mycloud. I removed the mycloud app, but I can't reinstall it, nor can I update the store. The strange thing is that the HDDs are constantly working (it sounds like), but the CPU load is now low (near zero). Resynchronization is not visible on the web interface. According to mdstat, the raid is fine (md0/2GB/, md1/2GB/, md2/~3TB/).

Unfortunately, I don't have any tools to check what is loading the disks.

Now I'll try the reverse order.

I will empty the disks (I'm here now, volume and disk group are deleted from web interface), factory reset, firmware upgrade (with the help of the command you sent). I add the disc only after that, but before that I do a full wipe (e.g. with dd).

Some information about the current state:

/ # mount

/proc on /proc type proc (rw)

/sys on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw)

ubi4:ubi_rootfs1 on /firmware/mnt/nand type ubifs (ro)

/dev/md0 on /firmware/mnt/sysdisk type ext4 (ro)

/firmware/mnt/sysdisk/sysdisk.img on /ram_bin type ext2 (ro)

/ram_bin/usr on /usr type none (ro,bind)

/ram_bin/lib/security on /lib/security type none (ro,bind)

/ram_bin/lib/modules on /lib/modules type none (ro,bind)

/ram_bin/lib/locale on /lib/locale type none (ro,bind)

/dev/ram0 on /tmp/tmpfs type tmpfs (rw,size=5m)

/tmp/tmpfs/usr_etc on /usr/local/etc type none (rw,bind)

ubi2:ubi_config on /etc/zyxel type ubifs (rw)

configfs on /sys/kernel/config type configfs (rw)

cat /proc/mdstat 

Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] 

md1 : active raid1 sdb2[0] sda2[1]

      1998784 blocks super 1.2 [2/2] [UU]

      

md0 : active raid1 sdb1[0] sda1[1]

      1997760 blocks super 1.2 [2/2] [UU]

      

unused devices: <none>

<div><div></div></div><div></div>

Is it normal to have raid1 arrays left on the disks even after deleting the volume group? Maybe it could be sysdisk.img, but what is md1?
This is why I want to wipe the disks with dd. Before that, I stop the block and take the disks out from under it.

1) I think I understand what you're writing, but I don't think I can do anything with the root file system.
2) now the /usr/ is read-only.
3) the /etc/zyxel/pkg_conf/status is now empty. After reconfigure HDD-s, I'll check it again.
4) I'll be watching.

Now I destroy the remaining raid arrays, do the wipe, then factory reset + reboot and firmware upgrade.

Adamyno

I wrote a long comment but it disappeared. I'm doing something now and I'll come back and report the results. Thanks for the info!

Adamyno

I can not remove the md1 array. And I dont know what is it.

/ # cat /proc/mdstat 

Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] 

md1 : active raid1 sdb2[0] sda2[1]

      1998784 blocks super 1.2 [2/2] [UU]

      

md0 : active raid1 sdb1[0] sda1[1]

      1997760 blocks super 1.2 [2/2] [UU]

      

unused devices: <none>

/ # 

/ # mount | grep md1

/ # mdadm -S /dev/md1

mdadm: Cannot get exclusive access to /dev/md1:Perhaps a running process, mounted filesystem or active volume group?

/ # vgdisplay 

/ # vgdisplay 

/ # lvdisplay 

/ # mdadm --detail /dev/md1

/dev/md1:

        Version : 1.2

  Creation Time : Wed Nov 17 15:30:18 2021

     Raid Level : raid1

     Array Size : 1998784 (1952.27 MiB 2046.75 MB)

  Used Dev Size : 1998784 (1952.27 MiB 2046.75 MB)

   Raid Devices : 2

  Total Devices : 2

    Persistence : Superblock is persistent

    Update Time : Sun Dec 11 05:14:40 2022

          State : clean 

 Active Devices : 2

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 0

           Name : NAS326:1  (local to host NAS326)

           UUID : 6f6bb48c:a6b00f34:3f0722db:8090b1d8

         Events : 2

    Number   Major   Minor   RaidDevice State

       0       8       18        0      active sync   /dev/sdb2

       1       8        2        1      active sync   /dev/sda2

/ # umount -l /dev/md1

umount: /dev/md1: not mounted

/ # lsof | grep md1

md1_raid1  1452        root  cwd       DIR        0,1        0     1 /

md1_raid1  1452        root  rtd       DIR        0,1        0     1 /

md1_raid1  1452        root  txt   unknown                           /proc/1452/exe

# fdisk -l

Disk /dev/loop0: 141 MiB, 147849216 bytes, 288768 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock0: 2 MiB, 2097152 bytes, 4096 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock1: 2 MiB, 2097152 bytes, 4096 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock2: 10 MiB, 10485760 bytes, 20480 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock3: 15 MiB, 15728640 bytes, 30720 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock4: 106 MiB, 111149056 bytes, 217088 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock5: 15 MiB, 15728640 bytes, 30720 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock6: 106 MiB, 111149056 bytes, 217088 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sda: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disklabel type: gpt

Disk identifier: EEF994D4-C51C-44BB-A609-7B2DD4400E51

Device       Start        End    Sectors  Size Type

/dev/sda1     2048    3999743    3997696  1.9G Linux RAID

/dev/sda2  3999744    7999487    3999744  1.9G Linux RAID

/dev/sda3  7999488 5860532223 5852532736  2.7T Linux RAID

Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disklabel type: gpt

Disk identifier: 2641CF03-3AFA-450A-908F-8635B380693B

Device       Start        End    Sectors  Size Type

/dev/sdb1     2048    3999743    3997696  1.9G Linux RAID

/dev/sdb2  3999744    7999487    3999744  1.9G Linux RAID

/dev/sdb3  7999488 5860532223 5852532736  2.7T Linux RAID

Disk /dev/md0: 1.9 GiB, 2045706240 bytes, 3995520 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/md1: 1.9 GiB, 2046754816 bytes, 3997568 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Adamyno

After I changed the working directory to /tmp/tmpfs/, the lsfof | grep md? are empty. Unfortunately the mdadm --stop is'nt working. The message unchanged. --force option is not hepled.

md1 was a swap partition. It is destroyed. Only md0 is there. I think, I will force destroy it with dd, after that I will remove the raid array.

Adamyno

After wiped HDDs, factory reset, readded disks, FW update... it works normally!