NAS326 recovery after ransomware infection
Adamyno
Posts: 8
I successfully got a ransomware infection. It was probably my fault that I got to this point. The last firmware update was about fixing a serious vulnerability (CVSS 9.8). The latest FW was on the device, but it is possible that the ransomware was already on the NAS before that. It is also possible that an application in an external repo was vulnerable. Maybe the Syncthing. I didn't do too much to be safe. I have been putting off the task for a long time. All I did was put the web interface on a non-default, high port number.
Most of my smaller files have the 0XXX extension. E.g. jpg, pdf, docx, etc. Videos and larger files are not affected (I think due to lack of resources). Presumably this is why I notice the infection late because the movies worked on the TV. Even when I noticed the infection, I only opened the shared folders to move a small file. that's when I saw that .TXT file in the root folder.
I wanted to try several applications that might be able to decrypt the encrypted files, but I can't do it because I would have to put the 3TB HDDs in a desktop machine in order to mount the RAID array. Unfortunately, such a large disc does not work properly via USB and I do not have a desktop computer. Another option is to copy the more important infected files somewhere and then try the tools (I don't believe in miracles), but as long as the NAS is running, it will encrypt continuously in its current state. As a root user, all that is visible in the "top" is that the cpu load is high, but it is not visible what is taking up the resources.
I was thinking how about doing a hard reset? I understand that then the system rewrites the configuration files to the disks and then even if the malware is there, it will not run. But that's just a guess. If, on the other hand, you have written yourself into the flash, then the reset is useless. How do I flash the latest firmware again? Does it even make sense? Any other ideas? Data is no longer important. The important thing is that the device can be used reliably again.
0
All Replies
-
I hardly dare to ask, did you already reboot the box?If you did, I'm aware of only 4 ways to get software running on boot. (In decreasing difficulty)
- Change the root filesystem
- Change the loop mounted file on /usr
- Use the package start mechanism
- Use /i-data/.system/zy-pkgs/USRPKG_DEPS_START
1) The rootfile system is embedded in the kernel as initramfs. While it is possible to change it, it involves compiling & flashing a new kernel. Which automatically means the malware has to have different versions for different hardware.2) The major part of the firmware is loopmounted on /usr/. The content is in one big file sysdisk.img containing an ext2 filesystem, and the file is located in /dev/md0 which is mounted on /firmware/mnt/sysdisk/. The md5sum of this file is calculated on boot, and compared to an md5sum stored in flash. If it doesn't match a fresh copy is extracted from a compressed blob in flash. (This is part of the update mechanism)sysdisk.img is mounted read-only. It is possible to remount it rw, but any change will be reverted on reboot, due to that md5sum. There is a way around, if some file exists (firmware/mnt/sysdisk/mount.sda1.rw.flag), sysdisk.img will be mounted rw, and the md5sum will not be calculated.So if /usr/ is writable, try to remove that flag file and reboot. If /usr/ is then ro, you probably removed your malware.3) The script which starts all packages (/etc/init.d/zypkg_controller.sh) contains next snippet:# - start to startup packages not in ${ZYPKG_DEPS} and ${USRPKG_DEPS_START}
cat /etc/zyxel/pkg_conf/status | grep Installed-Rule | awk '{print $2}' | while read zypkg; do
PKGName=`echo ${zypkg} | awk -F "/" '{print $5}'`
cat /i-data/.system/zy-pkgs/ZYPKG_DEPS | grep $PKGName > /dev/null 2>&1
if [ "$?" != "0" ]; then
echo ${Processed_Packages} | grep ${PKGName} > /dev/null 2>&1
if [ "$?" != "0" ]; then
EXEINIFILE=${zypkg}/etc/init.d/$PKGName
if [ ! -x ${EXEINIFILE} ]; then
write_log "---> Error: start-up program \"${zypkg}\" is not existed or not excutable"
else
$EXEINIFILE startup
if [ "$?" == "0" ]; then
write_log "---> start \"$PKGName\" successfully."
else
write_log "---> start \"$PKGName\" failed."
fi
fi
fi
fi
doneSo all lines in /etc/zyxel/pkg_conf/status containing 'Installed-Rule' will be read, and their second block is treated a directory. If that directory contains an executable script/binary /etc/init.d/<last-part-of-directoryname>, it is executed. My /etc/zyxel/pkg_conf/status contains a line 'Installed-Rule: /i-data/38aa86ff/.PKG/Tweaks/', which means /i-data/38aa86ff/.PKG/Tweaks/etc/init.d/Tweaks will be executed. (If it exists and is executable)Check your /etc/zyxel/pkg_conf/status for strange lines.4) Any executable script/binary in /i-data/.system/zy-pkgs/, which name is listed in /i-data/.system/zy-pkgs/USRPKG_DEPS_START, will be executed on boot. By default this file doesn't exist.It's certainly possible that there are more ways to get an executable started which I'm not aware of.AFAIK flashing the same firmware again can be done by executing 'echo 1 >/firmware/mnt/info/revision' which make the the updater think revision 1 is flashed. (In reality it's something like 52000)
1 -
The device has been rebooted (several times).I did a factory reset from the menu because it didn't work with the physical button.I tried reinstalling the current firmware (I don't know if it was successful because the web interface was not clear).After that, the transmission setting (9091) was still there under the upnp, which I couldn't even turn off, even though the transmission was theoretically no longer installed due to the factory reset.After that, I deleted the volume group via the web interface and emptied all the HDDs. I re-added the volumes to RAID1. After that I had an empty drive and I was able to turn everything off even under upnp. After a while, however, the anomalies came. I can't sync the device to mycloud. I removed the mycloud app, but I can't reinstall it, nor can I update the store. The strange thing is that the HDDs are constantly working (it sounds like), but the CPU load is now low (near zero). Resynchronization is not visible on the web interface. According to mdstat, the raid is fine (md0/2GB/, md1/2GB/, md2/~3TB/).Unfortunately, I don't have any tools to check what is loading the disks.Now I'll try the reverse order.I will empty the disks (I'm here now, volume and disk group are deleted from web interface), factory reset, firmware upgrade (with the help of the command you sent). I add the disc only after that, but before that I do a full wipe (e.g. with dd).Some information about the current state:
/ # mount/proc on /proc type proc (rw)/sys on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw)ubi4:ubi_rootfs1 on /firmware/mnt/nand type ubifs (ro)/dev/md0 on /firmware/mnt/sysdisk type ext4 (ro)/firmware/mnt/sysdisk/sysdisk.img on /ram_bin type ext2 (ro)/ram_bin/usr on /usr type none (ro,bind)/ram_bin/lib/security on /lib/security type none (ro,bind)/ram_bin/lib/modules on /lib/modules type none (ro,bind)/ram_bin/lib/locale on /lib/locale type none (ro,bind)/dev/ram0 on /tmp/tmpfs type tmpfs (rw,size=5m)/tmp/tmpfs/usr_etc on /usr/local/etc type none (rw,bind)ubi2:ubi_config on /etc/zyxel type ubifs (rw)configfs on /sys/kernel/config type configfs (rw)
cat /proc/mdstatPersonalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]md1 : active raid1 sdb2[0] sda2[1]1998784 blocks super 1.2 [2/2] [UU]md0 : active raid1 sdb1[0] sda1[1]1997760 blocks super 1.2 [2/2] [UU]unused devices: <none><div><div></div></div><div></div>
Is it normal to have raid1 arrays left on the disks even after deleting the volume group? Maybe it could be sysdisk.img, but what is md1?
This is why I want to wipe the disks with dd. Before that, I stop the block and take the disks out from under it.
1) I think I understand what you're writing, but I don't think I can do anything with the root file system.
2) now the /usr/ is read-only.
3) the /etc/zyxel/pkg_conf/status is now empty. After reconfigure HDD-s, I'll check it again.
4) I'll be watching.
Now I destroy the remaining raid arrays, do the wipe, then factory reset + reboot and firmware upgrade.0 -
I wrote a long comment but it disappeared. I'm doing something now and I'll come back and report the results. Thanks for the info!
0 -
I can not remove the md1 array. And I dont know what is it./ # cat /proc/mdstatPersonalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]md1 : active raid1 sdb2[0] sda2[1]1998784 blocks super 1.2 [2/2] [UU]md0 : active raid1 sdb1[0] sda1[1]1997760 blocks super 1.2 [2/2] [UU]unused devices: <none>/ #/ # mount | grep md1/ # mdadm -S /dev/md1mdadm: Cannot get exclusive access to /dev/md1:Perhaps a running process, mounted filesystem or active volume group?/ # vgdisplay/ # vgdisplay/ # lvdisplay/ # mdadm --detail /dev/md1/dev/md1:Version : 1.2Creation Time : Wed Nov 17 15:30:18 2021Raid Level : raid1Array Size : 1998784 (1952.27 MiB 2046.75 MB)Used Dev Size : 1998784 (1952.27 MiB 2046.75 MB)Raid Devices : 2Total Devices : 2Persistence : Superblock is persistentUpdate Time : Sun Dec 11 05:14:40 2022State : cleanActive Devices : 2Working Devices : 2Failed Devices : 0Spare Devices : 0Name : NAS326:1 (local to host NAS326)UUID : 6f6bb48c:a6b00f34:3f0722db:8090b1d8Events : 2Number Major Minor RaidDevice State0 8 18 0 active sync /dev/sdb21 8 2 1 active sync /dev/sda2/ # umount -l /dev/md1umount: /dev/md1: not mounted/ # lsof | grep md1md1_raid1 1452 root cwd DIR 0,1 0 1 /md1_raid1 1452 root rtd DIR 0,1 0 1 /md1_raid1 1452 root txt unknown /proc/1452/exe# fdisk -lDisk /dev/loop0: 141 MiB, 147849216 bytes, 288768 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mtdblock0: 2 MiB, 2097152 bytes, 4096 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mtdblock1: 2 MiB, 2097152 bytes, 4096 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mtdblock2: 10 MiB, 10485760 bytes, 20480 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mtdblock3: 15 MiB, 15728640 bytes, 30720 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mtdblock4: 106 MiB, 111149056 bytes, 217088 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mtdblock5: 15 MiB, 15728640 bytes, 30720 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mtdblock6: 106 MiB, 111149056 bytes, 217088 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/sda: 2.7 TiB, 3000592982016 bytes, 5860533168 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 4096 bytesDisklabel type: gptDisk identifier: EEF994D4-C51C-44BB-A609-7B2DD4400E51Device Start End Sectors Size Type/dev/sda1 2048 3999743 3997696 1.9G Linux RAID/dev/sda2 3999744 7999487 3999744 1.9G Linux RAID/dev/sda3 7999488 5860532223 5852532736 2.7T Linux RAIDDisk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 4096 bytesDisklabel type: gptDisk identifier: 2641CF03-3AFA-450A-908F-8635B380693BDevice Start End Sectors Size Type/dev/sdb1 2048 3999743 3997696 1.9G Linux RAID/dev/sdb2 3999744 7999487 3999744 1.9G Linux RAID/dev/sdb3 7999488 5860532223 5852532736 2.7T Linux RAIDDisk /dev/md0: 1.9 GiB, 2045706240 bytes, 3995520 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 4096 bytesDisk /dev/md1: 1.9 GiB, 2046754816 bytes, 3997568 sectorsUnits: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 4096 bytes0
-
After I changed the working directory to /tmp/tmpfs/, the lsfof | grep md? are empty. Unfortunately the mdadm --stop is'nt working. The message unchanged. --force option is not hepled.
md1 was a swap partition. It is destroyed. Only md0 is there. I think, I will force destroy it with dd, after that I will remove the raid array.0 -
After wiped HDDs, factory reset, readded disks, FW update... it works normally!0
Categories
- All Categories
- 415 Beta Program
- 2.4K Nebula
- 145 Nebula Ideas
- 96 Nebula Status and Incidents
- 5.6K Security
- 240 USG FLEX H Series
- 268 Security Ideas
- 1.4K Switch
- 71 Switch Ideas
- 1.1K Wireless
- 40 Wireless Ideas
- 6.3K Consumer Product
- 247 Service & License
- 386 News and Release
- 83 Security Advisories
- 29 Education Center
- 10 [Campaign] Zyxel Network Detective
- 3.2K FAQ
- 34 Documents
- 34 Nebula Monthly Express
- 83 About Community
- 72 Security Highlight