Reflash NAS540 via SD card slot? (Recover from probable Hack)

Clay_JF2019

I will have to order a compatible TTL level converter.
I guess that will give me some chance at seeing if its stable enough to recover.
Point me to how to run memory and CPU diagnostic on hardware first.
If hardware is bad then no point trying to get good flash.


But definitely was NEVER flashing correctly.
Never shut itself down even with hours of waiting.
Looked like a lot of cycling: booting up, doing something for 3-10 minutes then rebooting. 
Never any orange light that I recall.

Mijzelf

Point me to how to run memory and CPU diagnostic on hardware first.

The box runs barebox as bootloader (in opposite of bootbox, as I said earlier). That has a 'memtest' command, as you can read here. I don't know about CPU diagnostics.

Looked like a lot of cycling: booting up, doing something for 3-10 minutes then rebooting.

That's not flashing. I think a flash takes about a minute, and the boot upto the reading of the usb stick 30 seconds. That means the box should switch itself off after about 90 seconds.

I think you'd first use the serial connection to look at the boot process. Maybe it gives a clear reason for the reboot.

Clay_JF2019

Fixing this box will be delayed a couple weeks for USB to TTL 3.3V cable to arrive. 

But thinking about buying new NAS 540 to get drive back online then selling this box when firmware is working again. But is that worthwhile?

Is there a wiki page etc for how to remount a RAID 5 (4 disk) array from a compromised system in a secure way? Other than wiping all data,  that is. 

I am guessing that there is a possibility that the RAID 5 drives have malware on them ready to reload. Not big threat if infected firmware is required to load that. But if malware is stored as something uninfected firmware might load and execute from disk...

--- as NAS application if it was Zyxel specific attack (unlikely but possible)
--- not sure about what NAS stores on disk as far as mount points for CRON and other standard shell startup scripts. Much more likely depending on what "just supply an executable file with this name" hooks exist.

Anyways I would appreciate pointers to any material on how to recovery data from RAID 5 disks once a box with good firmware is available. That is instructions on how to do that without getting compromised by any malware left on disk. 

I am guessing that an extra single disk could be mounted as non-RAID to store recovery tools & then when ready a degraded RAID 5 array (3 of 4 disks) could be mounted and scanned etc. Once cleaned the extra disk would be removed and the remaining RAID 5 disk inserted for the rebuild process. That last RAID 5 disk no longer being capable of loading malware -- especially if it was wiped.

Clay_JF2019

Also NAS 540 runs an unmodified full version of BareBox?  

Just asking as a lot of embedded devices seem to customize basic distributions like this to save memory or alter features. Occasionally the altered software does not tell that features are not working because they have been removed -- just error or success completion without expected results.

Mijzelf

Is there a wiki page etc for how to remount a RAID 5 (4 disk) array from a compromised system in a secure way? Other than wiping all data,  that is.

Why and how is it compromised? To get the compromised content active it has to be executed somehow. I can identify 6 groups of 'executables' on the nas:

1. The bootloader in flash.
2. The kernel in flash.
3. The rootfs in flash.
4. A (readonly mounted) filesystem blob on the system volume. This contains the webinterface binaries and scripts.
   
5. Installable packages on the data volume.

If you take another NAS the first 3 groups are exchanged, and the 4th will be exchange either, as a script in the rootfs will check the md5sum of the whole blob, before mounting it. If the checksum doesn't fit to the one belonging to the active firmware , the blob will be exchanged by one from flash.

There is one exception, if the system volume contains a file mount.sda1.rw.flag, the blob is accepted (and mounted rw).

So if that flag lacks (the default situation), the blob will be exchanged, if compromised, and group 4 is also gone.

If some rogue executable on the data partition is using the package start mechanism, I don't think there is a way to safely put that volume on a maiden NAS. But it's a default Linux raid array, so you can assemble&mount it on any Linux box, and remove the packages and startfiles. As long as it isn't a ZyXEL nas, it wont start the rogue executable.

Also NAS 540 runs an unmodified full version of BareBox? 

Don't know. But seeing the amount of flash reserved for the bootloader (1MiB, http://zyxel.nas-central.org/wiki/Category:NAS540), I don't think ZyXEL used much time to strip down the bootloader.

Clay_JF2019

Awesome and pretty complete answer on secure remount. Your verification that the RAID disks can be mounted using another Linux machine is a huge relief.

I have dealt with RAID arrays were the manufacture insisted on stamping its brand into the process --- sometimes to the extent that loss of the hardware mention loss of data still on drives unless you paid big bucks to get a factory flashed replacement with the old "serial number" info.

I was also pretty concerned about CRON related stuff. Maybe that is part of the data blob along with web interface and would be wiped out. But I was pretty sure that once disks are installed new working entries for CRON are stored to disk. Plus every Linux kernel based system seems to use CRON for at least system maintenance even if CRON is not exposed in NAS 540 web interface by a package (I do not remember that part). 

In any case I have a way to recover data and maybe remove any bad stuff to remount the actual drives without wiping. Not all data was backed up elsewhere and much was not in its current form.
All assuming the drives did not get totally trashed (cross fingers). 

Linux RAID mount seems to be my next step.

Clay_JF2019

Mounted the NAS540 RAID5 drives under a Ubuntu server. Found 3 RAID partitions  MD0, MD1, MD2.   MD2 appears to be my ext4 DATA volume. MD1 is Linux swap formatted. Haven't looked deeply yet but the data volume appears intact. 

Is MD0 what you called the system volume? And is sysdisk.img what you were referring to as the system data blob?  If so is there some sort of pre-cautionary reformat or partition delete that I could do? If I understood you correctly a NAS540 with correctly working firmware would automatically be recreated that system volume from from scratch if it was missing. Would recreating or deleting the swap partition cause any problems? Or does BusyBox automatically sanitize the swap space on restart?

I am assuming that the main reconfiguration would then be recreating NAS users, groups, and file-groups access rights which would then be stored on the system volume between boots. I assume that is a relatively simple process that could easily be reapplied to the top data volume folders.

I am thinking this route because if I understood correctly the system blob is only deleted if the successfully rewritten firmware version is different than the firmware version that was in effect when the system blob was written. I had just updated to the new firmware a couple weeks before the intrusion and plan to restore to that version -- so versions would be equal. Thus manual delete might be good to make sure that system and swap partitions have no super clever hacks to carry over effects of intrusion after firmware is fixed.

Mijzelf

Is MD0 what you called the system volume?

Yes. 

And is sysdisk.img what you were referring to as the system data blob?

Yes, it contains an ext2 filesystem. You can loopmount it.

If I understood you correctly a NAS540 with correctly working firmware would automatically be recreated that system volume from from scratch if it was missing.

No, it won't recreate the volume, AFAIK, but it will recreate sysdisk.img. You can delete that file, a new one will be created.

About the swap, I don't think it will hurt to run mkswap on the volume (maybe preceded by a 'dd if=/dev/zero of=/dev/md1'), but it's not a good idea to delete the partition. I think the firmware will no longer recognize the disk as 'ZyXEL disk'.

I think the steps are:

delete the contents of md0 (sysdisk.img, and mount.sda1.rw.flag if it exists).

I don't think it's necessary to do anything on md1, but it won't hurt either.

on md2 rename .PKG and .system. In that case only the user data remains.

Clay_JF2019

Got the serial TTL level converter.  But got one big question after disassembling the NAS540 box to expose the main board and port.   I noted that the metal drive enclosure also provided heatsink capability that is now removed. 

So do I need temporary heatsinks for CPU and that other chip while communing with Busybox via serial and attempting to reflash? Or will things run slow (CPU on-die thermal protection slow down) and cool enough that no heatsinks are needed without disks connected and running full speed etc?

If I had something on hand suitable for temporary heatsinks I would just slap them on to be safe. I will need to order something if temps are suggested.

Mijzelf

It's some time ago I opened my 540. I can remember it was a pain to open housing, but I can't remember I was alarmed by some heatsink which was removed. I have booted the box while serial connected, as I somehow got this bootlog: http://zyxel.nas-central.org/wiki/Bootlog_(NAS540) and I'm quite sure I didn't add any extra heatsink. Just probe the SoC with your finger while running, I think you won't even notice it gets warm. Arm is cool.

There is a beeper on the board, which will beep if it gets to hot. Found that while struggling with the fan control. http://zyxel.nas-central.org/wiki/Fan_control_(NAS540) . You can also connect the fan, to keep an eye on the rpm.

Yet I must admit that I don't know where the temperature sensor is located. It's obviously on the I2c bus, and so probably not in the CPU housing. So on a naked PCB it might behave different.