USG ATP / FLEX high CPU usage -> reboot: doesn't come up, SYS blinking forever

OverWire

I'm managing some USG FLEX (100, 200, 500) in Nebula, for different customers.

Starting from yesterday March 10 at 21:30 UTC, all firewalls are using much CPU.

Today I have restarted 2 USG FLEX (a 100W and a 500) and are no more working now, they don't come up, don't boot.

Is someone experiencing this situation?

MarkT

Experiencing the same.

High usage on USG FLEX 100, 200, 700 and ATP 500 starting from March 3 at 21:30 UTC.

Tried to firmware upgrade the ATP 500 to 5.21, in hope for a solution - but the gateway did not come up again, only after a firmware recovery, but the high cpu usage remains. 

SLMark1

Yes, we lost a Flex 100 and 200 today. Both had been running fine, had to reboot each for different reasons, neither have come back online.

Blabababa

I have similar symptom on my device and Zyxel told me not to login to the device GUI after rebooting the device and they will release a patch for this issue soon. Try not to login to the device after rebooting and see if it helps

[Deleted User]

Dear Members.. Please create a ticket @ support.zyxel.eu

please use my name in the subject field:'Mark'

Then i will try to fix this asap
 :-)

SLMark1

Tickets have been opened for the two units I am working on however we just got an alert and there may be a third.

[Deleted User]

@SLMark1 Thanks for that...
At this moment, i think the most likely scenario is that 1 of the signatures is corrupted, 

You can recover the device with console (thanks to @SLMark1 for sharing this) Please find a complete guide here:

In standalone mode, you need to force or wait for the default time to update UTM signatures .
If the signatures are not updated, you should be able to safely reboot the device without issue's 
If you did update signatures, please do not reboot device! 

If you have the firewall in nebula, all signatures are immediately updated after reporting to NCC. So reboot here will cause the firewall to hang again.. 

If there is any news, you will get it asap from me 

MarkT

Thanks for the update @Mark_Zyxel.

Should I create a ticket for the devices with the corrupted signatures? Or was the ticket creation intended for those who have a device that hangs?

Furthermore - is this related to the unusual CPU usage?

OverWire

Hi @Mark_Zyxel, my ticket is in Italian, ID: #295265, appliances now have been recovered but are in NCC.

I will follow this thread.

Thanks.

Zyxel_Stanley

Hi @OverWire
If you updated signature and device is unable to boot up success, you can swap to the other partition to boot up your system.
(1) Reboot your device and enter to debug mode.
(2) Enter "atcd 1" or "atcd 2" to change system boots partition.
(3) Enter "atgof" to restart system again. (or power recycle again)

Lisgsarl

Hi all,

Since the 12.03.2022 at each task scheduled for the update of signatures, there is one or more processes blocked at 100%.

USG FLEX 700

Firmware Version

V5.21(ABWD.0) / 2022-02-26 05:59:56

On cli:
debug system ps
app_util --> freeze and 97% cpu

Then the only way to bypass the problem
I recovered from debug 2 times in a row this weekend for a zyxel usgflex700.

I tried to go back to the previous firmware version but no improvement.

Here is the workaround I'm using now, in the meantime

I use zyxel since 2000 after more than 20 years, I am disappointed .