USG ATP / FLEX high CPU usage -> reboot: doesn't come up, SYS blinking forever

OverWire
OverWire Posts: 3  Freshman Member
Friend Collector Sixth Anniversary
edited March 2022 in Nebula
I'm managing some USG FLEX (100, 200, 500) in Nebula, for different customers.
Starting from yesterday March 10 at 21:30 UTC, all firewalls are using much CPU.
Today I have restarted 2 USG FLEX (a 100W and a 500) and are no more working now, they don't come up, don't boot.
Is someone experiencing this situation?

«1

All Replies

  • MarkT
    MarkT Posts: 13  Freshman Member
    First Comment Friend Collector SurveyFeedback-2022-Sep Sixth Anniversary
    Experiencing the same.

    High usage on USG FLEX 100, 200, 700 and ATP 500 starting from March 3 at 21:30 UTC.

    Tried to firmware upgrade the ATP 500 to 5.21, in hope for a solution - but the gateway did not come up again, only after a firmware recovery, but the high cpu usage remains. 
  • SLMark1
    SLMark1 Posts: 19  Freshman Member
    First Comment Second Anniversary
    Yes, we lost a Flex 100 and 200 today. Both had been running fine, had to reboot each for different reasons, neither have come back online.
  • Blabababa
    Blabababa Posts: 151  Master Member
    5 Answers First Comment Friend Collector Sixth Anniversary
    I have similar symptom on my device and Zyxel told me not to login to the device GUI after rebooting the device and they will release a patch for this issue soon. Try not to login to the device after rebooting and see if it helps
  • [Deleted User]
    [Deleted User] Posts: 118  Ally Member
    5 Answers First Comment Friend Collector Fifth Anniversary
    edited March 2022
    Dear Members.. Please create a ticket @ support.zyxel.eu

    please use my name in the subject field:'Mark'

    Then i will try to fix this asap
     :-)

  • SLMark1
    SLMark1 Posts: 19  Freshman Member
    First Comment Second Anniversary
    edited March 2022
    Tickets have been opened for the two units I am working on however we just got an alert and there may be a third.
  • [Deleted User]
    [Deleted User] Posts: 118  Ally Member
    5 Answers First Comment Friend Collector Fifth Anniversary
    edited March 2022
    @SLMark1 Thanks for that...
    At this moment, i think the most likely scenario is that 1 of the signatures is corrupted, 

    You can recover the device with console (thanks to @SLMark1 for sharing this) Please find a complete guide here:

    In standalone mode, you need to force or wait for the default time to update UTM signatures .
    If the signatures are not updated, you should be able to safely reboot the device without issue's 
    If you did update signatures, please do not reboot device! 

    If you have the firewall in nebula, all signatures are immediately updated after reporting to NCC. So reboot here will cause the firewall to hang again.. 

    If there is any news, you will get it asap from me 

  • MarkT
    MarkT Posts: 13  Freshman Member
    First Comment Friend Collector SurveyFeedback-2022-Sep Sixth Anniversary
    Thanks for the update @Mark_Zyxel.

    Should I create a ticket for the devices with the corrupted signatures? Or was the ticket creation intended for those who have a device that hangs?

    Furthermore - is this related to the unusual CPU usage?
  • OverWire
    OverWire Posts: 3  Freshman Member
    Friend Collector Sixth Anniversary
    Hi @Mark_Zyxel, my ticket is in Italian, ID: #295265, appliances now have been recovered but are in NCC.
    I will follow this thread.
    Thanks.



  • Zyxel_Stanley
    Zyxel_Stanley Posts: 1,377  Zyxel Employee
    100 Answers 1000 Comments Friend Collector Seventh Anniversary
    Hi @OverWire
    If you updated signature and device is unable to boot up success, you can swap to the other partition to boot up your system.
    (1) Reboot your device and enter to debug mode.
    (2) Enter "atcd 1" or "atcd 2" to change system boots partition.
    (3) Enter "atgof" to restart system again. (or power recycle again)

  • Lisgsarl
    Lisgsarl Posts: 1
    Hi all,

    Since the 12.03.2022 at each task scheduled for the update of signatures, there is one or more processes blocked at 100%.

    USG FLEX 700
    Firmware Version
    V5.21(ABWD.0) / 2022-02-26 05:59:56



    On cli:
    debug system ps
    app_util --> freeze and 97% cpu

    Then the only way to bypass the problem
    I recovered from debug 2 times in a row this weekend for a zyxel usgflex700.

    I tried to go back to the previous firmware version but no improvement.

    Here is the workaround I'm using now, in the meantime



    I use zyxel since 2000 after more than 20 years, I am disappointed .





Categories

Nebula Tips & Tricks


    Read more

    Nebula Help Center

    EnglishTiếng ViệtРусскийไทย中文（台灣）