Forced Firmware Rollout

Pook

Anyone else seeing the appliances being upgraded this morning?

Dylan96

YES, our firewall and many of our client's firewalls got restarted and upgraded without ANY notice and OUTSIDE our configured schedule. What is going on?

Pook

As per https://support.zyxel.eu/hc/en-us/articles/11616709217810I was expecting there to be an update today, but not forced without notice!

Zyxel_Melen

Hi @Pook & @Dylan96,

We released the V5.36P2 firmware. But we didn't set up a forced upgrade.

Could you help to check the firmware version? If is V5.36P2, please help to enable Zyxel support and provide the org & site name. We will help to check the log.

Apologize for any inconvenience caused.

Pook

Hi Melen

I can confirm that is the firmware that has been rolled out across multiple Orgs we mange this morning. All sites are back online. Message me for info on an Org you can review.

MediaManaged

Had same scenario here, where two devices force upgraded this morning.

The force upgrade window as a whole is pretty short(6/8), if there is a mitigation available, I'd like to see this extended. Forced upgrades outside of maintenance windows are tough to schedule for 24/7 operations.

Pook

To make matters worse we are now facing issues of Firewalls not coming back online after applying Firmware (All Nebula mode Flex 200). Trying a factory reset now in the hopes we can get them back up and running.

Zyxel_Melen

Hi @Pook, @Dylan96, and @MediaManaged,

Thanks for bringing this to our attention.

Upon further examination, it has come to our attention that an error in our operations led to the accidental upgrade of your firewall. We sincerely apologize for any inconvenience this may have caused you.

If you encounter any problems after upgrading, please feel free to contact us.

Pook

Hi Melen, we are having many issues with appliances no longer booting with sys light flashing. I am currently on the Teams meeting with Tobias and he is going to send me the recovery procedures. But if you have access to them feel free to post here :-)

Dylan96

Good morning @Zyxel_Melen

Thank you for your time, i hope you understand this is unacceptable. We had 3 other firewalls that got auto updated, resulting in an unexpected downtime for our clients…

We hope this never happens again…

Zyxel_Melen

Hi @Dylan96,

We apologize for any inconvenience this may have caused. Please rest assured that we will take extra care with this particular aspect in the future.