Nebula Cloud Service Incident 2023-07-10

RasnoDigl

Good morning, I join pok, I can assure you that having over 120 nebula devices, I was able to test the efficiency of the Zyxel team; They are the best: fast, accurate, highly reliable and correct; I can assure you that in previous years I had to deal with very big Brends, who if it went well would answer you only if you had commercial contracts of a certain size, otherwise you would fall into oblivion!
Then we are all human and we make mistakes, only God is perfect! Good day everyone.

Zyxel_CSO

Root Cause and Immediate Actions:

To introduce new 17.10 security features, together with the needed upgrade of our key NCC components, an undetected bug emerged during the development phase due to limitations in accurately simulating real-world use cases at our development site. This led to the generation of incorrect VPN configurations and a disruption in the push of correct VPN configurations to affected devices.Our NCC team immediately took action to identify the root cause of this bug and diligently worked to resolve it. We want to assure you that we are taking this matter very seriously and are deploying all the necessary resources to fix it.

Preventive Measures and Enhanced Testing:

To prevent the recurrence of similar incidents in the future, we have implemented stringent measures to strengthen our testing and quality assurance processes. We are allocating additional resources to improve our development environment, allowing us to better uncover and address potential issues before they impact our customers.Moving forward, we are committed to conducting comprehensive reviews of our design and development processes, ensuring rigorous testing of our systems. By doing so, we are taking steps to minimize the chance of similar bugs causing problems.

We hereby sincerely apologize again for all those inconveniences brought to our product users.

Pook

Can we all just take a minute to appreciate that for a brief moment we were all united as we stood in our server rooms holding a WAN cable while counting up to 60. There is something beautiful about that. Anyway thanks ZyXel for taking those extra steps as to prevent incidents like this occurring less likely.

ITee

Zyxel_CSO

Hi @Pook @RasnoDigl @ITee

Thank you for your kind words.. We're delighted that our efforts have been appreciated and we'll continue to strive for excellence to serve you better.