Nebula Cloud Service Incident 2023-07-10

Zyxel_CSO

Dear Nebula Users,
Some of the Firewall VPN service may meet temporarily unavailable issue due to an outage incident.

Symptoms and Risks of the Issue:
The primary symptom of this incident is the non-functionality of the Nebula VPN service. You might have experienced difficulties in accessing and using our VPN service. We realize that this issue poses a significant inconvenience. Rest assured, our team is working around the clock to rectify the situation and restore normal service.
We sincerely apologize for any inconvenience caused by this issue. We commit to keeping you informed and will provide updates once the issue has been fully resolved.
Thank you for your understanding and patience.
Best regards,
Zyxel Nebula Support

Update on 11 July:

We had resolving the those devices that may have VPN connection issues by pushing the right configurations,. please kindly check if it's working right now. If the problem persists, please kindly provide your Org/Site name and Firewall Serial Number by private message for us. We will further assist on your cases.

Update on 14 July:

Root Cause and Immediate Actions:
To introduce new 17.10 security features, together with the needed upgrade of our key NCC components, an undetected bug emerged during the development phase due to limitations in accurately simulating real-world use cases at our development site. This led to the generation of incorrect VPN configurations and a disruption in the push of correct VPN configurations to affected devices.Our NCC team immediately took action to identify the root cause of this bug and diligently worked to resolve it. We want to assure you that we are taking this matter very seriously and are deploying all the necessary resources to fix it.

Preventive Measures and Enhanced Testing:
To prevent the recurrence of similar incidents in the future, we have implemented stringent measures to strengthen our testing and quality assurance processes. We are allocating additional resources to improve our development environment, allowing us to better uncover and address potential issues before they impact our customers.Moving forward, we are committed to conducting comprehensive reviews of our design and development processes, ensuring rigorous testing of our systems. By doing so, we are taking steps to minimize the chance of similar bugs causing problems.

We hereby sincerely apologize again for all those inconveniences brought to our product users.

Pook

Just to add we are seeing USG Flex firewalls show their configuration as "Not Up To Date", these appliances are also not pickup up any changes made to Nebula cloud settings.

asriggs

Zyxel has to get its VPN stuff situated, we have continuous issues not only with Nebula Site to Site, but remote VPN clients where we have to re-issue the VPN configuration profile after you push a firmware update. We rely on this over several sites across multiple clients and the support time we have to dedicate to something Zyxel screws up is pushing us to look at other vendors. Furthermore, when an issue like this happens - email the admins about the issue instead of posting on the forum.

Drjo_1

I find the following "strange" thing. Site-2-Site-Connection is up —- but not visible in Nebula.

I have connected to sites via site-2-site (not nebula). in the configuration I cannot see the settings or the exported sites nor the port.

BUT: the connection is up and running.

Hopefully the zyxel-team is getting the incident solved this night….

warm regards, jo

MarkSL

Can someone from Zyxel please post an update?

asriggs

This is a pretty critical issue, and Zyxel has been pretty quiet about it.

ITee

Hey Zyxel, is there any update on this critical issue? We are down and losing money each day we cant use our VPN services. Please update us on a timeline please. Thank you.

Zyxel_CSO

Dear Nebula Users,

We sincerely apologize for any inconvenience that you may have experienced. Please be assured that our team has been diligently working to address the situation.

Your concerns are our priority. We will keep you updated and will post further news every two hours, or sooner if available.

Please expect the next update by 4:00 (UTC+0), if not earlier.

Thank you for your patience and understanding.

Zyxel_CSO

Dear Nebula Users,

We want to let you know that our team is still diligently working to resolve this problem. We will continue to keep you updated on our progress.

The next update will be 6:00 8:00 (UTC+0), if not earlier.

Thank you for your patience and understanding.

Zyxel_CSO

Dear Nebula Users,

We would like to inform you that our team is persistently working to address the current issue. Rest assured, we will continually update you on our progress.

If you met VPN connection related symptom, please try to disconnect the WAN cable(s) from your firewall for approximately one minute (NO NEED to shutdown or power off your device) and reconnecting them. If the problem persists, please provide your Org/Site name and Firewall Serial Number by private message for further assistance.

Pook

Will a reboot still do the trick? I only ask as it is easier to perform a reboot on a remote site rather than asking an end user to pull the cable. I have users that will definitely pull the wrong one lol