USG 40 - active session full, Internet drops

kaika313

Hi,

I've been struggling with our USG 40 for a couple of weeks.
It runs firmware version V4.35(AALA.3) (yes it's old, I know) and something about two weeks ago all of a sudden it run 100% CPU and made Internet connection drop.
Looking throughout Zyxel community I've found that a possible solution was to disable SNMP and indeed it helped. As soon as I disabled it CPU worked normally.

But after that Active Sessions keep reaching 50000 making Internet unusable again. I need almost every 1-2 hours reboot firewall to flush the active session bar.

Under Security Policy--Session Control I have these settings:
UPD Session Time Out 28800 Seconds
Enable Session Limit: checked 
Default Session per Host: 0

I've tried also to uncheck Enable Session Limit but nothing changed.

What could be the cause of this behavior? How can I check the source of this strange traffic? Could it be a "hacking attack"?

Thank you

Kari

USG_User

With our USG110 the UDP Session Time Out is 60 s and the default sessions by host is limited to 1000. This is more than enough for standard applications. Commonly a single host with us is using not more than 180 concurent sessions (in max), but normally not more than 10.

Until now we experienced problems with session limit only one time where our programmers have implemented wrong code which was permanently enquiring the current time from an external time server by creating a new session every seconds. This led to an exceeding of session limit for that host and it was not longer able to use the internet. All other workstations were not affected.

You should check your session monitot to figure out which station is causing this high session usage.

p4_greg

@kaika313

The main problem is your UDP session timeout is set to 28800 seconds(8 hours), you need to decrease this to something more reasonable such as 60-120sec.

Your current setting will cause the firewall to keep an open session for *every* UDP connection for 8 hours before it times out. Each DNS request, etc is going to cause another session to populate in the session table. Loading a single web page can generate hundreds of DNS requests, so these sessions are going to build up FAST, maxing out the session limit and causing new sessions to fail.

USG_User

With our USG110 the UDP Session Time Out is 60 s and the default sessions by host is limited to 1000. This is more than enough for standard applications. Commonly a single host with us is using not more than 180 concurent sessions (in max), but normally not more than 10.

Until now we experienced problems with session limit only one time where our programmers have implemented wrong code which was permanently enquiring the current time from an external time server by creating a new session every seconds. This led to an exceeding of session limit for that host and it was not longer able to use the internet. All other workstations were not affected.

You should check your session monitot to figure out which station is causing this high session usage.

kaika313

Hi @USG_User,

thank you for your advice. I've tried to set the session limit to 1000 per host and it seems that for now, active sessions stay within a normal range. Some devices went far beyond 1000 sessions each so I think they were causing the USG to crash. Most of these sessions were multiple UDP connections toward Google. I've noticed also that there's a huge amount of warnings about certain external IP addresses reaching the maximum session limit per host (is it possible that these were the cause of our problem?) and a lot of alerts about "abnormal udp traffic detected, source port is zero, DROP". The last one worries me... 

Thank you

USG_User

Regarding the increasing amount of "anormal UDP or TCP traffic with source or destination port zero"... Yes we're experiencing it, too. But at the moment only 4-5 times per day.

There is an active post here:

https://community.zyxel.com/en/discussion/12638/where-to-disable-alert-log-for-abnormal-tcp-traffic-detected#latest

Maybe you could report your experiences that Zyxel is aware of it. Don't know whether this is going to become the next kind of "denial of service" attack.

Asking google should normally not lead to an increased number of sessions. One request and one answer from google. Or what are you doing else with google? Further, ordinary google enquiries by webbrowser should be carried out via TCP not UDP. There must be another application which is causing this huge amount of UDP packets.

You should made further evaluations of your log to discover which hosts are affected. Then you should switch on/off single applications to see which one is responsible for the high number of sessions.

p4_greg

@kaika313

The main problem is your UDP session timeout is set to 28800 seconds(8 hours), you need to decrease this to something more reasonable such as 60-120sec.

Your current setting will cause the firewall to keep an open session for *every* UDP connection for 8 hours before it times out. Each DNS request, etc is going to cause another session to populate in the session table. Loading a single web page can generate hundreds of DNS requests, so these sessions are going to build up FAST, maxing out the session limit and causing new sessions to fail.

p4_greg

USG_User said:

[...]

Asking google should normally not lead to an increased number of sessions. One request and one answer from google.

Each UDP request is going to remain in the session table for 8 hours with OP's current session timeout setting, causing the session limit to be hit. Also Chrome and some other browsers are using QUIC protocol which basically transfers all HTTP(S) data via UDP instead of TCP.

USG_User

p4_greg said:

Each UDP request is going to remain in the session table for 8 hours with OP's current session timeout setting, causing the session limit to be hit. Also Chrome and some other browsers are using QUIC protocol which basically transfers all HTTP(S) data via UDP instead of TCP.

We have outgoing UDP rules in place only for DNS, NTP, IP Phones, and some specalities, but never for ordinary web surfing. And this works fine since years. Ports 80, 8080, 443 are only allowed via TCP.

Further just checked my own active sessions within USG (originated from my IP). The longest sessions are valid for about 2500 seconds.

Presently 10 users are in the office, working in the internet, making IP phone calls, ... and we got 1075 open sessions.