USG Flex 200 UDP out-of-order frames

Chris2222

Running tests with iPerf3 in UDP mode, I'm seeing very high rate of out-of-order frames reported during download (in on wan, out on lan), with a some but less out-of-order on uploads. Have tested at multiple sites, multiple devices, multiple ISPs. The percentage ramps up way sooner than it should:

Download rate	out-of-order
10Mbps	0.6%
20Mbps	4.2%
50Mbps	12.6%
100Mbps	20.0%

The above table is for a link that can sustains 800Mbps with TCP traffic. We monitor CPU load on these devices and at 100Mbps there is still should be plenty of capacity.

This came up because testing WireGuard use through these links, and it was performing terribly. Wireguard uses UDP, and troubleshooting led to packet capture which showed lots out-of-order frames as the likely cause. Running the same types of tests through a other unrelated sites typically produce hardly any out-of-order frames (<.5%).

Could this be a firmware bug? All devices on current firmware: V5.42(ABUI.1)

Suggestions for configuration changes? Sites with a handful of IPSec nailed up site-to-site tunnels seem worse than those with only on-demand VPN users. Also, enabling "Monitor>Traffic Statistics>Collect Statistics" seems to make the problem a bit worse.

Zyxel_Judy

Hi @Chris2222 ,

It seems like the issue is caused by UDP Disperse — a multi-core packet distribution feature on the USG Flex 200. When enabled, UDP packets are spread across multiple CPU cores for parallel processing. Since each core finishes at a slightly different time, packets arrive at the LAN interface out of sequence. TCP traffic is unaffected because it has built-in sequencing and retransmission, but UDP (including WireGuard) has no such protection — hence the out-of-order frames you observed.

Please confirm whether UDP Disperse enable by run the following command on your USG FLEX 200 via CLI:
debug system no-udp-disperse show

If the result is no, please run the command:
debug system no-udp-disperse active

You can verify the change with:
debug system no-udp-disperse show
No UDP Disperse : Yes

Once No UDP Disperse shows Yes, UDP packets will be processed sequentially, and out-of-order frames should drop significantly. Please try it!

Zyxel_Judy

Hi @Chris2222 ,

It seems like the issue is caused by UDP Disperse — a multi-core packet distribution feature on the USG Flex 200. When enabled, UDP packets are spread across multiple CPU cores for parallel processing. Since each core finishes at a slightly different time, packets arrive at the LAN interface out of sequence. TCP traffic is unaffected because it has built-in sequencing and retransmission, but UDP (including WireGuard) has no such protection — hence the out-of-order frames you observed.

Please confirm whether UDP Disperse enable by run the following command on your USG FLEX 200 via CLI:
debug system no-udp-disperse show

If the result is no, please run the command:
debug system no-udp-disperse active

You can verify the change with:
debug system no-udp-disperse show
No UDP Disperse : Yes

Once No UDP Disperse shows Yes, UDP packets will be processed sequentially, and out-of-order frames should drop significantly. Please try it!

PeterUK

You would think this would be known by Apps in order to handle out-of-order frames better like I can't understand that this should be a some what simple fix? Just buffer all packets that are out-of-order for like 1000ms to have them rechecked as the order comes through for matching out-of-order that was ahead of time.

Chris2222

Much better…

Download rate/target	out-of-order
10Mbps	0.0%
20Mbps	0.0%
50Mbps	0.1%
100Mbps	0.1%
200Mbps	0.5%
500Mbps	9.3%
750Mbps	15.3%
1000Mbps	51.5%

(testing transits a Flex200, typical: iperf3 -c x.x.x.x -u -b 200M -R)

I'd still like less out-of-order, but this is probably reasonable for this class of hardware. Soon I'll be testing Wireguard tunnels again.

Now, how do I make the no-udp-disperse active persistent so that it lives through a device reboot?

Zyxel_Judy

Hi @Chris2222 ,

how do I make the no-udp-disperse active persistent so that it lives through a device reboot?

This CLI still remains after a device reboot.

p4_greg

Hi @Zyxel_Judy

Is there a similar command available on FLEX H series to control UDP multi-core processing?

Zyxel_Judy

Hi @p4_greg ,

There is currently no CLI command available to control UDP multi-core processing on the USG FLEX H series.

May we ask whether you are also experiencing UDP out-of-order frame issues on the USG FLEX H firewall? If yes, please share details, so we can help investigate further.

p4_greg

Hi @Zyxel_Judy

Currently I am not experiencing any issues that I can attribute to UDP out-of-order processing, but I was recently troubleshooting issues with VOIP RTP audio dropouts on a FLEX 100H, and having a way to control UDP processing could be a useful option to test. I just though having an option to control this might be helpful for troubleshooting in the future.

Zyxel_Judy

Hi @p4_greg ,

Thank you for the information.

Since USG FLEX and USG FLEX H are different platforms, USG FLEX H does not require the same mechanism, and there are no equivalent CLI commands available.

As for the VoIP RTP audio dropout issue on the FLEX 100H, if the problem persists, please feel free to open a post with detail information as typology, configuration and symptom at USG FLEX H Series - Zyxel Community and we'll be happy to assist.