USG Flex 200 UDP out-of-order frames

Options
Chris2222
Chris2222 Posts: 3 image  Freshman Member
First Comment
in Security

Running tests with iPerf3 in UDP mode, I'm seeing very high rate of out-of-order frames reported during download (in on wan, out on lan), with a some but less out-of-order on uploads. Have tested at multiple sites, multiple devices, multiple ISPs. The percentage ramps up way sooner than it should:

Download rate

out-of-order

10Mbps

0.6%

20Mbps

4.2%

50Mbps

12.6%

100Mbps

20.0%

The above table is for a link that can sustains 800Mbps with TCP traffic. We monitor CPU load on these devices and at 100Mbps there is still should be plenty of capacity.

This came up because testing WireGuard use through these links, and it was performing terribly. Wireguard uses UDP, and troubleshooting led to packet capture which showed lots out-of-order frames as the likely cause. Running the same types of tests through a other unrelated sites typically produce hardly any out-of-order frames (<.5%).

Could this be a firmware bug? All devices on current firmware: V5.42(ABUI.1)

Suggestions for configuration changes? Sites with a handful of IPSec nailed up site-to-site tunnels seem worse than those with only on-demand VPN users. Also, enabling "Monitor>Traffic Statistics>Collect Statistics" seems to make the problem a bit worse.

Accepted Solution

  • Zyxel_Judy
    Zyxel_Judy Posts: 2,542 image  Zyxel Employee
    Zyxel Certified Network Engineer Level 2 - Nebula Zyxel Certified Network Engineer Level 2 - Switch Zyxel Certified Network Engineer Level 2 - Security Zyxel Certified Network Engineer Level 1 - Nebula
    Answer ✓

    Hi @Chris2222 ,

    It seems like the issue is caused by UDP Disperse — a multi-core packet distribution feature on the USG Flex 200. When enabled, UDP packets are spread across multiple CPU cores for parallel processing. Since each core finishes at a slightly different time, packets arrive at the LAN interface out of sequence. TCP traffic is unaffected because it has built-in sequencing and retransmission, but UDP (including WireGuard) has no such protection — hence the out-of-order frames you observed.

    Please confirm whether UDP Disperse enable by run the following command on your USG FLEX 200 via CLI:
    debug system no-udp-disperse show

    If the result is no, please run the command:
    debug system no-udp-disperse active

    You can verify the change with:
    debug system no-udp-disperse show
    No UDP Disperse : Yes

    Once No UDP Disperse shows Yes, UDP packets will be processed sequentially, and out-of-order frames should drop significantly. Please try it!

    Zyxel_Judy

All Replies

  • Zyxel_Judy
    Zyxel_Judy Posts: 2,542 image  Zyxel Employee
    Zyxel Certified Network Engineer Level 2 - Nebula Zyxel Certified Network Engineer Level 2 - Switch Zyxel Certified Network Engineer Level 2 - Security Zyxel Certified Network Engineer Level 1 - Nebula
    Answer ✓

    Hi @Chris2222 ,

    It seems like the issue is caused by UDP Disperse — a multi-core packet distribution feature on the USG Flex 200. When enabled, UDP packets are spread across multiple CPU cores for parallel processing. Since each core finishes at a slightly different time, packets arrive at the LAN interface out of sequence. TCP traffic is unaffected because it has built-in sequencing and retransmission, but UDP (including WireGuard) has no such protection — hence the out-of-order frames you observed.

    Please confirm whether UDP Disperse enable by run the following command on your USG FLEX 200 via CLI:
    debug system no-udp-disperse show

    If the result is no, please run the command:
    debug system no-udp-disperse active

    You can verify the change with:
    debug system no-udp-disperse show
    No UDP Disperse : Yes

    Once No UDP Disperse shows Yes, UDP packets will be processed sequentially, and out-of-order frames should drop significantly. Please try it!

    Zyxel_Judy

  • PeterUK
    PeterUK Posts: 4,506 image  Guru Member
    250 Answers 2500 Comments Friend Collector Eighth Anniversary

    You would think this would be known by Apps in order to handle out-of-order frames better like I can't understand that this should be a some what simple fix? Just buffer all packets that are out-of-order for like 1000ms to have them rechecked as the order comes through for matching out-of-order that was ahead of time.

  • Chris2222
    Chris2222 Posts: 3 image  Freshman Member
    First Comment

    Much better…

    Download rate/target

    out-of-order

    10Mbps

    0.0%

    20Mbps

    0.0%

    50Mbps

    0.1%

    100Mbps

    0.1%

    200Mbps

    0.5%

    500Mbps

    9.3%

    750Mbps

    15.3%

    1000Mbps

    51.5%

    (testing transits a Flex200, typical: iperf3 -c x.x.x.x -u -b 200M -R)

    I'd still like less out-of-order, but this is probably reasonable for this class of hardware. Soon I'll be testing Wireguard tunnels again.

    Now, how do I make the no-udp-disperse active persistent so that it lives through a device reboot?

  • Zyxel_Judy
    Zyxel_Judy Posts: 2,542 image  Zyxel Employee
    Zyxel Certified Network Engineer Level 2 - Nebula Zyxel Certified Network Engineer Level 2 - Switch Zyxel Certified Network Engineer Level 2 - Security Zyxel Certified Network Engineer Level 1 - Nebula

    Hi @Chris2222 ,

    how do I make the no-udp-disperse active persistent so that it lives through a device reboot?

    This CLI still remains after a device reboot.

    Zyxel_Judy

  • p4_greg
    p4_greg Posts: 41 image  Freshman Member
    Network Detective-New Adventure Badge First Comment Friend Collector Fourth Anniversary

    Hi @Zyxel_Judy

    Is there a similar command available on FLEX H series to control UDP multi-core processing?

  • Zyxel_Judy
    Zyxel_Judy Posts: 2,542 image  Zyxel Employee
    Zyxel Certified Network Engineer Level 2 - Nebula Zyxel Certified Network Engineer Level 2 - Switch Zyxel Certified Network Engineer Level 2 - Security Zyxel Certified Network Engineer Level 1 - Nebula

    Hi @p4_greg ,

    There is currently no CLI command available to control UDP multi-core processing on the USG FLEX H series.

    May we ask whether you are also experiencing UDP out-of-order frame issues on the USG FLEX H firewall? If yes, please share details, so we can help investigate further.

    Zyxel_Judy

  • p4_greg
    p4_greg Posts: 41 image  Freshman Member
    Network Detective-New Adventure Badge First Comment Friend Collector Fourth Anniversary

    Hi @Zyxel_Judy

    Currently I am not experiencing any issues that I can attribute to UDP out-of-order processing, but I was recently troubleshooting issues with VOIP RTP audio dropouts on a FLEX 100H, and having a way to control UDP processing could be a useful option to test. I just though having an option to control this might be helpful for troubleshooting in the future.

  • Zyxel_Judy
    Zyxel_Judy Posts: 2,542 image  Zyxel Employee
    Zyxel Certified Network Engineer Level 2 - Nebula Zyxel Certified Network Engineer Level 2 - Switch Zyxel Certified Network Engineer Level 2 - Security Zyxel Certified Network Engineer Level 1 - Nebula
    edited May 26

    Hi @p4_greg ,

    Thank you for the information.

    Since USG FLEX and USG FLEX H are different platforms, USG FLEX H does not require the same mechanism, and there are no equivalent CLI commands available.

    As for the VoIP RTP audio dropout issue on the FLEX 100H, if the problem persists, please feel free to open a post with detail information as typology, configuration and symptom at USG FLEX H Series - Zyxel Community and we'll be happy to assist.

    Zyxel_Judy

Categories

Security Help Center

EnglishTiếng ViệtРусскийไทย中文（台灣）