Slowness on a new WAN setup with our zywall 110

My_IT_Hurts

Hello people,

I woud like to submit a weird issue that we're encountering.

We got a change in our WAN config a few days ago and have had issues since.

Basically, before our zywall 110 was handling 3 DSL lines (two 8Mb adsl and one 8Mb sdsl) , and we got an unlimited 4G link added to that (as optic fiber is just not coming anytime where we are...), so right now we have not yet cancelled the 2 adsl but the zywall is now just handling the 4G and the 8Mb SDSL.

Except that the users have been complaining about internet being slow and very problematic, even though speed tests are amazing for us considering our "history" (100Mbps DL and 70Mbps UL) and our download test was a debian iso at around 7MB/s.

I set up the 4G to handle most of the traffic (by setting the sdsl as passive in the trunk config), but the sdsl still handles the vpn connections or other nat related stuff.

After digging in, the issue is weird : most single files downloads (mostly pdfs) don't even manage to finish and clock around at 5Kb/s, many ending in failure in firefox or chrome). Trying to download a .msi from Eset or most files just don't work but some eventually end (like 2 hours to DL a 160MB file).

Web browsing is great , youtube works in HD with no buffering time, iso from debian still downloads at high speeds, it's just the pdf, and other file downloads that won't work. (seriously wtf)

Weirdly, uploading a 10MB pdf file attachment when sending from thunderbird works fine as 4G UL speed makes it almost instantaneous.

Now, while the zywall had always self-handled the dsl connections (all ppoe), the 4G is connected through our provider router on top and we just setup a small "vlanned" network through a switch between the two (so in network / interface the wan2 has a fixed lan IP and i just added that to the wan trunk)

So far nothing i tried works, the MTU or ingress/egress speed, the negociation speed...

But if i connect a pc right behind our provider router, then everything works amazing and iso files like the debian now even downloads at like 16MB/s (instead of 7MB/s). PDFs, msi or exe download fine this way.

The only plugin on the zywall is the content filter and disabling it doesn't change a thing.

i don't see any packet errors on the switch, everything is negociating at 1GB.

Thanks for reading, ideas are very welcome !

Best regards

My_IT_Hurts

i'll also point out that our 4G provider says he can't see what's wrong, everything is obviously perfect as far as their router goes

PeterUK

Do you have the anti-virus on in UTM profile?

My_IT_Hurts

Hi PeterUK, 

no we only have the content filter license.  It's got to be a detection process being activated.
i can clearly see when DL the msi from eset the speed starting at 2MB/s, and after 1 sec it immediately drops to zero, for a couple of seconds then goes up to 10Kb, for a while, then sometimes finishes if you're patient enough, sometimes drops.

This morning i've even had someone telling me he could log on specific website then not do anything on it like like fill and confirm some forms. As soon as i routed this website traffic to the sdsl, it was perfect again. 
Our AD handles the dns stuff so it's the same for both links (and i'm using the 4G link right now).

here is a screenshot (sorry, it's in french, but i think the firefox UI is the same everywhere^^)

gb5102

@My_IT_Hurts

To me this sounds like it could be an issue with the MTU size set on the Zywall interface connected to the LTE modem. You can try setting a lower MTU on the LTE interface, I would try setting it down to 1300 and see if it makes any difference.

If it helps, you can use the following method to determine the required MTU size: https://customer.cradlepoint.com/s/article/how-to-determine-the-mtu-size-of-the-lan-and-wan-of-your-router

PeterUK

Are the files that are downloading slowly by HTTP ?

What happens if you connect the 4G to a PC and download the same slow downloads?

My_IT_Hurts

@gb5102 : thanks, i didn't the value could be so low for 4G. One thing is that i suppose the provider router is supposed to has the specific 4g mtu setting ? Shouldn't our connection still be at 1500 for the MTU since it's router to router ?

edit : i completely forgot the -f flag in ping commands : https://kb.netgear.com/19863/Ping-Test-to-determine-Optimal-MTU-Size-on-Router
1400 is when it stops fragmenting, i just tried putting 1428 or 1400 but no chance.

@PeterUK : a pc directly connected to our provider router will download the problematic files fine at full speed.

By the way, i discovered that we had a cable in the loop (between our switch and our provider router...) that made the connection to 100M and not 1000M. So now a nperf or .iso files or wapt files (like https://store.wapt.fr/store/tis-chrome) also download at full speed up to 16MB/s and not 8MB/s like before.
But it has no impact on the .pdf .zip or exe/msi files

I finally got back a spare zywall 110 that was used on another site, so i'll just reset it to default, open firewall completely, make just the change needed to test and i'll see... This week-end i upgraded to latest firmware (4.65 aaaa.1) and rebooted it but also no change

PeterUK

Can you try the OPT port.

I can't think why the speed is being so odd a way out their thing could be the 4G provider is dong something with TTL on give routes I highly doubt it but you can try I mean I have seem routers edimax with a TTL option.

netsh interface ipv4 set global defaultcurhoplimit=65

netsh interface ipv4 set global defaultcurhoplimit=129

My_IT_Hurts

For the OPT port, you think there can be some profiling going on ?
Since the router is on another site, this test will have to wait until friday.

For the TTL, what is the default parameter ?

Thanks for the quick answers by the way

PeterUK

My_IT_Hurts said:

For the OPT port, you think there can be some profiling going on ?
Since the router is on another site, this test will have to wait until friday.

For the TTL, what is the default parameter ?

Thanks for the quick answers by the way

Just trying to be sure thats all it likely will not make a difference the default for windows is 128 the idea is with PC to 4G provider router your TTL is 128 but with it connected to the zywall you be TTL 127 so having your PC by 129 you get to the 4G provider router TTL 128 it should not make a difference however.
  

Zyxel_Stanley

Hi @My_IT_Hurts
You can have a test MTU size by mturouter.exe tool. You can download the test tool from this website. Run the test tool by CLI command, and it will auto test MTU size between client to server in whole the route path.

And also you can capture packets on LAN and WAN interface during download PDF file an share .cap files to us for further check.