Zyhel

NikitaHarashchuk

Hello, I need help configuring L2TP/IPSec VPN on Zyxel USG FLEX 100.

Problem description:

I configured L2TP VPN via the Quick Setup Wizard for remote access.
Using Windows 11 client, authentication method PSK.
When connecting, Windows shows:

"The L2TP connection attempt failed because of a security layer error occurred during initial negotiations with the remote computer."

In the Zyxel logs I can see the connection drops right after NONCE exchange in IKE phase (SA 1 never comes up).

Log fragment:

IKE SA [] is disconnected
185.152.139.170:500
23.139.82.62:47084
...
[INIT] Recv: [NOTIFY][NONCE]
Receiving IKEv2 request

What I have tried:

• Configured VPN via Wizard.
• Tested connection from LAN and external network — same result.
• Double-checked PSK, IKE and IPSec services are enabled.
• Allowed/forwarded UDP 500/4500/1701 on WAN interface.
• Followed all steps from winitpro.ru guide.

The strange part:

In Configuration > VPN > IPSec VPN > VPN Gateway, under Encryption, the only available option is DES.

AES128/256 and 3DES are completely missing.

The issue is that:

• Windows client by default uses DES/SHA1/DH2, so DES should match, but the tunnel never comes up.
• On older firmware versions (based on community feedback) AES/3DES could be selected.
• On my firmware (V5.40(ABUH.0), 2025-05-07) only DES is available.

Questions:

1. Is this a bug in current firmware V5.40, or is it intentional (DES-only left for compatibility)?
2. Is there a way to add AES/3DES to IPSec Proposal for L2TP via CLI?
3. Should I downgrade to V5.39 (or earlier) where AES/3DES was available?
4. Am I maybe looking in the wrong place, and encryption proposals are configured separately?

Configuration details:

• Model: USG FLEX 100
• Firmware: V5.40(ABUH.0) (2025-05-07)
• Standby: V5.39(ABUH.1) (2024-11-16)
• Client: Windows 11 (L2TP/IPSec PSK)

PeterUK

Not sure why you only see DES

Delete the Wizard setup and do it manually

windows default is
Phase 1
3DES SHA1
key group DH2
Phase 2
AES256 SHA1
PFS none

Phase 1 VPN gateway
negotiation mode = main

Phase 2 VPN connection
Remote Access (Server Role)
local policy 0.0.0.0
encapsulation Transport

setup L2TP over IPSec

Zyxel_Tina

Hi @NikitaHarashchuk,

We’ve tested the same model (USG FLEX 100) with the same firmware version V5.40(ABUH.0), and after configuring the VPN via the Wizard, going to Configuration > VPN > IPSec VPN > VPN Gateway, we can see the Phase 1 proposal is 3DES/SH1 and the full set of encryption options (DES/3DES/AES128/AES192/AES256).

To move forward, we recommend:

1. Backup your configuration first.
2. Double-check whether your current VPN settings fully match the wizard-created config.
3. If possible, try deleting the existing VPN configuration and recreating it from scratch using the Wizard.
4. If the issue still persists, please refer to the commands shown in the configuration image above and set them up via CLI.