USG FLEX 500H/uOS dynamic policy-based IPsec issue – RCO/R-CARD traffic works on VPN100/USG but fail

Updated info:

Subject: FLEX 500H / uOS – Dynamic policy-based IKEv1 IPsec return traffic issue with RCO/R-CARD UC devices

Hello Zyxel Support,

We are replacing an older Zyxel VPN100/ZLD firewall with a new Zyxel FLEX 500H running uOS/Nebula.

The old VPN100/ZLD firewall works with the same remote sites, same Teltonika RUT241 routers, same RCO/R-CARD server and same UC devices.

With the FLEX 500H/uOS, the IKEv1 policy-based IPsec tunnels establish, and the FLEX logs show accepted traffic from the central RCO server towards the remote UC devices. However, the RCO/R-CARD application does not work correctly.

Environment

Central firewall:

Zyxel FLEX 500H / uOS / Nebula
Public IP: 217.78.20.244
LAN subnet: 192.168.90.0/24
RCO/R-CARD server: 192.168.90.21

Remote site example:

Teltonika RUT241
Remote LAN: 192.168.174.0/24
UC device: 192.168.174.9
Remote WAN/public IP seen in logs: 83.255.16.246

The application traffic is RCO/R-CARD communication between:

192.168.90.21 <-> 192.168.174.9

Main observed UDP ports are:

UDP/1000
UDP/9000
UDP/9001

What we have confirmed

The IPsec tunnel is not simply down.

The FLEX log shows DPD activity with the remote peer, so the IKE tunnel was alive during the test window. It also logs accepted LAN-to-IPsec traffic from the RCO server 192.168.90.21 to the remote UC 192.168.174.9.

Example from the FLEX log:

2026-08-10 16:19:39–16:19:41
Source:      192.168.90.21
Destination: 192.168.174.9
Protocol:    UDP
Dst ports:   1000 / 9000
Rule:        TEST_RCO_to_ALL_VPN
Direction:   LAN to IPSec_VPN
Action:      ACCEPT

The same log window shows several accepted packets from 192.168.90.21 to 192.168.174.9 on UDP/1000 and UDP/9000.

The remote RUT also later shows a clean current IPsec state with the expected selectors:

192.168.174.0/24 === 192.168.90.0/24

and packet counters increasing in both directions.

Problem description

The important issue is not basic IKE establishment or a missing LAN-to-VPN allow rule.

The FLEX 500H logs show that traffic from the central RCO server is accepted and sent towards the UC device. However, the RCO application still fails, and in the exported FLEX log we do not see the expected matching UC-to-server application traffic being passed back from:

192.168.174.9 -> 192.168.90.21

The same RCO/UC communication works when the old VPN100/ZLD firewall is used as the central firewall.

Working hypothesis

Our current suspicion is that FLEX/uOS may be handling return traffic from dynamic policy-based IKEv1 peers differently than the older ZLD platform.

Possible areas to verify:

- Dynamic policy-based IKEv1 peer handling
- Traffic selector / Child SA matching for return traffic
- IPsec_VPN -> LAN zone classification
- Return-path handling from remote dynamic IPsec peer into LAN
- Difference between ZLD VPN100 and uOS FLEX 500H behavior

What we need help with

Please verify whether FLEX 500H/uOS supports this scenario in the same way as VPN100/ZLD:

Central firewall as dynamic policy-based IKEv1 responder/server side
Multiple dynamic remote peers
Local subnet: 192.168.90.0/24
Remote subnet example: 192.168.174.0/24
Application server: 192.168.90.21
Remote UC: 192.168.174.9
Application traffic: UDP/1000, UDP/9000, UDP/9001

We need to confirm whether the FLEX is:

1. Decrypting packets from the remote UC device.
2. Matching them to the correct Child SA / traffic selector.
3. Classifying them into the expected IPSec_VPN -> LAN path.
4. Passing them to the LAN server 192.168.90.21.

Please advise what debug commands, packet capture points, or support logs are required on FLEX/uOS to prove whether return packets from the remote UC are being received, decrypted and forwarded correctly.

The key comparison is:

Same remote Teltonika RUT241 + UC device works through old VPN100/ZLD.
Same setup through FLEX 500H/uOS establishes IPsec and logs accepted LAN-to-IPsec traffic, but RCO/R-CARD communication fails.

Best regards,

Hello,

We can place the RCO/R-CARD system behind the USG FLEX 500H again and run the real system.

With the old VPN100/ZLD firewall, the RCO/R-CARD system works correctly. In the working packet capture from VPN100/ZLD, we can see the normal real application dialogue on UDP/1000:

192.168.90.21:1000 -> 192.168.175.5:1000
192.168.175.5:1000 -> 192.168.90.21:1000

Our understanding is that the UC device sends a request to the RCO server on UDP/1000, and the RCO server answers.

With the USG FLEX 500H, the IPsec tunnel was established and basic connectivity worked. We could ping between the RCO server and the remote UC device, and manual test traffic could pass through the tunnel.

However, when running the real RCO/R-CARD system behind the FLEX 500H, we did not observe the expected real RCO/R-CARD UDP/1000 application dialogue:

192.168.175.5:1000 -> 192.168.90.21:1000
192.168.90.21:1000 -> 192.168.175.5:1000

The application communication does not start/work as it does with the old VPN100/ZLD firewall.

So we can reproduce the same setup and run the real RCO/R-CARD system behind the FLEX 500H again, but we cannot make the real application traffic pass correctly in both directions, because that is the issue we are troubleshooting.

The important difference is that with VPN100/ZLD we see the real two-way application dialogue on UDP/1000, including the UC device response back to the RCO server.

With FLEX 500H, the tunnel is up and the firewall logs accepted server-to-UC traffic, but the expected real application response/dialogue from the UC device back to the RCO server does not work the same way.

This has been seen with the Teltonika remote sites we have tested. We have also seen similar behavior with Nebula-managed remote devices, so it does not look like this is only a Teltonika-specific issue.

Please confirm if you want us to:

1. Put the RCO/R-CARD system behind the FLEX 500H again,
2. Run the real RCO/R-CARD test,
3. Notify you when the test is running or completed, so you can check the traffic on the FLEX 500H side.