USG60 <> USG200 Site2Site VPN stopped working after WAN IP change

stephan
stephan Posts: 31  Freshman Member
First Anniversary 10 Comments Friend Collector

We changed the WAN IP on our USG200 site today.

I thought I just needed to change the IP in the VPN configuration on both ends and it would work. For our other Site2Site VPN between two USG200s, this has worked. But the connection between the USG60 (site handle "LHS") and USG200 (site handle "MOL") do seem to connect as the icons show a "connected" status, but the USG200 is not receiving any packages from the USG60 (other direction lists traffic … see screenshot below). Regardless, the USG60 can not ping servers in the USG200 network anymore since the change.

I'll drop a couple of details following.

Here are the USG60 "LHS" gateway settings:

Here are the USG60 "LHS" connection settings:

Here are the USG200 "MOL" gateway settings

Here are the USG200 "MOL" connection settings

In the VPN monitor, the USG60 lists sent and received packages. Additionally, this connection has a greyed out "Connection Check" button.

In the VPN monitor of the USG200, it only lists sent packages but no received packages. The "Connection Check" button is working here, but I am getting a timeout.

Both show uptimes that are basically identical, making me think that the connection itself works.

I checked the PSKs and the respective WAN IPs, of course, and they seem to match. Curiously, the USG60 has a working VPN connection to our other office, which is also a USG200 (but the IP didn't change there). The settings for both connections are identical, aside from the different WAN targets and subnets at the target site.

The IKE logs show correct key handshakes. No errors in the IPsec logs, which I really do not understand. I must be missing something, but I can't figure out what. Anyone have any pointers?

Best Answers

  • PeterUK
    PeterUK Posts: 2,655  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    edited April 2023 Answer ✓

    Its sad that an ISP only thinks there are three protocols one uses 1, 6 and 17 and forgets 50 and 47😞
    If you put one end behind a NAT router then that should work to use UDP port 4500 (protocols 17).

  • stephan
    stephan Posts: 31  Freshman Member
    First Anniversary 10 Comments Friend Collector
    Answer ✓

    Okay these seem to have been transient issues.

    I restarted the USSG60 once again between my last post and now (it was restarted before my last post) without any configuration changes and it works now. Key handshake runs on USP4500 now as expected. All other stages of the VPN work.

    I think my USG 60 at the branch office had some latent borked configuration. After the first reboot, some changes from the past few months (without reboots) were gone. Nothing too major, but odd nevertheless. Good thing we are switching to a newer USG200 here soon.

    Special thanks to PeterUK who brought me on the right track.

  • PeterUK
    PeterUK Posts: 2,655  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    edited April 2023 Answer ✓

    No putting one behind  NAT will use port 4500 as the tunnel port 500 is used for exchange keys

    Can you check by packet capture on the USG you are sending and receiving either 4500 or protocol 50 when sending ping

    its also possible your ISP beyond support knows nothing of the block and just tell you ESP is not blocked

    Edit: I see its all working😁

«1

All Replies

  • mMontana
    mMontana Posts: 1,298  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer

    I'd go with these steps.

    1: disable gateways on both sides.
    2: double check both gateway sides
    3: at "the center" (usually, one is the hub, the other is the spoke) disable "Nailed up". IMO ony the spoke side should have "nailed up" enabled.
    4: keep connection check disabled on both sides
    5: enable the Hub side gateway, then the spoke side gateway. This should lead to a working VPN

    Side consideration: the new WAN ip… comes with a new connection and a new CPE? Are you sure that the "new connection" receives both ports UDP 500 and 4500?

  • PeterUK
    PeterUK Posts: 2,655  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    edited April 2023

    Do both USG have the WAN IPs on the WAN interfaces? or is one or both behind NAT?

    Any routing rules with red status?

  • stephan
    stephan Posts: 31  Freshman Member
    First Anniversary 10 Comments Friend Collector
    edited April 2023

    I tried this, no change.
    The hub had the IP change. Connectivity Check was never enabled on any of our VPN connections so far, afaik.

    "Are you sure that the "new connection" receives both ports UDP 500 and 4500?"
    No, but I strongly suspect so, because the other spoke (another USG200 at a 2nd branch, see initial post at the top) works fine. I only changed the changed WAN IP as appropriate for the VPN gateway settings, and it worked instantly. Both sides for the working spoke have Nailed Up enabled.

  • stephan
    stephan Posts: 31  Freshman Member
    First Anniversary 10 Comments Friend Collector

    Yes, both USGs have WAN IPs on the WAN Interfaces. Neither of them are behind a NAT.

    Where can I find if a routing rule has a red status? There is only one routing rule applicable to this VPN connection on the USG60 side, and it looks correct. This is the setting there:

    MOL_ALL_LANS just contains all the subnets on the MOL side. But the servers I am trying to reach there are in the primary subnet of the MOL USG200.

    The logs doesn't show errors for policy routes on either of the two sides.

  • PeterUK
    PeterUK Posts: 2,655  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    edited April 2023

    You can find the status of a routeing rule by looking at the rule list on the left with a light bulb icon normally its yellow.

    If both sides have the WAN IP you be using protocol 50 not UDP port 4500 so it might be your ISP is blocking this.

    Do a packet capture on the wan as you send pings to a target IP down the VPN both ends of the tunnel that allows ping you should see ESP going out if you don't see incoming then likely your ISP is blocking protocol 50.

  • stephan
    stephan Posts: 31  Freshman Member
    First Anniversary 10 Comments Friend Collector

    "You can find the status of a routeing rule by looking at the rule list on the left with a light bulb icon normally its yellow. "
    All rules are yellow, none are red. Is that what you mean.

    "If both sides have the WAN IP you be using protocol 50 not UDP port 4500 so it might be your ISP is blocking this."
    That might actually be it. I see the branch sending keys and the headquarters not picking it up. I'll talk to the ISP, as this must be new since we swapped IP as the tunnel was working before.

    I'll get back once I did this.

  • stephan
    stephan Posts: 31  Freshman Member
    First Anniversary 10 Comments Friend Collector

    "If both sides have the WAN IP you be using protocol 50 not UDP port 4500 so it might be your ISP is blocking this."

    ISP can't dig into this during the weekend. On the one branch that can connect, the FW has the WAN port set to an internal IP (which I guess is why VPN Tunnel there still works) and the IKE logs show key exchanges on port 4500.

    What options do I have? Can I force the Firewalls to use 4500 instead of protocol 50?

  • PeterUK
    PeterUK Posts: 2,655  Guru Member
    First Anniversary 10 Comments Friend Collector First Answer
    edited April 2023 Answer ✓

    Its sad that an ISP only thinks there are three protocols one uses 1, 6 and 17 and forgets 50 and 47😞
    If you put one end behind a NAT router then that should work to use UDP port 4500 (protocols 17).

  • stephan
    stephan Posts: 31  Freshman Member
    First Anniversary 10 Comments Friend Collector

    I'll talk more in detail with the ISP tomorrow. Especially also about why this was changed when we only changed our IP. Because the tunnel did work before the change.

  • stephan
    stephan Posts: 31  Freshman Member
    First Anniversary 10 Comments Friend Collector

    Talked to ISP: They do not block ESP.

    Unfortunately, it seems that the statement that UDP 4500 is used when one part is behind NAT is also not correct. I put the branch USG60 behind a local NAT and the firewalls still try to connect via UDP:500 to exchange keys. And I am observing very strange behavior.

    On the USG200 (main office, where the IP changed) This is showing connected (it's not). Disabling the gateway, disabling the connection and clicking on disconnect does not set the connection icon to disconnected. It stays connected.

    I had the same problem on the USG60 (branch office) and after a reboot the connection state is correctly displayed as disconnected.

    But even stranger. If I enable both gateways and connections, and click on connect on the USG200 on the main office side, the branch office USG60 receives IKE packets from the USG200.

    The blacked out IP is the new IP of the USG200 of the main office. The other is the now NATed IP of the USG60 at the branch. What is going on here?

Security Highlight