L2TP LDAP AAA Authentication fails in new Firmware V4.70(AALA.0) - rollback to V4.65(AAKY.0) =OK!

warwickt
warwickt Posts: 111  Ally Member
First Anniversary Friend Collector First Answer First Comment

Attention Zyxel lads 我的台湾朋友, this problem reproduceable on USG40 & USG60 with new Firmware update V4.70(AAKY.0)


Scenario

After Upgraded lab USG40 and USG60 lab routers from V4.65(AAKY.0) to V4.70(AAKY.0) over weekend and verification testing we found that........

Problem Issue

All L2TP client connections that require AAA LDAP authentication ( group ldap) are consistently failed in L2TP "authentication failed" on both USG40 and USG60 at Firmware V4.70(AAKY.0)
<local2.debug> usg40 msf-usg40-01 src="222.XX6.1XX.XXX:1701" dst="14.0.XXX.XXX:45716" msg="Message: PPP failure: Reason: Authentication failed" note="L2TP_LOG" user="unknown" devID="1c740dfec31c" cat="L2TP"


Refine:

It is only the L2TP client accounts requiring LDAP PAP authentication that are fail authentication.
  • :) all Local accounts [ the don't rely on AAA LDAP authentication ] in these routers are authenticated against the router's password = work correctly.

Reproduceable:

Yes, on USG40 and USG60 and only at firmware update V4.70(AAKY.0)

:# Workaround: (hmmm ...)

  1. add LDAP users as LOCAL users in router - (works but wont be doing that!)   :s
  2. rollback USG40, USG60 routers to firmware V4.65(AAKY.0)  - then L2TP AAA LDAP works 100% again!.  :/
Coarse Diags & Conclusion (and suspicion):
V4.70(AAKY.0) firmware for L2TP PPP subcomponent is not processing handoff to AAA LDAP correctly.

Controls:

  • LDAP server access available from USG40/ USG60 for both firmware levels : V4.65(AAKY.0) to V4.70(AAKY.0 .. example test LDAP account 'casper' resolves correctly to the external LDAP server from the USG routers themselves:
 Router> test aaa server ldap host ldapserver.lab.studio port 389&nbsp; base-dn dc=ldapserver,dc=lab,dc=studio bind-dn uid=diradmin_msf,cn=users,dc=ldapserver,dc=lab,dc=studio password XXXXXXXXXXXXXXXXXXXX login-name-attribute uid account casper
resolves correctly to the external LDAP server as:

<div>dn: uid=casper,cn=users,dc=ldapserver,dc=lab,dc=studio</div><div>cn: casper</div><div>uid: casper</div><div>uidNumber: 1027</div><div>sn: casper</div><div>objectClass: person</div><div>objectClass: inetOrgPerson</div><div>objectClass: organizationalPerson</div><div>objectClass: posixAccount</div><div>objectClass: shadowAccount</div><div>objectClass: top</div><div>objectClass: extensibleObject</div><div>gidNumber: 220</div><div>.... snip snip<span>
67122401325669019051132627070243663537895756704320186012999399913688</span><br></div><div>&nbsp;48707 root@LDAPSERVER.LAB.STUDIO:10.0.99.201</div><div>authAuthority: ;Kerberosv5;;casper@LDAPSERVER.LAB.STUDIO;LDAPSERVER.LAB.STUDIO;</div><div>altSecurityIdentities: Kerberos:casper@LDAPSERVER.LAB.STUDIO</div><div>apple-generateduid: 47319A36-323C-4A6A-AFDE-FBF36F2B394A</div><div>description: mocked up LDAP user for LDAP L2TP testing</div><div>loginShell: /bin/bash</div><div>.... snip snip&nbsp;</div><div>homeDirectory: /Network/Servers/LDAPSERVER.LAB.STUDIO/Volumes/ssh_group_home/casper</div><div>Router></div>
see attachments...

Operational Overview - L2TP AAA LDAP and Local Authentication = FAILURE

1. Basic stuff, all L2TP-over-IPSEC VPN connections pass through Phase 1 VPN Gateway then to Phase 2 VPN Connection and the tunnel is successuly built :
msg="Dynamic Tunnel [Client01_L2TP_usg40_GATEWAY:Client01_L2TP_usg40_CONNECTIO:0x0e359c53] built successfully" note="IKE_LOG" user="unknown" devID="1cffffffffff"

msg="Looking up IPSec SA for the L2TP tunnel" note="L2TP_LOG" user="unknown" devID="1cffffffffff" cat="L2TP"
msg="Fetching IPSec SA rule #354" note="L2TP_LOG" user="unknown" devID="1cffffffffff" cat="L2TP" 
msg="Creating outbound L2TP control traffic rule (depends on rule #354)" note="L2TP_LOG" user="unknown" devI
msg="L2TP control rule created: index: #362" note="L2TP_LOG" user="unknown" devID="1cffffffffff" cat="L2TP" 
msg="Tunnel Request Complete" note="L2TP_LOG" user="unknown" devID="1cffffffffff" cat="L2TP" 
msg="[Client01_L2TP_usg40_CONNECTIO(#3)]created incoming IPsec flow, idx: 24981" note="IPSec" user="unknown"
2. .. however as L2TP PAP processing is subsequently invoked that SHOULD be passed to the external LDAP server, 
msg="peer requests authentication protocol PAP" note="L2TP_LOG" .....
3. the LDAP request result (was it even called??) is deemed to have failed authentications as:
<div>msg="Remote L2TP peer 14.0.XXX.XXX:49299" note="L2TP_LOG" user="unknown" devID="1cfff</div><div>msg="Remote tunnel ID&nbsp; &nbsp; 26 session ID&nbsp; 3856" note="L2TP_LOG" user="unknown" devID="1</div><div>msg="&nbsp; User-name: <b>casper</b>" note="IKE_LOG" user="unknown" devID="1cffffffffff" cat="IKE</div>
unauthenticated resulter here----> 

msg="Message: PPP failure: Reason: Authentication failed" note="L2TP_LOG" user="un

 
then the tunnel is destroyed ... etc etc
msg="IPsec SA destroyed: ESP Inbound SPI: [9ee7dec7] Outbound SPI:

Attachments and Diags:
I have attached these logs and listings from the USG40 router .. the same basically for the USG60 router...

  • show version_l2tp-over-ipsec_aaa  group server ldap.log
  • test aaa server ldap host.log
  • USG_V4.70-AALA.0_L2TP_IPSEC_PH2_authentication_failure.log

.. Lastly


We will not be upgrading to  any other (including clients routers) to Firmware V4.70-AALA.0 until this is resolved due to the reliance of some routers we manager that use L2TP AAA LDAP authencation.

  • This circumstance is easy to reproduce. 
  • When available will also try on a USG20WVPN and a USG110 at thei respective updated   firmware levels.. ..these will likley be the same and not work wi the above.

Please let me know what other diagnostics you may require of us.

As always and from previous resopnsive and excellent support  back prior to 2018, I look forward to the prompt support from you engineering blokes at Zyxel TW 我的台湾朋友  B)
 
Kind Regards
Warwick
Hong Kong





Comments

  • Zyxel_Cooldia
    Zyxel_Cooldia Posts: 1,431  Zyxel Employee
    First Anniversary 10 Comments Friend Collector First Answer
    Hi @warwickt,
    Thanks for detailed information to speed up issue clarification.
    The issue also can be reproduced in lab.
    We are working on it, will get back to you as soon as possible.
  • warwickt
    warwickt Posts: 111  Ally Member
    First Anniversary Friend Collector First Answer First Comment
    Hi Zyxel_Cooldia thanks for the DM and the followup as expected form your excellent support service. <3

    Thanks also for the confirmation that this can be reproduces in your Zyxel labs..  =)

    We've also looked at a packet traces from the usg appliance the LAN  LDAP servers on same the LAN  and also over a VTI  for
    1. a test aaa command vs
    2. an L2TP AAA LDAp request 
    where (1.) is prolific and the (2.)  is non-existant at firmware V4.70-AALA.0 

    Please refer to my DM and please advise accordingly at your earliest convenience.\

    Many Thanks 
    Warwick
    Hong Kong
  • warwickt
    warwickt Posts: 111  Ally Member
    First Anniversary Friend Collector First Answer First Comment
    你好, G'day, Hallo fellow Zyxel forum denizens!  B)

    Update and Fix:

    I'm pleased to advise that thanks to the excellent bunch of blokes at Zyxel TW SW Engineering and Tech  Support (Zyxel_Cooldia)  have resolved this AAA LDAP Authentication issue that I reported above for firmware V4.70-AAKY.0 in Zyxel USG40 and USG60 appliances.

    One assumes Zyxel will roll this fix with the next release of firmware.

    We tested this against these Open
    LDAP Servers LOCAL and over an LDAP servers addressible over internal VTI links openldap24 (FreeBSD12.2 / 13 ;  & Linux(Debian 10))openldap26  (latest as at 2021-12) - FreeBSD 12.2 & 13OpenLDAP: slapd 2.4.28 for  MacOS Server (10.12.6 Server - ye old OpenDirectory)I suggest if you have this annoying issue, either
     - rollback to the V4.65 firmware 
     - or contact Zyxel_Cooldia for access to the Zyxel weekly firmware accumulated patches that ought to contain a fix for this issue.

    Lastly a HUGE THANK-YOU to Zyxel_Cooldia for his enthusiatic and highly profession attention to this request and his most splendid organistaion and DM communication .  <3 (Thanks Mate!)

    (For those that care): we noticed an extra set of 'debug' for category 'L2TP" .... nice one!  :3

    Warwick
    Hong Kong

Security Highlight