NWA50AX logging is buggy

StefanZ

Not sure if it fits the topic, but there is no general "bug" forum here.

The CEF formatted messages of the NWA50AX are a paint to parse – like many logs of Zyxel devices I might add!

<141>May 21 01:30:26 NWA50AX_A CEF:0|Zyxel|NWA50AX||0|System Monitoring|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=cpu=1,memory=40,sessions=40,uptime=9 days, 13:54:40

#1 Why is the msg field blank?

#2 There is no separator to the cpu field (should be a blank)

#3 Why on earth do we then get commas as separators for the rest of the log?

#4 And then the cherry on top is the uptime with a friggin' comma in the value!

Looking at this other message:

<141>May 21 01:50:40 NWA50AX_A CEF:0|Zyxel|NWA50AX||0|System Monitoring|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=hostname=NWA50AX_A,modelname=NWA50AX,firmwareversion=V6.29(ABYW.1)

… one can see that there is method to the madness!
Turns out, that the msg value is basically another list of key/value pairs! But comma separated. But WHY?

Same thing on the FLEX200, but with key:value pairs instead of key=value.
Separated with spaces – and value fields that sometimes contain – you guessed it? – SPACES!

name:vlan10 status:Down TxPkts:0 RxPkts:0 Colli.:0 TxB/s:0 RxB/s:0

Also: keys that contain slashes and dots – WHY?
Try to parse that into any system and predict what happens…

Zyxel_Kay

Hi @StefanZ

Regarding your problem,

#1 Why is the msg field blank?
#2 There is no separator to the cpu field (should be a blank)
#3 Why do we then get commas as separators for the rest of the log?

The msg field is not blank here, it contains the msg “cpu=1,memory=40,sessions=40,uptime=9 days, 13:54:40”. The cpu field appears as part of the msg field, therefore is no a distinct separator. In order to distinguish the rest of log (ex: cpu, memory, session, uptime, and etc,), we get commas for them.

#4 And then the cherry on top is the uptime with a friggin' comma in the value!

The comma in the value here is to differentiate the day and time components day and time.

To better assist you, we would appreciate some additional information:

1. Could you provide more details about the specific issue or challenge you are facing with your syslog server and the current AP CEF syslog format?
2. Which syslog server are you using? Knowing the specific syslog server you're working with would help us provide more targeted guidance/solution.

StefanZ

I am using a current Graylog server.

First issue is, that you cannot use the native CEF input – UDP RAW works, but you already lose some automatic pre-processing. I suspect that is because of the prepended facility code. That code could just live in its' own field.

Generally most of the Zyxel logs are formatted in a way that makes them hard to parse. And then they also differ from each other.

Often you also fail at just putting the important bit into a dedicated field:
"Limited-Admin admin(MAC:) from eap-cfg has logged out Device" (on FLEX200)
the rest of the fields do not tell me if someone logged in or out – I have to run some sort of RegEx or GROK to parse the "human readable" message. Let alone the text is missing the word "of".
Then on the NWA50AX you have Administrator admin http/https login. – same event, new message to parse, also zero additional fields that would help.

As for "message in a message with slightly different markup" – I figured that, but I fail to understand why one would want that?
This results in parsing errors, because the parser looks for = as a delimiter between key/value and finds that msg=next_key=text,another_key=12345 indeed means "msg field is empty" and "next_key value is 'text,'". So unless you replace all the commas with spaces before parsing, you end up with a value that has a trailing comma.

This might work, if you escape the delimiters in msg, but again: Why do that, if you can just have regular fields?

Once you start parsing the msg=, new problems arise…
<141>May 29 13:35:06 NWA50AX_A CEF:0|Zyxel|NWA50AX|6.29(ABYW.1)|0|INTERFACE STATISTICS|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=name=wlan-2-1,status=Up,TxPkts=11189,RxPkts=26895,Colli.=0,TxB/s=0,RxB/s=0

Here msg contains a key "name" – but the CEF header already contains a "name" field: "INTERFACE STATISTICS" (no idea why this one is in all caps).

So the event from the "name" field will be replaced with the interface "name" field, thereby screwing up the message classification / sorting. Renaming it to "interface_name" would help.

And lastly: Having dots or slashes in a key is quite the "inconvenience" in most common programming languages – to put it mildly! They serve zero purpose here, since people looking at those metrics surely will be able to deduce "Colli is an abbreviation for Collisions". But they make log ingestion a game of Russian roulette.

"The comma in the value here is to differentiate the day and time components day and time."

Well no – the commas separate your key/value pairs! You wrote that yourself.
In order to make the comma part of a value, you would have to escape it using a \ – although that might not even be possible or yield unpredictable results. But this would be no issue, if you just put everything into its' own field.

And lastly, there are messages that contain errors:

<141>May 18 22:56:28 usgflex200 CEF:0|ZyXEL|USG FLEX 200|5.36(ABUI.1)|0|System Monitoring|5|devID=XXXXXXXXX dvchost=usgflex200 msg=cpu:2,memory:39,sessions:237,uptime:7 days, 12:08:53 cat=System Monitoring ZYlevel=notice ZYnote=SYSTEM STATUS ZYcpu=2 ZYmem=39 ZYuptime=7 days, 12:08:5 ZYconnSession=237

The value of "ZYuptime" is missing the last seconds digit in every message.
And yes, here the msg= formatting switches over to key:value with a colon while still having a comma in one value – yet another special case to handle.

Some CEF headers are incomplete:

CEF:0|Zyxel|NWA50AX||0|System Monitoring|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=cpu=1,memory=39,sessions=39,uptime=22:41:19

Here the device_version key is missing – it exists in all/most other messages from that device.

And then there are redundant messages like this one:

<141>May 29 15:54:06 NWA50AX_A CEF:0|Zyxel|NWA50AX||0|System Monitoring|5|src=0.0.0.0 dst=0.0.0.0 spt=0 dpt=0 msg=hostname=NWA50AX_A,modelname=NWA50AX,firmwareversion=V6.29(ABYW.1)

Here the CEF header is missing the device_version once more, but has fields that will always be 0 – and a message body that just repeats the information that should be included in every header anyway (plus the hostname, but that one is included in the syslog envelope).

Zyxel_Kay

Hi @StefanZ

Thank you for providing us with such a detailed description of the issues you've encountered. We greatly appreciate your feedback and the time you've taken to outline each point.

Rest assured that we will promptly forward your concerns to our development team for further investigation and resolution.

StefanZ

Today I found another interesting one:

<141>Jun 1 11:05:14 usgflex200 CEF:0|ZyXEL|USG FLEX 200|5.36(ABUI.2)|0|Access Control|5|devID=XXXXXXXX […]

Now we get a double space behind the month's name – so it reads "Jun 1", instead of "Jun 01" or "Jun 1".

Zyxel_Kay

Hi @StefanZ

I apologize for the delay in responding. Regarding your inquiry about the CEF syslog of AP and Firewall, we have discussed it with the internal team and made updates to address the following issues:

1. Inconsistent Formatting between access point and firewall
   The msg field displayed in AP NWA50AX is in the format "key=value", whereas in firewall USG FLEX 200, it is "key:value". We have taken note of this inconsistency and raised a feature request to align the format.
2. Commas between system uptime
   Regarding the uptime= X days, XX:XX:XX, we will remove the comma "," between the day and time.
3. About the msg of "Limited-Admin admin(MAC:) from eap-cfg has logged out Device" couldn't show who is logged in/out
   For the AP, you can identify the login client by the source IP in the syslog.
   For the firewall, when a VPN client logs in/out of the device, its MAC address is not displayed. However, if the client logs in/out locally, the device's MAC address will be shown in the log.
   For example:
   1. Local host web login
      
   2. VPN host web login
      
4. Uses of dots or slashes in the key (ex: Colli. , TxB/s , RxB/s)
   We will raise a feature request to modify the abbreviation of "Collision". However, slashes are commonly used to indicate the rate, and we will retain their representation.
5. Missing message in the uptime and CEF header and duplicated information
   We have identified this issue and will work on fixing it.
6. Day is a single-digit number, there are double spaces, while for two-digit numbers, there is a single space
   We will address this issue and ensure uniformity by using a single space for both cases.

All the CEF syslog fixes and enhancements will be implemented starting from 2024/Q1.

Again, thank you for bringing these concerns to our attention, and we appreciate your patience as we work to improve the system.

StefanZ

Sorry for the even later reply… :-D

I just now realized, that I missed 3 firmware updates…

Good thing the new FW has an automatic reminder!

I will check the logging this week and give feedback.

Thanks!