USG20W-VPN frequent crashes

Morgan

I have a USG20W-VPN purchased Nov 2016, which has recently (just in the last week or so) started crashing regularly, several times per day (at least 11 times on 20 Apr).  I extracted the crash logs ("show software-watchdog-timer log" on the web console), and each crash is a bad memory reference:

<1>CPU 0 Unable to handle kernel paging request at virtual address 0000000000000080, epc == ffffffff80920744, ra == ffffffff809206fc

and each time prints a "Call Trace" that always begins

<4>[<ffffffff80920744>] kfree+0x74/0x138
<4>[<ffffffff80b49344>] __kfree_skb+0x14/0xd8

usually it continues

<4>[<ffffffffc086f5a0>] ol_tx_desc_frame_list_free+0xd0/0x148 [umac]
<4>[<ffffffffc0871970>] ol_tx_completion_handler+0x1c0/0x340 [umac]
<4>[<ffffffffc0877870>] htt_t2h_msg_handler_fast+0x1c0/0x208 [umac]
<4>[<ffffffffc08630f8>] CE_per_engine_service_each+0x1b0/0x5a8 [umac]
<4>[<ffffffffc0866e4c>] ath_tasklet+0x5c/0x100 [umac]

one time was

<4>[<ffffffff80ba830c>] ip_rcv_finish+0xbc/0x368
<4>[<ffffffff80b51250>] __netif_receive_skb_core+0x4e0/0x5f8
<4>[<ffffffff80c71a6c>] br_nf_pre_routing_finish+0x224/0x3f8
<4>[<ffffffffc00f75e8>] rtcompl_hook+0x330/0x498 [fastpath_kmodule]
<4>[<ffffffff80b87b24>] nf_iterate+0x11c/0x528
<4>[<ffffffff80b87fbc>] nf_hook_slow+0x8c/0x200
<4>[<ffffffff80c721c0>] br_nf_pre_routing+0x4d8/0x770
<4>[<ffffffff80b87b24>] nf_iterate+0x11c/0x528
<4>[<ffffffff80b87fbc>] nf_hook_slow+0x8c/0x200
<4>[<ffffffff80c6b540>] br_handle_frame+0x1f8/0x2c8
<4>[<ffffffff80b50fd0>] __netif_receive_skb_core+0x260/0x5f8
<4>[<ffffffff80aa6724>] cvm_oct_napi_poll_38+0x48c/0xa20

Until 20 Apr it was running firmware version 4.35(ABAR.0).  After it crashed several times on 20 Apr I upgraded it to version 4.35(ABAR.3), but it crashed several more times after the upgrade.

Bad firmware?  Corrupted firmware?  Failing hardware?

Zyxel_Charlie

@Morgan
Sorry for causing your inconvenience and bad user experience.
Can you private message the complete console dump information (during issue occur), diagnostic information(after issue happen) for our checking further?
Also, can I know what UTM service did you enable?

Zyxel_Charlie

@Morgan
Regarding to this case,
I have private message firmware to you. Please have a check.

Zac_C

I am having a very similar if not same issue (same virtual address). And downloaded the same firmware update. How would you like to handle this?

<1>CPU 0 Unable to handle kernel paging request at virtual address 0000000000000080, epc == ffffffff809206c4, ra == ffffffff8092067c

<4>Oops[#1]:

<4>CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: P         C O 3.10.87-rt80-Cavium-Octeon #2

<4>task: 8000000000348040 ti: 8000000000354000 task.ti: 8000000000354000

<4>$ 0   : 0000000000000000 0000000014009ce1 8000000002ba0bc0 8000000002ba0bc0

<4>$ 4   : 0000000000000000 00000000003b9668 0000000000010001 0000000000000001

<4>$ 8   : 0000000000000000 80000000770f4980 0000000000000018 0000000000000000

<4>$12   : ffffffffc08bf888 ffffffffc0870000 800000008e08ec80 0000000000000000

<4>$16   : 8000000077232080 80000000772cd000 0000000000000001 800000007822c800

<4>$20   : 0000000000000000 ffffffffc058c290 ffffffff808a0000 ffffffff808a0000

<4>$24   : 0000000000000005 ffffffff80b54a90

<4>$28   : 8000000000354000 8000000000357ac0 0000000000000000 ffffffff8092067c

<4>Hi    : 0000000000000000

<4>Lo    : 200000001f5e22a0

<4>epc   : ffffffff809206c4 kfree+0x74/0x138

<4>    Tainted: P         C O

<4>ra    : ffffffff8092067c kfree+0x2c/0x138

<4>Status: 14009ce2     KX SX UX KERNEL EXL

<4>Cause : 00800008

<4>BadVA : 0000000000000080

<4>PrId  : 000d9602 (Cavium Octeon III)

<4>Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) l2upframe(O) ath_pktlog(PO) umac(O) zyacl(O) ath_dev(PO) ath_dfs(PO) ath_spectral(PO) hst_tx99(PO) ath_rate_atheros(PO) ath_hal(PO) asf(PO) adf(PO) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) conntrack_flush(O) zld_utm_action(PO) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) nf_nat_ftp(O) nf_conntrack_ftp(O) zld_wireless(PO) conn_check(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) xt_dns(O) arpt_proxy(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O)

Morgan

That one-off firmware from April 28 has been running without a problem on my router, 77 days uptime as of now.  I recall there was an official firmware update released at the same time which probably had the same fix.  I would expect that current firmware would not suffer from this problem.

Zyxel_Cooldia

Hi @Zac_C,

Sorry for causing your inconvenience and bad user experience.

Can you send me the complete console log and diag-info via private message?

I would like confirm something more.