USG310 reboots & sometimes hangs up

bszl · March 2023

Hello,

Hi,I have a problem with the USG310. In my opinion, this is a software bug. I test the setup on two physical devices and the symptoms are the same. The device restarts every few days(~1/7days). Here are the logs part:

debug ,default , , , , , kernel: [536754.180111] Log Date: 1679963744, fwVersion=Firmware Version: 4.73(AAPJ.0)|2022-11-18 09:07:22, exception_event=3
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.180120] Firmware Version: 4.73(AAPJ.0)|2022-11-18 09:07:22
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.180124] Kernel Info Collector: detect system hanged [Core5], softlockup_hang_count = 8
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.180129] BUG: soft lockup - CPU#5 stuck for 31s! [kworker/5:1:8922]
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.186994] CPU: 5 PID: 8922 Comm: kworker/5:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187015] Workqueue: events fqdnObject_work_queue_func [fqdn_object]
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187020] task: 80000000d4f07500 ti: 80000000be300000 task.ti: 80000000be300000
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187023] $ 0 : 0000000000000000 0000000000000000 80000000ce2b4780 0000000000000000
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187031] $ 4 : 0000000000000000 ffffffffffffffff 80000000d786dc80 80000000be302950
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187038] $ 8 : 0000000000000000 0000000000000001 0000000000000004 0000000000000001
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187046] $12 : 80000000be302980 000000000000001f 000000000000003f 0000000000000590
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187053] $16 : 80000000ce1c6d80 0000000000000000 0000000000000003 80000000be302950
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187061] $20 : fffffffffffffff5 80000000be302a30 800000010ee8e200 0000000000000002
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187068] $24 : ffffffff80d265f0 00000000000000fd2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187076] $28 : 80000000be300000 80000000be3028f0 800000010ee8e180 ffffffff80baa12c
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187084] Hi : 0000000000000167
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187087] Lo : 5555555555555609
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187097] epc : ffffffff80baa13c fib_rules_lookup+0x134/0x1d8
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187100] Tainted: P C O
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187106] ra : ffffffff80baa12c fib_rules_lookup+0x124/0x1d8
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187109] Status: 10009ce3 KX SX UX KERNEL EXL IE
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187117] Cause : 40808000
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187121] PrId : 000d9202 (Cavium Octeon II)
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187339] Process kworker/5:1 (pid: 8922, threadinfo=80000000be300000, task=80000000d4f07500, tls=0000000000000000)
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] Stack : 80000000ce1c6d80 0000000080e10000 80000000be302a30 ffffffff80e68e00
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] ffffffffffffff80 800000010924a000 0000000000000000 0000000000000006
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] 0000000000200000 ffffffffc0780000 80000000c583be08 ffffffff80c38240
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] 0000000000000000 80000000be302980 0000000000000000 0000000100000000
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] ffffffff80e68e00 ffffffff80be71bc 8000000000000000 0000000000000000
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] 0000000000000000 80000000ea238808 000000000000002c 80000000e661edd0
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] 80000000be302a30 0000000000000000 ffffffff80e68e00 80000000eabd4000
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] 0000000000000000 ffffffffc077da60 0000000000200000 ffffffffc0780000
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] 80000000c583be08 ffffffff80be751c 80000000be302a70 80000000bb1ab980
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] 80000000be600128 ffffffff80c3f79c 000000000000781b ffffffffc0780370
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187342] ...
2023-03-28 02:35:44, , , debug ,default , , , , , kernel: [536754.187399] Call Trace: