[2023-05-01 08:29:35.105] [253279.100970] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:29:35.121] [253279.110137] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:29:35.121] [253279.116769] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:29:35.137] [253279.124347] $ 0 : 0000000000000000 0000000014009ce1 80000000243e98a0 0000000000000001 [2023-05-01 08:29:35.137] [253279.132502] $ 4 : 0000000000000f61 0000000000000038 0000000000000000 0000000000000001 [2023-05-01 08:29:35.152] [253279.140655] $ 8 : 0000000000000000 0000000000000001 80000001f4996092 0000000000000000 [2023-05-01 08:29:35.152] [253279.148811] $12 : 0000000000000000 ffffffff80a489bc 000000000000003f 0000000000000000 [2023-05-01 08:29:35.168] [253279.156968] $16 : 80000001f6788008 000000000000000f 80000001ee7f1c08 80000001f686e540 [2023-05-01 08:29:35.168] [253279.165125] $20 : ffffffffc295ba40 ffffffffc295ba80 0000000000200000 ffffffffc2960000 [2023-05-01 08:29:35.183] [253279.173279] $24 : 0000000000000000 ffffffffc2f03450 [2023-05-01 08:29:35.183] [253279.181434] $28 : 80000001f6e30000 80000001f6e33780 800000019b343008 ffffffffc29d5a30 [2023-05-01 08:29:35.199] [253279.189590] Hi : 0000000000000519 [2023-05-01 08:29:35.199] [253279.193255] Lo : 55555555555557e2 [2023-05-01 08:29:35.199] [253279.196980] epc : ffffffffc29d5a30 ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [2023-05-01 08:29:35.215] [253279.204905] Tainted: P C O [2023-05-01 08:29:35.215] [253279.208776] ra : ffffffffc29d5a30 ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [2023-05-01 08:29:35.230] [253279.216700] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:29:35.230] [253279.221835] Cause : 40808000 [2023-05-01 08:29:35.230] [253279.224806] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:29:35.230] [253279.229513] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:29:35.543] [253279.530179] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:29:35.543] [253279.540885] Stack : 0000000000200000 000000000000000f 000000000000000f 0000000000000001 [2023-05-01 08:29:35.558] 0000000000000000 80000001ee7f1dd0 0000000000000000 ffffffff00000001 [2023-05-01 08:29:35.558] 0000000000000001 80000001d79b3db8 ffffffffffffffff 0000000000000000 [2023-05-01 08:29:35.574] 80000001f6a0e058 0000000000000002 80000001f6a0e000 800000018ffb3000 [2023-05-01 08:29:35.574] ffffffffc295bbd8 80000001e9867000 0000000000000001 80000001f49960c4 [2023-05-01 08:29:35.574] 0000000000000001 ffffffffc2962338 0000000000000001 0000000000000000 [2023-05-01 08:29:35.590] 80000001eb6e1500 0000000000000000 80000001ea255980 80000001e417f800 [2023-05-01 08:29:35.590] 80000000243e3de0 000000000000003f 800000018ffb3000 ffffffffc2960000 [2023-05-01 08:29:35.605] ffffffffc2a5c838 ffffffffc2a5cac8 0000000000000034 00000000000f4240 [2023-05-01 08:29:35.605] ffffffff80e00550 0000000000000006 0000000000000001 ffffffffc2a5c568 [2023-05-01 08:29:35.605] ... [2023-05-01 08:29:35.621] [253279.606633] Call Trace: [2023-05-01 08:29:35.621] [253279.609210] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [2023-05-01 08:29:35.621] [253279.616850] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:29:35.637] [253279.624901] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:29:35.637] [253279.632054] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:29:35.652] [253279.637809] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:29:35.652] [253279.643302] [] do_softirq+0x68/0x70 [2023-05-01 08:29:35.652] [253279.648447] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:29:35.668] [253279.654040] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:29:35.668] [253279.661981] [] fqdnObject_work_queue_func+0xf8/0x390 [fqdn_object] [2023-05-01 08:29:35.683] [253279.669828] [] process_one_work+0x178/0x3e0 [2023-05-01 08:29:35.683] [253279.675670] [] worker_thread+0x140/0x3c8 [2023-05-01 08:29:35.683] [253279.681249] [] kthread+0xa4/0xb0 [2023-05-01 08:29:35.699] [253279.686135] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:29:35.699] [253279.692321] [2023-05-01 08:29:35.699] [253279.693899] [2023-05-01 08:29:35.699] Code: de0400b8 0060f809 dca60070 <0280f809> de040030 de5e01e8 17c0fff0 00000000 02a0f809 [2023-05-01 08:29:35.715] [253279.704034] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:29:35.715] [253279.713186] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:29:35.730] [253279.719817] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:29:35.730] 0000000000000000 0000000000000000 000000000000cb62 00000000000002ee [2023-05-01 08:29:35.746] ffffffff859a07f8 ffffffff80ec0000 00000000000002ee 0000000000000002 [2023-05-01 08:29:35.746] 0000000000000002 ffffffff859b23f0 ffffffff859e0000 0000000000000000 [2023-05-01 08:29:35.761] ffffffff80eb0428 ffffffff80eb0000 80000001f6e335f0 80000000243e1688 [2023-05-01 08:29:35.761] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:29:35.761] 0000000000000060 0000000000000001 0000000000000000 0000000000000001 [2023-05-01 08:29:35.777] 80000001f6e30000 80000001f6e33280 0000e654a3bc2123 ffffffff808f739c [2023-05-01 08:29:35.777] 0000000000000004 0000000000000000 00000000000002ed 6a6563745f776f72 [2023-05-01 08:29:35.793] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:29:35.793] ... [2023-05-01 08:29:35.793] [253279.785540] Call Trace: [2023-05-01 08:29:35.793] [253279.788080] [] show_stack+0x6c/0xf8 [2023-05-01 08:29:35.808] [253279.793229] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:29:35.808] [253279.799159] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:29:35.808] [253279.805261] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:29:35.824] [253279.811188] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:29:35.824] [253279.817201] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:29:35.824] [253279.823564] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:29:35.840] [253279.829317] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:29:35.840] [253279.835160] [] do_IRQ+0x18/0x30 [2023-05-01 08:29:35.855] [253279.839961] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:29:35.855] [253279.845713] [2023-05-01 08:30:09.271] [253313.268460] Log Date: 1682886610, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:30:09.286] [253313.279000] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:30:09.286] [253313.285100] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 2 [2023-05-01 08:30:09.302] [253313.293460] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:30:09.302] [253313.300080] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:30:09.614] [253313.600781] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:30:09.614] [253313.609947] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:30:09.630] [253313.616579] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:30:09.630] [253313.624157] $ 0 : 0000000000000000 0000000014009ce1 80000001f4eb2300 0000000000000016 [2023-05-01 08:30:09.646] [253313.632312] $ 4 : 80000001f4eb2300 0000000000000000 80000001f4c14c00 ffffffffc29620f8 [2023-05-01 08:30:09.646] [253313.640467] $ 8 : 0000000000000000 0000000000000001 80000001ea69b092 0000000000000000 [2023-05-01 08:30:09.661] [253313.648620] $12 : 0000000000000000 ffffffff80a489bc 000000000000003f 0000000000000000 [2023-05-01 08:30:09.661] [253313.656778] $16 : 800000018ffb3000 ffffffffc2960000 ffffffffc2a5c838 ffffffffc2a5cac8 [2023-05-01 08:30:09.677] [253313.664933] $20 : 000000be789a995e 00000000000f4240 ffffffff80e00550 0000000000000006 [2023-05-01 08:30:09.677] [253313.673086] $24 : 0000000000000000 ffffffffc2f03450 [2023-05-01 08:30:09.693] [253313.681241] $28 : 80000001f6e30000 80000001f6e338c0 0000000000000001 ffffffffc2a5c568 [2023-05-01 08:30:09.693] [253313.689398] Hi : 0000000000000519 [2023-05-01 08:30:09.693] [253313.693064] Lo : 55555555555557e2 [2023-05-01 08:30:09.708] [253313.696749] epc : ffffffffc2a5c544 vti_xmit_queue_handler+0xcc/0x170 [zld_vti] [2023-05-01 08:30:09.708] [253313.704241] Tainted: P C O [2023-05-01 08:30:09.708] [253313.708097] ra : ffffffffc2a5c568 vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:30:09.724] [253313.715588] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:30:09.724] [253313.720723] Cause : 40808000 [2023-05-01 08:30:09.724] [253313.723692] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:30:09.739] [253313.728400] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:30:10.036] [253314.029081] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:30:10.052] [253314.039787] Stack : 80000001f4eb2300 80000001e8963980 00000016243e3d00 80000000243e3de0 [2023-05-01 08:30:10.052] 80000000243e4b78 80000000243e4b70 0000000000000002 0000000000000000 [2023-05-01 08:30:10.067] ffffffff80e00538 ffffffff808a29b4 0000000000000001 ffffffff858c5870 [2023-05-01 08:30:10.067] ffffffff80eaf000 0000000000000007 0000000000000030 ffffffff80eaf030 [2023-05-01 08:30:10.067] 0000000000000101 ffffffff808a2ef8 000000000000000a 0000000101820687 [2023-05-01 08:30:10.083] 0000000004208060 80000001f45e20e8 0000000000000001 ffffffffc2a40000 [2023-05-01 08:30:10.083] 0000000000000000 80000001eb852a00 ffffffffc2a42c08 ffffffffc2a40000 [2023-05-01 08:30:10.099] 0000000000000005 80000001f6e33b68 80000001f6e33c90 ffffffff808a3008 [2023-05-01 08:30:10.099] ffffffff80f00000 ffffffff808a3168 80000001f4445a80 ffffffffc2a424a8 [2023-05-01 08:30:10.099] 000000000000001e 80000001f6e33b78 80000001f6e33b68 80000001eb852a00 [2023-05-01 08:30:10.114] ... [2023-05-01 08:30:10.114] [253314.105525] Call Trace: [2023-05-01 08:30:10.114] [253314.108065] [] vti_xmit_queue_handler+0xcc/0x170 [zld_vti] [2023-05-01 08:30:10.114] [253314.115218] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:30:10.130] [253314.120975] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:30:10.130] [253314.126473] [] do_softirq+0x68/0x70 [2023-05-01 08:30:10.146] [253314.131618] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:30:10.146] [253314.137216] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:30:10.146] [253314.145156] [] fqdnObject_work_queue_func+0x2f8/0x390 [fqdn_object] [2023-05-01 08:30:10.161] [253314.153091] [] process_one_work+0x178/0x3e0 [2023-05-01 08:30:10.161] [253314.158933] [] worker_thread+0x140/0x3c8 [2023-05-01 08:30:10.177] [253314.164514] [] kthread+0xa4/0xb0 [2023-05-01 08:30:10.177] [253314.169400] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:30:10.177] [253314.175586] [2023-05-01 08:30:10.177] [253314.177165] [2023-05-01 08:30:10.192] Code: 2463ffff afa30010 dc460000 fc400000 fc400008 fcc30008 fc660000 dc500020 [2023-05-01 08:30:10.192] [253314.187296] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:30:10.208] [253314.196448] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:30:10.208] [253314.203078] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:30:10.224] 0000000000000000 0000000000000000 000000000000d034 000000000000032e [2023-05-01 08:30:10.224] ffffffff859a07f8 ffffffff80ec0000 000000000000032e 0000000000000002 [2023-05-01 08:30:10.224] 0000000000000002 ffffffff859b5438 ffffffff859e0000 0000000000000000 [2023-05-01 08:30:10.239] ffffffff80eb0428 ffffffff80eb0000 80000001f6e33730 80000000243e1688 [2023-05-01 08:30:10.239] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:30:10.255] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:30:10.255] 80000001f6e30000 80000001f6e333c0 0000e65cac185d6f ffffffff808f739c [2023-05-01 08:30:10.255] 0000000000000004 0000000000000000 000000000000032d 6a6563745f776f72 [2023-05-01 08:30:10.271] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:30:10.271] ... [2023-05-01 08:30:10.271] [253314.268809] Call Trace: [2023-05-01 08:30:10.271] [253314.271346] [] show_stack+0x6c/0xf8 [2023-05-01 08:30:10.286] [253314.276495] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:30:10.286] [253314.282423] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:30:10.302] [253314.288525] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:30:10.302] [253314.294453] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:30:10.302] [253314.300466] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:30:10.317] [253314.306829] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:30:10.317] [253314.312583] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:30:10.317] [253314.318425] [] do_IRQ+0x18/0x30 [2023-05-01 08:30:10.333] [253314.323230] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:30:10.333] [253314.328981] [2023-05-01 08:30:43.280] [253347.268459] Log Date: 1682886644, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:30:43.280] [253347.278999] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:30:43.295] [253347.285100] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 3 [2023-05-01 08:30:43.295] [253347.293460] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:30:43.311] [253347.300081] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:30:43.608] [253347.600789] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:30:43.624] [253347.609957] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:30:43.624] [253347.616590] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:30:43.624] [253347.624167] $ 0 : 0000000000000000 0000000014009ce1 000000000000000a 000000000000000a [2023-05-01 08:30:43.639] [253347.632325] $ 4 : 80000001d79b3db8 000000000000000a 80000001ee7f1c08 00000000000000e0 [2023-05-01 08:30:43.655] [253347.640482] $ 8 : 0000000001200000 0000000000000001 ffffffffc2347600 80000001e8601c00 [2023-05-01 08:30:43.655] [253347.648638] $12 : 0000000000000000 0000000000000000 0000000000000002 6db6db6db6db6db7 [2023-05-01 08:30:43.655] [253347.656793] $16 : 80000001d79b3db8 80000001f48260a0 80000001ee7f1c08 800000018bcef208 [2023-05-01 08:30:43.670] [253347.664949] $20 : ffffffffc295bbd8 0000000000000288 ffffffff80d34f00 8000000186d58000 [2023-05-01 08:30:43.686] [253347.673106] $24 : 0000000000000000 ffffffffc295ba80 [2023-05-01 08:30:43.686] [253347.681262] $28 : 80000001f6e30000 80000001f6e33030 0000000000000000 ffffffffc29b1104 [2023-05-01 08:30:43.702] [253347.689418] Hi : 0000000000000002 [2023-05-01 08:30:43.702] [253347.693082] Lo : 00000000000000af [2023-05-01 08:30:43.702] [253347.696801] epc : ffffffffc29a8830 fastpath_execute_transform_step+0x28/0x5a0 [quicksec] [2023-05-01 08:30:43.717] [253347.705162] Tainted: P C O [2023-05-01 08:30:43.717] [253347.709035] ra : ffffffffc29b1104 ssh_fastpath_transform_out_finish+0x48c/0x998 [quicksec] [2023-05-01 08:30:43.717] [253347.717656] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:30:43.733] [253347.722790] Cause : 40808000 [2023-05-01 08:30:43.733] [253347.725759] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:30:43.733] [253347.730466] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:30:44.030] [253348.031155] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:30:44.045] [253348.041862] Stack : ffffffffc295bbd8 80000001d79b3db8 80000001d79b3db8 80000001f48260a0 [2023-05-01 08:30:44.061] 80000001e40bf398 ffffffffc29b1104 0000000000000000 0000000000000000 [2023-05-01 08:30:44.061] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:30:44.077] 0000000000000000 0000000000000002 0000000000000588 0000000000000001 [2023-05-01 08:30:44.077] 0000000000000590 80000001e8601c00 80000001e4283708 80000001e842a000 [2023-05-01 08:30:44.077] 8000000186d58080 00000000000005b0 0000000000000008 80000001ea92b080 [2023-05-01 08:30:44.092] ffffffff80d34f00 ffffffffc29d3874 8000000186d58080 80000001e842a000 [2023-05-01 08:30:44.092] 00000000000005b0 ffffffffc2344838 80000001e8601c00 800000000c509852 [2023-05-01 08:30:44.092] 000000a0000005b0 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:30:44.108] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:30:44.108] ... [2023-05-01 08:30:44.108] [253348.107618] Call Trace: [2023-05-01 08:30:44.123] [253348.110199] [] fastpath_execute_transform_step+0x28/0x5a0 [quicksec] [2023-05-01 08:30:44.123] [253348.118279] [] ssh_fastpath_transform_out_finish+0x48c/0x998 [quicksec] [2023-05-01 08:30:44.139] [253348.126618] [] ocf_cb+0xb4/0x128 [quicksec] [2023-05-01 08:30:44.139] [253348.132546] [] octo_process.part.4+0xc8/0xa38 [cryptocteon] [2023-05-01 08:30:44.139] [253348.139800] [] crypto_invoke+0x60/0x168 [2023-05-01 08:30:44.155] [253348.145293] [] crypto_dispatch+0x214/0x360 [2023-05-01 08:30:44.155] [253348.151080] [] ssh_hwaccel_perform_ipsec+0x31c/0x5b8 [quicksec] [2023-05-01 08:30:44.170] [253348.158723] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [2023-05-01 08:30:44.170] [253348.166626] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [2023-05-01 08:30:44.186] [253348.174268] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:30:44.186] [253348.181909] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:30:44.202] [253348.189959] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:30:44.202] [253348.197111] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:30:44.202] [253348.202865] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:30:44.217] [253348.208358] [] do_softirq+0x68/0x70 [2023-05-01 08:30:44.217] [253348.213504] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:30:44.233] [253348.219097] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:30:44.233] [253348.227051] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [2023-05-01 08:30:44.248] [253348.235001] [] process_one_work+0x178/0x3e0 [2023-05-01 08:30:44.248] [253348.240847] [] worker_thread+0x140/0x3c8 [2023-05-01 08:30:44.248] [253348.246428] [] kthread+0xa4/0xb0 [2023-05-01 08:30:44.264] [253348.251315] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:30:44.264] [253348.257502] [2023-05-01 08:30:44.264] [253348.259080] [2023-05-01 08:30:44.264] Code: 0080802d 14a3010f ffb10018 <8e050238> 24020001 de110018 10a2007c 8e060000 10a00024 [2023-05-01 08:30:44.280] [253348.269217] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:30:44.280] [253348.278379] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:30:44.295] [253348.285010] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:30:44.295] 0000000000000000 0000000000000000 000000000000d52e 0000000000000379 [2023-05-01 08:30:44.311] ffffffff859a07f8 ffffffff80ec0000 0000000000000379 0000000000000002 [2023-05-01 08:30:44.311] 0000000000000002 ffffffff859b8838 ffffffff859e0000 0000000000000000 [2023-05-01 08:30:44.311] ffffffff80eb0428 ffffffff80eb0000 80000001f6e32ea0 80000000243e1688 [2023-05-01 08:30:44.327] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:30:44.327] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:30:44.342] 80000001f6e30000 80000001f6e32b30 0000e66496a73117 ffffffff808f739c [2023-05-01 08:30:44.342] 0000000000000004 0000000000000000 0000000000000378 6a6563745f776f72 [2023-05-01 08:30:44.342] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:30:44.358] ... [2023-05-01 08:30:44.358] [253348.350751] Call Trace: [2023-05-01 08:30:44.358] [253348.353288] [] show_stack+0x6c/0xf8 [2023-05-01 08:30:44.358] [253348.358436] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:30:44.373] [253348.364366] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:30:44.373] [253348.370468] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:30:44.389] [253348.376395] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:30:44.389] [253348.382410] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:30:44.389] [253348.388773] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:30:44.405] [253348.394528] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:30:44.405] [253348.400369] [] do_IRQ+0x18/0x30 [2023-05-01 08:30:44.405] [253348.405169] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:30:44.420] [253348.410921] [2023-05-01 08:30:44.436] [253348.433822] [zy_adp]tcp syn flush flow!! [2023-05-01 08:31:08.431] [253372.425316] unpack_message_rr fail ret=-31 [2023-05-01 08:32:58.770] [253482.768459] Log Date: 1682886780, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:32:58.785] [253482.779185] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:32:58.785] [253482.785285] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [2023-05-01 08:32:58.801] [253482.793646] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:32:58.801] [253482.800266] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:32:59.098] [253483.100965] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:32:59.113] [253483.110132] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:32:59.113] [253483.116763] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:32:59.129] [253483.124340] $ 0 : 0000000000000000 0000000054009ce1 00000000000005a4 ffffffffc0020000 [2023-05-01 08:32:59.129] [253483.132494] $ 4 : 00000000000005a4 6db6db6db6db6db7 12e436e900000000 0000000000000534 [2023-05-01 08:32:59.144] [253483.140651] $ 8 : ffffffffc2340000 0000000000000001 ffffffffc2347600 80000001e8601c00 [2023-05-01 08:32:59.160] [253483.148807] $12 : 0000000000000000 0000000000000000 0000000000000002 6db6db6db6db6db7 [2023-05-01 08:32:59.160] [253483.156960] $16 : 80000001f6e33120 0000000000000534 80000001eb6c2118 000000000000052c [2023-05-01 08:32:59.176] [253483.165115] $20 : 0000000000000000 6db6db6db6db6db7 8000000000000000 000000000000000c [2023-05-01 08:32:59.176] [253483.173270] $24 : 0000000000000000 8000000000000000 [2023-05-01 08:32:59.191] [253483.181423] $28 : 80000001f6e30000 80000001f6e32e90 ffffffff859e9900 ffffffffc233de98 [2023-05-01 08:32:59.191] [253483.189579] Hi : 0000000000000002 [2023-05-01 08:32:59.191] [253483.193244] Lo : 00000000000000af [2023-05-01 08:32:59.207] [253483.196953] epc : ffffffffc233e710 octo_des_cbc_sha1_encrypt+0x9c0/0x1018 [cryptocteon] [2023-05-01 08:32:59.207] [253483.205227] Tainted: P C O [2023-05-01 08:32:59.207] [253483.209087] ra : ffffffffc233de98 octo_des_cbc_sha1_encrypt+0x148/0x1018 [cryptocteon] [2023-05-01 08:32:59.223] [253483.217359] Status: 54009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:32:59.223] [253483.222496] Cause : 40808000 [2023-05-01 08:32:59.223] [253483.225466] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:32:59.238] [253483.230174] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:32:59.535] [253483.530855] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:32:59.551] [253483.541562] Stack : 0000000000000000 0000000014009ce1 0000000000010000 0000000017c917c9 [2023-05-01 08:32:59.551] 80000001e4109888 0000000000000001 00000000000000f0 0000000000000000 [2023-05-01 08:32:59.566] 0000000001200000 0000000000000001 ffffffffc2347600 80000001e8601c00 [2023-05-01 08:32:59.566] 0000000000000000 0000000000000000 0000000000000002 6db6db6db6db6db7 [2023-05-01 08:32:59.566] 80000001e4109888 80000001e40bf398 80000001ee7f1c20 80000001ec364008 [2023-05-01 08:32:59.582] ffffffffc295bbd8 0000000000000288 ffffffff80d34f00 8000000186d58000 [2023-05-01 08:32:59.582] 0000000000000000 ffffffffc29d37c0 800000018bceb008 ffffffff80bbb7a0 [2023-05-01 08:32:59.598] 80000001f6e30000 80000001f6e33020 0000000000000000 ffffffffc295ba5c [2023-05-01 08:32:59.598] 0000000014009ce3 0000000000000002 00000000000000af ffffffff80bbb9b8 [2023-05-01 08:32:59.598] 0000000000800800 ffffffff80d34df8 0000000000000000 0000000000000000 [2023-05-01 08:32:59.613] ... [2023-05-01 08:32:59.613] [253483.607339] Call Trace: [2023-05-01 08:32:59.613] [253483.609896] [] octo_des_cbc_sha1_encrypt+0x9c0/0x1018 [cryptocteon] [2023-05-01 08:32:59.629] [253483.617879] [] octo_process.part.4+0x48c/0xa38 [cryptocteon] [2023-05-01 08:32:59.629] [253483.625219] [] crypto_invoke+0x60/0x168 [2023-05-01 08:32:59.629] [253483.630712] [] crypto_dispatch+0x214/0x360 [2023-05-01 08:32:59.644] [253483.636516] [] ssh_hwaccel_perform_ipsec+0x31c/0x5b8 [quicksec] [2023-05-01 08:32:59.644] [253483.644158] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [2023-05-01 08:32:59.660] [253483.652061] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [2023-05-01 08:32:59.660] [253483.659702] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:32:59.676] [253483.667340] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:32:59.676] [253483.675390] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:32:59.691] [253483.682545] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:32:59.691] [253483.688303] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:32:59.691] [253483.693801] [] do_softirq+0x68/0x70 [2023-05-01 08:32:59.707] [253483.698949] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:32:59.707] [253483.704551] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:32:59.722] [253483.712491] [] fqdnObject_work_queue_func+0x2f8/0x390 [fqdn_object] [2023-05-01 08:32:59.722] [253483.720426] [] process_one_work+0x178/0x3e0 [2023-05-01 08:32:59.722] [253483.726267] [] worker_thread+0x140/0x3c8 [2023-05-01 08:32:59.738] [253483.731848] [] kthread+0xa4/0xb0 [2023-05-01 08:32:59.738] [253483.736735] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:32:59.754] [253483.742921] [2023-05-01 08:32:59.754] [253483.744499] [2023-05-01 08:32:59.754] Code: 654a7600 088cf9e0 3c08c234 <14e00017> 66520004 26940001 00141178 0289202a 1080002a [2023-05-01 08:32:59.754] [253483.754628] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:32:59.769] [253483.763781] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:32:59.769] [253483.770412] Stack : 0000000000000001 0000000054001ce0 0000000000000000 0000000000000007 [2023-05-01 08:32:59.785] 0000000000000000 0000000000000000 000000000000da28 00000000000003c4 [2023-05-01 08:32:59.785] ffffffff859a07f8 ffffffff80ec0000 00000000000003c4 0000000000000002 [2023-05-01 08:32:59.801] 0000000000000002 ffffffff859bbbe8 ffffffff859e0000 0000000000000000 [2023-05-01 08:32:59.801] ffffffff80eb0428 ffffffff80eb0000 80000001f6e32d00 80000000243e1688 [2023-05-01 08:32:59.801] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:32:59.816] 0000000000000060 0000000000000001 0000000000000000 0000000000000001 [2023-05-01 08:32:59.816] 80000001f6e30000 80000001f6e32990 0000e68423151b00 ffffffff808f739c [2023-05-01 08:32:59.832] 0000000000000004 0000000000000000 00000000000003c3 6a6563745f776f72 [2023-05-01 08:32:59.832] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:32:59.832] ... [2023-05-01 08:32:59.847] [253483.836142] Call Trace: [2023-05-01 08:32:59.847] [253483.838680] [] show_stack+0x6c/0xf8 [2023-05-01 08:32:59.847] [253483.843829] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:32:59.847] [253483.849759] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:32:59.863] [253483.855860] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:32:59.863] [253483.861789] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:32:59.879] [253483.867803] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:32:59.879] [253483.874165] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:32:59.879] [253483.879920] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:32:59.894] [253483.885762] [] do_IRQ+0x18/0x30 [2023-05-01 08:32:59.894] [253483.890562] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:32:59.894] [253483.896316] [2023-05-01 08:33:05.487] [253489.480089] unpack_message_rr fail ret=-31unpack_message_rr fail ret=-31unpack_message_rr fail ret=-31 [2023-05-01 08:33:40.777] [253524.768463] Log Date: 1682886822, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:33:40.777] [253524.779191] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:33:40.793] [253524.785292] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 2 [2023-05-01 08:33:40.793] [253524.793653] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:33:40.808] [253524.800273] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:33:41.105] [253525.100959] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:33:41.105] [253525.110120] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:33:41.121] [253525.116752] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:33:41.121] [253525.124328] $ 0 : 0000000000000000 000000000000001c 0000000000010000 0000000000000000 [2023-05-01 08:33:41.136] [253525.132483] $ 4 : 000000000000c22c 0000000000000000 0000000000000001 ffffffffc29b0000 [2023-05-01 08:33:41.136] [253525.140636] $ 8 : 0000000000000000 0000000000000001 0000000000000000 0000000000000000 [2023-05-01 08:33:41.152] [253525.148795] $12 : ffffffffc2970000 80000001f6e3363c 00000000000061da 0000000000000000 [2023-05-01 08:33:41.168] [253525.156951] $16 : 80000001e41096c8 8000000188a4f608 80000001ee7f1c08 80000001ee7f1c08 [2023-05-01 08:33:41.168] [253525.165107] $20 : 000000000000000b ffffffffc29dda80 80000001d79b3db8 80000001cd3804d8 [2023-05-01 08:33:41.168] [253525.173262] $24 : 0000000000000000 ffffffffc29a8808 [2023-05-01 08:33:41.183] [253525.181418] $28 : 80000001f6e30000 80000001f6e33680 80000001cd3803c8 ffffffff80d34fcc [2023-05-01 08:33:41.199] [253525.189574] Hi : 0000000000000007 [2023-05-01 08:33:41.199] [253525.193239] Lo : 0000000000001fff [2023-05-01 08:33:41.199] [253525.196910] epc : ffffffff80d352a0 _raw_write_lock_bh+0xc0/0xe0 [2023-05-01 08:33:41.199] [253525.203095] Tainted: P C O [2023-05-01 08:33:41.215] [253525.206937] ra : ffffffff80d34fcc _raw_spin_lock_bh+0x14/0x48 [2023-05-01 08:33:41.215] [253525.213035] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:33:41.215] [253525.218171] Cause : 40808000 [2023-05-01 08:33:41.230] [253525.221141] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:33:41.230] [253525.225849] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:33:41.527] [253525.526506] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:33:41.543] [253525.537212] Stack : 80000001e41096c8 ffffffffc295babc 80000001d79b3db8 ffffffffc29a8d1c [2023-05-01 08:33:41.543] 7d2f0000f6788008 0000000000000003 8000000188a4f608 80000001f6788008 [2023-05-01 08:33:41.558] 0000000000000003 ffffffffc29d4e9c 0000000000000000 ffffffff80c19808 [2023-05-01 08:33:41.558] ac150063f0742000 80000001f6a0e000 80000001e9b34a80 8000000188a4f608 [2023-05-01 08:33:41.574] 80000001ee7f1c28 0000000000000000 8000000188a4f608 0000000000000000 [2023-05-01 08:33:41.574] 0000000000000001 ffffffffc2950000 80000001f6788008 000000000000000f [2023-05-01 08:33:41.574] 80000001ee7f1c08 80000001f686e540 ffffffffc295ba40 ffffffffc295ba80 [2023-05-01 08:33:41.589] ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 [2023-05-01 08:33:41.589] 0000000000200000 000000000000000f 000000000000000f 0000000000000001 [2023-05-01 08:33:41.605] 0000000000000000 80000001ee7f1dd0 0000000000000000 ffffffff00000001 [2023-05-01 08:33:41.605] ... [2023-05-01 08:33:41.605] [253525.602942] Call Trace: [2023-05-01 08:33:41.605] [253525.605480] [] _raw_write_lock_bh+0xc0/0xe0 [2023-05-01 08:33:41.621] [253525.611367] [] ssh_kernel_mutex_lock_bh_i+0x1c/0x40 [quicksec] [2023-05-01 08:33:41.621] [253525.618921] [] fastpath_execute_transform_step+0x514/0x5a0 [quicksec] [2023-05-01 08:33:41.636] [253525.627087] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [2023-05-01 08:33:41.636] [253525.634728] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:33:41.652] [253525.642367] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:33:41.652] [253525.650418] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:33:41.668] [253525.657570] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:33:41.668] [253525.663325] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:33:41.668] [253525.668818] [] do_softirq+0x68/0x70 [2023-05-01 08:33:41.683] [253525.673964] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:33:41.683] [253525.679813] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [2023-05-01 08:33:41.683] [253525.687493] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [2023-05-01 08:33:41.699] [253525.695427] [] process_one_work+0x178/0x3e0 [2023-05-01 08:33:41.699] [253525.701268] [] worker_thread+0x140/0x3c8 [2023-05-01 08:33:41.714] [253525.706848] [] kthread+0xa4/0xb0 [2023-05-01 08:33:41.714] [253525.711736] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:33:41.714] [253525.717922] [2023-05-01 08:33:41.730] [253525.719500] [2023-05-01 08:33:41.730] Code: 1000ff3e 24030000 30631fff <00031940> 1460ffff 2463ffff 96030002 1064ff4f 00831823 [2023-05-01 08:33:41.730] [253525.729627] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:33:41.746] [253525.738778] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:33:41.746] [253525.745410] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:33:41.761] 0000000000000000 0000000000000000 000000000000df10 000000000000040b [2023-05-01 08:33:41.761] ffffffff859a07f8 ffffffff80ec0000 000000000000040b 0000000000000002 [2023-05-01 08:33:41.761] 0000000000000002 ffffffff859bee90 ffffffff859e0000 0000000000000000 [2023-05-01 08:33:41.777] ffffffff80eb0428 ffffffff80eb0000 80000001f6e334f0 80000000243e1688 [2023-05-01 08:33:41.777] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:33:41.793] 0000000000000060 0000000000000001 0000000000000000 0000000000000001 [2023-05-01 08:33:41.793] 80000001f6e30000 80000001f6e33180 0000e68dea7a41a2 ffffffff808f739c [2023-05-01 08:33:41.793] 0000000000000004 0000000000000000 000000000000040a 6a6563745f776f72 [2023-05-01 08:33:41.808] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:33:41.808] ... [2023-05-01 08:33:41.808] [253525.811126] Call Trace: [2023-05-01 08:33:41.824] [253525.813664] [] show_stack+0x6c/0xf8 [2023-05-01 08:33:41.824] [253525.818812] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:33:41.824] [253525.824742] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:33:41.839] [253525.830844] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:33:41.839] [253525.836773] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:33:41.839] [253525.842787] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:33:41.855] [253525.849150] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:33:41.855] [253525.854905] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:33:41.871] [253525.860746] [] do_IRQ+0x18/0x30 [2023-05-01 08:33:41.871] [253525.865546] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:33:41.871] [253525.871297] [2023-05-01 08:34:14.771] [253558.768459] Log Date: 1682886856, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:34:14.786] [253558.779000] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:34:14.786] [253558.785100] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 3 [2023-05-01 08:34:14.802] [253558.793460] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:34:14.802] [253558.800081] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:34:15.099] [253559.100776] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:34:15.114] [253559.109943] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:34:15.114] [253559.116576] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:34:15.130] [253559.124152] $ 0 : 0000000000000000 0000000000000000 80000001a946b940 00000000000001f2 [2023-05-01 08:34:15.130] [253559.132309] $ 4 : ffffffff80eeab00 00000000000001f2 80000001a9430f00 80000001f6e335a0 [2023-05-01 08:34:15.146] [253559.140464] $ 8 : 0000000000000000 0000000008008000 0000000000000002 0000000000000001 [2023-05-01 08:34:15.146] [253559.148621] $12 : 80000001f6e335d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [2023-05-01 08:34:15.161] [253559.156777] $16 : 80000001f6e33680 0000000000000000 0000000000000003 80000001f6e335a0 [2023-05-01 08:34:15.161] [253559.164934] $20 : fffffffffffffff5 80000001f6e33680 80000001fbb17a40 0000000000000002 [2023-05-01 08:34:15.177] [253559.173094] $24 : 0000000000000000 ffffffffc295bc30 [2023-05-01 08:34:15.177] [253559.181251] $28 : 80000001f6e30000 80000001f6e33520 80000001fbb179c0 ffffffff80c621b0 [2023-05-01 08:34:15.192] [253559.189408] Hi : 00000000000004ff [2023-05-01 08:34:15.192] [253559.193073] Lo : 55555555555557d5 [2023-05-01 08:34:15.192] [253559.196743] epc : ffffffff80c545a8 fib_get_table+0x38/0x50 [2023-05-01 08:34:15.208] [253559.202494] Tainted: P C O [2023-05-01 08:34:15.208] [253559.206337] ra : ffffffff80c621b0 fib4_rule_action+0x58/0xa0 [2023-05-01 08:34:15.208] [253559.212349] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:34:15.224] [253559.217485] Cause : 40808000 [2023-05-01 08:34:15.224] [253559.220454] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:34:15.224] [253559.225162] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:34:15.520] [253559.525839] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:34:15.536] [253559.536546] Stack : 80000001f6e335a0 ffffffffc29b2644 80000001a9430f00 ffffffff80bb659c [2023-05-01 08:34:15.552] 80000001a9430f00 f032c135ac157464 80000001f6e33680 ffffffff80eeab00 [2023-05-01 08:34:15.552] ffffffffffffff80 80000001f0742000 0000000000000000 0000000000000008 [2023-05-01 08:34:15.552] 0000000000200000 ffffffffc2960000 800000018bce8608 ffffffff80c62020 [2023-05-01 08:34:15.567] 0000000000000000 80000001f6e335d0 0000000000000000 0000000100000000 [2023-05-01 08:34:15.567] ffffffff80eeab00 ffffffff80c10bcc 8000000100000000 0000000000000000 [2023-05-01 08:34:15.583] 0000000000000000 80000001eb678300 0000000006000000 ffffffffc2960000 [2023-05-01 08:34:15.583] 80000001f6e33680 0000000000000000 ffffffff80eeab00 80000001f6a0e000 [2023-05-01 08:34:15.599] 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 [2023-05-01 08:34:15.599] 800000018bce8608 ffffffff80c10f2c 80000001f6e336c0 80000001ea411b00 [2023-05-01 08:34:15.599] ... [2023-05-01 08:34:15.599] [253559.602284] Call Trace: [2023-05-01 08:34:15.614] [253559.604822] [] fib_get_table+0x38/0x50 [2023-05-01 08:34:15.614] [253559.610231] [] fib4_rule_action+0x58/0xa0 [2023-05-01 08:34:15.614] [253559.615894] [2023-05-01 08:34:15.614] [253559.617472] [2023-05-01 08:34:15.614] Code: 10400006 00000000 8c430010 <1465fffb> 00000000 03e00008 00000000 03e00008 0000102d [2023-05-01 08:34:15.630] [253559.627599] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:34:15.645] [253559.636762] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:34:15.645] [253559.643393] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:34:15.645] 0000000000000000 0000000000000000 000000000000e3d2 0000000000000442 [2023-05-01 08:34:15.661] ffffffff859a07f8 ffffffff80ec0000 0000000000000442 0000000000000002 [2023-05-01 08:34:15.661] 0000000000000002 ffffffff859c1c18 ffffffff859e0000 0000000000000000 [2023-05-01 08:34:15.677] ffffffff80eb0428 ffffffff80eb0000 80000001f6e33390 80000000243e1688 [2023-05-01 08:34:15.677] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:34:15.677] 0000000000000060 0000000000000001 0000000000000000 0000000000000001 [2023-05-01 08:34:15.692] 80000001f6e30000 80000001f6e33020 0000e695d509140d ffffffff808f739c [2023-05-01 08:34:15.692] 0000000000000004 0000000000000000 0000000000000441 6a6563745f776f72 [2023-05-01 08:34:15.708] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:34:15.708] ... [2023-05-01 08:34:15.708] [253559.709119] Call Trace: [2023-05-01 08:34:15.708] [253559.711656] [] show_stack+0x6c/0xf8 [2023-05-01 08:34:15.724] [253559.716804] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:34:15.724] [253559.722734] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:34:15.724] [253559.728835] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:34:15.739] [253559.734762] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:34:15.739] [253559.740776] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:34:15.755] [253559.747140] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:34:15.755] [253559.752895] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:34:15.755] [253559.758738] [] do_IRQ+0x18/0x30 [2023-05-01 08:34:15.770] [253559.763538] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:34:15.770] [253559.769289] [2023-05-01 08:34:48.264] [253592.268458] Log Date: 1682886889, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:34:48.280] [253592.278999] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:34:48.280] [253592.285098] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 4 [2023-05-01 08:34:48.295] [253592.293459] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:34:48.295] [253592.300079] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:34:48.608] [253592.600750] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:34:48.608] [253592.609917] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:34:48.624] [253592.616551] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:34:48.624] [253592.624128] $ 0 : 0000000000000000 0000000000000008 0000000000000001 0000000000000006 [2023-05-01 08:34:48.639] [253592.632283] $ 4 : 0000000000000014 0000000000000578 80000001f6e336a0 80000001f6e336a8 [2023-05-01 08:34:48.639] [253592.640439] $ 8 : 80000001ea686000 80000001f6e337e0 ffffffff80c13b00 ffffffff80000000 [2023-05-01 08:34:48.655] [253592.648597] $12 : 0000000000000010 ffffffff80a489b4 ffffffff858d4910 0000000000001080 [2023-05-01 08:34:48.655] [253592.656755] $16 : 80000001ea619b00 0000000000000014 80000001f6e336a0 80000001f6e336a8 [2023-05-01 08:34:48.670] [253592.664912] $20 : 80000001ea619b00 ffffffff80eb7e90 0000000000000000 ffffffff80c13b00 [2023-05-01 08:34:48.670] [253592.673067] $24 : 0000000000000004 ffffffffc24cc000 [2023-05-01 08:34:48.686] [253592.681226] $28 : 80000001f6e30000 80000001f6e33600 80000001f6e337e0 ffffffff80bda7f8 [2023-05-01 08:34:48.686] [253592.689382] Hi : 000000000000004c [2023-05-01 08:34:48.702] [253592.693047] Lo : 0000000000100fd5 [2023-05-01 08:34:48.702] [253592.696716] epc : ffffffff80c6ab04 ipv4_get_l4proto+0x7c/0xe0 [2023-05-01 08:34:48.702] [253592.702728] Tainted: P C O [2023-05-01 08:34:48.702] [253592.706572] ra : ffffffff80bda7f8 nf_conntrack_in+0x98/0x860 [2023-05-01 08:34:48.717] [253592.712584] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:34:48.717] [253592.717718] Cause : 40808000 [2023-05-01 08:34:48.717] [253592.720688] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:34:48.733] [253592.725396] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:34:49.030] [253593.026047] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:34:49.045] [253593.036754] Stack : 80000001e417f800 0000000000000000 ffffffff858d4900 80000001ea784900 [2023-05-01 08:34:49.045] ffffffff80eab568 80000001ea619b00 0000000000000000 0000000000000002 [2023-05-01 08:34:49.045] ffffffff80eeab00 ffffffff80bda7f8 80000001e417f800 ffffffffc2364630 [2023-05-01 08:34:49.061] 80000001ea633500 800000018fd1c000 80000001f0742000 ac14105a00000000 [2023-05-01 08:34:49.061] 0000000000000000 00160002ac14005a 0000000000000000 0000000000000000 [2023-05-01 08:34:49.077] 0000001400000003 0600000000000013 0000000000000012 0000000000000000 [2023-05-01 08:34:49.077] 0000000000000000 0000000000000000 ffffffff80eb0000 0000000000000000 [2023-05-01 08:34:49.092] 80000001ea619b00 80000001e417f800 0000000000000000 ffffffff80c13b00 [2023-05-01 08:34:49.092] 80000001f6e337e0 ffffffff80bd2e74 ffffffff80eb7f00 ffffffffc29653f4 [2023-05-01 08:34:49.092] 0000000000000000 ffffffff80eb2070 ffffffff80000000 ffffffff858d4930 [2023-05-01 08:34:49.108] ... [2023-05-01 08:34:49.108] [253593.102479] Call Trace: [2023-05-01 08:34:49.108] [253593.105018] [] ipv4_get_l4proto+0x7c/0xe0 [2023-05-01 08:34:49.108] [253593.110688] [] nf_conntrack_in+0x98/0x860 [2023-05-01 08:34:49.123] [253593.116354] [] nf_iterate+0x13c/0x590 [2023-05-01 08:34:49.123] [253593.121673] [] nf_hook_slow+0x8c/0x1f8 [2023-05-01 08:34:49.123] [253593.127081] [] ip_rcv+0x2f8/0x410 [2023-05-01 08:34:49.139] [253593.132055] [] __netif_receive_skb_core+0x500/0x618 [2023-05-01 08:34:49.139] [253593.138591] [] process_backlog+0xa8/0x190 [2023-05-01 08:34:49.139] [253593.144257] [] net_rx_action+0x14c/0x228 [2023-05-01 08:34:49.155] [253593.149840] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:34:49.155] [253593.155333] [] do_softirq+0x68/0x70 [2023-05-01 08:34:49.155] [253593.160478] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:34:49.170] [253593.166341] [] fqdnObject_packetQ_pop+0x64/0x120 [fqdn_object] [2023-05-01 08:34:49.170] [253593.173847] [] fqdnObject_work_queue_func+0x40/0x390 [fqdn_object] [2023-05-01 08:34:49.186] [253593.181694] [] process_one_work+0x178/0x3e0 [2023-05-01 08:34:49.186] [253593.187535] [] worker_thread+0x140/0x3c8 [2023-05-01 08:34:49.202] [253593.193116] [] kthread+0xa4/0xb0 [2023-05-01 08:34:49.202] [253593.198002] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:34:49.202] [253593.204188] [2023-05-01 08:34:49.202] [253593.205766] [2023-05-01 08:34:49.217] Code: ae510000 90630009 a2630000 <8e0300a0> 8e440000 0064202b 2403ffff 0064100b 0040202d [2023-05-01 08:34:49.217] [253593.215894] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:34:49.233] [253593.225047] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:34:49.233] [253593.231679] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:34:49.248] 0000000000000000 0000000000000000 000000000000e8c0 0000000000000488 [2023-05-01 08:34:49.248] ffffffff859a07f8 ffffffff80ec0000 0000000000000488 0000000000000002 [2023-05-01 08:34:49.248] 0000000000000002 ffffffff859c4db0 ffffffff859e0000 0000000000000000 [2023-05-01 08:34:49.264] ffffffff80eb0428 ffffffff80eb0000 80000001f6e33470 80000000243e1688 [2023-05-01 08:34:49.264] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:34:49.280] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:34:49.280] 80000001f6e30000 80000001f6e33100 0000e69da1ca8231 ffffffff808f739c [2023-05-01 08:34:49.280] 0000000000000004 0000000000000000 0000000000000487 6a6563745f776f72 [2023-05-01 08:34:49.295] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:34:49.295] ... [2023-05-01 08:34:49.295] [253593.297398] Call Trace: [2023-05-01 08:34:49.295] [253593.299936] [] show_stack+0x6c/0xf8 [2023-05-01 08:34:49.311] [253593.305087] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:34:49.311] [253593.311017] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:34:49.311] [253593.317118] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:34:49.327] [253593.323047] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:34:49.327] [253593.329061] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:34:49.342] [253593.335423] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:34:49.342] [253593.341177] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:34:49.342] [253593.347020] [] do_IRQ+0x18/0x30 [2023-05-01 08:34:49.358] [253593.351820] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:34:49.358] [253593.357572] [2023-05-01 08:35:20.008] [253624.003911] unpack_message_rr fail ret=-31 [2023-05-01 08:35:53.767] [253657.768459] Log Date: 1682886955, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:35:53.783] [253657.779183] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:35:53.783] [253657.785284] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [2023-05-01 08:35:53.798] [253657.793644] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:35:53.798] [253657.800265] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:35:54.095] [253658.100951] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:35:54.111] [253658.110118] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:35:54.111] [253658.116750] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:35:54.127] [253658.124327] $ 0 : 0000000000000000 0000000000000000 0000000000000001 80000001f52ad092 [2023-05-01 08:35:54.127] [253658.132483] $ 4 : 80000001f0676000 8000000184251500 0000000000000000 00000000000005be [2023-05-01 08:35:54.142] [253658.140637] $ 8 : ffffffffffffff97 00000000000005b0 80000001f52ad0a0 0000000000000000 [2023-05-01 08:35:54.142] [253658.148794] $12 : 0000000000000001 0000000000000000 000000000000003f 0000000000000000 [2023-05-01 08:35:54.158] [253658.156953] $16 : 8000000184251500 80000001f0676000 8000000184251500 800000017fdf0000 [2023-05-01 08:35:54.158] [253658.165109] $20 : ffffffffc295bbd8 0000000000000001 ffffffff80c19570 ffffffffc2960000 [2023-05-01 08:35:54.173] [253658.173267] $24 : 0000000000000000 ffffffffc2f03450 [2023-05-01 08:35:54.173] [253658.181424] $28 : 80000001f6e30000 80000001f6e33660 800000019b343008 ffffffff80b9fa94 [2023-05-01 08:35:54.189] [253658.189579] Hi : 0000000000000519 [2023-05-01 08:35:54.189] [253658.193244] Lo : 55555555555557e2 [2023-05-01 08:35:54.189] [253658.196916] epc : ffffffff80b98028 netdev_pick_tx+0x20/0xf8 [2023-05-01 08:35:54.205] [253658.202755] Tainted: P C O [2023-05-01 08:35:54.205] [253658.206596] ra : ffffffff80b9fa94 dev_queue_xmit+0x4c/0x4d8 [2023-05-01 08:35:54.220] [253658.212521] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:35:54.220] [253658.217658] Cause : 40808000 [2023-05-01 08:35:54.220] [253658.220628] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:35:54.220] [253658.225336] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:35:54.517] [253658.526012] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:35:54.533] [253658.536718] Stack : 0000000000000004 8000000184251500 80000001f0676000 80000001f0676000 [2023-05-01 08:35:54.548] 800000017fdf0000 ffffffff80b9fa94 0000000000000002 ffffffff80ba8364 [2023-05-01 08:35:54.548] 80000001e8550600 80000001f0676000 8000000184251500 800000017fdf0000 [2023-05-01 08:35:54.564] ffffffffc295bbd8 0000000000000001 ffffffff80c19570 ffffffff80c19800 [2023-05-01 08:35:54.564] c0a80063f0676000 80000001f6a0e000 8000000184251500 800000019b343008 [2023-05-01 08:35:54.564] 80000001f0676000 80000001f6a0e000 ffffffffc295bbd8 ffffffffc296535c [2023-05-01 08:35:54.580] 0000000000000001 ffffffffc2950000 80000001f6788008 000000000000000f [2023-05-01 08:35:54.580] 80000001f6788008 000000000000000f 80000001ee7f1c08 80000001f686e540 [2023-05-01 08:35:54.595] ffffffffc295ba40 ffffffffc295ba80 0000000000200000 ffffffffc29d5a30 [2023-05-01 08:35:54.595] 0000000000200000 000000000000000f 000000000000000f 0000000000000001 [2023-05-01 08:35:54.595] ... [2023-05-01 08:35:54.595] [253658.602445] Call Trace: [2023-05-01 08:35:54.611] [253658.604986] [] netdev_pick_tx+0x20/0xf8 [2023-05-01 08:35:54.611] [253658.610480] [] dev_queue_xmit+0x4c/0x4d8 [2023-05-01 08:35:54.611] [253658.616064] [] ip_finish_output+0x290/0x490 [2023-05-01 08:35:54.626] [253658.621960] [] ssh_interceptor_send+0x6ec/0xed8 [quicksec] [2023-05-01 08:35:54.626] [253658.629168] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [2023-05-01 08:35:54.642] [253658.636808] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:35:54.642] [253658.644859] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:35:54.658] [253658.652011] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:35:54.658] [253658.657764] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:35:54.658] [253658.663256] [] do_softirq+0x68/0x70 [2023-05-01 08:35:54.673] [253658.668402] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:35:54.673] [253658.674255] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [2023-05-01 08:35:54.689] [253658.681936] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [2023-05-01 08:35:54.689] [253658.689870] [] process_one_work+0x178/0x3e0 [2023-05-01 08:35:54.689] [253658.695711] [] worker_thread+0x140/0x3c8 [2023-05-01 08:35:54.705] [253658.701291] [] kthread+0xa4/0xb0 [2023-05-01 08:35:54.705] [253658.706177] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:35:54.720] [253658.712362] [2023-05-01 08:35:54.720] [253658.713940] [2023-05-01 08:35:54.720] Code: ffbf0028 ffb30020 ffb10010 <8c83030c> 10620018 0080802d dc820190 dc420028 10400020 [2023-05-01 08:35:54.720] [253658.724075] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:35:54.736] [253658.733228] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:35:54.736] [253658.739858] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:35:54.751] 0000000000000000 0000000000000000 000000000000eda6 00000000000004cf [2023-05-01 08:35:54.751] ffffffff859a07f8 ffffffff80ec0000 00000000000004cf 0000000000000002 [2023-05-01 08:35:54.767] 0000000000000002 ffffffff859c7fd8 ffffffff859e0000 0000000000000000 [2023-05-01 08:35:54.767] ffffffff80eb0428 ffffffff80eb0000 80000001f6e334d0 80000000243e1688 [2023-05-01 08:35:54.767] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:35:54.783] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:35:54.783] 80000001f6e30000 80000001f6e33160 0000e6ace1e5303c ffffffff808f739c [2023-05-01 08:35:54.798] 0000000000000004 0000000000000000 00000000000004ce 6a6563745f776f72 [2023-05-01 08:35:54.798] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:35:54.798] ... [2023-05-01 08:35:54.814] [253658.805588] Call Trace: [2023-05-01 08:35:54.814] [253658.808127] [] show_stack+0x6c/0xf8 [2023-05-01 08:35:54.814] [253658.813275] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:35:54.814] [253658.819205] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:35:54.830] [253658.825307] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:35:54.830] [253658.831234] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:35:54.830] [253658.837249] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:35:54.845] [253658.843612] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:35:54.845] [253658.849366] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:35:54.861] [253658.855209] [] do_IRQ+0x18/0x30 [2023-05-01 08:35:54.861] [253658.860010] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:35:54.861] [253658.865761] [2023-05-01 08:36:35.759] [253699.768459] Log Date: 1682886997, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:36:35.775] [253699.779000] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:36:35.790] [253699.785100] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [2023-05-01 08:36:35.790] [253699.793461] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:36:35.806] [253699.800081] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:36:36.103] [253700.100782] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:36:36.103] [253700.109950] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:36:36.118] [253700.116583] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:36:36.118] [253700.124160] $ 0 : 0000000000000000 0000000000000000 80000001f072f9c0 0000000000000000 [2023-05-01 08:36:36.134] [253700.132315] $ 4 : 0000000000000000 ffffffffffffffff 80000001a0609e40 80000001f6e335a0 [2023-05-01 08:36:36.134] [253700.140469] $ 8 : 0000000000000000 00000000582a0000 0000000000000001 0000000000000001 [2023-05-01 08:36:36.150] [253700.148624] $12 : 80000001f6e335d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [2023-05-01 08:36:36.150] [253700.156778] $16 : 80000001f072f9c0 0000000000000000 0000000000000003 80000001f6e335a0 [2023-05-01 08:36:36.165] [253700.164931] $20 : fffffffffffffff5 80000001f6e33680 80000001fbb17a40 0000000000000002 [2023-05-01 08:36:36.165] [253700.173085] $24 : 0000000000000000 ffffffffc295bc30 [2023-05-01 08:36:36.181] [253700.181240] $28 : 80000001f6e30000 80000001f6e33540 80000001fbb179c0 ffffffff80bb659c [2023-05-01 08:36:36.181] [253700.189395] Hi : 0000000000000519 [2023-05-01 08:36:36.197] [253700.193061] Lo : 55555555555557e2 [2023-05-01 08:36:36.197] [253700.196732] epc : ffffffff80bb64dc fib_rules_lookup+0x64/0x1d8 [2023-05-01 08:36:36.197] [253700.202832] Tainted: P C O [2023-05-01 08:36:36.212] [253700.206673] ra : ffffffff80bb659c fib_rules_lookup+0x124/0x1d8 [2023-05-01 08:36:36.212] [253700.212858] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:36:36.212] [253700.217995] Cause : 40808000 [2023-05-01 08:36:36.228] [253700.220966] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:36:36.228] [253700.225674] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:36:36.525] [253700.526354] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:36:36.540] [253700.537062] Stack : 80000001f072f9c0 f032f8b7c0a81064 80000001f6e33680 ffffffff80eeab00 [2023-05-01 08:36:36.540] ffffffffffffff80 80000001f0676000 0000000000000000 0000000000000009 [2023-05-01 08:36:36.556] 0000000000200000 ffffffffc2960000 8000000188a4f608 ffffffff80c62020 [2023-05-01 08:36:36.556] 0000000000000000 80000001f6e335d0 0000000000000000 0000000100000000 [2023-05-01 08:36:36.556] ffffffff80eeab00 ffffffff80c10bcc 8000000100000000 0000000000000000 [2023-05-01 08:36:36.571] 0000000000000000 80000001ea632300 0000000006000000 ffffffffc2960000 [2023-05-01 08:36:36.571] 80000001f6e33680 0000000000000000 ffffffff80eeab00 80000001f6a0e000 [2023-05-01 08:36:36.587] 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 [2023-05-01 08:36:36.587] 8000000188a4f608 ffffffff80c10f2c 80000001f6e336c0 80000001f46a0900 [2023-05-01 08:36:36.587] 80000001ebcbc0a0 ffffffff80c6957c 000000000000781b ffffffffc295e390 [2023-05-01 08:36:36.603] ... [2023-05-01 08:36:36.603] [253700.602790] Call Trace: [2023-05-01 08:36:36.603] [253700.605332] [] fib_rules_lookup+0x64/0x1d8 [2023-05-01 08:36:36.603] [253700.611088] [] __fib_lookup+0x30/0x50 [2023-05-01 08:36:36.618] [253700.616408] [] __ip_route_output_key+0x52c/0x868 [2023-05-01 08:36:36.618] [253700.622683] [] ip_route_output_flow+0x24/0x78 [2023-05-01 08:36:36.634] [253700.628698] [] nf_ip_route+0x1c/0x48 [2023-05-01 08:36:36.634] [253700.633985] [] ssh_interceptor_reroute_skb_ipv4+0x108/0x368 [quicksec] [2023-05-01 08:36:36.634] [253700.642231] [] ssh_interceptor_send+0x134/0xed8 [quicksec] [2023-05-01 08:36:36.650] [253700.649438] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [2023-05-01 08:36:36.650] [253700.657078] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:36:36.665] [253700.665128] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:36:36.665] [253700.672280] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:36:36.681] [253700.678035] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:36:36.681] [253700.683528] [] do_softirq+0x68/0x70 [2023-05-01 08:36:36.681] [253700.688674] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:36:36.696] [253700.694266] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:36:36.696] [253700.702206] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [2023-05-01 08:36:36.712] [253700.710141] [] process_one_work+0x178/0x3e0 [2023-05-01 08:36:36.712] [253700.715982] [] worker_thread+0x140/0x3c8 [2023-05-01 08:36:36.728] [253700.721562] [] kthread+0xa4/0xb0 [2023-05-01 08:36:36.728] [253700.726448] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:36:36.728] [253700.732633] [2023-05-01 08:36:36.728] [253700.734211] [2023-05-01 08:36:36.743] Code: 2414fff5 00000000 8e030014 <10600013> 00000000 8ea50004 10a30010 0000102d 8e030028 [2023-05-01 08:36:36.743] [253700.744346] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:36:36.759] [253700.753505] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:36:36.759] [253700.760135] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:36:36.775] 0000000000000000 0000000000000000 000000000000f290 0000000000000518 [2023-05-01 08:36:36.775] ffffffff859a07f8 ffffffff80ec0000 0000000000000518 0000000000000002 [2023-05-01 08:36:36.775] 0000000000000002 ffffffff859cb2b8 ffffffff859e0000 0000000000000000 [2023-05-01 08:36:36.790] ffffffff80eb0428 ffffffff80eb0000 80000001f6e333b0 80000000243e1688 [2023-05-01 08:36:36.790] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:36:36.806] 0000000000000060 0000000000000001 0000000000000000 3c00000007bc6707 [2023-05-01 08:36:36.806] 80000001f6e30000 80000001f6e33040 0000e6b6a94a55e5 ffffffff808f739c [2023-05-01 08:36:36.806] 0000000000000004 0000000000000000 0000000000000517 6a6563745f776f72 [2023-05-01 08:36:36.821] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:36:36.821] ... [2023-05-01 08:36:36.821] [253700.825866] Call Trace: [2023-05-01 08:36:36.821] [253700.828403] [] show_stack+0x6c/0xf8 [2023-05-01 08:36:36.837] [253700.833552] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:36:36.837] [253700.839481] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:36:36.853] [253700.845583] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:36:36.853] [253700.851510] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:36:36.853] [253700.857525] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:36:36.868] [253700.863888] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:36:36.868] [253700.869641] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:36:36.868] [253700.875484] [] do_IRQ+0x18/0x30 [2023-05-01 08:36:36.884] [253700.880285] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:36:36.884] [253700.886035] [2023-05-01 08:37:10.268] [253734.268458] Log Date: 1682887031, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:37:10.284] [253734.278999] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:37:10.284] [253734.285099] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 2 [2023-05-01 08:37:10.299] [253734.293459] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:37:10.299] [253734.300079] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:37:10.596] [253734.600776] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:37:10.612] [253734.609942] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:37:10.612] [253734.616574] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:37:10.627] [253734.624151] $ 0 : 0000000000000000 0000000014009ce1 80000001f643e608 ffffd55555550000 [2023-05-01 08:37:10.627] [253734.632305] $ 4 : 80000001ee7f1c08 00000000000007a6 0000000000000001 00000000000007a6 [2023-05-01 08:37:10.643] [253734.640461] $ 8 : 0000000000000001 0000000000000001 80000001e842a080 80000001e842a080 [2023-05-01 08:37:10.643] [253734.648617] $12 : 0000000042497dab 0000000000000000 0000000000000002 6db6db6db6db6db7 [2023-05-01 08:37:10.659] [253734.656772] $16 : 0000000000000001 80000001d79b3db8 80000001ee7f1c20 80000001ec364008 [2023-05-01 08:37:10.659] [253734.664927] $20 : 000000000000000b ffffffffc29dda80 80000001d79b3db8 800000019b343028 [2023-05-01 08:37:10.674] [253734.673080] $24 : 0000000000000000 ffffffffc295ba80 [2023-05-01 08:37:10.674] [253734.681235] $28 : 80000001f6e30000 80000001f6e33680 80000001ca7e2808 ffffffffc29ad194 [2023-05-01 08:37:10.690] [253734.689392] Hi : 0000000000000002 [2023-05-01 08:37:10.690] [253734.693057] Lo : 00000000000000af [2023-05-01 08:37:10.690] [253734.696782] epc : ffffffffc29bacb4 swi_fastpath_get_trd_lock+0x34/0xd8 [quicksec] [2023-05-01 08:37:10.706] [253734.704534] Tainted: P C O [2023-05-01 08:37:10.706] [253734.708405] ra : ffffffffc29ad194 fastpath_update_statistics_counters+0xfc/0x258 [quicksec] [2023-05-01 08:37:10.721] [253734.717112] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:37:10.721] [253734.722250] Cause : 40808000 [2023-05-01 08:37:10.721] [253734.725221] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:37:10.721] [253734.729929] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:37:11.034] [253735.030609] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:37:11.034] [253735.041316] Stack : 800000019b343008 80000001f6788008 80000001ee7f1c08 ffffffffc29ad194 [2023-05-01 08:37:11.049] 000000000000000b 800000019b343008 80000001f6788008 0000000000000002 [2023-05-01 08:37:11.049] 80000001ee7f1c08 ffffffffc29d4c1c 0000000000000000 ffffffff80c19808 [2023-05-01 08:37:11.065] c0a80063f0676000 80000001f6a0e000 80000001ea21d200 800000019b343008 [2023-05-01 08:37:11.065] 80000001ee7f1c28 0000000000000000 800000019b343008 0000000000000000 [2023-05-01 08:37:11.065] 0000000000000001 ffffffffc2950000 80000001f6788008 000000000000000f [2023-05-01 08:37:11.080] 80000001ee7f1c08 80000001f686e540 ffffffffc295ba40 ffffffffc295ba80 [2023-05-01 08:37:11.080] ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 [2023-05-01 08:37:11.096] 0000000000200000 000000000000000f 000000000000000f 0000000000000001 [2023-05-01 08:37:11.096] 0000000000000000 80000001ee7f1dd0 0000000000000000 ffffffff00000001 [2023-05-01 08:37:11.112] ... [2023-05-01 08:37:11.112] [253735.107056] Call Trace: [2023-05-01 08:37:11.112] [253735.109638] [] swi_fastpath_get_trd_lock+0x34/0xd8 [quicksec] [2023-05-01 08:37:11.112] [253735.117108] [] fastpath_update_statistics_counters+0xfc/0x258 [quicksec] [2023-05-01 08:37:11.127] [253735.125535] [] fastpath_packet_continue+0x3f4/0x1038 [quicksec] [2023-05-01 08:37:11.127] [253735.133177] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:37:11.143] [253735.140816] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:37:11.143] [253735.148867] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:37:11.159] [253735.156020] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:37:11.159] [253735.161775] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:37:11.159] [253735.167267] [] do_softirq+0x68/0x70 [2023-05-01 08:37:11.174] [253735.172411] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:37:11.174] [253735.178264] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [2023-05-01 08:37:11.190] [253735.185944] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [2023-05-01 08:37:11.190] [253735.193878] [] process_one_work+0x178/0x3e0 [2023-05-01 08:37:11.205] [253735.199720] [] worker_thread+0x140/0x3c8 [2023-05-01 08:37:11.205] [253735.205301] [] kthread+0xa4/0xb0 [2023-05-01 08:37:11.205] [253735.210179] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:37:11.221] [253735.216366] [2023-05-01 08:37:11.221] [253735.217944] [2023-05-01 08:37:11.221] Code: dc820050 34635555 00031c38 <34635555> 00031c78 3463aaab 00e3001d 00001810 0003197a [2023-05-01 08:37:11.221] [253735.228075] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:37:11.237] [253735.237228] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:37:11.237] [253735.243860] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:37:11.252] 0000000000000000 0000000000000000 000000000000f77a 000000000000055d [2023-05-01 08:37:11.252] ffffffff859a07f8 ffffffff80ec0000 000000000000055d 0000000000000002 [2023-05-01 08:37:11.268] 0000000000000002 ffffffff859ce4d8 ffffffff859e0000 0000000000000000 [2023-05-01 08:37:11.268] ffffffff80eb0428 ffffffff80eb0000 80000001f6e334f0 80000000243e1688 [2023-05-01 08:37:11.268] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:37:11.284] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:37:11.284] 80000001f6e30000 80000001f6e33180 0000e6beb1a68ea8 ffffffff808f739c [2023-05-01 08:37:11.299] 0000000000000004 0000000000000000 000000000000055c 6a6563745f776f72 [2023-05-01 08:37:11.299] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:37:11.299] ... [2023-05-01 08:37:11.315] [253735.309592] Call Trace: [2023-05-01 08:37:11.315] [253735.312130] [] show_stack+0x6c/0xf8 [2023-05-01 08:37:11.315] [253735.317280] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:37:11.315] [253735.323209] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:37:11.330] [253735.329311] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:37:11.330] [253735.335239] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:37:11.346] [253735.341254] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:37:11.346] [253735.347616] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:37:11.346] [253735.353371] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:37:11.362] [253735.359215] [] do_IRQ+0x18/0x30 [2023-05-01 08:37:11.362] [253735.364016] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:37:11.362] [253735.369766] [2023-05-01 08:37:44.262] [253768.268458] Log Date: 1682887065, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:37:44.277] [253768.278998] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:37:44.277] [253768.285098] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 3 [2023-05-01 08:37:44.293] [253768.293460] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:37:44.293] [253768.300081] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:37:44.605] [253768.600763] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:37:44.605] [253768.609924] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:37:44.621] [253768.616555] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:37:44.621] [253768.624132] $ 0 : 0000000000000000 0000000000000008 800000018bfd8000 0000000000000017 [2023-05-01 08:37:44.637] [253768.632287] $ 4 : ffffffffc2386bc8 0000000000000017 ffffffffac14005a 0000000000000040 [2023-05-01 08:37:44.637] [253768.640441] $ 8 : 0000000000000000 80000001ef12a600 0000000000000001 0000000000000004 [2023-05-01 08:37:44.652] [253768.648594] $12 : 0000000000000004 80000001f6e33222 ffffffff858d4908 0000000000000000 [2023-05-01 08:37:44.652] [253768.656750] $16 : 0000000000000000 0000000000000001 ffffffffac14005a 0000000000000040 [2023-05-01 08:37:44.668] [253768.664907] $20 : ffffffffc2380000 0000000000000000 ffffffffc238c3f8 ffffffffc2380000 [2023-05-01 08:37:44.668] [253768.673063] $24 : 0000000000000002 ffffffff80a50a40 [2023-05-01 08:37:44.683] [253768.681219] $28 : 80000001f6e30000 80000001f6e33220 8000000184049200 ffffffffc238ab2c [2023-05-01 08:37:44.683] [253768.689375] Hi : 000000000000005e [2023-05-01 08:37:44.699] [253768.693041] Lo : 0000000001c6251d [2023-05-01 08:37:44.699] [253768.696734] epc : ffffffffc238814c find_rpif_country_info+0x6c/0x90 [nf_report] [2023-05-01 08:37:44.699] [253768.704313] Tainted: P C O [2023-05-01 08:37:44.699] [253768.708172] ra : ffffffffc238ab2c country_traffic_statistics+0xb4/0x538 [nf_report] [2023-05-01 08:37:44.715] [253768.716184] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:37:44.715] [253768.721320] Cause : 40808000 [2023-05-01 08:37:44.715] [253768.724289] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:37:44.730] [253768.728997] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:37:45.027] [253769.029649] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:37:45.043] [253769.040354] Stack : 0000000000000000 0000000000000000 000000003269b10c 0000000000000000 [2023-05-01 08:37:45.043] 0000000000000000 0000005ee417f800 800000018ffb3000 80000001e417f800 [2023-05-01 08:37:45.058] 80000001f473e0e8 80000001e9867000 ffffffffc2380000 0000000000000000 [2023-05-01 08:37:45.058] ffffffffc238c3f8 ffffffffc2380000 8000000184049200 ffffffffc23904cc [2023-05-01 08:37:45.058] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:37:45.074] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:37:45.074] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:37:45.090] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:37:45.090] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:37:45.090] 0000000000000000 0000000000000000 0000000000000001 80000001e417f800 [2023-05-01 08:37:45.105] ... [2023-05-01 08:37:45.105] [253769.106104] Call Trace: [2023-05-01 08:37:45.105] [253769.108659] [] find_rpif_country_info+0x6c/0x90 [nf_report] [2023-05-01 08:37:45.121] [253769.115945] [] country_traffic_statistics+0xb4/0x538 [nf_report] [2023-05-01 08:37:45.121] [253769.123665] [] FH_traffic_report_ipv4_hook+0x264/0xeb8 [nf_report] [2023-05-01 08:37:45.136] [253769.131559] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [2023-05-01 08:37:45.136] [253769.138725] [] nf_iterate+0x13c/0x590 [2023-05-01 08:37:45.136] [253769.144045] [] nf_hook_slow+0x8c/0x1f8 [2023-05-01 08:37:45.152] [253769.149454] [] ip_forward+0x45c/0x488 [2023-05-01 08:37:45.152] [253769.154806] [] rtcompl_hook+0x338/0x4a0 [fastpath_kmodule] [2023-05-01 08:37:45.168] [253769.161970] [] nf_iterate+0x13c/0x590 [2023-05-01 08:37:45.168] [253769.167288] [] nf_hook_slow+0x8c/0x1f8 [2023-05-01 08:37:45.168] [253769.172695] [] ip_rcv+0x2f8/0x410 [2023-05-01 08:37:45.168] [253769.177669] [] __netif_receive_skb_core+0x500/0x618 [2023-05-01 08:37:45.183] [253769.184206] [] process_backlog+0xa8/0x190 [2023-05-01 08:37:45.183] [253769.189873] [] net_rx_action+0x14c/0x228 [2023-05-01 08:37:45.199] [253769.195456] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:37:45.199] [253769.200948] [] do_softirq+0x68/0x70 [2023-05-01 08:37:45.199] [253769.206094] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:37:45.215] [253769.211946] [] fqdnObject_packetQ_pop+0x64/0x120 [fqdn_object] [2023-05-01 08:37:45.215] [253769.219452] [] fqdnObject_work_queue_func+0x40/0x390 [fqdn_object] [2023-05-01 08:37:45.230] [253769.227298] [] process_one_work+0x178/0x3e0 [2023-05-01 08:37:45.230] [253769.233140] [] worker_thread+0x140/0x3c8 [2023-05-01 08:37:45.230] [253769.238721] [] kthread+0xa4/0xb0 [2023-05-01 08:37:45.246] [253769.243607] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:37:45.246] [253769.249793] [2023-05-01 08:37:45.246] [253769.251371] [2023-05-01 08:37:45.246] Code: 088e205a 00000000 8c430024 <10650006> 00000000 dc420000 1482fffb 00000000 03e00008 [2023-05-01 08:37:45.261] [253769.261500] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:37:45.261] [253769.270650] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:37:45.277] [253769.277280] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:37:45.277] 0000000000000000 0000000000000000 000000000000fc68 00000000000005a9 [2023-05-01 08:37:45.293] ffffffff859a07f8 ffffffff80ec0000 00000000000005a9 0000000000000002 [2023-05-01 08:37:45.293] 0000000000000002 ffffffff859d1880 ffffffff859e0000 0000000000000000 [2023-05-01 08:37:45.308] ffffffff80eb0428 ffffffff80eb0000 80000001f6e33090 80000000243e1688 [2023-05-01 08:37:45.308] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:37:45.308] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:37:45.324] 80000001f6e30000 80000001f6e32d20 0000e6c69c3562ee ffffffff808f739c [2023-05-01 08:37:45.324] 0000000000000004 0000000000000000 00000000000005a8 6a6563745f776f72 [2023-05-01 08:37:45.339] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:37:45.339] ... [2023-05-01 08:37:45.339] [253769.343009] Call Trace: [2023-05-01 08:37:45.339] [253769.345546] [] show_stack+0x6c/0xf8 [2023-05-01 08:37:45.355] [253769.350694] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:37:45.355] [253769.356623] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:37:45.355] [253769.362725] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:37:45.371] [253769.368652] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:37:45.371] [253769.374667] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:37:45.371] [253769.381029] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:37:45.386] [253769.386783] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:37:45.386] [253769.392624] [] do_IRQ+0x18/0x30 [2023-05-01 08:37:45.402] [253769.397425] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:37:45.402] [253769.403177] [2023-05-01 08:38:18.771] [253802.768458] Log Date: 1682887100, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:38:18.771] [253802.778999] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:38:18.786] [253802.785099] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 4 [2023-05-01 08:38:18.786] [253802.793459] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:38:18.802] [253802.800080] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:38:19.099] [253803.100761] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:38:19.114] [253803.109927] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:38:19.114] [253803.116561] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:38:19.114] [253803.124137] $ 0 : 0000000000000000 0000000054009ce1 00000000000005a4 ffffffffc0020000 [2023-05-01 08:38:19.130] [253803.132294] $ 4 : 00000000000005a4 6db6db6db6db6db7 c920b2ef00000000 0000000000000188 [2023-05-01 08:38:19.130] [253803.140449] $ 8 : ffffffffc2340000 0000000000000001 ffffffffc2347600 80000001e8c42200 [2023-05-01 08:38:19.145] [253803.148604] $12 : 0000000000000000 0000000000000000 0000000000000002 6db6db6db6db6db7 [2023-05-01 08:38:19.161] [253803.156759] $16 : 80000001f6e33120 000000000000018c 80000001ee79e4c4 0000000000000180 [2023-05-01 08:38:19.161] [253803.164914] $20 : 0000000000000000 6db6db6db6db6db7 8000000000000000 000000000000000c [2023-05-01 08:38:19.177] [253803.173068] $24 : 0000000000000000 8000000000000000 [2023-05-01 08:38:19.177] [253803.181223] $28 : 80000001f6e30000 80000001f6e32e90 ffffffff859e9900 ffffffffc233de98 [2023-05-01 08:38:19.192] [253803.189378] Hi : 0000000000000002 [2023-05-01 08:38:19.192] [253803.193043] Lo : 00000000000000af [2023-05-01 08:38:19.192] [253803.196755] epc : ffffffffc233e784 octo_des_cbc_sha1_encrypt+0xa34/0x1018 [cryptocteon] [2023-05-01 08:38:19.208] [253803.205029] Tainted: P C O [2023-05-01 08:38:19.208] [253803.208889] ra : ffffffffc233de98 octo_des_cbc_sha1_encrypt+0x148/0x1018 [cryptocteon] [2023-05-01 08:38:19.208] [253803.217162] Status: 54009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:38:19.224] [253803.222294] Cause : 40808000 [2023-05-01 08:38:19.224] [253803.225264] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:38:19.224] [253803.229971] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:38:19.520] [253803.530614] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:38:19.536] [253803.541321] Stack : 0000000000000000 ffffffff80c19808 ac14100d80c19570 80000001e417f800 [2023-05-01 08:38:19.552] 0000000000000002 0000000000000004 ffffffff80c19570 80000001e417f800 [2023-05-01 08:38:19.552] 0000000000000000 ffffffffc2364630 00000000644ed1bc 00000000136b230f [2023-05-01 08:38:19.552] ffffffffc2380000 80000001f5c18800 80000001e417f800 80000001e8e2f0e8 [2023-05-01 08:38:19.567] 0000000000000000 0000000000000000 0000000000000000 0000000000000009 [2023-05-01 08:38:19.567] 0000000000000008 0000000000000004 0000000000000004 0000000000000000 [2023-05-01 08:38:19.583] ffffffff80eb0000 0000000000000000 0000000000000120 ffffffff80ae5ae8 [2023-05-01 08:38:19.583] 0000000000000001 80000001ea78b000 80000001f6598e68 ffffffffffff8110 [2023-05-01 08:38:19.583] 80000001f0cc0c00 0000000000000000 80000001f6598000 80000001f5c53200 [2023-05-01 08:38:19.599] ffffffff80d78988 0000000000000ff3 0000000000008100 80000001ea78b0d6 [2023-05-01 08:38:19.599] ... [2023-05-01 08:38:19.599] [253803.607089] Call Trace: [2023-05-01 08:38:19.614] [253803.609647] [] octo_des_cbc_sha1_encrypt+0xa34/0x1018 [cryptocteon] [2023-05-01 08:38:19.614] [253803.617630] [] octo_process.part.4+0x48c/0xa38 [cryptocteon] [2023-05-01 08:38:19.630] [253803.624970] [] crypto_invoke+0x60/0x168 [2023-05-01 08:38:19.630] [253803.630463] [] crypto_dispatch+0x214/0x360 [2023-05-01 08:38:19.630] [253803.636266] [] ssh_hwaccel_perform_ipsec+0x31c/0x5b8 [quicksec] [2023-05-01 08:38:19.645] [253803.643909] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [2023-05-01 08:38:19.645] [253803.651812] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [2023-05-01 08:38:19.661] [253803.659453] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:38:19.661] [253803.667092] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:38:19.677] [253803.675143] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:38:19.677] [253803.682294] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:38:19.692] [253803.688049] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:38:19.692] [253803.693541] [] do_softirq+0x68/0x70 [2023-05-01 08:38:19.692] [253803.698686] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:38:19.708] [253803.704278] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:38:19.708] [253803.712218] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [2023-05-01 08:38:19.723] [253803.720153] [] process_one_work+0x178/0x3e0 [2023-05-01 08:38:19.723] [253803.725993] [] worker_thread+0x140/0x3c8 [2023-05-01 08:38:19.723] [253803.731573] [] kthread+0xa4/0xb0 [2023-05-01 08:38:19.739] [253803.736459] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:38:19.739] [253803.742645] [2023-05-01 08:38:19.739] [253803.744223] [2023-05-01 08:38:19.739] Code: 14400019 dfa201d0 2627fffc <04e1ffe2> 00e0882d 8c6272b0 1040fff7 3c0280d3 ffa30230 [2023-05-01 08:38:19.755] [253803.754353] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:38:19.755] [253803.763517] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:38:19.770] [253803.770151] Stack : 0000000000000001 0000000054001ce0 0000000000000000 0000000000000007 [2023-05-01 08:38:19.770] 0000000000000000 0000000000000000 000000000000014e 00000000000005f2 [2023-05-01 08:38:19.786] ffffffff859a07f8 ffffffff80ec0000 00000000000005f2 0000000000000002 [2023-05-01 08:38:19.786] 0000000000000002 ffffffff859d4bd0 ffffffff859e0000 0000000000000000 [2023-05-01 08:38:19.786] ffffffff80eb0428 ffffffff80eb0000 80000001f6e32d00 80000000243e1688 [2023-05-01 08:38:19.802] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:38:19.802] 0000000000000060 0000000000000001 0000000000000000 3c00000007bc0687 [2023-05-01 08:38:19.817] 80000001f6e30000 80000001f6e32990 0000e6cea4919add ffffffff808f739c [2023-05-01 08:38:19.817] 0000000000000004 0000000000000000 00000000000005f1 6a6563745f776f72 [2023-05-01 08:38:19.833] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:38:19.833] ... [2023-05-01 08:38:19.833] [253803.835868] Call Trace: [2023-05-01 08:38:19.833] [253803.838406] [] show_stack+0x6c/0xf8 [2023-05-01 08:38:19.848] [253803.843554] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:38:19.848] [253803.849485] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:38:19.848] [253803.855586] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:38:19.864] [253803.861514] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:38:19.864] [253803.867528] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:38:19.864] [253803.873891] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:38:19.880] [253803.879646] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:38:19.880] [253803.885488] [] do_IRQ+0x18/0x30 [2023-05-01 08:38:19.880] [253803.890288] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:38:19.895] [253803.896040] [2023-05-01 08:38:53.264] [253837.268461] Log Date: 1682887134, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:38:53.280] [253837.279003] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:38:53.280] [253837.285106] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 5 [2023-05-01 08:38:53.295] [253837.293469] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:38:53.295] [253837.300090] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:38:53.592] [253837.600787] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:38:53.608] [253837.609954] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:38:53.608] [253837.616587] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:38:53.623] [253837.624165] $ 0 : 0000000000000000 000000000000001c 0000000000010000 000000000000e959 [2023-05-01 08:38:53.623] [253837.632322] $ 4 : 000000000000e95a 0000000000000000 0000000000000001 ffffffffc29b0000 [2023-05-01 08:38:53.639] [253837.640478] $ 8 : 0000000000000000 0000000000000001 0000000000000000 0000000000000000 [2023-05-01 08:38:53.639] [253837.648633] $12 : ffffffffc2970000 80000001f6e3363c 00000000000061da 0000000000000000 [2023-05-01 08:38:53.654] [253837.656789] $16 : 80000001e41096c8 8000000188a4f608 80000001ee7f1c08 80000001ee7f1c08 [2023-05-01 08:38:53.654] [253837.664945] $20 : 000000000000000b ffffffffc29dda80 80000001d79b3db8 80000001ca7e2918 [2023-05-01 08:38:53.670] [253837.673102] $24 : 0000000000000000 ffffffffc29a8808 [2023-05-01 08:38:53.686] [253837.681257] $28 : 80000001f6e30000 80000001f6e33680 80000001ca7e2808 ffffffff80d34fcc [2023-05-01 08:38:53.686] [253837.689412] Hi : 0000000000000007 [2023-05-01 08:38:53.686] [253837.693077] Lo : 0000000000001fff [2023-05-01 08:38:53.686] [253837.696747] epc : ffffffff80d352b0 _raw_write_lock_bh+0xd0/0xe0 [2023-05-01 08:38:53.701] [253837.702934] Tainted: P C O [2023-05-01 08:38:53.701] [253837.706776] ra : ffffffff80d34fcc _raw_spin_lock_bh+0x14/0x48 [2023-05-01 08:38:53.701] [253837.712875] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:38:53.717] [253837.718011] Cause : 40808000 [2023-05-01 08:38:53.717] [253837.720981] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:38:53.717] [253837.725689] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:38:54.029] [253838.026341] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:38:54.029] [253838.037048] Stack : 80000001e41096c8 ffffffffc295babc 80000001d79b3db8 ffffffffc29a8d1c [2023-05-01 08:38:54.045] 0c360000f6788008 0000000000000003 8000000188a4f608 80000001f6788008 [2023-05-01 08:38:54.045] 0000000000000003 ffffffffc29d4e9c 0000000000000000 ffffffff80c19808 [2023-05-01 08:38:54.061] c0a80063f0676000 80000001f6a0e000 80000001eba52780 8000000188a4f608 [2023-05-01 08:38:54.061] 80000001ee7f1c28 0000000000000000 8000000188a4f608 0000000000000000 [2023-05-01 08:38:54.061] 0000000000000001 ffffffffc2950000 80000001f6788008 000000000000000f [2023-05-01 08:38:54.076] 80000001ee7f1c08 80000001f686e540 ffffffffc295ba40 ffffffffc295ba80 [2023-05-01 08:38:54.076] ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 [2023-05-01 08:38:54.092] 0000000000200000 000000000000000f 000000000000000f 0000000000000001 [2023-05-01 08:38:54.092] 0000000000000000 80000001ee7f1dd0 0000000000000000 ffffffff00000001 [2023-05-01 08:38:54.092] ... [2023-05-01 08:38:54.108] [253838.102776] Call Trace: [2023-05-01 08:38:54.108] [253838.105314] [] _raw_write_lock_bh+0xd0/0xe0 [2023-05-01 08:38:54.108] [253838.111202] [] ssh_kernel_mutex_lock_bh_i+0x1c/0x40 [quicksec] [2023-05-01 08:38:54.108] [253838.118755] [] fastpath_execute_transform_step+0x514/0x5a0 [quicksec] [2023-05-01 08:38:54.123] [253838.126920] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [2023-05-01 08:38:54.123] [253838.134561] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:38:54.139] [253838.142201] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:38:54.154] [253838.150251] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:38:54.154] [253838.157403] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:38:54.154] [253838.163156] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:38:54.170] [253838.168649] [] do_softirq+0x68/0x70 [2023-05-01 08:38:54.170] [253838.173793] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:38:54.170] [253838.179382] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:38:54.186] [253838.187322] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [2023-05-01 08:38:54.186] [253838.195256] [] process_one_work+0x178/0x3e0 [2023-05-01 08:38:54.201] [253838.201098] [] worker_thread+0x140/0x3c8 [2023-05-01 08:38:54.201] [253838.206679] [] kthread+0xa4/0xb0 [2023-05-01 08:38:54.201] [253838.211565] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:38:54.217] [253838.217750] [2023-05-01 08:38:54.217] [253838.219328] [2023-05-01 08:38:54.217] Code: 1460ffff 2463ffff 96030002 <1064ff4f> 00831823 1000fff8 2463ffff 67bdffe0 24025401 [2023-05-01 08:38:54.232] [253838.229458] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:38:54.232] [253838.238609] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:38:54.248] [253838.245239] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:38:54.248] 0000000000000000 0000000000000000 000000000000063a 0000000000000638 [2023-05-01 08:38:54.248] ffffffff859a07f8 ffffffff80ec0000 0000000000000638 0000000000000002 [2023-05-01 08:38:54.264] 0000000000000002 ffffffff859d7e08 ffffffff859e0000 0000000000000000 [2023-05-01 08:38:54.264] ffffffff80eb0428 ffffffff80eb0000 80000001f6e334f0 80000000243e1688 [2023-05-01 08:38:54.279] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:38:54.279] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:38:54.295] 80000001f6e30000 80000001f6e33180 0000e6d6acedd59b ffffffff808f739c [2023-05-01 08:38:54.295] 0000000000000004 0000000000000000 0000000000000637 6a6563745f776f72 [2023-05-01 08:38:54.295] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:38:54.311] ... [2023-05-01 08:38:54.311] [253838.310980] Call Trace: [2023-05-01 08:38:54.311] [253838.313517] [] show_stack+0x6c/0xf8 [2023-05-01 08:38:54.311] [253838.318665] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:38:54.326] [253838.324594] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:38:54.326] [253838.330696] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:38:54.326] [253838.336623] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:38:54.342] [253838.342638] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:38:54.342] [253838.349000] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:38:54.357] [253838.354755] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:38:54.357] [253838.360597] [] do_IRQ+0x18/0x30 [2023-05-01 08:38:54.357] [253838.365397] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:38:54.373] [253838.371148] [2023-05-01 08:38:56.904] [253840.904942] unpack_message_rr fail ret=-31 [2023-05-01 08:39:30.757] [253874.768459] Log Date: 1682887172, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:39:30.772] [253874.779184] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:39:30.788] [253874.785286] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 6 [2023-05-01 08:39:30.788] [253874.793649] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:39:30.804] [253874.800270] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:39:31.100] [253875.100928] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:39:31.100] [253875.110094] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:39:31.116] [253875.116727] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:39:31.116] [253875.124304] $ 0 : 0000000000000000 0000000000000008 ffffffffffffff38 ffffffff80eb2070 [2023-05-01 08:39:31.132] [253875.132459] $ 4 : ffffffff80eb7f00 ffffffff80eb7f00 0000000000000000 ffffffff858d4930 [2023-05-01 08:39:31.132] [253875.140615] $ 8 : ffffffff80c13b00 80000001f6e337a0 ffffffff80c13b00 ffffffff80000000 [2023-05-01 08:39:31.147] [253875.148770] $12 : 0000000000000010 ffffffff80a489b4 ffffffff858d4910 0000000000001080 [2023-05-01 08:39:31.147] [253875.156925] $16 : 0000000000000000 0000000000000000 ffffffff80eb0000 80000001f6e337a0 [2023-05-01 08:39:31.163] [253875.165081] $20 : 80000001f47faf00 80000001e417f800 0000000000000000 ffffffff80c13b00 [2023-05-01 08:39:31.163] [253875.173236] $24 : 0000000000000004 ffffffffc24cc000 [2023-05-01 08:39:31.179] [253875.181391] $28 : 80000001f6e30000 80000001f6e336d0 0000000000000000 ffffffff80bd2e74 [2023-05-01 08:39:31.179] [253875.189546] Hi : 000000000000004c [2023-05-01 08:39:31.194] [253875.193211] Lo : 0000000000100fd5 [2023-05-01 08:39:31.194] [253875.196881] epc : ffffffff80bd2df4 nf_iterate+0xbc/0x590 [2023-05-01 08:39:31.194] [253875.202460] Tainted: P C O [2023-05-01 08:39:31.194] [253875.206306] ra : ffffffff80bd2e74 nf_iterate+0x13c/0x590 [2023-05-01 08:39:31.210] [253875.211971] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:39:31.210] [253875.217105] Cause : 40808000 [2023-05-01 08:39:31.210] [253875.220075] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:39:31.225] [253875.224783] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:39:31.522] [253875.525446] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:39:31.538] [253875.536154] Stack : ffffffff80eb7f00 0000000000000000 0000000000000000 ffffffff80eb2070 [2023-05-01 08:39:31.538] ffffffff80000000 ffffffff858d4930 0000000000000000 0000000000000000 [2023-05-01 08:39:31.538] 0000000000000000 0000000000000000 ffffffff858d4900 0000000000000000 [2023-05-01 08:39:31.553] ffffffff80000000 ffffffff858d4930 0000000000000000 80000001f47faf00 [2023-05-01 08:39:31.553] ffffffff80c13b00 0000000000000000 80000001e417f800 ffffffff80eb2070 [2023-05-01 08:39:31.569] 0000000000000002 ffffffff80eb7e10 ffffffff80000000 ffffffff80bd3354 [2023-05-01 08:39:31.569] ffffffff80c13b00 0000000000000000 ffffffff80eb7f00 ffffffff80eb2070 [2023-05-01 08:39:31.585] 80000001f47faf00 ffffffff80eb0428 ffffffff80eeab00 80000001e417f800 [2023-05-01 08:39:31.585] 80000001ea5eb0c4 0000000000000000 ffffffff80eb0428 ffffffff80eb7e10 [2023-05-01 08:39:31.585] 0000000000000001 ffffffff80c14638 ffffffff80eb0e28 80000001e417f800 [2023-05-01 08:39:31.600] ... [2023-05-01 08:39:31.600] [253875.601876] Call Trace: [2023-05-01 08:39:31.600] [253875.604415] [] nf_iterate+0xbc/0x590 [2023-05-01 08:39:31.600] [253875.609647] [] nf_hook_slow+0x8c/0x1f8 [2023-05-01 08:39:31.616] [253875.615055] [] ip_rcv+0x2f8/0x410 [2023-05-01 08:39:31.616] [253875.620029] [] __netif_receive_skb_core+0x500/0x618 [2023-05-01 08:39:31.616] [253875.626571] [] process_backlog+0xa8/0x190 [2023-05-01 08:39:31.632] [253875.632244] [] net_rx_action+0x14c/0x228 [2023-05-01 08:39:31.632] [253875.637829] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:39:31.632] [253875.643322] [] do_softirq+0x68/0x70 [2023-05-01 08:39:31.647] [253875.648467] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:39:31.647] [253875.654330] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [2023-05-01 08:39:31.663] [253875.662009] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [2023-05-01 08:39:31.663] [253875.669943] [] process_one_work+0x178/0x3e0 [2023-05-01 08:39:31.678] [253875.675784] [] worker_thread+0x140/0x3c8 [2023-05-01 08:39:31.678] [253875.681365] [] kthread+0xa4/0xb0 [2023-05-01 08:39:31.678] [253875.686252] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:39:31.694] [253875.692438] [2023-05-01 08:39:31.694] [253875.694016] [2023-05-01 08:39:31.694] Code: ffa00010 ffa30050 8ca20028 0043102a 14400066 3c06858d 64c643b8 8cd10040 [2023-05-01 08:39:31.694] [253875.704145] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:39:31.710] [253875.713296] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:39:31.710] [253875.719927] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:39:31.725] 0000000000000000 0000000000000000 0000000000000b18 000000000000067d [2023-05-01 08:39:31.725] ffffffff859a07f8 ffffffff80ec0000 000000000000067d 0000000000000002 [2023-05-01 08:39:31.741] 0000000000000002 ffffffff859daf48 ffffffff859e0000 0000000000000000 [2023-05-01 08:39:31.741] ffffffff80eb0428 ffffffff80eb0000 80000001f6e33540 80000000243e1688 [2023-05-01 08:39:31.741] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:39:31.757] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:39:31.757] 80000001f6e30000 80000001f6e331d0 0000e6df681a6bc8 ffffffff808f739c [2023-05-01 08:39:31.772] 0000000000000004 0000000000000000 000000000000067c 6a6563745f776f72 [2023-05-01 08:39:31.772] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:39:31.772] ... [2023-05-01 08:39:31.788] [253875.785646] Call Trace: [2023-05-01 08:39:31.788] [253875.788183] [] show_stack+0x6c/0xf8 [2023-05-01 08:39:31.788] [253875.793330] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:39:31.788] [253875.799259] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:39:31.803] [253875.805360] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:39:31.803] [253875.811288] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:39:31.819] [253875.817301] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:39:31.819] [253875.823664] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:39:31.819] [253875.829418] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:39:31.835] [253875.835265] [] do_IRQ+0x18/0x30 [2023-05-01 08:39:31.835] [253875.840070] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:39:31.835] [253875.845824] [2023-05-01 08:39:44.660] [253888.662144] unpack_message_rr fail ret=-31 [2023-05-01 08:40:17.763] [253921.768465] Log Date: 1682887219, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:40:17.779] [253921.779192] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:40:17.779] [253921.785292] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 7 [2023-05-01 08:40:17.795] [253921.793654] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:40:17.795] [253921.800275] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:40:18.091] [253922.100958] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:40:18.107] [253922.110118] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:40:18.107] [253922.116752] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:40:18.123] [253922.124329] $ 0 : 0000000000000000 000000000000001c 0000000000010000 0000000000000000 [2023-05-01 08:40:18.123] [253922.132483] $ 4 : 000000000000ed63 0000000000000000 0000000000000001 ffffffffc29b0000 [2023-05-01 08:40:18.138] [253922.140640] $ 8 : 0000000000000000 0000000000000001 0000000000000000 0000000000000000 [2023-05-01 08:40:18.138] [253922.148793] $12 : ffffffffc2970000 80000001f6e3363c 00000000000061da 0000000000000000 [2023-05-01 08:40:18.154] [253922.156949] $16 : 80000001e41096c8 800000019b3cbc08 80000001ee7f1c08 80000001ee7f1c08 [2023-05-01 08:40:18.154] [253922.165105] $20 : 000000000000000b ffffffffc29dda80 80000001d79b3db8 80000001ca7e2918 [2023-05-01 08:40:18.169] [253922.173261] $24 : 0000000000000004 ffffffffc29a8808 [2023-05-01 08:40:18.169] [253922.181414] $28 : 80000001f6e30000 80000001f6e33680 80000001ca7e2808 ffffffff80d34fcc [2023-05-01 08:40:18.185] [253922.189570] Hi : 0000000000000007 [2023-05-01 08:40:18.185] [253922.193236] Lo : 0000000000001fff [2023-05-01 08:40:18.185] [253922.196906] epc : ffffffff80d352a4 _raw_write_lock_bh+0xc4/0xe0 [2023-05-01 08:40:18.201] [253922.203092] Tainted: P C O [2023-05-01 08:40:18.201] [253922.206933] ra : ffffffff80d34fcc _raw_spin_lock_bh+0x14/0x48 [2023-05-01 08:40:18.201] [253922.213032] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:40:18.216] [253922.218168] Cause : 40808000 [2023-05-01 08:40:18.216] [253922.221138] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:40:18.216] [253922.225847] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:40:18.529] [253922.526494] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:40:18.529] [253922.537201] Stack : 80000001e41096c8 ffffffffc295babc 80000001d79b3db8 ffffffffc29a8d1c [2023-05-01 08:40:18.544] 32c80000f6788008 0000000000000003 800000019b3cbc08 80000001f6788008 [2023-05-01 08:40:18.544] 0000000000000003 ffffffffc29d4e9c ffffffffc2362230 000000001403c76a [2023-05-01 08:40:18.560] 0000000000000000 ffffffff80eb2070 ffffffff80000000 ffffffff858d4930 [2023-05-01 08:40:18.560] 80000001ee7f1c28 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:18.560] 0000000000000001 0000000000000000 80000001f6788008 000000000000000f [2023-05-01 08:40:18.576] 80000001ee7f1c08 80000001f686e540 ffffffffc295ba40 ffffffffc295ba80 [2023-05-01 08:40:18.576] ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 [2023-05-01 08:40:18.591] 0000000000200000 000000000000000f 000000000000000f 0000000000000001 [2023-05-01 08:40:18.591] 0000000000000000 80000001ee7f1dd0 80000001ebc1a180 ffffffff00000001 [2023-05-01 08:40:18.591] ... [2023-05-01 08:40:18.591] [253922.602928] Call Trace: [2023-05-01 08:40:18.607] [253922.605464] [] _raw_write_lock_bh+0xc4/0xe0 [2023-05-01 08:40:18.607] [253922.611352] [] ssh_kernel_mutex_lock_bh_i+0x1c/0x40 [quicksec] [2023-05-01 08:40:18.607] [253922.618906] [] fastpath_execute_transform_step+0x514/0x5a0 [quicksec] [2023-05-01 08:40:18.623] [253922.627070] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [2023-05-01 08:40:18.623] [253922.634712] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:40:18.638] [253922.642352] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:40:18.638] [253922.650402] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:40:18.654] [253922.657554] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:40:18.654] [253922.663307] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:40:18.669] [253922.668800] [] do_softirq+0x68/0x70 [2023-05-01 08:40:18.669] [253922.673945] [] local_bh_enable+0xa8/0xb0 [2023-05-01 08:40:18.669] [253922.679536] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [2023-05-01 08:40:18.685] [253922.687477] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [2023-05-01 08:40:18.685] [253922.695412] [] process_one_work+0x178/0x3e0 [2023-05-01 08:40:18.701] [253922.701253] [] worker_thread+0x140/0x3c8 [2023-05-01 08:40:18.701] [253922.706834] [] kthread+0xa4/0xb0 [2023-05-01 08:40:18.701] [253922.711720] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:40:18.716] [253922.717905] [2023-05-01 08:40:18.716] [253922.719483] [2023-05-01 08:40:18.716] Code: 24030000 30631fff 00031940 <1460ffff> 2463ffff 96030002 1064ff4f 00831823 1000fff8 [2023-05-01 08:40:18.716] [253922.729610] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:40:18.732] [253922.738761] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:40:18.732] [253922.745391] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:40:18.747] 0000000000000000 0000000000000000 0000000000000ffc 00000000000006c4 [2023-05-01 08:40:18.747] ffffffff859a07f8 ffffffff80ec0000 00000000000006c4 0000000000000002 [2023-05-01 08:40:18.763] 0000000000000002 ffffffff859de1b0 ffffffff859e0000 0000000000000000 [2023-05-01 08:40:18.763] ffffffff80eb0428 ffffffff80eb0000 80000001f6e334f0 80000000243e1688 [2023-05-01 08:40:18.779] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:40:18.779] 0000000000000060 0000000000000001 0000000000000000 3c00000007904e87 [2023-05-01 08:40:18.779] 80000001f6e30000 80000001f6e33180 0000e6ea59858993 ffffffff808f739c [2023-05-01 08:40:18.794] 0000000000000004 0000000000000000 00000000000006c3 6a6563745f776f72 [2023-05-01 08:40:18.794] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:40:18.810] ... [2023-05-01 08:40:18.810] [253922.811125] Call Trace: [2023-05-01 08:40:18.810] [253922.813662] [] show_stack+0x6c/0xf8 [2023-05-01 08:40:18.810] [253922.818810] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:40:18.826] [253922.824739] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:40:18.826] [253922.830841] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:40:18.826] [253922.836768] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:40:18.841] [253922.842783] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:40:18.841] [253922.849146] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:40:18.857] [253922.854899] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:40:18.857] [253922.860741] [] do_IRQ+0x18/0x30 [2023-05-01 08:40:18.857] [253922.865541] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:40:18.872] [253922.871293] [2023-05-01 08:40:52.257] [253956.268458] Log Date: 1682887253, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:40:52.272] [253956.278999] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:40:52.272] [253956.285098] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 8 [2023-05-01 08:40:52.288] [253956.293460] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:40:52.288] [253956.300080] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:40:52.600] [253956.600779] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:40:52.600] [253956.609947] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:40:52.616] [253956.616579] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:40:52.616] [253956.624156] $ 0 : 0000000000000000 0000000054009ce1 00000000000005a4 ffffffffc0020000 [2023-05-01 08:40:52.632] [253956.632312] $ 4 : 00000000000005a4 6db6db6db6db6db7 c920b2ef00000000 000000000000055c [2023-05-01 08:40:52.632] [253956.640467] $ 8 : ffffffffc2340000 0000000000000001 ffffffffc2347600 80000001e8c42200 [2023-05-01 08:40:52.647] [253956.648623] $12 : 0000000000000000 0000000000000000 0000000000000002 6db6db6db6db6db7 [2023-05-01 08:40:52.647] [253956.656776] $16 : 80000001f6e33160 000000000000055c 80000001ea5f80f0 0000000000000554 [2023-05-01 08:40:52.663] [253956.664929] $20 : 0000000000000000 6db6db6db6db6db7 8000000000000000 000000000000000c [2023-05-01 08:40:52.663] [253956.673082] $24 : 0000000000000000 8000000000000000 [2023-05-01 08:40:52.678] [253956.681240] $28 : 80000001f6e30000 80000001f6e32ed0 ffffffff859e9900 ffffffffc233de98 [2023-05-01 08:40:52.678] [253956.689394] Hi : 0000000000000002 [2023-05-01 08:40:52.694] [253956.693057] Lo : 00000000000000af [2023-05-01 08:40:52.694] [253956.696767] epc : ffffffffc233e710 octo_des_cbc_sha1_encrypt+0x9c0/0x1018 [cryptocteon] [2023-05-01 08:40:52.694] [253956.705041] Tainted: P C O [2023-05-01 08:40:52.710] [253956.708900] ra : ffffffffc233de98 octo_des_cbc_sha1_encrypt+0x148/0x1018 [cryptocteon] [2023-05-01 08:40:52.710] [253956.717173] Status: 54009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:40:52.710] [253956.722309] Cause : 40808000 [2023-05-01 08:40:52.725] [253956.725280] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:40:52.725] [253956.729988] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:40:53.022] [253957.030656] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:40:53.038] [253957.041363] Stack : 8000000186d58000 80000001e842a000 0000000000000068 ffffffffc2344838 [2023-05-01 08:40:53.038] 80000001e8e65600 800000000c574c32 000000c400000068 0000000000000000 [2023-05-01 08:40:53.053] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:53.053] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:53.069] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:53.069] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:53.069] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:53.085] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:53.085] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [2023-05-01 08:40:53.100] 0000000000000000 0000000000000000 [2023-05-01 08:40:53.100] [253957.103441] Call Trace: [2023-05-01 08:40:53.100] [253957.105998] [] octo_des_cbc_sha1_encrypt+0x9c0/0x1018 [cryptocteon] [2023-05-01 08:40:53.116] [253957.113980] [] octo_process.part.4+0x48c/0xa38 [cryptocteon] [2023-05-01 08:40:53.116] [253957.121320] [] crypto_invoke+0x60/0x168 [2023-05-01 08:40:53.116] [253957.126813] [] crypto_dispatch+0x214/0x360 [2023-05-01 08:40:53.132] [253957.132616] [] ssh_hwaccel_perform_ipsec+0x31c/0x5b8 [quicksec] [2023-05-01 08:40:53.132] [253957.140260] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [2023-05-01 08:40:53.147] [253957.148164] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [2023-05-01 08:40:53.147] [253957.155806] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [2023-05-01 08:40:53.163] [253957.163446] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:40:53.163] [253957.171497] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:40:53.178] [253957.178649] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:40:53.178] [253957.184402] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:40:53.178] [253957.189894] [] do_softirq+0x68/0x70 [2023-05-01 08:40:53.194] [253957.195039] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:40:53.194] [253957.200895] [] fqdnObject_packetQ_pop+0x64/0x120 [fqdn_object] [2023-05-01 08:40:53.194] [253957.208415] [] fqdnObject_work_queue_func+0x40/0x390 [fqdn_object] [2023-05-01 08:40:53.210] [253957.216275] [] process_one_work+0x178/0x3e0 [2023-05-01 08:40:53.210] [253957.222117] [] worker_thread+0x140/0x3c8 [2023-05-01 08:40:53.225] [253957.227698] [] kthread+0xa4/0xb0 [2023-05-01 08:40:53.225] [253957.232585] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:40:53.225] [253957.238771] [2023-05-01 08:40:53.241] [253957.240350] [2023-05-01 08:40:53.241] Code: 654a7600 088cf9e0 3c08c234 <14e00017> 66520004 26940001 00141178 0289202a 1080002a [2023-05-01 08:40:53.241] [253957.250485] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:40:53.256] [253957.259638] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:40:53.256] [253957.266269] Stack : 0000000000000001 0000000054001ce0 0000000000000000 0000000000000007 [2023-05-01 08:40:53.272] 0000000000000000 0000000000000000 00000000000014de 000000000000070d [2023-05-01 08:40:53.272] ffffffff859a07f8 ffffffff80ec0000 000000000000070d 0000000000000002 [2023-05-01 08:40:53.288] 0000000000000002 ffffffff859a16a8 ffffffff859e0000 0000000000000000 [2023-05-01 08:40:53.288] ffffffff80eb0428 ffffffff80eb0000 80000001f6e32d40 80000000243e1688 [2023-05-01 08:40:53.288] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:40:53.303] 0000000000000060 0000000000000001 0000000000000000 3c00000007bc0687 [2023-05-01 08:40:53.303] 80000001f6e30000 80000001f6e329d0 0000e6f261e1b905 ffffffff808f739c [2023-05-01 08:40:53.319] 0000000000000004 0000000000000000 000000000000070c 6a6563745f776f72 [2023-05-01 08:40:53.319] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:40:53.319] ... [2023-05-01 08:40:53.335] [253957.331995] Call Trace: [2023-05-01 08:40:53.335] [253957.334533] [] show_stack+0x6c/0xf8 [2023-05-01 08:40:53.335] [253957.339682] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:40:53.335] [253957.345611] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:40:53.350] [253957.351713] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:40:53.350] [253957.357641] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:40:53.350] [253957.363656] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:40:53.366] [253957.370018] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:40:53.366] [253957.375772] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:40:53.381] [253957.381615] [] do_IRQ+0x18/0x30 [2023-05-01 08:40:53.381] [253957.386415] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:40:53.381] [253957.392166] [2023-05-01 08:41:26.766] [253990.768460] Log Date: 1682887288, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:41:26.766] [253990.779000] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:41:26.781] [253990.785101] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 9 [2023-05-01 08:41:26.781] [253990.793461] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:41:26.797] [253990.800081] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:41:27.094] [253991.100755] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:41:27.109] [253991.109922] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:41:27.109] [253991.116556] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:41:27.125] [253991.124135] $ 0 : 0000000000000000 0000000000000000 80000001a94d13c0 0000000000000000 [2023-05-01 08:41:27.125] [253991.132292] $ 4 : 0000000000000000 ffffffffffffffff 80000001a0609e40 80000001f6e335a0 [2023-05-01 08:41:27.141] [253991.140449] $ 8 : 0000000000000000 00000000582a0000 0000000000000001 0000000000000001 [2023-05-01 08:41:27.141] [253991.148607] $12 : 80000001f6e335d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [2023-05-01 08:41:27.156] [253991.156764] $16 : 80000001a94d13c0 0000000000000000 0000000000000003 80000001f6e335a0 [2023-05-01 08:41:27.156] [253991.164921] $20 : fffffffffffffff5 80000001f6e33680 80000001fbb17a40 0000000000000002 [2023-05-01 08:41:27.172] [253991.173077] $24 : 0000000000000000 ffffffffc295bc30 [2023-05-01 08:41:27.172] [253991.181232] $28 : 80000001f6e30000 80000001f6e33540 80000001fbb179c0 ffffffff80bb659c [2023-05-01 08:41:27.187] [253991.189389] Hi : 0000000000000519 [2023-05-01 08:41:27.187] [253991.193054] Lo : 55555555555557e2 [2023-05-01 08:41:27.187] [253991.196727] epc : ffffffff80bb64dc fib_rules_lookup+0x64/0x1d8 [2023-05-01 08:41:27.203] [253991.202827] Tainted: P C O [2023-05-01 08:41:27.203] [253991.206669] ra : ffffffff80bb659c fib_rules_lookup+0x124/0x1d8 [2023-05-01 08:41:27.203] [253991.212855] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:41:27.203] [253991.217989] Cause : 40808000 [2023-05-01 08:41:27.219] [253991.220959] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:41:27.219] [253991.225666] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:41:27.516] [253991.526325] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:41:27.531] [253991.537032] Stack : 80000001a94d13c0 f03202c4c0a81064 80000001f6e33680 ffffffff80eeab00 [2023-05-01 08:41:27.531] ffffffffffffff80 80000001f0676000 0000000000000000 0000000000000009 [2023-05-01 08:41:27.547] 0000000000200000 ffffffffc2960000 8000000186ed4808 ffffffff80c62020 [2023-05-01 08:41:27.547] 0000000000000000 80000001f6e335d0 0000000000000000 0000000100000000 [2023-05-01 08:41:27.562] ffffffff80eeab00 ffffffff80c10bcc 8000000100000000 0000000000000000 [2023-05-01 08:41:27.562] 0000000000000000 80000001e8ad1380 0000000006000000 ffffffffc2960000 [2023-05-01 08:41:27.562] 80000001f6e33680 0000000000000000 ffffffff80eeab00 80000001f6a0e000 [2023-05-01 08:41:27.578] 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 [2023-05-01 08:41:27.578] 8000000186ed4808 ffffffff80c10f2c 80000001f6e336c0 80000001e8da2300 [2023-05-01 08:41:27.594] 80000001ea1560a0 ffffffff80c6957c 000000000000781b ffffffffc295e390 [2023-05-01 08:41:27.594] ... [2023-05-01 08:41:27.594] [253991.602756] Call Trace: [2023-05-01 08:41:27.594] [253991.605297] [] fib_rules_lookup+0x64/0x1d8 [2023-05-01 08:41:27.609] [253991.611053] [] __fib_lookup+0x30/0x50 [2023-05-01 08:41:27.609] [253991.616375] [] __ip_route_output_key+0x52c/0x868 [2023-05-01 08:41:27.609] [253991.622650] [] ip_route_output_flow+0x24/0x78 [2023-05-01 08:41:27.625] [253991.628663] [] nf_ip_route+0x1c/0x48 [2023-05-01 08:41:27.625] [253991.633950] [] ssh_interceptor_reroute_skb_ipv4+0x108/0x368 [quicksec] [2023-05-01 08:41:27.640] [253991.642196] [] ssh_interceptor_send+0x134/0xed8 [quicksec] [2023-05-01 08:41:27.640] [253991.649403] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [2023-05-01 08:41:27.656] [253991.657043] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [2023-05-01 08:41:27.656] [253991.665093] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [2023-05-01 08:41:27.672] [253991.672244] [] __tasklet_action+0xac/0x1b8 [2023-05-01 08:41:27.672] [253991.677997] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:41:27.672] [253991.683490] [] do_softirq+0x68/0x70 [2023-05-01 08:41:27.687] [253991.688635] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:41:27.687] [253991.694488] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [2023-05-01 08:41:27.703] [253991.702167] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [2023-05-01 08:41:27.703] [253991.710101] [] process_one_work+0x178/0x3e0 [2023-05-01 08:41:27.703] [253991.715942] [] worker_thread+0x140/0x3c8 [2023-05-01 08:41:27.719] [253991.721522] [] kthread+0xa4/0xb0 [2023-05-01 08:41:27.719] [253991.726408] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:41:27.719] [253991.732595] [2023-05-01 08:41:27.734] [253991.734173] [2023-05-01 08:41:27.734] Code: 2414fff5 00000000 8e030014 <10600013> 00000000 8ea50004 10a30010 0000102d 8e030028 [2023-05-01 08:41:27.734] [253991.744301] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:41:27.750] [253991.753454] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:41:27.750] [253991.760087] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:41:27.765] 0000000000000000 0000000000000000 00000000000019cc 0000000000000756 [2023-05-01 08:41:27.765] ffffffff859a07f8 ffffffff80ec0000 0000000000000756 0000000000000002 [2023-05-01 08:41:27.781] 0000000000000002 ffffffff859a4988 ffffffff859e0000 0000000000000000 [2023-05-01 08:41:27.781] ffffffff80eb0428 ffffffff80eb0000 80000001f6e333b0 80000000243e1688 [2023-05-01 08:41:27.781] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:41:27.797] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:41:27.797] 80000001f6e30000 80000001f6e33040 0000e6fa6a3df5b4 ffffffff808f739c [2023-05-01 08:41:27.812] 0000000000000004 0000000000000000 0000000000000755 6a6563745f776f72 [2023-05-01 08:41:27.812] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:41:27.812] ... [2023-05-01 08:41:27.828] [253991.825808] Call Trace: [2023-05-01 08:41:27.828] [253991.828347] [] show_stack+0x6c/0xf8 [2023-05-01 08:41:27.828] [253991.833495] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:41:27.828] [253991.839425] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:41:27.844] [253991.845526] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:41:27.844] [253991.851455] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:41:27.844] [253991.857469] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:41:27.859] [253991.863832] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:41:27.859] [253991.869585] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:41:27.875] [253991.875429] [] do_IRQ+0x18/0x30 [2023-05-01 08:41:27.875] [253991.880229] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:41:27.875] [253991.885980] [2023-05-01 08:42:00.759] [254024.768459] Log Date: 1682887322, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:42:00.775] [254024.779000] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:42:00.775] [254024.785100] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 10 [2023-05-01 08:42:00.790] [254024.793548] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1154] [2023-05-01 08:42:00.790] [254024.800168] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:42:01.087] [254025.100855] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:42:01.103] [254025.110022] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:42:01.103] [254025.116654] task: 80000001f686e540 ti: 80000001f6e30000 task.ti: 80000001f6e30000 [2023-05-01 08:42:01.118] [254025.124231] $ 0 : 0000000000000000 0000000000000000 0000000000000102 00000000000006d8 [2023-05-01 08:42:01.118] [254025.132387] $ 4 : 0000000026399949 000000386ddd5d4a 000000386ddd57d2 800000018ffb3000 [2023-05-01 08:42:01.134] [254025.140542] $ 8 : 0000000000000001 0000000000000001 0000000000000002 7674693300000000 [2023-05-01 08:42:01.134] [254025.148697] $12 : 0000000000000001 80000001ea3ec740 ffffffff858d4908 0000000000000890 [2023-05-01 08:42:01.150] [254025.156852] $16 : ffffffff80f00000 80000001f072fb40 ffffffff80c15b90 800000018ffb3000 [2023-05-01 08:42:01.165] [254025.165008] $20 : 80000001e417f800 80000001e98671b0 ffffffff858d4908 c000000009fc86d8 [2023-05-01 08:42:01.165] [254025.173163] $24 : 0000000000000002 000000000000000a [2023-05-01 08:42:01.181] [254025.181318] $28 : 80000001f6e30000 80000001f6e33340 ffffffff80eabcd8 ffffffff80c6d18c [2023-05-01 08:42:01.181] [254025.189473] Hi : 0000000002c72420 [2023-05-01 08:42:01.181] [254025.193138] Lo : c49ba5e3574449c0 [2023-05-01 08:42:01.197] [254025.196811] epc : ffffffff808a3104 local_bh_enable+0x44/0xb0 [2023-05-01 08:42:01.197] [254025.202736] Tainted: P C O [2023-05-01 08:42:01.197] [254025.206578] ra : ffffffff80c6d18c ipt_do_table+0x404/0x4e0 [2023-05-01 08:42:01.212] [254025.212416] Status: 14009ce3 KX SX UX KERNEL EXL IE [2023-05-01 08:42:01.212] [254025.217549] Cause : 40808000 [2023-05-01 08:42:01.212] [254025.220519] PrId : 000d9703 (Cavium Octeon III) [2023-05-01 08:42:01.212] [254025.225227] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [2023-05-01 08:42:01.525] [254025.525894] Process kworker/7:1 (pid: 1154, threadinfo=80000001f6e30000, task=80000001f686e540, tls=0000000000000000) [2023-05-01 08:42:01.525] [254025.536601] Stack : 80000001f5180480 ffffffff80c6d18c 8000000186c68628 0000000000000002 [2023-05-01 08:42:01.540] 80000001e417f800 800000018ffb3000 80000001f5180480 80000001888acaf0 [2023-05-01 08:42:01.540] 800000018ffb3000 0000000000000002 c000000009fc8000 80000001f6e333d0 [2023-05-01 08:42:01.556] 0000000000000000 80000001f5180480 ffffffff80eabcb8 0000000000000001 [2023-05-01 08:42:01.556] 80000001e417f800 800000018ffb3000 000000000000000a 0000000000000002 [2023-05-01 08:42:01.556] ffffffff80c15b90 800000018ffb3000 80000001e417f800 80000001e98671b0 [2023-05-01 08:42:01.571] ffffffff858d4908 80000001f5180480 ffffffff80eabcd8 ffffffffc23644c4 [2023-05-01 08:42:01.571] ffffffff80c13b00 ffffffff808d8298 00000000644ed29a 0000000015b3ca40 [2023-05-01 08:42:01.587] 80000001f6e334a0 ffffffffc2913558 0000000000000000 0000000000000000 [2023-05-01 08:42:01.587] 80000001f6e334a0 000000000000000a 0000000000000009 0000000000000002 [2023-05-01 08:42:01.587] ... [2023-05-01 08:42:01.603] [254025.602322] Call Trace: [2023-05-01 08:42:01.603] [254025.604862] [] local_bh_enable+0x44/0xb0 [2023-05-01 08:42:01.603] [254025.610444] [] ipt_do_table+0x404/0x4e0 [2023-05-01 08:42:01.603] [254025.615999] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [2023-05-01 08:42:01.618] [254025.623163] [] nf_iterate+0x13c/0x590 [2023-05-01 08:42:01.618] [254025.628483] [] nf_hook_slow+0x8c/0x1f8 [2023-05-01 08:42:01.634] [254025.633891] [] ip_forward+0x45c/0x488 [2023-05-01 08:42:01.634] [254025.639245] [] rtcompl_hook+0x338/0x4a0 [fastpath_kmodule] [2023-05-01 08:42:01.634] [254025.646408] [] nf_iterate+0x13c/0x590 [2023-05-01 08:42:01.650] [254025.651727] [] nf_hook_slow+0x8c/0x1f8 [2023-05-01 08:42:01.650] [254025.657134] [] ip_rcv+0x2f8/0x410 [2023-05-01 08:42:01.650] [254025.662108] [] __netif_receive_skb_core+0x500/0x618 [2023-05-01 08:42:01.665] [254025.668651] [] process_backlog+0xa8/0x190 [2023-05-01 08:42:01.665] [254025.674324] [] net_rx_action+0x14c/0x228 [2023-05-01 08:42:01.665] [254025.679910] [] __do_softirq+0x1c8/0x210 [2023-05-01 08:42:01.681] [254025.685404] [] do_softirq+0x68/0x70 [2023-05-01 08:42:01.681] [254025.690549] [] local_bh_enable_ip+0xa8/0xb0 [2023-05-01 08:42:01.681] [254025.696411] [] fqdnObject_syncQ_exist_check+0xb8/0x2d0 [fqdn_object] [2023-05-01 08:42:01.696] [254025.704438] [] fqdnObject_work_queue_func+0x1d0/0x390 [fqdn_object] [2023-05-01 08:42:01.712] [254025.712373] [] process_one_work+0x178/0x3e0 [2023-05-01 08:42:01.712] [254025.718214] [] worker_thread+0x140/0x3c8 [2023-05-01 08:42:01.712] [254025.723795] [] kthread+0xa4/0xb0 [2023-05-01 08:42:01.728] [254025.728680] [] ret_from_kernel_thread+0x14/0x1c [2023-05-01 08:42:01.728] [254025.734866] [2023-05-01 08:42:01.728] [254025.736444] [2023-05-01 08:42:01.728] Code: 10400013 9f820020 8f820024 <2442ffff> af820024 dfbf0008 dfb00000 03e00008 67bd0010 [2023-05-01 08:42:01.743] [254025.746573] CPU: 7 PID: 1154 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [2023-05-01 08:42:01.743] [254025.755725] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [2023-05-01 08:42:01.759] [254025.762357] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 [2023-05-01 08:42:01.759] 0000000000000000 0000000000000000 0000000000001eb8 00000000000007a1 [2023-05-01 08:42:01.775] ffffffff859a07f8 ffffffff80ec0000 00000000000007a1 0000000000000002 [2023-05-01 08:42:01.775] 0000000000000002 ffffffff859a7c90 ffffffff859e0000 0000000000000000 [2023-05-01 08:42:01.775] ffffffff80eb0428 ffffffff80eb0000 80000001f6e331b0 80000000243e1688 [2023-05-01 08:42:01.790] 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 [2023-05-01 08:42:01.790] 0000000000000060 0000000000000001 0000000000000000 3c00000007c1ee87 [2023-05-01 08:42:01.806] 80000001f6e30000 80000001f6e32e40 0000e70254ccc7b1 ffffffff808f739c [2023-05-01 08:42:01.806] 0000000000000004 0000000000000000 00000000000007a0 6a6563745f776f72 [2023-05-01 08:42:01.806] 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 [2023-05-01 08:42:01.821] ... [2023-05-01 08:42:01.821] [254025.828077] Call Trace: [2023-05-01 08:42:01.821] [254025.830614] [] show_stack+0x6c/0xf8 [2023-05-01 08:42:01.821] [254025.835763] [] watchdog_timer_fn+0x24c/0x288 [2023-05-01 08:42:01.837] [254025.841693] [] __run_hrtimer.isra.20+0x60/0xf8 [2023-05-01 08:42:01.837] [254025.847794] [] hrtimer_interrupt+0x138/0x310 [2023-05-01 08:42:01.853] [254025.853722] [] c0_compare_interrupt+0x68/0x98 [2023-05-01 08:42:01.853] [254025.859736] [] handle_irq_event_percpu+0x88/0x218 [2023-05-01 08:42:01.853] [254025.866099] [] handle_percpu_irq+0x84/0xb8 [2023-05-01 08:42:01.868] [254025.871852] [] generic_handle_irq+0x38/0x50 [2023-05-01 08:42:01.868] [254025.877694] [] do_IRQ+0x18/0x30 [2023-05-01 08:42:01.868] [254025.882495] [] plat_irq_dispatch+0x98/0xb8 [2023-05-01 08:42:01.884] [254025.888247] [2023-05-01 08:42:35.253] [254059.268459] Log Date: 1682887356, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [2023-05-01 08:42:35.268] [254059.278999] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [2023-05-01 08:42:35.284] [254059.285099] Kernel Info Collector: detect system hanged [Core7], store information in disk. [2023-05-01 08:42:35.284] [254059.293547] Mem-Info: [2023-05-01 08:42:35.284] [254059.295910] DMA32 per-cpu: [2023-05-01 08:42:35.284] [254059.298705] CPU 0: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.299] [254059.303588] CPU 1: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.299] [254059.308471] CPU 2: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.299] [254059.313353] CPU 3: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.315] [254059.318236] CPU 4: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.315] [254059.323119] CPU 5: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.315] [254059.328002] CPU 6: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.331] [254059.332884] CPU 7: hi: 186, btch: 31 usd: 0 [2023-05-01 08:42:35.331] [254059.337766] Normal per-cpu: [2023-05-01 08:42:35.331] [254059.340649] CPU 0: hi: 186, btch: 31 usd: 163 [2023-05-01 08:42:35.331] [254059.345531] CPU 1: hi: 186, btch: 31 usd: 171 [2023-05-01 08:42:35.346] [254059.350414] CPU 2: hi: 186, btch: 31 usd: 91 [2023-05-01 08:42:35.346] [254059.355297] CPU 3: hi: 186, btch: 31 usd: 24 [2023-05-01 08:42:35.346] [254059.360179] CPU 4: hi: 186, btch: 31 usd: 165 [2023-05-01 08:42:35.362] [254059.365062] CPU 5: hi: 186, btch: 31 usd: 135 [2023-05-01 08:42:35.362] [254059.369945] CPU 6: hi: 186, btch: 31 usd: 59 [2023-05-01 08:42:35.362] [254059.374827] CPU 7: hi: 186, btch: 31 usd: 157 [2023-05-01 08:42:35.377] [254059.379716] active_anon:130639 inactive_anon:12363 isolated_anon:0 [2023-05-01 08:42:35.377] [254059.379716] active_file:14762 inactive_file:54365 isolated_file:0 [2023-05-01 08:42:35.377] [254059.379716] unevictable:64 dirty:0 writeback:0 unstable:0 [2023-05-01 08:42:35.393] [254059.379716] free:1433375 slab_reclaimable:15696 slab_unreclaimable:102463 [2023-05-01 08:42:35.393] [254059.379716] mapped:20734 shmem:14156 pagetables:2429 bounce:0 [2023-05-01 08:42:35.409] [254059.379716] free_cma:0 [2023-05-01 08:42:35.409] [254059.413219] DMA32 free:3450144kB min:4848kB low:6060kB high:7272kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3656832kB managed:3382808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [2023-05-01 08:42:35.440] [254059.452875] lowmem_reserve[]: 0 4290 4290 [2023-05-01 08:42:35.456] [254059.457033] Normal free:2283356kB min:6300kB low:7872kB high:9448kB active_anon:522556kB inactive_anon:49452kB active_file:59048kB inactive_file:217460kB unevictable:256kB isolated(anon):0kB isolated(file):0kB present:4393976kB managed:4393976kB mlocked:256kB dirty:0kB writeback:0kB mapped:82936kB shmem:56624kB slab_reclaimable:62784kB slab_unreclaimable:409852kB kernel_stack:32688kB pagetables:9716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [2023-05-01 08:42:35.487] [254059.500774] lowmem_reserve[]: 0 0 0 [2023-05-01 08:42:35.502] [254059.504396] DMA32: 2*4kB (M) 3*8kB (M) 2*16kB (M) 1*32kB (M) 3*64kB (UM) 2*128kB (UM) 3*256kB (UM) 2*512kB (UM) 5*1024kB (UM) 3*2048kB (UM) 839*4096kB (UMR) = 3450144kB [2023-05-01 08:42:35.518] [254059.519804] Normal: 722*4kB (EM) 512*8kB (UEM) 68*16kB (UEM) 35*32kB (EM) 26*64kB (UM) 15*128kB (UM) 12*256kB (M) 24*512kB (UM) 14*1024kB (UM) 6*2048kB (EM) 544*4096kB (MR) = 2282984kB [2023-05-01 08:42:35.534] [254059.536613] 83283 total pagecache pages [2023-05-01 08:42:35.534] [254059.540542] 0 pages in swap cache [2023-05-01 08:42:35.534] [254059.543947] Swap cache stats: add 0, delete 0, find 0/0 [2023-05-01 08:42:35.534] [254059.549264] Free swap = 0kB [2023-05-01 08:42:35.549] [254059.552233] Total swap = 0kB [2023-05-01 08:42:35.674] [254059.690831] 2015232 pages RAM [2023-05-01 08:42:35.690] [254059.693890] 67000 pages reserved [2023-05-01 08:42:35.690] [254059.697206] 137954 pages shared [2023-05-01 08:42:35.690] [254059.700436] 471401 pages non-shared [2023-05-01 08:42:36.049] [254060.065032] KIC: Rebooting in 10 seconds...