[20965.955291] Log Date: 1682632180, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [20965.965929] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [20965.971942] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [20965.980216] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [20965.986749] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [20966.287345] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [20966.296420] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [20966.302966] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [20966.310456] $ 0 : 0000000000000000 0000000014009ce1 80000001e42caca8 ffffffffddff0000 [20966.318524] $ 4 : 0000000000000f33 80000001be57fad8 80000001f6495d08 00000000000000e8 [20966.326590] $ 8 : 0000000001200000 0000000000000001 ffffffffc2347600 8000000184fff200 [20966.334657] $12 : 0000000000000000 0000000000000000 0000000000000002 6db6db6db6db6db7 [20966.342726] $16 : 80000001dff52008 80000001be57fad8 80000001dff52020 80000001f6226008 [20966.350796] $20 : ffffffffc295bbd8 0000000000000288 ffffffff80d34f00 8000000184ecb000 [20966.358861] $24 : 0000000000000000 ffffffffc29d37c0 [20966.366928] $28 : 80000001f6e54000 80000001f6e57050 0000000000000000 ffffffffc29b4ac8 [20966.374997] Hi : 0000000000000002 [20966.378575] Lo : 00000000000000af [20966.382202] epc : ffffffffc29b2f50 ssh_fastpath_tc_lru_insert+0xc8/0x118 [quicksec] [20966.390040] Tainted: P C O [20966.393825] ra : ffffffffc29b4ac8 ssh_fastpath_release_transform_context+0x128/0x130 [quicksec] [20966.402793] Status: 14009ce3 KX SX UX KERNEL EXL IE [20966.407843] Cause : 40808000 [20966.410726] PrId : 000d9703 (Cavium Octeon III) [20966.415346] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [20966.715915] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [20966.726534] Stack : 80000001d79dbdb8 80000001eb6700a0 80000001dff52008 ffffffffc29b4ac8 ffffffffc295bbd8 80000001d79dbdb8 80000001f3e8c0a0 80000001be57fad8 80000001ea008808 ffffffffc29b10f0 0000000000000000 0000000000000000 80000001f57cc730 ffffffff808d0900 00000000000003ff 0000000000000000 0000000000000038 ffffffff80eae018 0000000000000588 0000000000000001 0000000000000590 8000000184fff200 80000001e42ca908 8000000188acd000 8000000184ecb080 00000000000005b0 0000000000000008 80000001f1dd4d80 ffffffff80d34f00 ffffffffc29d3874 8000000184ecb080 8000000188acd000 00000000000005b0 ffffffffc2344838 8000000184fff200 800000000c4e7ea2 000000a0000005b0 0000000000000000 0000000000000000 0000000000000000 ... [20966.792189] Call Trace: [20966.794667] [] ssh_fastpath_tc_lru_insert+0xc8/0x118 [quicksec] [20966.802225] [] ssh_fastpath_release_transform_context+0x128/0x130 [quicksec] [20966.810912] [] ssh_fastpath_transform_out_finish+0x478/0x998 [quicksec] [20966.819165] [] ocf_cb+0xb4/0x128 [quicksec] [20966.824991] [] octo_process.part.4+0xc8/0xa38 [cryptocteon] [20966.832157] [] crypto_invoke+0x60/0x168 [20966.837564] [] crypto_dispatch+0x214/0x360 [20966.843264] [] ssh_hwaccel_perform_ipsec+0x31c/0x5b8 [quicksec] [20966.850820] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [20966.858638] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [20966.866192] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [20966.873744] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [20966.881707] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [20966.888770] [] __tasklet_action+0xac/0x1b8 [20966.894438] [] __do_softirq+0x1c8/0x210 [20966.899844] [] do_softirq+0x68/0x70 [20966.904902] [] local_bh_enable_ip+0xa8/0xb0 [20966.910665] [] fqdnObject_packetQ_pop+0x64/0x120 [fqdn_object] [20966.918085] [] fqdnObject_work_queue_func+0x40/0x390 [fqdn_object] [20966.925845] [] process_one_work+0x178/0x3e0 [20966.931598] [] worker_thread+0x140/0x3c8 [20966.937092] [] kthread+0xa4/0xb0 [20966.941891] [] ret_from_kernel_thread+0x14/0x1c [20966.947990] [20966.949480] Code: 0062102d 3c03ddff 8c440000 <3463ffff> 14830004 dfbf0018 8ca30000 ac430000 dfbf0018 [20966.959526] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [20966.968590] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [20966.975134] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000002c0c 00000000000002b3 ffffffff859a07f8 ffffffff80ec0000 00000000000002b3 0000000000000002 0000000000000002 ffffffff859aed50 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e56ec0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e56b50 0000130b2d058414 ffffffff808f739c 0000000000000004 0000000000000000 00000000000002b2 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [20967.040776] Call Trace: [20967.043227] [] show_stack+0x6c/0xf8 [20967.048287] [] watchdog_timer_fn+0x24c/0x288 [20967.054130] [] __run_hrtimer.isra.20+0x60/0xf8 [20967.060145] [] hrtimer_interrupt+0x138/0x310 [20967.065986] [] c0_compare_interrupt+0x68/0x98 [20967.071913] [] handle_irq_event_percpu+0x88/0x218 [20967.078190] [] handle_percpu_irq+0x84/0xb8 [20967.083857] [] generic_handle_irq+0x38/0x50 [20967.089612] [] do_IRQ+0x18/0x30 [20967.094324] [] plat_irq_dispatch+0x98/0xb8 [20967.099989] [20967.295570] [zy_adp]tcp syn flush flow!! [21084.455290] Log Date: 1682632298, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21084.465744] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21084.471758] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [21084.480034] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21084.486570] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21084.787201] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21084.796281] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21084.802827] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21084.810317] $ 0 : 0000000000000000 000000000000001c 020c49ba5e353f7d 0000000000000000 [21084.818386] $ 4 : 80000001dff52008 80000001d79dbed0 80000001d79dbdb8 0000000000000000 [21084.826452] $ 8 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [21084.834518] $12 : ffffffffc2970000 000000000000002c 00000000000327bd 0000000000000000 [21084.842589] $16 : 80000001d79dbdb8 0000000000000288 80000001d79dbed0 80000001d79dbdb8 [21084.850657] $20 : ffffffffc295bbd8 0000000000000000 80000001dff52008 80000001dff52008 [21084.858726] $24 : 0000000000000000 ffffffffc29af750 [21084.866792] $28 : 80000001f6e54000 80000001f6e57260 80000001d79dbed0 ffffffffc29af860 [21084.874861] Hi : 000000000000051a [21084.878439] Lo : aaaaaaaaaaaaad38 [21084.882072] epc : ffffffffc29b42e8 ssh_fastpath_get_transform_context+0x58/0x550 [quicksec] [21084.890606] Tainted: P C O [21084.894393] ra : ffffffffc29af860 ssh_fastpath_transform_in+0x110/0xe90 [quicksec] [21084.902231] Status: 14009ce3 KX SX UX KERNEL EXL IE [21084.907281] Cause : 40808000 [21084.910163] PrId : 000d9703 (Cavium Octeon III) [21084.914785] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21085.215400] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21085.226020] Stack : 0000000000000000 0000000000000009 80000001d79dbdb8 0000000000000288 80000001d79dbed0 80000001dff52008 ffffffffc295bbd8 ffffffffc29a8808 80000001dff52008 80000001bfa87a58 80000001bfa87948 ffffffffc29af860 80000001f6e573a0 ffffffff80bd2e74 ffffffffc2362290 0000000000000000 0000000000000000 ffffffff80eb20b0 ffffffff80000000 ffffffff858d4940 0000000000000004 0000000000000000 0000000000000000 0000000000000000 ffffffff858d4910 0000000000000000 80000001f3066480 ffffffff80c6d18c 0000000000000004 80000001f3066480 ffffffff80c19570 8000000197e64800 0000000000000000 ffffffff80eb20b0 0000000000000002 80000001f3066480 ffffffff80000000 ffffffff80bd3354 0000000000000034 00000000f3066480 ... [21085.291681] Call Trace: [21085.294176] [] ssh_fastpath_get_transform_context+0x58/0x550 [quicksec] [21085.302431] [] ssh_fastpath_transform_in+0x110/0xe90 [quicksec] [21085.309991] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [21085.317551] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21085.325105] [] FH_ssh_interceptor_encrypted_packet_in+0x1c4/0x290 [quicksec] [21085.333811] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [21085.340898] [] nf_iterate+0x13c/0x590 [21085.346131] [] nf_hook_slow+0x8c/0x1f8 [21085.351453] [] ip_rcv+0x2f8/0x410 [21085.356340] [] __netif_receive_skb_core+0x500/0x618 [21085.362790] [] process_backlog+0xa8/0x190 [21085.368372] [] net_rx_action+0x14c/0x228 [21085.373868] [] __do_softirq+0x1c8/0x210 [21085.379275] [] do_softirq+0x68/0x70 [21085.384333] [] local_bh_enable+0xa8/0xb0 [21085.389847] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [21085.397703] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [21085.405553] [] process_one_work+0x178/0x3e0 [21085.411308] [] worker_thread+0x140/0x3c8 [21085.416801] [] kthread+0xa4/0xb0 [21085.421600] [] ret_from_kernel_thread+0x14/0x1c [21085.427699] [21085.429190] Code: 00c0982d ffb10018 90ad008c <90ab008d> 93c300b1 90a500b0 000d6e38 93cf008f 000b5c38 [21085.439239] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21085.448317] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21085.454863] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 000000000000310a 00000000000002fe ffffffff859a07f8 ffffffff80ec0000 00000000000002fe 0000000000000002 0000000000000002 ffffffff859b20f8 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e570d0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007ec7b07 80000001f6e54000 80000001f6e56d60 00001326c42c0524 ffffffff808f739c 0000000000000004 0000000000000000 00000000000002fd 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21085.520503] Call Trace: [21085.522955] [] show_stack+0x6c/0xf8 [21085.528016] [] watchdog_timer_fn+0x24c/0x288 [21085.533859] [] __run_hrtimer.isra.20+0x60/0xf8 [21085.539874] [] hrtimer_interrupt+0x138/0x310 [21085.545715] [] c0_compare_interrupt+0x68/0x98 [21085.551641] [] handle_irq_event_percpu+0x88/0x218 [21085.557918] [] handle_percpu_irq+0x84/0xb8 [21085.563584] [] generic_handle_irq+0x38/0x50 [21085.569342] [] do_IRQ+0x18/0x30 [21085.574055] [] plat_irq_dispatch+0x98/0xb8 [21085.579720] [21110.200253] unpack_message_rr fail ret=-31 [21143.455289] Log Date: 1682632357, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21143.465928] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21143.471943] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 2 [21143.480217] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21143.486752] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21143.787368] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21143.796448] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21143.802994] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21143.810484] $ 0 : 0000000000000000 0000000014009ce1 80000001ef15cd40 ffffffffc2f91110 [21143.818553] $ 4 : ffffffffc260f2e0 ffffffff858d4cc8 80000001eb96a460 0000000000000001 [21143.826622] $ 8 : ffffffff80c13b00 ffffffffc24e0fe8 0000000000000000 0000000000000000 [21143.834687] $12 : 0000000000000000 6574683200000000 0000000000000000 0000000000000000 [21143.842759] $16 : 0000000000000000 0000000000000000 ffffffff80eb0000 000000000000000b [21143.850828] $20 : 80000001ebce4d80 80000001a2eb6800 0000000000000000 ffffffff80c13b00 [21143.858894] $24 : 0000000000000000 ffffffff80be0d60 [21143.866960] $28 : 80000001f6e54000 80000001f6e576d0 80000001f6e577a0 ffffffff80bd2eec [21143.875029] Hi : 0a3d70a3d4933cea [21143.878607] Lo : 5999999992d25828 [21143.882190] epc : ffffffff80805394 rtcompl_rtc_hookmap_lookup+0x64/0xa0 [21143.888986] Tainted: P C O [21143.892741] ra : ffffffff80bd2eec nf_iterate+0x1b4/0x590 [21143.898319] Status: 14009ce3 KX SX UX KERNEL EXL IE [21143.903370] Cause : 40808000 [21143.906254] PrId : 000d9703 (Cavium Octeon III) [21143.910875] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21144.211481] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21144.222101] Stack : ffffffffc260f2e0 0000000000000000 80000001eb96a460 ffffffff80eb2070 ffffffff80000000 ffffffff858d4930 0000000000000000 0000000000000001 ffffffffffffff9e 0000000000000a02 ffffffff858d4900 0000000000000002 0000000040000001 ffffffff858d4930 0000000000000000 80000001ebce4d80 ffffffff80c13b00 0000000000000000 80000001a2eb6800 ffffffff80eb2070 0000000000000002 ffffffff80eb7e10 ffffffff80000000 ffffffff80bd3354 ffffffff80c13b00 0000000000000000 ffffffffc260f2e0 ffffffff80eb2070 80000001ebce4d80 ffffffff80eb0428 ffffffff80eeab00 80000001a2eb6800 80000001eb7c50c4 0000000000000000 ffffffff80eb0428 ffffffff80eb7e10 0000000000000001 ffffffff80c14638 ffffffff80eb0e28 80000001a2eb6800 ... [21144.287744] Call Trace: [21144.290198] [] rtcompl_rtc_hookmap_lookup+0x64/0xa0 [21144.296653] [] nf_iterate+0x1b4/0x590 [21144.301890] [] nf_hook_slow+0x8c/0x1f8 [21144.307212] [] ip_rcv+0x2f8/0x410 [21144.312099] [] __netif_receive_skb_core+0x500/0x618 [21144.318550] [] process_backlog+0xa8/0x190 [21144.324130] [] net_rx_action+0x14c/0x228 [21144.329626] [] __do_softirq+0x1c8/0x210 [21144.335032] [] do_softirq+0x68/0x70 [21144.340090] [] local_bh_enable_ip+0xa8/0xb0 [21144.345867] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21144.353459] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21144.361307] [] process_one_work+0x178/0x3e0 [21144.367061] [] worker_thread+0x140/0x3c8 [21144.372555] [] kthread+0xa4/0xb0 [21144.377355] [] ret_from_kernel_thread+0x14/0x1c [21144.383453] [21144.384943] Code: 1045000d 00000000 dc430010 <14640007> 00000000 082014f2 00000000 00000000 dc430010 [21144.394985] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21144.404055] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21144.410599] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 00000000000035f6 0000000000000344 ffffffff859a07f8 ffffffff80ec0000 0000000000000344 0000000000000002 0000000000000002 ffffffff859b5298 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57540 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007c0bfc7 80000001f6e54000 80000001f6e571d0 0000133480d89352 ffffffff808f739c 0000000000000004 0000000000000000 0000000000000343 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21144.476241] Call Trace: [21144.478691] [] show_stack+0x6c/0xf8 [21144.483752] [] watchdog_timer_fn+0x24c/0x288 [21144.489596] [] __run_hrtimer.isra.20+0x60/0xf8 [21144.495611] [] hrtimer_interrupt+0x138/0x310 [21144.501455] [] c0_compare_interrupt+0x68/0x98 [21144.507386] [] handle_irq_event_percpu+0x88/0x218 [21144.513665] [] handle_percpu_irq+0x84/0xb8 [21144.519336] [] generic_handle_irq+0x38/0x50 [21144.525094] [] do_IRQ+0x18/0x30 [21144.529808] [] plat_irq_dispatch+0x98/0xb8 [21144.535474] [21179.955290] Log Date: 1682632394, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21179.965744] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21179.971758] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [21179.980033] BUG: soft lockup - CPU#7 stuck for 31s! [ksoftirqd/7:51] [21179.986393] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21180.286976] CPU: 7 PID: 51 Comm: ksoftirqd/7 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21180.295858] task: 80000001fc1bdb20 ti: 80000001fc1f8000 task.ti: 80000001fc1f8000 [21180.303348] $ 0 : 0000000000000000 0000000000000000 0000000020005000 0000000000000800 [21180.311417] $ 4 : 0000000000000000 80000001ebd54000 0000000000008900 00000000000080a8 [21180.319485] $ 8 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [21180.327553] $12 : 0000000000000000 0000000000000000 000000000000003f 0000000000000000 [21180.335622] $16 : 80000001ea705b00 0000000000000000 80000001ebd54000 80000001aa3c4c00 [21180.343691] $20 : ffffffff80d92990 ffffffff80eb0000 0000000000000000 ffffffffc2980000 [21180.351760] $24 : 0000000000000000 ffffffffc2f03450 [21180.359827] $28 : 80000001fc1f8000 80000001fc1fba10 0000000000000007 ffffffff80b9f5b4 [21180.367895] Hi : 0000000000000004 [21180.371473] Lo : 666666666666666d [21180.375056] epc : ffffffff80b9ec64 netif_skb_dev_features+0xbc/0xc8 [21180.381504] Tainted: P C O [21180.385257] ra : ffffffff80b9f5b4 dev_hard_start_xmit+0x84/0x518 [21180.391530] Status: 14009ce3 KX SX UX KERNEL EXL IE [21180.396577] Cause : c0808000 [21180.399459] PrId : 000d9703 (Cavium Octeon III) [21180.404079] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21180.704616] Process ksoftirqd/7 (pid: 51, threadinfo=80000001fc1f8000, task=80000001fc1bdb20, tls=0000000000000000) [21180.715062] Stack : ffffffff80eb0000 0000000000000000 80000001aa3c4c00 800000019d5f5000 800000019d5f509c 80000001ebd54000 0000000000000010 80000001ea705b00 0000000000000000 ffffffffc2980000 0000000000000007 ffffffff80bbb7a0 ffffffff80f00000 800000019d5f5000 0000000000000040 ffffffff80f00000 0000000000000001 ffffffff80e35030 0000000000000000 ffffffff80bbb9b8 80000001ebd54000 800000018bce54a8 80000001ea705b00 800000019d5f5000 80000001ebd54000 80000001aa3c4c00 0000000000000000 ffffffff80b9fcf8 80000001ea705b00 800000018bce5400 800000018bce5400 80000001ebd54000 80000001ea705b00 80000001e4030b00 800000018bce54d8 0000000000000001 ffffffff80c19570 ffffffff80c19800 ac150063ebd54000 80000001f5c6e000 ... [21180.780694] Call Trace: [21180.783145] [] netif_skb_dev_features+0xbc/0xc8 [21180.789247] [] dev_hard_start_xmit+0x84/0x518 [21180.795174] [] sch_direct_xmit+0x110/0x240 [21180.800839] [] __qdisc_run+0xe8/0x188 [21180.806070] [] dev_queue_xmit+0x2b0/0x4d8 [21180.811653] [] ip_finish_output+0x290/0x490 [21180.817456] [] ssh_interceptor_send+0x6ec/0xed8 [quicksec] [21180.824576] [] ssh_engine_process_asynch_packets+0x158/0x2e8 [quicksec] [21180.832824] [] ssh_kernel_timeout_cb+0xdc/0x140 [quicksec] [21180.839912] [] call_timer_fn.isra.26+0x24/0x88 [21180.845926] [] run_timer_softirq+0x1f4/0x270 [21180.851768] [] __do_softirq+0x1c8/0x210 [21180.857174] [] run_ksoftirqd+0x40/0x60 [21180.862494] [] smpboot_thread_fn+0x1d0/0x1d8 [21180.868334] [] kthread+0xa4/0xb0 [21180.873134] [] ret_from_kernel_thread+0x14/0x1c [21180.879233] [21180.880724] Code: 7c840f03 1480fff7 00000000 <03e00008> 00000000 00000000 67bdffc0 00a0382d 3c0580e0 [21180.890765] CPU: 7 PID: 51 Comm: ksoftirqd/7 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21180.899645] Stack : ffffffff80ec0000 0000000014001ce0 0000000004208040 0000000000000007 0000000000000000 0000000000000000 ffffffff859e0000 0000000000000387 0000000000000004 0000000000000000 0000000000000387 332e31302e38372d 0000000000000005 0000000000000009 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001fc1fb880 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000005 0000000000000001 0000000000000000 3c0000000790bf47 80000001fc1f8000 80000001fc1fb510 0000133d006a5fe4 ffffffff808f739c 80000001fc1bdb20 ffffffff80dfe628 0000000000000007 0000000000000033 0000000000000000 ffffffff8087d21c 0000000000000000 0000000000000000 ... [21180.965275] Call Trace: [21180.967725] [] show_stack+0x6c/0xf8 [21180.972787] [] watchdog_timer_fn+0x24c/0x288 [21180.978629] [] __run_hrtimer.isra.20+0x60/0xf8 [21180.984643] [] hrtimer_interrupt+0x138/0x310 [21180.990484] [] c0_compare_interrupt+0x68/0x98 [21180.996410] [] handle_irq_event_percpu+0x88/0x218 [21181.002687] [] handle_percpu_irq+0x84/0xb8 [21181.008354] [] generic_handle_irq+0x38/0x50 [21181.014108] [] do_IRQ+0x18/0x30 [21181.018821] [] plat_irq_dispatch+0x98/0xb8 [21181.024486] [21213.955291] Log Date: 1682632428, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21213.965745] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21213.971759] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 2 [21213.980035] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21213.986568] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21214.287196] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21214.296277] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21214.302823] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21214.310314] $ 0 : 0000000000000000 0000000054009ce1 80000001ebeca600 ffffffffc0020000 [21214.318383] $ 4 : ffffffffe8be9a16 0000000000000006 800000000c328c30 0000000000000050 [21214.326451] $ 8 : 0000000000000001 0000000000000001 0000000000000590 8000000184fff200 [21214.334520] $12 : 0000000000000001 0000000000000048 0000000000000001 6db6db6db6db6db7 [21214.342589] $16 : 80000001f6e57120 0000000000000050 80000001ebeca5fc 0000000000000000 [21214.350657] $20 : 0000000000000000 0000000000000000 0000000000000001 0000000000000001 [21214.358725] $24 : 0000000000000048 8000000000000000 [21214.366793] $28 : 80000001f6e54000 80000001f6e56e90 ffffffff859e9900 ffffffffc233de98 [21214.374861] Hi : 0000000000000002 [21214.378438] Lo : 00000000000000af [21214.382069] epc : ffffffffc233e1a8 octo_des_cbc_sha1_encrypt+0x458/0x1018 [cryptocteon] [21214.390255] Tainted: P C O [21214.394032] ra : ffffffffc233de98 octo_des_cbc_sha1_encrypt+0x148/0x1018 [cryptocteon] [21214.402218] Status: 54009ce3 KX SX UX KERNEL EXL IE [21214.407266] Cause : 40808000 [21214.410150] PrId : 000d9703 (Cavium Octeon III) [21214.414770] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21214.715363] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21214.725983] Stack : 0000000000000000 ffffffff80c19808 ac14102e80c19570 80000001a2eb6800 0000000000000002 0000000000000004 ffffffff80c19570 80000001a2eb6800 0000000000000000 ffffffffc2364630 00000000644aeeeb 000000003b0bd261 ffffffffc2380000 80000001f6394000 80000001a2eb6800 80000001ebdbf0e8 0000000000000000 0000000000000000 0000000000000000 0000000000000009 0000000000000008 0000000000000004 0000000000000004 0000000000000000 ffffffff80eb0000 0000000000000000 80000001fb04e300 0000000000000000 80000001a2eb6800 ffffffff80c19570 80000001f6e57060 ffffffff80bd2e74 ffffffffc2362290 0000000000000000 0000000000000000 ffffffff80eb20b0 ffffffff80000000 ffffffff858d4940 0000000000000004 0000000000000000 ... [21214.791662] Call Trace: [21214.794134] [] octo_des_cbc_sha1_encrypt+0x458/0x1018 [cryptocteon] [21214.802052] [] octo_process.part.4+0x48c/0xa38 [cryptocteon] [21214.809322] [] crypto_invoke+0x60/0x168 [21214.814729] [] crypto_dispatch+0x214/0x360 [21214.820456] [] ssh_hwaccel_perform_ipsec+0x31c/0x5b8 [quicksec] [21214.828036] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [21214.835876] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [21214.843430] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21214.850986] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21214.858952] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21214.866019] [] __tasklet_action+0xac/0x1b8 [21214.871686] [] __do_softirq+0x1c8/0x210 [21214.877092] [] do_softirq+0x68/0x70 [21214.882150] [] local_bh_enable_ip+0xa8/0xb0 [21214.887919] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21214.895512] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21214.903360] [] process_one_work+0x178/0x3e0 [21214.909114] [] worker_thread+0x140/0x3c8 [21214.914608] [] kthread+0xa4/0xb0 [21214.919408] [] ret_from_kernel_thread+0x14/0x1c [21214.925507] [21214.926998] Code: 0046102d cc5c0000 8c440000 <1a60005d> afa401cc 2673fff8 1a800050 00000000 2694fff8 [21214.937044] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21214.946116] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21214.952661] Stack : 0000000000000001 0000000054001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000003fb2 00000000000003d0 ffffffff859a07f8 ffffffff80ec0000 00000000000003d0 0000000000000002 0000000000000002 ffffffff859bb6c0 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e56d00 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 0000000000000001 80000001f6e54000 80000001f6e56990 00001344eaf934a1 ffffffff808f739c 0000000000000004 0000000000000000 00000000000003cf 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21215.018295] Call Trace: [21215.020746] [] show_stack+0x6c/0xf8 [21215.025808] [] watchdog_timer_fn+0x24c/0x288 [21215.031653] [] __run_hrtimer.isra.20+0x60/0xf8 [21215.037671] [] hrtimer_interrupt+0x138/0x310 [21215.043516] [] c0_compare_interrupt+0x68/0x98 [21215.049447] [] handle_irq_event_percpu+0x88/0x218 [21215.055725] [] handle_percpu_irq+0x84/0xb8 [21215.061395] [] generic_handle_irq+0x38/0x50 [21215.067152] [] do_IRQ+0x18/0x30 [21215.071866] [] plat_irq_dispatch+0x98/0xb8 [21215.077531] [21215.091124] [zy_adp]tcp syn flush flow!! [21248.455290] Log Date: 1682632462, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21248.465743] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21248.471757] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 3 [21248.480031] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21248.486565] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21248.787173] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21248.796247] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21248.802792] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21248.810281] $ 0 : 0000000000000000 0000000000000004 80000001ef9ca500 0000000000000024 [21248.818350] $ 4 : ffffffffc2386ad0 0000000000000017 80000001e944c0a5 0000000000000039 [21248.826419] $ 8 : 0000000000000039 0000000000000000 0000000000000024 6574683200000000 [21248.834490] $12 : 0000000000000000 80000001f6e57246 ffffffff858d4908 0000000000000a40 [21248.842557] $16 : 80000001a2eb6800 ffffffff80a48900 00000000000004ec 80000001e944c09f [21248.850626] $20 : 0000000000000000 0000000057faf7b7 80000001fb79a000 ffffffffc2380000 [21248.858694] $24 : 0000000000000002 0000000000000025 [21248.866762] $28 : 80000001f6e54000 80000001f6e57210 80000001f0ff7e00 ffffffffc238c484 [21248.874830] Hi : 0000000000000011 [21248.878408] Lo : 000000000032c5f0 [21248.882013] epc : ffffffffc2388078 rpFindif+0x78/0x90 [nf_report] [21248.888288] Tainted: P C O [21248.892060] ra : ffffffffc238c484 rpIpv4Hit+0x8c/0x4d0 [nf_report] [21248.898507] Status: 14009ce3 KX SX UX KERNEL EXL IE [21248.903557] Cause : 40808000 [21248.906440] PrId : 000d9703 (Cavium Octeon III) [21248.911060] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21249.211624] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21249.222244] Stack : 0000000000000000 ffffffff80c19808 ac14100680c19570 80000001a2eb6800 0000000000000002 0000000000000000 0000000000009570 00000000a2eb6800 0000000000000000 0000000000000000 80000001fb79a000 80000001a2eb6800 80000001819e30e8 80000001e944c000 ffffffffc2380000 0000000000000003 ffffffffc238c3f8 ffffffffc239047c 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... [21249.287890] Call Trace: [21249.290357] [] rpFindif+0x78/0x90 [nf_report] [21249.296340] [] rpIpv4Hit+0x8c/0x4d0 [nf_report] [21249.302496] [] FH_traffic_report_ipv4_hook+0x214/0xeb8 [nf_report] [21249.310303] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [21249.317382] [] nf_iterate+0x13c/0x590 [21249.322614] [] nf_hook_slow+0x8c/0x1f8 [21249.327936] [] ip_forward+0x45c/0x488 [21249.333203] [] rtcompl_hook+0x338/0x4a0 [fastpath_kmodule] [21249.340280] [] nf_iterate+0x13c/0x590 [21249.345511] [] nf_hook_slow+0x8c/0x1f8 [21249.350831] [] ip_rcv+0x2f8/0x410 [21249.355719] [] __netif_receive_skb_core+0x500/0x618 [21249.362169] [] process_backlog+0xa8/0x190 [21249.367749] [] net_rx_action+0x14c/0x228 [21249.373243] [] __do_softirq+0x1c8/0x210 [21249.378649] [] do_softirq+0x68/0x70 [21249.383706] [] local_bh_enable_ip+0xa8/0xb0 [21249.389472] [] fqdnObject_packetQ_pop+0x64/0x120 [fqdn_object] [21249.396892] [] fqdnObject_work_queue_func+0x40/0x390 [fqdn_object] [21249.404652] [] process_one_work+0x178/0x3e0 [21249.410406] [] worker_thread+0x140/0x3c8 [21249.415900] [] kthread+0xa4/0xb0 [21249.420700] [] ret_from_kernel_thread+0x14/0x1c [21249.426799] [21249.428291] Code: 10650006 00000000 dc420000 <1482fffb> 00000000 03e00008 0000102d 03e00008 00000000 [21249.438331] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21249.447395] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21249.453938] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 00000000000044a2 000000000000041d ffffffff859a07f8 ffffffff80ec0000 000000000000041d 0000000000000002 0000000000000002 ffffffff859bea58 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57080 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e56d10 0000134cf3556cb6 ffffffff808f739c 0000000000000004 0000000000000000 000000000000041c 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21249.519573] Call Trace: [21249.522023] [] show_stack+0x6c/0xf8 [21249.527084] [] watchdog_timer_fn+0x24c/0x288 [21249.532926] [] __run_hrtimer.isra.20+0x60/0xf8 [21249.538941] [] hrtimer_interrupt+0x138/0x310 [21249.544781] [] c0_compare_interrupt+0x68/0x98 [21249.550709] [] handle_irq_event_percpu+0x88/0x218 [21249.556985] [] handle_percpu_irq+0x84/0xb8 [21249.562651] [] generic_handle_irq+0x38/0x50 [21249.568406] [] do_IRQ+0x18/0x30 [21249.573120] [] plat_irq_dispatch+0x98/0xb8 [21249.578785] [21306.455290] Log Date: 1682632520, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21306.465747] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21306.471762] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [21306.480039] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21306.486575] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21306.787187] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21306.796262] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21306.802807] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21306.810298] $ 0 : 0000000000000000 000000000000001c 0000000000010000 0000000000000001 [21306.818368] $ 4 : 000000000000546e 0000000000000000 0000000000000001 ffffffffc29b0000 [21306.826439] $ 8 : 0000000000000000 0000000000000001 0000000000000000 0000000000000000 [21306.834505] $12 : ffffffffc2970000 80000001f6e5763c 000000000002dae2 0000000000000000 [21306.842573] $16 : 80000001e42caa48 800000019d7ea008 80000001dff52008 80000001dff52008 [21306.850640] $20 : 000000000000000b ffffffffc29dda80 80000001d79dbdb8 80000001c52e1518 [21306.858708] $24 : 0000000000000000 ffffffffc29a8808 [21306.866775] $28 : 80000001f6e54000 80000001f6e57680 80000001c52e1408 ffffffff80d34fcc [21306.874843] Hi : 0000000000000007 [21306.878420] Lo : 0000000000001fff [21306.882004] epc : ffffffff80d352b8 _raw_write_lock_bh+0xd8/0xe0 [21306.888104] Tainted: P C O [21306.891858] ra : ffffffff80d34fcc _raw_spin_lock_bh+0x14/0x48 [21306.897870] Status: 14009ce3 KX SX UX KERNEL EXL IE [21306.902920] Cause : 40808000 [21306.905803] PrId : 000d9703 (Cavium Octeon III) [21306.910423] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21307.210977] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21307.221597] Stack : 80000001e42caa48 ffffffffc295babc 80000001d79dbdb8 ffffffffc29a8d1c 1f8a0000f5c6c008 0000000000000003 800000019d7ea008 80000001f5c6c008 0000000000000003 ffffffffc29d4e9c 0000000000000000 ffffffff80c19808 ac150063ebd54000 80000001f5c6e000 80000001eafa8d80 800000019d7ea008 80000001dff52028 0000000000000000 800000019d7ea008 0000000000000000 0000000000000001 ffffffffc2950000 80000001f5c6c008 000000000000000f 80000001dff52008 80000001f57cc6e0 ffffffffc295ba40 ffffffffc295ba80 ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 0000000000200000 000000000000000f 000000000000000f 0000000000000001 0000000000000000 80000001dff521d0 0000000000000000 ffffffff00000001 ... [21307.287243] Call Trace: [21307.289693] [] _raw_write_lock_bh+0xd8/0xe0 [21307.295493] [] ssh_kernel_mutex_lock_bh_i+0x1c/0x40 [quicksec] [21307.302960] [] fastpath_execute_transform_step+0x514/0x5a0 [quicksec] [21307.311038] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [21307.318594] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21307.326147] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21307.334110] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21307.341175] [] __tasklet_action+0xac/0x1b8 [21307.346842] [] __do_softirq+0x1c8/0x210 [21307.352249] [] do_softirq+0x68/0x70 [21307.357307] [] local_bh_enable_ip+0xa8/0xb0 [21307.363071] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21307.370663] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21307.378510] [] process_one_work+0x178/0x3e0 [21307.384264] [] worker_thread+0x140/0x3c8 [21307.389758] [] kthread+0xa4/0xb0 [21307.394558] [] ret_from_kernel_thread+0x14/0x1c [21307.400656] [21307.402146] Code: 96030002 1064ff4f 00831823 <1000fff8> 2463ffff 67bdffe0 24025401 ffb00008 ffbf0018 [21307.412187] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21307.421252] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21307.427796] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000004990 0000000000000463 ffffffff859a07f8 ffffffff80ec0000 0000000000000463 0000000000000002 0000000000000002 ffffffff859c1c90 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e574f0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bec947 80000001f6e54000 80000001f6e57180 0000135a74672c8d ffffffff808f739c 0000000000000004 0000000000000000 0000000000000462 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21307.493455] Call Trace: [21307.495905] [] show_stack+0x6c/0xf8 [21307.500966] [] watchdog_timer_fn+0x24c/0x288 [21307.506809] [] __run_hrtimer.isra.20+0x60/0xf8 [21307.512824] [] hrtimer_interrupt+0x138/0x310 [21307.518665] [] c0_compare_interrupt+0x68/0x98 [21307.524592] [] handle_irq_event_percpu+0x88/0x218 [21307.530868] [] handle_percpu_irq+0x84/0xb8 [21307.536536] [] generic_handle_irq+0x38/0x50 [21307.542290] [] do_IRQ+0x18/0x30 [21307.547003] [] plat_irq_dispatch+0x98/0xb8 [21307.552668] [21340.955291] Log Date: 1682632555, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21340.965745] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21340.971759] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 2 [21340.980033] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21340.986569] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21341.287193] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21341.296274] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21341.302820] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21341.310310] $ 0 : 0000000000000000 0000000014009ce1 0000000000000168 ffffffff85a14850 [21341.318379] $ 4 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [21341.326445] $ 8 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [21341.334512] $12 : 80000001eaf75000 8000000189f57500 0000000000000000 0000000000000000 [21341.342578] $16 : 80000001f370d980 8000000197ee9780 8000000189d95000 ffffffff80eb0428 [21341.350646] $20 : 80000000243ea5d4 c000000000924a18 ffffffff858c67e0 c0000000009249a8 [21341.358716] $24 : 0000000000000000 ffffffff80bdf0a0 [21341.366786] $28 : 80000001f6e54000 80000001f6e575a0 80000001eaf750e8 ffffffff80c6d0b0 [21341.374854] Hi : 0a3d70a3d1f4f0bb [21341.378432] Lo : f33333332538a042 [21341.382015] epc : ffffffff80c6d080 ipt_do_table+0x2f8/0x4e0 [21341.387768] Tainted: P C O [21341.391521] ra : ffffffff80c6d0b0 ipt_do_table+0x328/0x4e0 [21341.397273] Status: 14009ce3 KX SX UX KERNEL EXL IE [21341.402323] Cause : 40808000 [21341.405206] PrId : 000d9703 (Cavium Octeon III) [21341.409827] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21341.710441] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21341.721062] Stack : ffffffff80eb5b80 c000000000924888 8000000189d95000 0000000000000000 0000000000000014 0000000002000000 0000000000000001 ffffffff85a14850 c000000000918000 0000000000000000 0000000000000000 80000001877a9c60 8000000197ee97a0 0000000000000000 8000000189d95000 0000000000000000 80000001f370d980 80000001ea107800 80000001ea107800 0000000000000000 8000000189d95000 0000000000000000 0000000000000000 ffffffff80c13b00 80000001f6e577b0 ffffffffc24e16b8 ffffffff80eb0000 0000000000000001 000000000000002c ffffffffc28492dc 0000000000000000 0000000000000000 ffffffff80eb0000 0000000000000009 80000001f370d980 8000000189d95000 0000000000000000 ffffffff80c13b00 80000001f6e577b0 ffffffff80bd2e74 ... [21341.786702] Call Trace: [21341.789154] [] ipt_do_table+0x2f8/0x4e0 [21341.794611] [] ipt_zymark_hook+0x178/0x4d0 [iptable_zymark] [21341.801774] [] nf_iterate+0x13c/0x590 [21341.807007] [] nf_hook_slow+0x8c/0x1f8 [21341.812327] [] ip_rcv+0x2f8/0x410 [21341.817214] [] __netif_receive_skb_core+0x500/0x618 [21341.823665] [] process_backlog+0xa8/0x190 [21341.829246] [] net_rx_action+0x14c/0x228 [21341.834741] [] __do_softirq+0x1c8/0x210 [21341.840146] [] do_softirq+0x68/0x70 [21341.845205] [] local_bh_enable_ip+0xa8/0xb0 [21341.850976] [] fqdnObject_syncQ_exist_check+0xb8/0x2d0 [fqdn_object] [21341.858917] [] fqdnObject_work_queue_func+0x1d0/0x390 [fqdn_object] [21341.866767] [] process_one_work+0x178/0x3e0 [21341.872520] [] worker_thread+0x140/0x3c8 [21341.878014] [] kthread+0xa4/0xb0 [21341.882814] [] ret_from_kernel_thread+0x14/0x1c [21341.888913] [21341.890403] Code: 1522ffca 66f50070 96e20058 <02e2102d> 02a2202b 10800012 00000000 dea20008 66a40020 [21341.900453] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21341.909523] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21341.916068] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000004e70 00000000000004a9 ffffffff859a07f8 ffffffff80ec0000 00000000000004a9 0000000000000002 0000000000000002 ffffffff859c4e50 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57410 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 0000000000000001 80000001f6e54000 80000001f6e570a0 000013627cc36ab3 ffffffff808f739c 0000000000000004 0000000000000000 00000000000004a8 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21341.981712] Call Trace: [21341.984164] [] show_stack+0x6c/0xf8 [21341.989225] [] watchdog_timer_fn+0x24c/0x288 [21341.995070] [] __run_hrtimer.isra.20+0x60/0xf8 [21342.001084] [] hrtimer_interrupt+0x138/0x310 [21342.006926] [] c0_compare_interrupt+0x68/0x98 [21342.012853] [] handle_irq_event_percpu+0x88/0x218 [21342.019130] [] handle_percpu_irq+0x84/0xb8 [21342.024798] [] generic_handle_irq+0x38/0x50 [21342.030555] [] do_IRQ+0x18/0x30 [21342.035268] [] plat_irq_dispatch+0x98/0xb8 [21342.040932] [21375.455290] Log Date: 1682632589, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21375.465744] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21375.471757] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 3 [21375.480032] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21375.486565] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21375.787185] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21375.796264] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21375.802810] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21375.810299] $ 0 : 0000000000000000 0000000000000000 80000001b6da6840 0000000000000000 [21375.818365] $ 4 : 0000000000000000 ffffffffffffffff 80000001a944e180 80000001f6e575a0 [21375.826433] $ 8 : 0000000000000000 00000000582a0000 0000000000000001 0000000000000001 [21375.834498] $12 : 80000001f6e575d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [21375.842567] $16 : 80000001b6da6840 0000000000000000 0000000000000003 80000001f6e575a0 [21375.850634] $20 : fffffffffffffff5 80000001f6e57680 80000001fba60a40 0000000000000002 [21375.858700] $24 : 0000000000000000 ffffffffc295bc30 [21375.866766] $28 : 80000001f6e54000 80000001f6e57540 80000001fba609c0 ffffffff80bb659c [21375.874834] Hi : 000000000000051a [21375.878412] Lo : aaaaaaaaaaaaad38 [21375.881996] epc : ffffffff80bb64dc fib_rules_lookup+0x64/0x1d8 [21375.888008] Tainted: P C O [21375.891763] ra : ffffffff80bb659c fib_rules_lookup+0x124/0x1d8 [21375.897862] Status: 14009ce3 KX SX UX KERNEL EXL IE [21375.902910] Cause : 40808000 [21375.905794] PrId : 000d9703 (Cavium Octeon III) [21375.910415] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21376.211016] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21376.221637] Stack : 80000001b6da6840 f0320996c0a81064 80000001f6e57680 ffffffff80eeab00 ffffffffffffff80 80000001a2eb3000 0000000000000000 0000000000000009 0000000000200000 ffffffffc2960000 800000018bd3da08 ffffffff80c62020 0000000000000000 80000001f6e575d0 0000000000000000 0000000100000000 ffffffff80eeab00 ffffffff80c10bcc 8000000100000000 0000000000000000 0000000000000000 80000001f45bc480 0000000006000000 ffffffffc2960000 80000001f6e57680 0000000000000000 ffffffff80eeab00 80000001f5c6e000 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 800000018bd3da08 ffffffff80c10f2c 80000001f6e576c0 80000001f4110c00 800000018421c0a0 ffffffff80c6957c 000000000000781b ffffffffc295e390 ... [21376.287272] Call Trace: [21376.289726] [] fib_rules_lookup+0x64/0x1d8 [21376.295396] [] __fib_lookup+0x30/0x50 [21376.300631] [] __ip_route_output_key+0x52c/0x868 [21376.306819] [] ip_route_output_flow+0x24/0x78 [21376.312746] [] nf_ip_route+0x1c/0x48 [21376.317945] [] ssh_interceptor_reroute_skb_ipv4+0x108/0x368 [quicksec] [21376.326104] [] ssh_interceptor_send+0x134/0xed8 [quicksec] [21376.333223] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [21376.340779] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21376.348744] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21376.355813] [] __tasklet_action+0xac/0x1b8 [21376.361484] [] __do_softirq+0x1c8/0x210 [21376.366891] [] do_softirq+0x68/0x70 [21376.371949] [] local_bh_enable_ip+0xa8/0xb0 [21376.377721] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21376.385314] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21376.393161] [] process_one_work+0x178/0x3e0 [21376.398916] [] worker_thread+0x140/0x3c8 [21376.404410] [] kthread+0xa4/0xb0 [21376.409210] [] ret_from_kernel_thread+0x14/0x1c [21376.415309] [21376.416800] Code: 2414fff5 00000000 8e030014 <10600013> 00000000 8ea50004 10a30010 0000102d 8e030028 [21376.426849] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21376.435919] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21376.442463] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 000000000000535c 00000000000004f2 ffffffff859a07f8 ffffffff80ec0000 00000000000004f2 0000000000000002 0000000000000002 ffffffff859c8130 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e573b0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c0000000790ca47 80000001f6e54000 80000001f6e57040 0000136a851fa1f5 ffffffff808f739c 0000000000000004 0000000000000000 00000000000004f1 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21376.508107] Call Trace: [21376.510558] [] show_stack+0x6c/0xf8 [21376.515620] [] watchdog_timer_fn+0x24c/0x288 [21376.521463] [] __run_hrtimer.isra.20+0x60/0xf8 [21376.527478] [] hrtimer_interrupt+0x138/0x310 [21376.533320] [] c0_compare_interrupt+0x68/0x98 [21376.539247] [] handle_irq_event_percpu+0x88/0x218 [21376.545524] [] handle_percpu_irq+0x84/0xb8 [21376.551191] [] generic_handle_irq+0x38/0x50 [21376.556950] [] do_IRQ+0x18/0x30 [21376.561666] [] plat_irq_dispatch+0x98/0xb8 [21376.567331] [21409.455290] Log Date: 1682632623, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21409.465743] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21409.471756] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 4 [21409.480030] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21409.486564] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21409.787178] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21409.796259] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21409.802804] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21409.810293] $ 0 : 0000000000000000 0000000000000000 ffffffffc28640b0 0000000000000001 [21409.818360] $ 4 : 80000001f6d869c0 0000000000000001 0000000000000000 0000000000000000 [21409.826427] $ 8 : 0000000000000001 0000000000000001 0000000000000004 ffffffff80000000 [21409.834495] $12 : 80000001f6e575d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [21409.842560] $16 : 0000000000000004 0000000000000000 ffffffff80eb0000 80000001f6e576c0 [21409.850627] $20 : 80000001f4d58a80 0000000000000000 80000001a2eb3000 ffffffff80c19570 [21409.858694] $24 : 0000000000000000 ffffffffc295bc30 [21409.866761] $28 : 80000001f6e54000 80000001f6e575f0 0000000000000000 ffffffff80bd2e74 [21409.874828] Hi : 000000000000051a [21409.878406] Lo : aaaaaaaaaaaaad38 [21409.881989] epc : ffffffff80bd2f9c nf_iterate+0x264/0x590 [21409.887568] Tainted: P C O [21409.891322] ra : ffffffff80bd2e74 nf_iterate+0x13c/0x590 [21409.896898] Status: 14009ce3 KX SX UX KERNEL EXL IE [21409.901949] Cause : 40808000 [21409.904832] PrId : 000d9703 (Cavium Octeon III) [21409.909454] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21410.210054] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21410.220674] Stack : 80000001f6d869c0 ffffffffc2960000 0000000000000000 ffffffff80eb20b0 ffffffff80000000 ffffffff858d4940 0000000000000004 0000000000000000 0000000000000000 0000000000000000 ffffffff858d4910 0000000000000003 0000000000200000 ffffffffc2960000 0000000000000004 80000001f4d58a80 ffffffff80c19570 80000001a2eb3000 0000000000000000 ffffffff80eb20b0 0000000000000002 ffffffffc2960000 ffffffff80000000 ffffffff80bd3354 0000000000000000 0000000000000000 80000001f6d869c0 ffffffffc29d4b8c 80000001f4d58a80 80000001a2df2808 80000001a2eb3000 80000001f5c6e000 ffffffffc295bbd8 ffffffffc295ba80 ffffffff80c19570 ffffffffc2960000 80000001a2df2808 ffffffffc29657a4 0000000000000001 ffffffffc2950000 ... [21410.286313] Call Trace: [21410.288766] [] nf_iterate+0x264/0x590 [21410.293998] [] nf_hook_slow+0x8c/0x1f8 [21410.299373] [] ssh_interceptor_send+0xb34/0xed8 [quicksec] [21410.306511] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [21410.314081] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21410.322043] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21410.329109] [] __tasklet_action+0xac/0x1b8 [21410.334776] [] __do_softirq+0x1c8/0x210 [21410.340183] [] do_softirq+0x68/0x70 [21410.345241] [] local_bh_enable+0xa8/0xb0 [21410.350751] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [21410.358605] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [21410.366455] [] process_one_work+0x178/0x3e0 [21410.372210] [] worker_thread+0x140/0x3c8 [21410.377703] [] kthread+0xa4/0xb0 [21410.382503] [] ret_from_kernel_thread+0x14/0x1c [21410.388602] [21410.390094] Code: dc440000 0040982d dc820000 ffa20000 dfa40000 fe640000 1483ff90 0080282d [21410.400144] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21410.409214] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21410.415758] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000005844 0000000000000537 ffffffff859a07f8 ffffffff80ec0000 0000000000000537 0000000000000002 0000000000000002 ffffffff859cb2d0 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57460 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e570f0 000013726fae7666 ffffffff808f739c 0000000000000004 0000000000000000 0000000000000536 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21410.481393] Call Trace: [21410.483844] [] show_stack+0x6c/0xf8 [21410.488906] [] watchdog_timer_fn+0x24c/0x288 [21410.494749] [] __run_hrtimer.isra.20+0x60/0xf8 [21410.500763] [] hrtimer_interrupt+0x138/0x310 [21410.506608] [] c0_compare_interrupt+0x68/0x98 [21410.512539] [] handle_irq_event_percpu+0x88/0x218 [21410.518816] [] handle_percpu_irq+0x84/0xb8 [21410.524485] [] generic_handle_irq+0x38/0x50 [21410.530242] [] do_IRQ+0x18/0x30 [21410.534954] [] plat_irq_dispatch+0x98/0xb8 [21410.540620] [21443.955289] Log Date: 1682632658, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21443.965744] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21443.971757] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 5 [21443.980032] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21443.986566] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21444.287181] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21444.296255] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21444.302801] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21444.310292] $ 0 : 0000000000000000 000000000000001c 0000000000000001 0000000000000001 [21444.318362] $ 4 : 0000000000000000 00000000000d52b4 0000000000000000 0000000000000001 [21444.326430] $ 8 : 000000000000004f 0000000000697579 ffffffff90b93000 fffffffffca37c00 [21444.334499] $12 : ffffffffc2970000 80000001f6e5763c 0000000000063c9c 0000000000000000 [21444.342566] $16 : 800000018bd3f208 80000001f5c6c008 0000000000000003 80000001dff52008 [21444.350637] $20 : 000000000000000b ffffffffc29dda80 80000001d79dbdb8 0000000000000404 [21444.358704] $24 : 0000000000000000 ffffffff80d35000 [21444.366772] $28 : 80000001f6e54000 80000001f6e576d0 80000001c1c19908 ffffffffc29d4984 [21444.374840] Hi : 00000000000dd87d [21444.378418] Lo : 0000000000000c8e [21444.382044] epc : ffffffffc29d4a08 fastpath_packet_continue+0x1e0/0x1038 [quicksec] [21444.389883] Tainted: P C O [21444.393666] ra : ffffffffc29d4984 fastpath_packet_continue+0x15c/0x1038 [quicksec] [21444.401504] Status: 14009ce3 KX SX UX KERNEL EXL IE [21444.406554] Cause : 40808000 [21444.409437] PrId : 000d9703 (Cavium Octeon III) [21444.414057] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21444.714624] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21444.725244] Stack : 0000000000000000 ffffffff80c19808 ac150063ebd54000 80000001f5c6e000 80000001ea4f8480 800000018bd3f208 80000001ebd54000 0000000000000000 800000018bd3f208 0000000000000000 0000000000000001 ffffffffc2950000 80000001f5c6c008 000000000000000d 80000001dff52008 80000001f57cc6e0 ffffffffc295ba40 ffffffffc295ba80 ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 0000000000200000 000000000000000d 000000000000000d 0000000000000001 0000000000000000 80000001dff521c0 0000000000000000 ffffffff00000001 0000000000000001 80000001d79dbdb8 ffffffffffffffff 0000000000000000 80000001f5c6e058 0000000000000002 80000001f5c6e000 8000000197e64800 ffffffffc295bbd8 80000001eaceb400 ... [21444.790893] Call Trace: [21444.793371] [] fastpath_packet_continue+0x1e0/0x1038 [quicksec] [21444.800926] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21444.808478] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21444.816441] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21444.823507] [] __tasklet_action+0xac/0x1b8 [21444.829175] [] __do_softirq+0x1c8/0x210 [21444.834581] [] do_softirq+0x68/0x70 [21444.839639] [] local_bh_enable+0xa8/0xb0 [21444.845141] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [21444.852994] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [21444.860841] [] process_one_work+0x178/0x3e0 [21444.866594] [] worker_thread+0x140/0x3c8 [21444.872087] [] kthread+0xa4/0xb0 [21444.876887] [] ret_from_kernel_thread+0x14/0x1c [21444.882985] [21444.884477] Code: 148002c1 67c40110 8ec40030 <1080008b> 00000000 8ec40000 e8810088 00000000 10e0020a [21444.894521] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21444.903586] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21444.910129] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000005d1c 000000000000057a ffffffff859a07f8 ffffffff80ec0000 000000000000057a 0000000000000002 0000000000000002 ffffffff859ce438 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57540 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e571d0 0000137a780aaef6 ffffffff808f739c 0000000000000004 0000000000000000 0000000000000579 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21444.975774] Call Trace: [21444.978225] [] show_stack+0x6c/0xf8 [21444.983285] [] watchdog_timer_fn+0x24c/0x288 [21444.989128] [] __run_hrtimer.isra.20+0x60/0xf8 [21444.995143] [] hrtimer_interrupt+0x138/0x310 [21445.000984] [] c0_compare_interrupt+0x68/0x98 [21445.006911] [] handle_irq_event_percpu+0x88/0x218 [21445.013188] [] handle_percpu_irq+0x84/0xb8 [21445.018855] [] generic_handle_irq+0x38/0x50 [21445.024609] [] do_IRQ+0x18/0x30 [21445.029323] [] plat_irq_dispatch+0x98/0xb8 [21445.034987] [21478.455289] Log Date: 1682632692, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21478.465742] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21478.471755] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 6 [21478.480029] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21478.486563] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21478.787171] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21478.796248] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21478.802794] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21478.810285] $ 0 : 0000000000000000 0000000000000008 0000000000000001 0000000000000017 [21478.818352] $ 4 : ffffffffc253e180 80000001f6e57228 ffffffffac14005a 0000000000000578 [21478.826419] $ 8 : 0000000000000001 80000001ef10c600 0000000000000001 0000000000000004 [21478.834485] $12 : 0000000000000004 80000001f6e57222 ffffffff858d4908 0000000000000000 [21478.842552] $16 : 0000000000000010 ffffffffac14005a ffffffffc2540000 0000000000000578 [21478.850619] $20 : 80000001ef908500 0000000000000003 ffffffffc238c3f8 ffffffffc2380000 [21478.858685] $24 : 0000000000000002 ffffffff80a50a40 [21478.866750] $28 : 80000001f6e54000 80000001f6e57200 80000001840ae900 ffffffffc254ae2c [21478.874819] Hi : 0000000000000048 [21478.878396] Lo : 0000000001c624f3 [21478.881979] epc : ffffffff80d35014 _raw_read_lock+0x14/0x28 [21478.887731] Tainted: P C O [21478.891528] ra : ffffffffc254ae2c is_geoip4_reserved+0x34/0x110 [geoip_database] [21478.899193] Status: 14009ce3 KX SX UX KERNEL EXL IE [21478.904244] Cause : 40808000 [21478.907127] PrId : 000d9703 (Cavium Octeon III) [21478.911748] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21479.212320] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21479.222941] Stack : 0000000000000000 0000000000000001 ffffffffac14005a ffffffffc238aba8 0000000000000000 0000000000000000 000000001dd722ac 0000000000000000 0000000000000001 00000048fb79a000 80000001a2eb6800 80000001fb79a000 80000001f608e0c4 80000001eb554800 ffffffffc2380000 0000000000000003 ffffffffc238c3f8 ffffffffc2380000 80000001840ae900 ffffffffc23904ec 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... [21479.288574] Call Trace: [21479.291027] [] _raw_read_lock+0x14/0x28 [21479.296506] [] is_geoip4_reserved+0x34/0x110 [geoip_database] [21479.303882] [] country_traffic_statistics+0x130/0x538 [nf_report] [21479.311602] [] FH_traffic_report_ipv4_hook+0x284/0xeb8 [nf_report] [21479.319409] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [21479.326487] [] nf_iterate+0x13c/0x590 [21479.331720] [] nf_hook_slow+0x8c/0x1f8 [21479.337041] [] ip_forward+0x45c/0x488 [21479.342308] [] rtcompl_hook+0x338/0x4a0 [fastpath_kmodule] [21479.349385] [] nf_iterate+0x13c/0x590 [21479.354617] [] nf_hook_slow+0x8c/0x1f8 [21479.359937] [] ip_rcv+0x2f8/0x410 [21479.364824] [] __netif_receive_skb_core+0x500/0x618 [21479.371273] [] process_backlog+0xa8/0x190 [21479.376853] [] net_rx_action+0x14c/0x228 [21479.382348] [] __do_softirq+0x1c8/0x210 [21479.387753] [] do_softirq+0x68/0x70 [21479.392812] [] local_bh_enable_ip+0xa8/0xb0 [21479.398580] [] fqdnObject_packetQ_pop+0x64/0x120 [fqdn_object] [21479.405999] [] fqdnObject_work_queue_func+0x40/0x390 [fqdn_object] [21479.413759] [] process_one_work+0x178/0x3e0 [21479.419513] [] worker_thread+0x140/0x3c8 [21479.425007] [] kthread+0xa4/0xb0 [21479.429807] [] ret_from_kernel_thread+0x14/0x1c [21479.435906] [21479.437398] Code: 00000000 24420001 e0820000 <1040fffa> 00000000 03e00008 00000000 00000000 41626000 [21479.447445] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21479.456522] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21479.463069] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 000000000000620e 00000000000005c7 ffffffff859a07f8 ffffffff80ec0000 00000000000005c7 0000000000000002 0000000000000002 ffffffff859d1808 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57070 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007ec7b07 80000001f6e54000 80000001f6e56d00 000013828066e8f5 ffffffff808f739c 0000000000000004 0000000000000000 00000000000005c6 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21479.528704] Call Trace: [21479.531156] [] show_stack+0x6c/0xf8 [21479.536218] [] watchdog_timer_fn+0x24c/0x288 [21479.542062] [] __run_hrtimer.isra.20+0x60/0xf8 [21479.548076] [] hrtimer_interrupt+0x138/0x310 [21479.553918] [] c0_compare_interrupt+0x68/0x98 [21479.559845] [] handle_irq_event_percpu+0x88/0x218 [21479.566122] [] handle_percpu_irq+0x84/0xb8 [21479.571790] [] generic_handle_irq+0x38/0x50 [21479.577546] [] do_IRQ+0x18/0x30 [21479.582260] [] plat_irq_dispatch+0x98/0xb8 [21479.587925] [21509.416333] unpack_message_rr fail ret=-31unpack_message_rr fail ret=-31 [21543.955290] Log Date: 1682632758, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21543.965929] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21543.971943] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 7 [21543.980217] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21543.986752] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21544.287380] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21544.296460] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21544.303007] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21544.310498] $ 0 : 0000000000000000 000000000000001c 0000000000143401 0000000000000000 [21544.318568] $ 4 : 80000001d79dbdb8 80000001d79dbdb8 0000000000000001 0000000000000014 [21544.326637] $ 8 : 0000000000000000 0000000000000000 0000000006d7ac00 fffffffffa890800 [21544.334707] $12 : ffffffffc2970000 80000001f6e573dc 0000000000017d56 0000000000000000 [21544.342777] $16 : 80000001d79dbdb8 0000000000000001 80000001dff52008 80000001dff52008 [21544.350847] $20 : 000000000000000b ffffffffc29dda80 80000001d79dbdb8 80000001c70e3458 [21544.358919] $24 : 0000000000000004 ffffffffc29a8808 [21544.366986] $28 : 80000001f6e54000 80000001f6e57410 80000001c70e3348 ffffffffc29a8b28 [21544.375055] Hi : 000000000008df41 [21544.378634] Lo : 000000000000101e [21544.382271] epc : ffffffffc29acd48 fastpath_decrement_ttl+0x20/0x208 [quicksec] [21544.389763] Tainted: P C O [21544.393550] ra : ffffffffc29a8b28 fastpath_execute_transform_step+0x320/0x5a0 [quicksec] [21544.401910] Status: 14009ce3 KX SX UX KERNEL EXL IE [21544.406958] Cause : 40808000 [21544.409841] PrId : 000d9703 (Cavium Octeon III) [21544.414462] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21544.715054] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21544.725674] Stack : 000000000008df41 80000001d79dbdb8 800000018bdb6a08 80000001dff52008 80000001dff52008 ffffffffc29a8b28 f90b00000000000b ffffffffc29dda80 800000018bdb6a08 80000001f5c6c008 0000000000000003 ffffffffc29d4e9c 80000001eb58d500 800000018bdb6a08 0000000000000000 80000001f5c6e000 ffffffffc295bbd8 ffffffff80c13b00 80000001dff52028 0000000000000000 800000018bdb6a08 0000000000000000 0000000000000001 80000001f0312000 80000001f5c6c008 000000000000000f 80000001dff52008 80000001f57cc6e0 ffffffffc295ba40 ffffffffc295ba80 ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 0000000000000000 000000000000000f 000000000000000f 0000000000000001 0000000000000000 80000001dff521d0 ... [21544.791319] Call Trace: [21544.793815] [] fastpath_decrement_ttl+0x20/0x208 [quicksec] [21544.801050] [] fastpath_execute_transform_step+0x320/0x5a0 [quicksec] [21544.809154] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [21544.816728] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21544.824301] [] FH_ssh_interceptor_encrypted_packet_in+0x1c4/0x290 [quicksec] [21544.833028] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [21544.840117] [] nf_iterate+0x13c/0x590 [21544.845349] [] nf_hook_slow+0x8c/0x1f8 [21544.850671] [] ip_rcv+0x2f8/0x410 [21544.855559] [] __netif_receive_skb_core+0x500/0x618 [21544.862008] [] process_backlog+0xa8/0x190 [21544.867589] [] net_rx_action+0x14c/0x228 [21544.873085] [] __do_softirq+0x1c8/0x210 [21544.878492] [] do_softirq+0x68/0x70 [21544.883550] [] local_bh_enable_ip+0xa8/0xb0 [21544.889323] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21544.896916] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21544.904764] [] process_one_work+0x178/0x3e0 [21544.910518] [] worker_thread+0x140/0x3c8 [21544.916012] [] kthread+0xa4/0xb0 [21544.920813] [] ret_from_kernel_thread+0x14/0x1c [21544.926912] [21544.928404] Code: ffb30020 ffb20018 8c820000 0080802d dfbf0028 0220102d dfb30020 dfb20018 [21544.938454] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21544.947523] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21544.954067] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000006706 0000000000000612 ffffffff859a07f8 ffffffff80ec0000 0000000000000612 0000000000000002 0000000000000002 ffffffff859d4bc0 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57280 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e56f10 [21545.003365] Call Trace: [21545.005818] [] show_stack+0x6c/0xf8 [21545.010879] [] watchdog_timer_fn+0x24c/0x288 [21545.016722] [] __run_hrtimer.isra.20+0x60/0xf8 [21545.022736] [] hrtimer_interrupt+0x138/0x310 [21545.028580] [] c0_compare_interrupt+0x68/0x98 [21545.034510] [] handle_irq_event_percpu+0x88/0x218 [21545.040790] [] handle_percpu_irq+0x84/0xb8 [21545.046460] [] generic_handle_irq+0x38/0x50 [21545.052220] [] do_IRQ+0x18/0x30 [21545.056935] [] plat_irq_dispatch+0x98/0xb8 [21545.062601] [21567.263276] unpack_message_rr fail ret=-31[21599.588432] ignore icmp time exceed [21599.588520] ignore icmp time exceed [21600.955291] Log Date: 1682632815, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21600.965930] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21600.971944] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 8 [21600.980219] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21600.986754] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21601.287389] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21601.296469] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21601.303015] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21601.310506] $ 0 : 0000000000000000 0000000000000000 0000000000000007 80000001fbb16ee0 [21601.318573] $ 4 : 0000000000000038 80000001d79da638 0000000000000000 0000000000000000 [21601.326639] $ 8 : 0000000000000010 0000000000000001 0000000000000000 0000000000000001 [21601.334710] $12 : 0000000000000000 ffffffff80a489bc 000000000000003f 0000000000000000 [21601.342778] $16 : 80000001d79dbdb8 80000001f5c6c008 0000000000000007 80000001f5c6c030 [21601.350844] $20 : ffffffffc295bbd8 80000001dff52008 ffffffffddffffff 8000000198fc8208 [21601.358911] $24 : 0000000000000000 ffffffffc29d5860 [21601.366979] $28 : 80000001f6e54000 80000001f6e57730 0000000000000000 ffffffffc29d735c [21601.375049] Hi : 000000000000051a [21601.378627] Lo : aaaaaaaaaaaaad38 [21601.382263] epc : ffffffffc29d7380 ssh_engine_alloc_pc+0x70/0x1e8 [quicksec] [21601.389494] Tainted: P C O [21601.393291] ra : ffffffffc29d735c ssh_engine_alloc_pc+0x4c/0x1e8 [quicksec] [21601.400522] Status: 14009ce3 KX SX UX KERNEL EXL IE [21601.405571] Cause : 40808000 [21601.408454] PrId : 000d9703 (Cavium Octeon III) [21601.413075] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21601.713681] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21601.724303] Stack : 80000001f5c6c008 000000000000000f 80000001f5c6c008 80000001f5c6c008 000000000000000f 80000001dff52008 80000001f57cc6e0 ffffffffc295ba40 ffffffffc295ba80 ffffffffc29d5998 0000000000200000 000000000000000f 000000000000000f 0000000000000001 0000000000000000 80000001dff521d0 0000000000000000 ffffffff00000001 0000000000000001 80000001d79dbdb8 ffffffffffffffff 0000000000000000 80000001f5c6e058 0000000000000002 80000001f5c6e000 80000001fb79a000 ffffffffc295bbd8 80000001eb554800 0000000000000001 8000000181ad10c4 0000000000000001 ffffffffc2962338 0000000000000001 ffffffff80c14638 80000001ea441080 0000000000000000 800000017bb75108 ffffffffc295ba80 80000000243e3de0 000000000000003f ... [21601.789956] Call Trace: [21601.792452] [] ssh_engine_alloc_pc+0x70/0x1e8 [quicksec] [21601.799399] [] ssh_engine_packet_handler+0x138/0x3e8 [quicksec] [21601.806952] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21601.814917] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21601.821984] [] __tasklet_action+0xac/0x1b8 [21601.827651] [] __do_softirq+0x1c8/0x210 [21601.833058] [] do_softirq+0x68/0x70 [21601.838116] [] local_bh_enable_ip+0xa8/0xb0 [21601.843885] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21601.851480] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21601.859332] [] process_one_work+0x178/0x3e0 [21601.865086] [] worker_thread+0x140/0x3c8 [21601.870580] [] kthread+0xa4/0xb0 [21601.875380] [] ret_from_kernel_thread+0x14/0x1c [21601.881480] [21601.882970] Code: 12000018 00000000 de050028 <00129238> fc650000 ae020070 dea20010 0052902d 8e420054 [21601.893017] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21601.902086] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21601.908631] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000006bc9 0000000000000658 ffffffff859a07f8 ffffffff80ec0000 0000000000000658 0000000000000002 0000000000000002 ffffffff859d7cd8 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e575a0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e57230 0000139f05f8904e ffffffff808f739c 0000000000000004 0000000000000000 0000000000000657 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21601.974264] Call Trace: [21601.976717] [] show_stack+0x6c/0xf8 [21601.981781] [] watchdog_timer_fn+0x24c/0x288 [21601.987626] [] __run_hrtimer.isra.20+0x60/0xf8 [21601.993645] [] hrtimer_interrupt+0x138/0x310 [21601.999489] [] c0_compare_interrupt+0x68/0x98 [21602.005418] [] handle_irq_event_percpu+0x88/0x218 [21602.011695] [] handle_percpu_irq+0x84/0xb8 [21602.017365] [] generic_handle_irq+0x38/0x50 [21602.023121] [] do_IRQ+0x18/0x30 [21602.027834] [] plat_irq_dispatch+0x98/0xb8 [21602.033499] [21634.955290] Log Date: 1682632849, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21634.965745] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21634.971759] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 9 [21634.980033] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21634.986567] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21635.287177] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21635.296257] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21635.302803] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21635.310294] $ 0 : 0000000000000000 0000000000000000 8000000189f9bb40 0000000000000000 [21635.318363] $ 4 : 0000000000000000 ffffffffffffffff 80000001a944e180 80000001f6e575a0 [21635.326431] $ 8 : 0000000000000000 00000000582a0000 0000000000000001 0000000000000001 [21635.334500] $12 : 80000001f6e575d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [21635.342569] $16 : 8000000189f9bb40 0000000000000000 0000000000000003 80000001f6e575a0 [21635.350638] $20 : fffffffffffffff5 80000001f6e57680 80000001fba60a40 0000000000000002 [21635.358707] $24 : 0000000000000000 ffffffffc295bc30 [21635.366776] $28 : 80000001f6e54000 80000001f6e57540 80000001fba609c0 ffffffff80bb659c [21635.374844] Hi : 000000000000051a [21635.378422] Lo : aaaaaaaaaaaaad38 [21635.382006] epc : ffffffff80bb64dc fib_rules_lookup+0x64/0x1d8 [21635.388019] Tainted: P C O [21635.391774] ra : ffffffff80bb659c fib_rules_lookup+0x124/0x1d8 [21635.397873] Status: 14009ce3 KX SX UX KERNEL EXL IE [21635.402923] Cause : 40808000 [21635.405806] PrId : 000d9703 (Cavium Octeon III) [21635.410426] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21635.711020] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21635.721640] Stack : 8000000189f9bb40 f0320cb2c0a81064 80000001f6e57680 ffffffff80eeab00 ffffffffffffff80 80000001a2eb3000 0000000000000000 0000000000000009 0000000000200000 ffffffffc2960000 800000019d7ea008 ffffffff80c62020 0000000000000000 80000001f6e575d0 0000000000000000 0000000100000000 ffffffff80eeab00 ffffffff80c10bcc 8000000100000000 0000000000000000 0000000000000000 80000001ea27ad80 0000000006000000 ffffffffc2960000 80000001f6e57680 0000000000000000 ffffffff80eeab00 80000001f5c6e000 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 800000019d7ea008 ffffffff80c10f2c 80000001f6e576c0 80000001ec3a6c00 80000001f61e10a0 ffffffff80c6957c 000000000000781b ffffffffc295e390 ... [21635.787277] Call Trace: [21635.789731] [] fib_rules_lookup+0x64/0x1d8 [21635.795402] [] __fib_lookup+0x30/0x50 [21635.800639] [] __ip_route_output_key+0x52c/0x868 [21635.806831] [] ip_route_output_flow+0x24/0x78 [21635.812762] [] nf_ip_route+0x1c/0x48 [21635.817965] [] ssh_interceptor_reroute_skb_ipv4+0x108/0x368 [quicksec] [21635.826146] [] ssh_interceptor_send+0x134/0xed8 [quicksec] [21635.833288] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [21635.840843] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21635.848815] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21635.855882] [] __tasklet_action+0xac/0x1b8 [21635.861549] [] __do_softirq+0x1c8/0x210 [21635.866955] [] do_softirq+0x68/0x70 [21635.872013] [] local_bh_enable_ip+0xa8/0xb0 [21635.877781] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21635.885375] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21635.893222] [] process_one_work+0x178/0x3e0 [21635.898976] [] worker_thread+0x140/0x3c8 [21635.904470] [] kthread+0xa4/0xb0 [21635.909270] [] ret_from_kernel_thread+0x14/0x1c [21635.915369] [21635.916861] Code: 2414fff5 00000000 8e030014 <10600013> 00000000 8ea50004 10a30010 0000102d 8e030028 [21635.926911] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21635.935982] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21635.942526] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 00000000000070b9 00000000000006a1 ffffffff859a07f8 ffffffff80ec0000 00000000000006a1 0000000000000002 0000000000000002 ffffffff859dafb8 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e573b0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e57040 000013a6f087646d ffffffff808f739c 0000000000000004 0000000000000000 00000000000006a0 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21636.008170] Call Trace: [21636.010621] [] show_stack+0x6c/0xf8 [21636.015683] [] watchdog_timer_fn+0x24c/0x288 [21636.021527] [] __run_hrtimer.isra.20+0x60/0xf8 [21636.027544] [] hrtimer_interrupt+0x138/0x310 [21636.033388] [] c0_compare_interrupt+0x68/0x98 [21636.039320] [] handle_irq_event_percpu+0x88/0x218 [21636.045599] [] handle_percpu_irq+0x84/0xb8 [21636.051269] [] generic_handle_irq+0x38/0x50 [21636.057028] [] do_IRQ+0x18/0x30 [21636.061743] [] plat_irq_dispatch+0x98/0xb8 [21636.067408] [21669.455289] Log Date: 1682632883, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21669.465743] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21669.471756] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 1 [21669.480029] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21669.486562] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21669.787182] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21669.796262] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21669.802809] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21669.810299] $ 0 : 0000000000000000 0000000014009ce1 ffffffffc29d75b8 0000000000000000 [21669.818366] $ 4 : 80000001f5c6c008 80000001d79dbdb8 0000000000000001 00000000000007ae [21669.826433] $ 8 : 0000000000000001 0000000000000001 8000000188acd080 8000000188acd080 [21669.834500] $12 : 0000000006365121 0000000000000000 0000000000000002 6db6db6db6db6db7 [21669.842567] $16 : 0000000000000000 80000001f5c6c008 0000000000000002 80000001dff52008 [21669.850636] $20 : 000000000000000b ffffffffc29dda80 80000001d79dbdb8 800000018bdb6a28 [21669.858707] $24 : 0000000000000000 ffffffffc295ba80 [21669.866776] $28 : 80000001f6e54000 80000001f6e576d0 80000001c62f4c48 ffffffffc29d4b8c [21669.874846] Hi : 000000000000051e [21669.878424] Lo : aaaaaaaaaaaaad3a [21669.882063] epc : ffffffffc29d75b8 ssh_engine_free_pc+0x0/0x158 [quicksec] [21669.889120] Tainted: P C O [21669.892932] ra : ffffffffc29d4b8c fastpath_packet_continue+0x364/0x1038 [quicksec] [21669.900772] Status: 14009ce3 KX SX UX KERNEL EXL IE [21669.905822] Cause : 40808000 [21669.908705] PrId : 000d9703 (Cavium Octeon III) [21669.913326] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21670.213934] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21670.224554] Stack : 0000000000000000 ffffffff80c19808 ac150063ebd54000 80000001f5c6e000 80000001eae58a80 800000018bdb6a08 80000001dff52028 0000000000000000 800000018bdb6a08 0000000000000000 0000000000000001 0000000000000000 80000001f5c6c008 000000000000000f 80000001dff52008 80000001f57cc6e0 ffffffffc295ba40 ffffffffc295ba80 ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 0000000000200000 000000000000000f 000000000000000f 0000000000000001 0000000000000000 80000001dff521d0 0000000000000000 ffffffff00000001 0000000000000001 80000001d79dbdb8 ffffffffffffffff 0000000000000000 80000001f5c6e058 0000000000000002 80000001f5c6e000 8000000197e64800 ffffffffc295bbd8 8000000198e43c00 ... [21670.290199] Call Trace: [21670.292693] [] ssh_engine_free_pc+0x0/0x158 [quicksec] [21670.299464] [] fastpath_packet_continue+0x364/0x1038 [quicksec] [21670.307019] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21670.314572] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21670.322538] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21670.329610] [] __tasklet_action+0xac/0x1b8 [21670.335278] [] __do_softirq+0x1c8/0x210 [21670.340688] [] do_softirq+0x68/0x70 [21670.345749] [] local_bh_enable+0xa8/0xb0 [21670.351267] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [21670.359132] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [21670.366996] [] process_one_work+0x178/0x3e0 [21670.372753] [] worker_thread+0x140/0x3c8 [21670.378249] [] kthread+0xa4/0xb0 [21670.383049] [] ret_from_kernel_thread+0x14/0x1c [21670.389148] [21670.390638] Code: 67bd0010 08a75d55 fc800050 <67bdffc0> 3c02c296 ffb30020 64930028 ffbf0038 6442bc28 [21670.400686] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21670.409762] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21670.416306] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000007599 00000000000006e5 ffffffff859a07f8 ffffffff80ec0000 00000000000006e5 0000000000000002 0000000000000002 ffffffff859de160 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57540 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c0000000790cac7 80000001f6e54000 80000001f6e571d0 000013aef8e39d50 ffffffff808f739c 0000000000000004 0000000000000000 00000000000006e4 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21670.481931] Call Trace: [21670.484383] [] show_stack+0x6c/0xf8 [21670.489444] [] watchdog_timer_fn+0x24c/0x288 [21670.495286] [] __run_hrtimer.isra.20+0x60/0xf8 [21670.501301] [] hrtimer_interrupt+0x138/0x310 [21670.507143] [] c0_compare_interrupt+0x68/0x98 [21670.513071] [] handle_irq_event_percpu+0x88/0x218 [21670.519347] [] handle_percpu_irq+0x84/0xb8 [21670.525015] [] generic_handle_irq+0x38/0x50 [21670.530770] [] do_IRQ+0x18/0x30 [21670.535484] [] plat_irq_dispatch+0x98/0xb8 [21670.541149] [21703.955289] Log Date: 1682632918, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21703.965743] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21703.971758] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 2 [21703.980033] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21703.986567] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21704.287187] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21704.296267] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21704.302813] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21704.310302] $ 0 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [21704.318369] $ 4 : 800000019d7eba08 0000000000000014 800000019d7eba28 000000000000000b [21704.326436] $ 8 : 0000000000000000 0000000000000001 0000000000000000 0000000000000001 [21704.334505] $12 : 0000000000000000 ffffffff80a489bc ffffffff858d4900 0000000000000000 [21704.342574] $16 : 80000001d79dbdb8 80000001ea6ea0c4 800000019d7eba08 80000001f5c6c008 [21704.350643] $20 : 000000000000000b ffffffffc295c790 80000001d79dbdb8 800000019d7eba28 [21704.358712] $24 : 0000000000000000 ffffffffc29d5860 [21704.366781] $28 : 80000001f6e54000 80000001f6e57400 ffffffffc29d5e28 ffffffffc29d5ee4 [21704.374851] Hi : 0000000000028e03 [21704.378429] Lo : 0000000000000da9 [21704.382067] epc : ffffffffc29d5efc fastpath_packet_context_pullup+0xd4/0xd58 [quicksec] [21704.390254] Tainted: P C O [21704.394041] ra : ffffffffc29d5ee4 fastpath_packet_context_pullup+0xbc/0xd58 [quicksec] [21704.402226] Status: 14009ce3 KX SX UX KERNEL EXL IE [21704.407275] Cause : 40808000 [21704.410158] PrId : 000d9703 (Cavium Octeon III) [21704.414779] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21704.715374] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21704.725994] Stack : 80000001dff52008 ffffffffc29a815c ffffffffc29ba030 80000001d79dbdb8 800000019d7eba08 80000001dff52008 800000019d7eba08 80000001f5c6c008 0000000000000003 80000001dff52008 000000000000000b ffffffffc29dda80 80000001d79dbdb8 800000019d7eba28 ffffffffc29d5e28 ffffffffc29d48d8 800000019d7eba08 8000000197e64800 80000001f5c6e000 ffffffffc295bbd8 ffffffffc295ba80 ffffffff80b9db70 80000001f6e574b8 0000000000000000 800000019d7eba08 0000000000000000 0000000000000000 80000001f30970ec 80000001f5c6c008 000000000000000f 80000001dff52008 80000001f57cc6e0 ffffffffc295ba40 ffffffffc295ba80 ffffffffddffffff ffffffffc2960000 0000000000000000 ffffffffc29d59f0 0000000000000000 000000000000000f ... [21704.791651] Call Trace: [21704.794146] [] fastpath_packet_context_pullup+0xd4/0xd58 [quicksec] [21704.802050] [] fastpath_packet_continue+0xb0/0x1038 [quicksec] [21704.809519] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21704.817072] [] FH_ssh_interceptor_encrypted_packet_in+0x1c4/0x290 [quicksec] [21704.825777] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [21704.832856] [] nf_iterate+0x13c/0x590 [21704.838089] [] nf_hook_slow+0x8c/0x1f8 [21704.843409] [] ip_rcv+0x2f8/0x410 [21704.848296] [] __netif_receive_skb_core+0x500/0x618 [21704.854747] [] process_backlog+0xa8/0x190 [21704.860327] [] net_rx_action+0x14c/0x228 [21704.865823] [] __do_softirq+0x1c8/0x210 [21704.871229] [] do_softirq+0x68/0x70 [21704.876287] [] local_bh_enable_ip+0xa8/0xb0 [21704.882059] [] fqdnObject_syncQ_exist_check+0xb8/0x2d0 [fqdn_object] [21704.890001] [] fqdnObject_work_queue_func+0x1d0/0x390 [fqdn_object] [21704.897849] [] process_one_work+0x178/0x3e0 [21704.903604] [] worker_thread+0x140/0x3c8 [21704.909098] [] kthread+0xa4/0xb0 [21704.913897] [] ret_from_kernel_thread+0x14/0x1c [21704.919997] [21704.921487] Code: 704218b2 a6020074 92220006 <92240007> 92340002 92230003 00021200 00441025 3042ffff [21704.931535] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21704.940605] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21704.947149] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000007a81 000000000000072e ffffffff859a07f8 ffffffff80ec0000 000000000000072e 0000000000000002 0000000000000002 ffffffff859a16b0 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57270 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 0000000000000001 80000001f6e54000 80000001f6e56f00 000013b7013fd7de ffffffff808f739c [21704.999441] Call Trace: [21705.001892] [] show_stack+0x6c/0xf8 [21705.006954] [] watchdog_timer_fn+0x24c/0x288 [21705.012797] [] __run_hrtimer.isra.20+0x60/0xf8 [21705.018811] [] hrtimer_interrupt+0x138/0x310 [21705.024653] [] c0_compare_interrupt+0x68/0x98 [21705.030580] [] handle_irq_event_percpu+0x88/0x218 [21705.036856] [] handle_percpu_irq+0x84/0xb8 [21705.042524] [] generic_handle_irq+0x38/0x50 [21705.048280] [] do_IRQ+0x18/0x30 [21705.052994] [] plat_irq_dispatch+0x98/0xb8 [21705.058659] [21737.955292] Log Date: 1682632952, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21737.965746] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21737.971759] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 3 [21737.980033] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21737.986567] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21738.287198] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21738.296276] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21738.302822] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21738.310312] $ 0 : 0000000000000000 0000000054009ce1 800000019d49e258 ffffffffc0020000 [21738.318380] $ 4 : 305177cce6d7fef3 0000000000000004 800000000b1f5290 00000000000003f8 [21738.326450] $ 8 : 0000000000000001 0000000000000001 0000000000000590 800000018be72200 [21738.334517] $12 : 0000000000000001 00000000000003f0 0000000000000004 6db6db6db6db6db7 [21738.342584] $16 : 80000001f6e57130 00000000000003f8 800000019d49e254 0000000000000000 [21738.350649] $20 : 0000000000000000 0000000000000000 0000000000000001 0000000000000001 [21738.358718] $24 : 00000000000003f0 8000000000000000 [21738.366787] $28 : 80000001f6e54000 80000001f6e56ea0 ffffffff859e9900 ffffffffc233de98 [21738.374855] Hi : 0000000000000002 [21738.378432] Lo : 00000000000000af [21738.382061] epc : ffffffffc233e330 octo_des_cbc_sha1_encrypt+0x5e0/0x1018 [cryptocteon] [21738.390248] Tainted: P C O [21738.394023] ra : ffffffffc233de98 octo_des_cbc_sha1_encrypt+0x148/0x1018 [cryptocteon] [21738.402208] Status: 54009ce3 KX SX UX KERNEL EXL IE [21738.407257] Cause : 40808000 [21738.410140] PrId : 000d9703 (Cavium Octeon III) [21738.414760] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21738.715360] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21738.725980] Stack : 8000000184ecb000 8000000188acd000 0000000000000078 ffffffffc2344838 80000001e950d600 800000000c4ce39a 000000c400000078 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... [21738.791670] Call Trace: [21738.794141] [] octo_des_cbc_sha1_encrypt+0x5e0/0x1018 [cryptocteon] [21738.802040] [] octo_process.part.4+0x48c/0xa38 [cryptocteon] [21738.809293] [] crypto_invoke+0x60/0x168 [21738.814700] [] crypto_dispatch+0x214/0x360 [21738.820419] [] ssh_hwaccel_perform_ipsec+0x31c/0x5b8 [quicksec] [21738.827976] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [21738.835795] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [21738.843349] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21738.850903] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21738.858867] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21738.865931] [] __tasklet_action+0xac/0x1b8 [21738.871598] [] __do_softirq+0x1c8/0x210 [21738.877005] [] do_softirq+0x68/0x70 [21738.882064] [] local_bh_enable_ip+0xa8/0xb0 [21738.887834] [] fqdnObject_syncQ_exist_check+0xb8/0x2d0 [fqdn_object] [21738.895776] [] fqdnObject_work_queue_func+0x1d0/0x390 [fqdn_object] [21738.903623] [] process_one_work+0x178/0x3e0 [21738.909378] [] worker_thread+0x140/0x3c8 [21738.914872] [] kthread+0xa4/0xb0 [21738.919675] [] ret_from_kernel_thread+0x14/0x1c [21738.925775] [21738.927266] Code: 00000000 dfa401c8 48a44088 <48240088> ffa401c8 25adfff8 088cf86d 000d602a 00000000 [21738.937315] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21738.946389] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21738.952934] Stack : 0000000000000001 0000000054001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000007f4f 0000000000000777 ffffffff859a07f8 ffffffff80ec0000 0000000000000777 0000000000000002 0000000000000002 ffffffff859a4970 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e56d10 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007ec7b07 80000001f6e54000 80000001f6e569a0 000013beebceac56 ffffffff808f739c 0000000000000004 0000000000000000 0000000000000776 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21739.018577] Call Trace: [21739.021029] [] show_stack+0x6c/0xf8 [21739.026091] [] watchdog_timer_fn+0x24c/0x288 [21739.031935] [] __run_hrtimer.isra.20+0x60/0xf8 [21739.037949] [] hrtimer_interrupt+0x138/0x310 [21739.043790] [] c0_compare_interrupt+0x68/0x98 [21739.049718] [] handle_irq_event_percpu+0x88/0x218 [21739.055994] [] handle_percpu_irq+0x84/0xb8 [21739.061661] [] generic_handle_irq+0x38/0x50 [21739.067417] [] do_IRQ+0x18/0x30 [21739.072131] [] plat_irq_dispatch+0x98/0xb8 [21739.077797] [21755.450910] unpack_message_rr fail ret=-31 [21788.455290] Log Date: 1682633002, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21788.465929] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21788.471942] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 4 [21788.480216] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21788.486750] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21788.787356] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21788.796436] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21788.802982] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21788.810473] $ 0 : 0000000000000000 0000000000000000 0000000000000026 0000000000000010 [21788.818542] $ 4 : 80000001f6e57320 0000000000000000 0000000000000000 00000015800092dd [21788.826610] $ 8 : ffffffff80c15b90 0000000000000001 000000000000000f 7674693300000000 [21788.834680] $12 : 0000000000000001 ffffffff80a489bc ffffffff858d4908 00000000000008f0 [21788.842749] $16 : 0000000000000011 0000000000000002 ffffffff80c15b90 80000001fb79a000 [21788.850818] $20 : 80000001a2eb6800 80000001eb5549b0 ffffffff858d4908 80000001ebe3ca80 [21788.858887] $24 : 0000000000000002 0000000000000001 [21788.866957] $28 : 80000001f6e54000 80000001f6e57420 ffffffff80eabd38 ffffffffc23644c4 [21788.875026] Hi : 0000000003df18f3 [21788.878604] Lo : d70a3d70a86ff7df [21788.882230] epc : ffffffffc2364460 rtcompl_hook+0x168/0x4a0 [fastpath_kmodule] [21788.889634] Tainted: P C O [21788.893406] ra : ffffffffc23644c4 rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [21788.900809] Status: 14009ce3 KX SX UX KERNEL EXL IE [21788.905857] Cause : 40808000 [21788.908739] PrId : 000d9703 (Cavium Octeon III) [21788.913360] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21789.213940] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21789.224560] Stack : 80000001850c6e08 ffffffff808d8298 00000000644af12a 000000001e3ef2f1 80000001f6e574a0 ffffffffc2913558 0000000000000000 0000000000000000 80000001f6e574a0 0000000000000010 000000000000000f 0000000000000002 0000000000000002 0000000000000000 ffffffff80eb0000 0000000000000000 80000001ebe3ca80 80000001a2eb6800 80000001fb79a000 ffffffff80c15b90 80000001f6e575a0 ffffffff80bd2e74 ffffffffc2362260 80000001ea6290f8 0000000000000000 ffffffff80eb2090 ffffffff80000000 ffffffff858d4938 0000000000000002 0000000000000000 0000000000000000 0000000000000000 ffffffff858d4908 0000000000000000 80000001a2eb6800 ffffffffc2924744 0000000000000002 80000001ebe3ca80 ffffffff80c15b90 80000001fb79a000 ... [21789.290242] Call Trace: [21789.292711] [] rtcompl_hook+0x168/0x4a0 [fastpath_kmodule] [21789.299791] [] nf_iterate+0x13c/0x590 [21789.305024] [] nf_hook_slow+0x8c/0x1f8 [21789.310346] [] ip_forward+0x45c/0x488 [21789.315620] [] rtcompl_hook+0x338/0x4a0 [fastpath_kmodule] [21789.322697] [] nf_iterate+0x13c/0x590 [21789.327929] [] nf_hook_slow+0x8c/0x1f8 [21789.333249] [] ip_rcv+0x2f8/0x410 [21789.338135] [] __netif_receive_skb_core+0x500/0x618 [21789.344586] [] process_backlog+0xa8/0x190 [21789.350166] [] net_rx_action+0x14c/0x228 [21789.355662] [] __do_softirq+0x1c8/0x210 [21789.361067] [] do_softirq+0x68/0x70 [21789.366125] [] local_bh_enable_ip+0xa8/0xb0 [21789.371897] [] fqdnObject_syncQ_exist_check+0xb8/0x2d0 [fqdn_object] [21789.379847] [] fqdnObject_work_queue_func+0x1d0/0x390 [fqdn_object] [21789.387696] [] process_one_work+0x178/0x3e0 [21789.393450] [] worker_thread+0x140/0x3c8 [21789.398943] [] kthread+0xa4/0xb0 [21789.403743] [] ret_from_kernel_thread+0x14/0x1c [21789.409842] [21789.411332] Code: 26100001 dea70000 2603ffff <00035000> 01473816 c8e0fff7 0200c82d 001879b8 3c0480eb [21789.421376] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21789.430444] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21789.436988] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000008447 00000000000007c1 ffffffff859a07f8 ffffffff80ec0000 00000000000007c1 0000000000000002 0000000000000002 ffffffff859a7c48 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57290 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007f09dc7 [21789.482955] Call Trace: [21789.485406] [] show_stack+0x6c/0xf8 [21789.490468] [] watchdog_timer_fn+0x24c/0x288 [21789.496311] [] __run_hrtimer.isra.20+0x60/0xf8 [21789.502326] [] hrtimer_interrupt+0x138/0x310 [21789.508167] [] c0_compare_interrupt+0x68/0x98 [21789.514095] [] handle_irq_event_percpu+0x88/0x218 [21789.520371] [] handle_percpu_irq+0x84/0xb8 [21789.526037] [] generic_handle_irq+0x38/0x50 [21789.531792] [] do_IRQ+0x18/0x30 [21789.536505] [] plat_irq_dispatch+0x98/0xb8 [21789.542169] [21822.955289] Log Date: 1682633037, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21822.965744] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21822.971758] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 5 [21822.980032] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21822.986566] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21823.287159] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21823.296239] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21823.302786] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21823.310276] $ 0 : 0000000000000000 0000000000000001 00000001001ab905 0000000000000000 [21823.318347] $ 4 : 0000000000000001 8000000181afbb00 80000001a2eb6800 0000000000000000 [21823.326416] $ 8 : ffffffff80c13b00 0000000000000010 000000000000000b 000000000000000c [21823.334485] $12 : 0000000000000001 0000000000000000 ffffffff858d4900 00000000000000b0 [21823.342553] $16 : 8000000181afbb00 8000000184e7bd88 80000001eb554800 0000000000000000 [21823.350620] $20 : 0000000000000000 80000001a2eb6800 0000000000000000 0000000000000001 [21823.358690] $24 : 0000000000000000 000000000000000c [21823.366759] $28 : 80000001f6e54000 80000001f6e575f0 ffffffff80eab4f8 ffffffffc24e1054 [21823.374826] Hi : 000000000000004c [21823.378404] Lo : 0000000000100fd5 [21823.382020] epc : ffffffffc24e1084 FH_ipt_zymark_hook+0x9c/0x558 [iptable_zymark] [21823.389685] Tainted: P C O [21823.393454] ra : ffffffffc24e1054 FH_ipt_zymark_hook+0x6c/0x558 [iptable_zymark] [21823.401119] Status: 14009ce3 KX SX UX KERNEL EXL IE [21823.406165] Cause : 40808000 [21823.409047] PrId : 000d9703 (Cavium Octeon III) [21823.413668] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21823.714252] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21823.724873] Stack : ffffffff80c13b00 0000000000000000 80000001a2eb6800 80000001eb5549a0 000000000000000c 0000000000000000 ffffffff80c13b00 0000000000000000 80000001a2eb6800 80000001eb5549a0 ffffffff858d4900 8000000181afbb00 ffffffff80eab4f8 ffffffffc23644c4 80000001f6394000 80000001850c6e08 0000000000000000 ac14105a00000000 0000000000000000 00160002ac14005a 0000000000000000 0000000000000000 0000001400000003 000000000000000c 000000000000000b 0000000000000000 0000000000000000 0000000000000000 ffffffff80eb0000 0000000000000000 8000000181afbb00 80000001a2eb6800 0000000000000000 ffffffff80c13b00 80000001f6e577e0 ffffffff80bd2e74 ffffffffc2362230 80000001f0ff1c80 0000000000000000 ffffffff80eb2070 ... [21823.790555] Call Trace: [21823.793020] [] FH_ipt_zymark_hook+0x9c/0x558 [iptable_zymark] [21823.800396] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [21823.807475] [] nf_iterate+0x13c/0x590 [21823.812707] [] nf_hook_slow+0x8c/0x1f8 [21823.818029] [] ip_rcv+0x2f8/0x410 [21823.822916] [] __netif_receive_skb_core+0x500/0x618 [21823.829367] [] process_backlog+0xa8/0x190 [21823.834947] [] net_rx_action+0x14c/0x228 [21823.840442] [] __do_softirq+0x1c8/0x210 [21823.845848] [] do_softirq+0x68/0x70 [21823.850906] [] local_bh_enable_ip+0xa8/0xb0 [21823.856678] [] fqdnObject_packetQ_pop+0x64/0x120 [fqdn_object] [21823.864096] [] fqdnObject_work_queue_func+0x40/0x390 [fqdn_object] [21823.871856] [] process_one_work+0x178/0x3e0 [21823.877610] [] worker_thread+0x140/0x3c8 [21823.883103] [] kthread+0xa4/0xb0 [21823.887903] [] ret_from_kernel_thread+0x14/0x1c [21823.894001] [21823.895492] Code: 00641824 14600034 2e770001 <24130001> 7ef7f803 0257102d 9042006c 2443ff02 2c630002 [21823.905534] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21823.914603] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21823.921148] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000008905 0000000000000807 ffffffff859a07f8 ffffffff80ec0000 0000000000000807 0000000000000002 0000000000000002 ffffffff859aad48 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57460 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e570f0 000013d2b633bd7e ffffffff808f739c 0000000000000004 0000000000000000 0000000000000806 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21823.986796] Call Trace: [21823.989248] [] show_stack+0x6c/0xf8 [21823.994311] [] watchdog_timer_fn+0x24c/0x288 [21824.000156] [] __run_hrtimer.isra.20+0x60/0xf8 [21824.006173] [] hrtimer_interrupt+0x138/0x310 [21824.012015] [] c0_compare_interrupt+0x68/0x98 [21824.017943] [] handle_irq_event_percpu+0x88/0x218 [21824.024220] [] handle_percpu_irq+0x84/0xb8 [21824.029889] [] generic_handle_irq+0x38/0x50 [21824.035646] [] do_IRQ+0x18/0x30 [21824.040360] [] plat_irq_dispatch+0x98/0xb8 [21824.046024] [21824.420331] unpack_message_rr fail ret=-31 [21857.455291] Log Date: 1682633071, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21857.465930] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21857.471943] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 6 [21857.480218] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21857.486753] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21857.787382] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21857.796461] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21857.803006] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21857.810496] $ 0 : 0000000000000000 0000000000000000 8000000189f9b300 0000000000000000 [21857.818564] $ 4 : 0000000000000000 ffffffffffffffff 80000001a944e180 80000001f6e575a0 [21857.826631] $ 8 : 0000000000000000 00000000582a0000 0000000000000001 0000000000000001 [21857.834699] $12 : 80000001f6e575d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [21857.842770] $16 : 8000000189f9b300 0000000000000000 0000000000000003 80000001f6e575a0 [21857.850838] $20 : fffffffffffffff5 80000001f6e57680 80000001fba60a40 0000000000000002 [21857.858908] $24 : 0000000000000000 ffffffffc295bc30 [21857.866975] $28 : 80000001f6e54000 80000001f6e57540 80000001fba609c0 ffffffff80bb659c [21857.875043] Hi : 000000000000051a [21857.878620] Lo : aaaaaaaaaaaaad38 [21857.882204] epc : ffffffff80bb64dc fib_rules_lookup+0x64/0x1d8 [21857.888217] Tainted: P C O [21857.891971] ra : ffffffff80bb659c fib_rules_lookup+0x124/0x1d8 [21857.898070] Status: 14009ce3 KX SX UX KERNEL EXL IE [21857.903119] Cause : 40808000 [21857.906002] PrId : 000d9703 (Cavium Octeon III) [21857.910622] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21858.211232] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21858.221853] Stack : 8000000189f9b300 f032fe52c0a81064 80000001f6e57680 ffffffff80eeab00 ffffffffffffff80 80000001a2eb3000 0000000000000000 0000000000000009 0000000000200000 ffffffffc2960000 800000019d7eba08 ffffffff80c62020 0000000000000000 80000001f6e575d0 0000000000000000 0000000100000000 ffffffff80eeab00 ffffffff80c10bcc 8000000100000000 0000000000000000 0000000000000000 80000001e9daaf00 0000000006000000 ffffffffc2960000 80000001f6e57680 0000000000000000 ffffffff80eeab00 80000001f5c6e000 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 800000019d7eba08 ffffffff80c10f2c 80000001f6e576c0 80000001eb935500 80000001840b90a0 ffffffff80c6957c 000000000000781b ffffffffc295e390 ... [21858.287493] Call Trace: [21858.289947] [] fib_rules_lookup+0x64/0x1d8 [21858.295617] [] __fib_lookup+0x30/0x50 [21858.300851] [] __ip_route_output_key+0x52c/0x868 [21858.307040] [] ip_route_output_flow+0x24/0x78 [21858.312967] [] nf_ip_route+0x1c/0x48 [21858.318168] [] ssh_interceptor_reroute_skb_ipv4+0x108/0x368 [quicksec] [21858.326328] [] ssh_interceptor_send+0x134/0xed8 [quicksec] [21858.333448] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [21858.341002] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21858.348966] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21858.356030] [] __tasklet_action+0xac/0x1b8 [21858.361697] [] __do_softirq+0x1c8/0x210 [21858.367104] [] do_softirq+0x68/0x70 [21858.372162] [] local_bh_enable+0xa8/0xb0 [21858.377670] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [21858.385524] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [21858.393371] [] process_one_work+0x178/0x3e0 [21858.399125] [] worker_thread+0x140/0x3c8 [21858.404618] [] kthread+0xa4/0xb0 [21858.409419] [] ret_from_kernel_thread+0x14/0x1c [21858.415518] [21858.417010] Code: 2414fff5 00000000 8e030014 <10600013> 00000000 8ea50004 10a30010 0000102d 8e030028 [21858.427055] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21858.436130] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21858.442677] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000008ded 0000000000000851 ffffffff859a07f8 ffffffff80ec0000 0000000000000851 0000000000000002 0000000000000002 ffffffff859ae058 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e573b0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007ec7b07 80000001f6e54000 80000001f6e57040 000013dabe8ff668 ffffffff808f739c 0000000000000004 0000000000000000 0000000000000850 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21858.508316] Call Trace: [21858.510768] [] show_stack+0x6c/0xf8 [21858.515830] [] watchdog_timer_fn+0x24c/0x288 [21858.521674] [] __run_hrtimer.isra.20+0x60/0xf8 [21858.527688] [] hrtimer_interrupt+0x138/0x310 [21858.533530] [] c0_compare_interrupt+0x68/0x98 [21858.539458] [] handle_irq_event_percpu+0x88/0x218 [21858.545735] [] handle_percpu_irq+0x84/0xb8 [21858.551402] [] generic_handle_irq+0x38/0x50 [21858.557159] [] do_IRQ+0x18/0x30 [21858.561874] [] plat_irq_dispatch+0x98/0xb8 [21858.567538] [21891.455289] Log Date: 1682633105, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21891.465744] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21891.471759] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 7 [21891.480036] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21891.486572] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21891.787213] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21891.796293] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21891.802839] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21891.810329] $ 0 : 0000000000000000 0000000000000000 800000018bd11cc0 0000000000000000 [21891.818398] $ 4 : 0000000000000000 ffffffffffffffff 80000001a944e180 80000001f6e575a0 [21891.826466] $ 8 : 0000000000000000 00000000582a0000 0000000000000001 0000000000000001 [21891.834533] $12 : 80000001f6e575d0 000000000000001f 000000000000003f 6db6db6db6db6db7 [21891.842602] $16 : 800000018bd11cc0 0000000000000000 0000000000000003 80000001f6e575a0 [21891.850671] $20 : fffffffffffffff5 80000001f6e57680 80000001fba60a40 0000000000000002 [21891.858740] $24 : 0000000000000000 ffffffffc295bc30 [21891.866807] $28 : 80000001f6e54000 80000001f6e57540 80000001fba609c0 ffffffff80bb659c [21891.874876] Hi : 000000000000051a [21891.878454] Lo : aaaaaaaaaaaaad38 [21891.882039] epc : ffffffff80bb64dc fib_rules_lookup+0x64/0x1d8 [21891.888052] Tainted: P C O [21891.891806] ra : ffffffff80bb659c fib_rules_lookup+0x124/0x1d8 [21891.897906] Status: 14009ce3 KX SX UX KERNEL EXL IE [21891.902954] Cause : 40808000 [21891.905838] PrId : 000d9703 (Cavium Octeon III) [21891.910458] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21892.211077] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21892.221698] Stack : 800000018bd11cc0 f03203c4c0a81064 80000001f6e57680 ffffffff80eeab00 ffffffffffffff80 80000001a2eb3000 0000000000000000 0000000000000009 0000000000200000 ffffffffc2960000 80000001f4045008 ffffffff80c62020 0000000000000000 80000001f6e575d0 0000000000000000 0000000100000000 ffffffff80eeab00 ffffffff80c10bcc 8000000100000000 0000000000000000 0000000000000000 80000001ec3a6900 0000000006000000 ffffffffc2960000 80000001f6e57680 0000000000000000 ffffffff80eeab00 80000001f5c6e000 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 80000001f4045008 ffffffff80c10f2c 80000001f6e576c0 80000001f461a900 80000001f46420a0 ffffffff80c6957c 000000000000781b ffffffffc295e390 ... [21892.287336] Call Trace: [21892.289790] [] fib_rules_lookup+0x64/0x1d8 [21892.295460] [] __fib_lookup+0x30/0x50 [21892.300694] [] __ip_route_output_key+0x52c/0x868 [21892.306882] [] ip_route_output_flow+0x24/0x78 [21892.312809] [] nf_ip_route+0x1c/0x48 [21892.318009] [] ssh_interceptor_reroute_skb_ipv4+0x108/0x368 [quicksec] [21892.326168] [] ssh_interceptor_send+0x134/0xed8 [quicksec] [21892.333289] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [21892.340841] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21892.348805] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21892.355869] [] __tasklet_action+0xac/0x1b8 [21892.361536] [] __do_softirq+0x1c8/0x210 [21892.366941] [] do_softirq+0x68/0x70 [21892.371999] [] local_bh_enable_ip+0xa8/0xb0 [21892.377767] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21892.385361] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21892.393207] [] process_one_work+0x178/0x3e0 [21892.398961] [] worker_thread+0x140/0x3c8 [21892.404455] [] kthread+0xa4/0xb0 [21892.409254] [] ret_from_kernel_thread+0x14/0x1c [21892.415353] [21892.416844] Code: 2414fff5 00000000 8e030014 <10600013> 00000000 8ea50004 10a30010 0000102d 8e030028 [21892.426890] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21892.435969] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21892.442515] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 00000000000092d9 000000000000089a ffffffff859a07f8 ffffffff80ec0000 000000000000089a 0000000000000002 0000000000000002 ffffffff859b1338 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e573b0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c0000000790ca47 80000001f6e54000 80000001f6e57040 000013e2a91eca40 ffffffff808f739c 0000000000000004 0000000000000000 0000000000000899 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21892.508151] Call Trace: [21892.510603] [] show_stack+0x6c/0xf8 [21892.515665] [] watchdog_timer_fn+0x24c/0x288 [21892.521508] [] __run_hrtimer.isra.20+0x60/0xf8 [21892.527522] [] hrtimer_interrupt+0x138/0x310 [21892.533364] [] c0_compare_interrupt+0x68/0x98 [21892.539291] [] handle_irq_event_percpu+0x88/0x218 [21892.545568] [] handle_percpu_irq+0x84/0xb8 [21892.551235] [] generic_handle_irq+0x38/0x50 [21892.556991] [] do_IRQ+0x18/0x30 [21892.561705] [] plat_irq_dispatch+0x98/0xb8 [21892.567370] [21925.955291] Log Date: 1682633140, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21925.965745] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21925.971761] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 8 [21925.980037] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21925.986573] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21926.287186] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21926.296267] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21926.302813] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21926.310303] $ 0 : 0000000000000000 0000000000000000 8000000197f02f80 ffffffffc0a81064 [21926.318372] $ 4 : ffffffff80eeab00 ffffffffc0a81064 0000000000000000 0000000000000001 [21926.326439] $ 8 : ffffffffc0a81064 0000000000000001 8000000188acd080 8000000188acd080 [21926.334505] $12 : 0000000000000000 ffffffff80a489bc 0000000000000002 6db6db6db6db6db7 [21926.342571] $16 : 80000001f6e57680 0000000000000000 ffffffffffffffea 80000001f5c6e000 [21926.350636] $20 : 0000000000000000 0000000000000009 0000000000200000 ffffffffc2960000 [21926.358704] $24 : 0000000000000000 ffffffffc295bc30 [21926.366771] $28 : 80000001f6e54000 80000001f6e57560 800000018bd94808 ffffffff80c10a08 [21926.374838] Hi : 000000000000051a [21926.378416] Lo : aaaaaaaaaaaaad38 [21926.381999] epc : ffffffff80c4b23c __ip_dev_find+0x44/0x170 [21926.387752] Tainted: P C O [21926.391508] ra : ffffffff80c10a08 __ip_route_output_key+0x368/0x868 [21926.398042] Status: 14009ce3 KX SX UX KERNEL EXL IE [21926.403091] Cause : 40808000 [21926.405974] PrId : 000d9703 (Cavium Octeon III) [21926.410595] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21926.711194] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21926.721816] Stack : 421e075ba2eb3000 80000001aa3c5a00 ffffffff80d92990 0000000000000000 0000000000000000 ffffffff80cf1b44 80000001e9dabb00 80000001e9dabb00 0000000020005000 ffffffff80b9f830 ffffffff80eb0000 80000001f6e57680 ffffffff80eeab00 ffffffff80c10a08 8000000100000000 0000000000000000 0000000000000000 80000001e9dabb00 0000000006000000 ffffffffc2960000 80000001f6e57680 0000000000000000 ffffffff80eeab00 80000001f5c6e000 0000000000000000 ffffffffc295ba80 0000000000200000 ffffffffc2960000 800000018bd94808 ffffffff80c10f2c 80000001f6e576c0 80000001ee75a780 80000001eb78c0a0 ffffffff80c6957c 000000000000781b ffffffffc295e390 0000000900000001 0000000000003200 00000000c0a81064 c0a8006300000000 ... [21926.787453] Call Trace: [21926.789906] [] __ip_dev_find+0x44/0x170 [21926.795316] [] __ip_route_output_key+0x368/0x868 [21926.801505] [] ip_route_output_flow+0x24/0x78 [21926.807432] [] nf_ip_route+0x1c/0x48 [21926.812631] [] ssh_interceptor_reroute_skb_ipv4+0x108/0x368 [quicksec] [21926.820791] [] ssh_interceptor_send+0x134/0xed8 [quicksec] [21926.827913] [] ssh_engine_packet_handler+0x1d0/0x3e8 [quicksec] [21926.835465] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21926.843428] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21926.850494] [] __tasklet_action+0xac/0x1b8 [21926.856162] [] __do_softirq+0x1c8/0x210 [21926.861568] [] do_softirq+0x68/0x70 [21926.866626] [] local_bh_enable_ip+0xa8/0xb0 [21926.872395] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [21926.879990] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [21926.887841] [] process_one_work+0x178/0x3e0 [21926.893594] [] worker_thread+0x140/0x3c8 [21926.899088] [] kthread+0xa4/0xb0 [21926.903887] [] ret_from_kernel_thread+0x14/0x1c [21926.909987] [21926.911478] Code: 1040000d 00000000 8c430030 <14650007> 00a0382d 08312cd5 dc420018 00000000 8c430030 [21926.921527] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21926.930603] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21926.937149] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 00000000000097c5 00000000000008e2 ffffffff859a07f8 ffffffff80ec0000 00000000000008e2 0000000000000002 0000000000000002 ffffffff859b45e0 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e573d0 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e57060 000013eab17b0317 ffffffff808f739c 0000000000000004 0000000000000000 00000000000008e1 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [21927.002784] Call Trace: [21927.005236] [] show_stack+0x6c/0xf8 [21927.010299] [] watchdog_timer_fn+0x24c/0x288 [21927.016142] [] __run_hrtimer.isra.20+0x60/0xf8 [21927.022157] [] hrtimer_interrupt+0x138/0x310 [21927.027999] [] c0_compare_interrupt+0x68/0x98 [21927.033926] [] handle_irq_event_percpu+0x88/0x218 [21927.040203] [] handle_percpu_irq+0x84/0xb8 [21927.045871] [] generic_handle_irq+0x38/0x50 [21927.051627] [] do_IRQ+0x18/0x30 [21927.056340] [] plat_irq_dispatch+0x98/0xb8 [21927.062004] [21947.682516] unpack_message_rr fail ret=-31 [21980.955291] Log Date: 1682633195, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [21980.965929] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [21980.971944] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 9 [21980.980217] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [21980.986751] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21981.287364] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21981.296444] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21981.302990] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [21981.310481] $ 0 : 0000000000000000 0000000014009ce1 8000000184ecb080 ffffffff80eb0428 [21981.318551] $ 4 : 8000000184ecb080 0000000000000000 0000000000000048 800000009eb1c000 [21981.326619] $ 8 : 000000000d72ba4c 8000000184ecb100 ffffffff80eb0428 0000000000000001 [21981.334689] $12 : 000000000d72ba4d ffffffff80a489b8 0000000000000000 0000000000000000 [21981.342757] $16 : 8000000184ecb080 0000000000000000 8000000188acd000 ffffffff85a00000 [21981.350826] $20 : ffffffffc295ba80 ffffffffc295ba40 ffffffffc2960000 80000001be57f538 [21981.358894] $24 : 0000000000000020 ffffffffc29b1610 [21981.366962] $28 : 80000001f6e54000 80000001f6e57460 80000001dff40808 ffffffff80a1a548 [21981.375032] Hi : 0000000000000002 [21981.378610] Lo : 00000000000000af [21981.382192] epc : ffffffff80a48900 memset+0x0/0x24 [21981.387161] Tainted: P C O [21981.390915] ra : ffffffff80a1a548 crypto_getreq+0x78/0xd8 [21981.396579] Status: 14009ce3 KX SX UX KERNEL EXL IE [21981.401624] Cause : 40808000 [21981.404507] PrId : 000d9703 (Cavium Octeon III) [21981.409127] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [21981.709696] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [21981.720316] Stack : 0000000000000120 80000001e42ca888 ffffffffc2950000 ffffffffc2950000 800000019d7eba08 ffffffffc29d3cf0 ffffffffc29b0c78 80000001d79dbdb8 000000000000001c 0000000000000588 0000000000000014 0000000000000590 00000000000005a4 ffffffffc29b44f0 0000000000000000 0000000000000000 80000001d79dbed0 80000001dff52008 0000000000000000 ffffffffc295bbd8 80000001dff52008 0000000000000024 0000000000000024 80000001be57f538 80000001d79dbdb8 ffffffffc29b2644 80000001d79dbdb8 ffffffff80b9fc7c 452005b0e1874000 f032115cc0a81064 c0a80063d0c5f548 04d61262330d010f 42298db4a2eb3000 80000001aa3c5a00 ffffffff80d92990 0000000000000000 0000000000000000 ffffffff80cf1b44 80000001f61fe180 80000001f61fe180 ... [21981.785956] Call Trace: [21981.788407] [] memset+0x0/0x24 [21981.793031] [] crypto_getreq+0x78/0xd8 [21981.798409] [] ssh_hwaccel_perform_ipsec+0x198/0x5b8 [quicksec] [21981.805968] [] ssh_fastpath_transform_out+0x1034/0x15e0 [quicksec] [21981.813784] [] fastpath_packet_continue+0x674/0x1038 [quicksec] [21981.821343] [] ssh_engine_packet_handler+0x190/0x3e8 [quicksec] [21981.828895] [] ssh_interceptor_vti_packet_out+0x240/0x8f8 [quicksec] [21981.836860] [] vti_xmit_queue_handler+0xf0/0x170 [zld_vti] [21981.843925] [] __tasklet_action+0xac/0x1b8 [21981.849591] [] __do_softirq+0x1c8/0x210 [21981.854998] [] do_softirq+0x68/0x70 [21981.860056] [] local_bh_enable+0xa8/0xb0 [21981.865569] [] fqdnObject_packetQ_reinject+0xc0/0x2c8 [fqdn_object] [21981.873433] [] fqdnObject_work_queue_func+0x1e4/0x390 [fqdn_object] [21981.881283] [] process_one_work+0x178/0x3e0 [21981.887038] [] worker_thread+0x140/0x3c8 [21981.892531] [] kthread+0xa4/0xb0 [21981.897330] [] ret_from_kernel_thread+0x14/0x1c [21981.903429] [21981.904920] Code: 00000000 00000000 00000000 <10a00008> 0080102d 30a500ff 00056a38 00ad2825 00056c38 [21981.914963] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [21981.924035] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [21981.930581] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 0000000000009cb1 000000000000092a ffffffff859a07f8 ffffffff80ec0000 000000000000092a 0000000000000002 0000000000000002 ffffffff859b7858 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e572d0 80000000243e1688 [21981.963917] Call Trace: [21981.966370] [] show_stack+0x6c/0xf8 [21981.971432] [] watchdog_timer_fn+0x24c/0x288 [21981.977275] [] __run_hrtimer.isra.20+0x60/0xf8 [21981.983290] [] hrtimer_interrupt+0x138/0x310 [21981.989130] [] c0_compare_interrupt+0x68/0x98 [21981.995058] [] handle_irq_event_percpu+0x88/0x218 [21982.001334] [] handle_percpu_irq+0x84/0xb8 [21982.007001] [] generic_handle_irq+0x38/0x50 [21982.012756] [] do_IRQ+0x18/0x30 [21982.017470] [] plat_irq_dispatch+0x98/0xb8 [21982.023133] [22001.185162] unpack_message_rr fail ret=-31unpack_message_rr fail ret=-31 [22040.955290] Log Date: 1682633255, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [22040.965929] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [22040.971944] Kernel Info Collector: detect system hanged [Core7], softlockup_hang_count = 10 [22040.980306] BUG: soft lockup - CPU#7 stuck for 31s! [kworker/7:1:1153] [22040.986839] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [22041.287465] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [22041.296544] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [22041.303091] task: 80000001f57cc6e0 ti: 80000001f6e54000 task.ti: 80000001f6e54000 [22041.310581] $ 0 : 0000000000000000 0000000000000000 ffffffff80c0f068 ffffffffffffffff [22041.318651] $ 4 : 8000000180dca900 0000000000000000 0000000000000000 0000000000000000 [22041.326719] $ 8 : ffffffff80c13b00 0000000000000535 0000000000000011 000000000000000d [22041.334788] $12 : 0000000000000001 ffffffff80a489bc ffffffff858d4900 0000000000000110 [22041.342857] $16 : 80000001840af080 0000000000000000 80000001f34773c8 0000000000000001 [22041.350925] $20 : 80000001a2eb6800 80000001eb5549a0 ffffffff858d4900 80000001840af080 [22041.358996] $24 : 0000000000000000 0000000000000012 [22041.367064] $28 : 80000001f6e54000 80000001f6e575c0 ffffffff80eab558 ffffffff80be22f0 [22041.375132] Hi : 00000000000e8ab9 [22041.378710] Lo : 374bc6a7efaef6e9 [22041.382294] epc : ffffffff80c0f068 ipv4_dst_check+0x0/0x38 [22041.387959] Tainted: P C O [22041.391715] ra : ffffffff80be22f0 nf_rtcache_in+0x250/0x320 [22041.397553] Status: 14009ce3 KX SX UX KERNEL EXL IE [22041.402601] Cause : 40808000 [22041.405484] PrId : 000d9703 (Cavium Octeon III) [22041.410105] Modules linked in: fastpath_prearray(PO) zyDPM_ctrl(PO) adt7463(O) option cdc_acm huawei_cdc_ncm cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether sierra usb_wwan usbserial cls_user(O) kbwm(PO) sdwan_kbwm(PO) bonding kuser_info(PO) zy_mss(O) xt_zy_TCPMSS(O) zld_wdt(O) nf_nat_sip(O) nf_conntrack_sip(O) ip_vs_ftp df_discover_srv_module(O) conntrack_flush(O) zld_zydns(PO) zld_secure_dns(PO) sec_dns_genmd5(O) ZyAntiVirus(O) ZyMatcher_HFA(PO) zld_fileidentify(PO) ZyFHM_module(PO) zld_av_module_wbl(PO) zld_zyidp(PO) dpi_mem(PO) zld_utm_action(PO) zld_utm_dispatcher(O) zld_filepump(O) zld_ir_filter(PO) zld_ir_db(PO) zld_cloud_query(O) as_kmodule(O) qsearch_bm(O) qsearch_skeleton(O) qsearch(O) zld_malsite_db(PO) ZyMatcher(PO) zld_vti(PO) zld_ioctl(PO) fqdn_object(PO) zld_ftps_alg_helper(O) nf_nat_ftp(O) nf_conntrack_ftp(O) conn_check(O) trigger_tunnel(PO) path_measure(O) quicksec(PO) ilb_llf(PO) ilb_wrr(PO) ilb_dns(O) zld_adp(PO) zy_adp(PO) cvm_hfa_lib(PO) dns_mgmt_filter_statistics(PO) dns_filter_statistics(PO) zyav_statistics(PO) app_statistics(PO) idp_statistics(PO) xt_dns(O) arpt_proxy(O) xt_zycloud_passthrough(O) xt_zydns_passthrough(O) iptable_zynac(PO) iptable_nat_over_ipsec(O) vpn_concentrator6(O) vpn_concentrator(O) ip6table_vpnid(O) iptable_vpnid(O) zy_ipsec_conn(O) xt_TUNNELID(O) xt_tunnelid(O) xt_zysession_limit(O) xt_zysession_login(O) ta_block(O) xt_zyusergroup(PO) user_group(PO) xt_zyappgroup(PO) app_group(PO) xt_zyzone(PO) ip6table_zyfilter6(O) iptable_zyfilter(O) iptable_sdwan_filter(O) xt_ZYDROP(O) xt_ZYACCEPT(O) xt_ZYFIRE(O) xt_SECURE_POLICY(O) xt_zyislocal(O) xt_asymmetrical_route(O) xt_zyfromlocal(O) policy_reset(O) routing_alive(PO) doll_netdev(PO) klink_updown(O) cfilter_kmodule(PO) zld_rfebl(PO) zld_sslinsp(PO) cryptosoft(O) zld_aead_ciphers(O) zld_dns_filter_post_route_hook_dummy(PO) zld_zyidp_post_proc(PO) zld_utm_manager(PO) zld_cdr(O) configfs_utm(PO) configfs(PO) zld_memory_limiter(PO) ZyFSD(PO) ZyParser_POP3(PO) ZyParser_SMTP(PO) ZyParser_FTP(PO) ZyParser_HTTP(PO) ZyParser(PO) zypktorder(PO) zyinetpkt(O) zld_arp_seal(PO) zld_arp(PO) ipmacbinding(PO) xt_zysso_nonhttptarget(PO) xt_zysso_httptarget(PO) xt_ZYSSO6(PO) xt_ZYSSO(PO) zy_reset(O) hook_zyfrag_ipv6(O) hook_zydefrag_ipv6(O) ip6table_zymark(O) iptable_zymark(O) xt_MARKBWM(O) usb_reset(O) hook_zyping(O) iptable_zyssu(O) xt_zyvpnid_check(PO) xt_zysession_status_update(PO) xt_zydev(PO) xt_BUILTIN_SERVICE(O) zld_forward_hook(O) zld_route_multipath(PO) sdwan_customize_proc(PO) ipt_ZYDNAT(O) ipt_ZYNETMAP(O) ipt_ZYNOLSNAT(O) xt_nat_loopback(O) xt_set(O) ip_set_zyport(O) ip_set_zyip(O) ip_set_list_set(O) ip_set_hash_netportnet(O) ip_set_hash_netport(O) ip_set_hash_netnet(O) ip_set_hash_netiface(O) ip_set_hash_net(O) ip_set_hash_ipportnet(O) ip_set_hash_ipportip(O) ip_set_hash_ipport(O) ip_set_hash_ipmark(O) ip_set_hash_ip(O) ip_set_bitmap_port(O) ip_set_bitmap_ipmac(O) ip_set_bitmap_ip(O) ip_set(O) xt_geoip(O) sslvpn(O) zld_devinet(O) nf_traffic_detect(O) nf_report(O) xt_traffic_flow(O) xt_deviceid(O) devidentify(O) fastpath_kmodule(PO) zld_pkt_manager(PO) zyiface_lib_module(O) zy_bridge_iface(PO) cryptocteon(PO) zld_alg_sip_log(PO) zld_disklog(O) zld_conntrack_data(O) idx_aliasing_disable_kernel(PO) zyklog_kmodule(PO) geoip_database(PO) linux_bcm_diag_full(PO) linux_uk_proxy(PO) linux_kernel_bde(PO) zld_mrd(PO) sw_cn60xx(PO) switchdev_char(PO) switchdev(PO) platform_support(PO) [22041.710716] Process kworker/7:1 (pid: 1153, threadinfo=80000001f6e54000, task=80000001f57cc6e0, tls=0000000000000000) [22041.721336] Stack : 80000001a2eb6800 80000001eb5549a0 000000000000000f 0000000000000000 0000000000000012 0000000000000000 ffffffff80c13b00 0000000000000000 80000001a2eb6800 80000001eb5549a0 ffffffff858d4900 ffffffffc23644c4 ffffffff80eab568 ffffffff80c161b4 0000000000000002 ac14105a00000000 0000000000000000 00160002ac14005a 0000000000000000 0000000000000000 0000001400000003 0000000000000012 0000000000000011 0000000000000000 0000000000000000 0000000000000000 ffffffff80eb0000 0000000000000000 80000001840af080 80000001a2eb6800 0000000000000000 ffffffff80c13b00 80000001f6e577a0 ffffffff80bd2e74 ffffffffc2362230 0000000000000000 0000000000000000 ffffffff80eb2070 ffffffff80000000 ffffffff858d4930 ... [22041.786970] Call Trace: [22041.789424] [] ipv4_dst_check+0x0/0x38 [22041.794746] [] nf_rtcache_in+0x250/0x320 [22041.800307] [] rtcompl_hook+0x1cc/0x4a0 [fastpath_kmodule] [22041.807405] [] nf_iterate+0x13c/0x590 [22041.812639] [] nf_hook_slow+0x8c/0x1f8 [22041.817964] [] ip_rcv+0x2f8/0x410 [22041.822852] [] __netif_receive_skb_core+0x500/0x618 [22041.829308] [] process_backlog+0xa8/0x190 [22041.834894] [] net_rx_action+0x14c/0x228 [22041.840393] [] __do_softirq+0x1c8/0x210 [22041.845799] [] do_softirq+0x68/0x70 [22041.850858] [] local_bh_enable_ip+0xa8/0xb0 [22041.856633] [] fqdnObject_cache_update+0x200/0x348 [fqdn_object] [22041.864227] [] fqdnObject_work_queue_func+0x150/0x390 [fqdn_object] [22041.872074] [] process_one_work+0x178/0x3e0 [22041.877828] [] worker_thread+0x140/0x3c8 [22041.883321] [] kthread+0xa4/0xb0 [22041.888122] [] ret_from_kernel_thread+0x14/0x1c [22041.894221] [22041.895713] Code: a480001e 03e00008 ac800020 <0080102d> 2403fffe 84840066 10830008 3c0380ef 8c440098 [22041.905759] CPU: 7 PID: 1153 Comm: kworker/7:1 Tainted: P C O 3.10.87-rt80-Cavium-Octeon #2 [22041.914827] Workqueue: events fqdnObject_work_queue_func [fqdn_object] [22041.921372] Stack : 0000000000000001 0000000014001ce0 0000000000000000 0000000000000007 0000000000000000 0000000000000000 000000000000a16d 0000000000000972 ffffffff859a07f8 ffffffff80ec0000 0000000000000972 0000000000000002 0000000000000002 ffffffff859ba920 ffffffff859e0000 0000000000000000 ffffffff80eb0428 ffffffff80eb0000 80000001f6e57430 80000000243e1688 0000000000000000 ffffffff858d4398 000000000000001f 000000001dcd6500 0000000000000060 0000000000000001 0000000000000000 3c00000007bf8947 80000001f6e54000 80000001f6e570c0 000014057803c187 ffffffff808f739c 0000000000000004 0000000000000000 0000000000000971 6a6563745f776f72 0000000000000000 ffffffff8087d21c 0000000000000000 6576656e74730000 ... [22041.987008] Call Trace: [22041.989460] [] show_stack+0x6c/0xf8 [22041.994521] [] watchdog_timer_fn+0x24c/0x288 [22042.000365] [] __run_hrtimer.isra.20+0x60/0xf8 [22042.006381] [] hrtimer_interrupt+0x138/0x310 [22042.012222] [] c0_compare_interrupt+0x68/0x98 [22042.018150] [] handle_irq_event_percpu+0x88/0x218 [22042.024426] [] handle_percpu_irq+0x84/0xb8 [22042.030099] [] generic_handle_irq+0x38/0x50 [22042.035858] [] do_IRQ+0x18/0x30 [22042.040573] [] plat_irq_dispatch+0x98/0xb8 [22042.046240] [22075.455290] Log Date: 1682633289, fwVersion=Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57, exception_event=3 [22075.465743] Firmware Version: 5.35(ABIQ.0)|2022-12-23 02:29:57 [22075.471757] Kernel Info Collector: detect system hanged [Core7], store information in disk. [22075.480118] Mem-Info: [22075.482392] DMA32 per-cpu: [22075.485102] CPU 0: hi: 186, btch: 31 usd: 0 [22075.489898] CPU 1: hi: 186, btch: 31 usd: 0 [22075.494694] CPU 2: hi: 186, btch: 31 usd: 0 [22075.499490] CPU 3: hi: 186, btch: 31 usd: 0 [22075.504286] CPU 4: hi: 186, btch: 31 usd: 0 [22075.509082] CPU 5: hi: 186, btch: 31 usd: 0 [22075.513878] CPU 6: hi: 186, btch: 31 usd: 0 [22075.518675] CPU 7: hi: 186, btch: 31 usd: 0 [22075.523470] Normal per-cpu: [22075.526265] CPU 0: hi: 186, btch: 31 usd: 163 [22075.531061] CPU 1: hi: 186, btch: 31 usd: 86 [22075.535857] CPU 2: hi: 186, btch: 31 usd: 93 [22075.540653] CPU 3: hi: 186, btch: 31 usd: 145 [22075.545450] CPU 4: hi: 186, btch: 31 usd: 96 [22075.550246] CPU 5: hi: 186, btch: 31 usd: 9 [22075.555042] CPU 6: hi: 186, btch: 31 usd: 85 [22075.559840] CPU 7: hi: 186, btch: 31 usd: 24 [22075.564642] active_anon:129057 inactive_anon:12375 isolated_anon:0 [22075.564642] active_file:9045 inactive_file:53068 isolated_file:0 [22075.564642] unevictable:6 dirty:0 writeback:0 unstable:0 [22075.564642] free:1462384 slab_reclaimable:10808 slab_unreclaimable:89548 [22075.564642] mapped:20691 shmem:13896 pagetables:2421 bounce:0 [22075.564642] free_cma:0 [22075.597363] DMA32 free:3450144kB min:4848kB low:6060kB high:7272kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3656832kB managed:3382808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [22075.636934] lowmem_reserve[]: 0 4290 4290 [22075.641004] Normal free:2399392kB min:6300kB low:7872kB high:9448kB active_anon:516228kB inactive_anon:49500kB active_file:36180kB inactive_file:212272kB unevictable:24kB isolated(anon):0kB isolated(file):0kB present:4393976kB managed:4393976kB mlocked:24kB dirty:0kB writeback:0kB mapped:82764kB shmem:55584kB slab_reclaimable:43232kB slab_unreclaimable:358192kB kernel_stack:23440kB pagetables:9684kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [22075.684486] lowmem_reserve[]: 0 0 0 [22075.688021] DMA32: 2*4kB (M) 3*8kB (M) 2*16kB (M) 1*32kB (M) 3*64kB (UM) 2*128kB (UM) 3*256kB (UM) 2*512kB (UM) 5*1024kB (UM) 3*2048kB (UM) 839*4096kB (UMR) = 3450144kB [22075.703344] Normal: 552*4kB (UEM) 344*8kB (EM) 165*16kB (UEM) 62*32kB (UM) 572*64kB (UEM) 415*128kB (UM) 141*256kB (UEM) 51*512kB (UM) 13*1024kB (EM) 12*2048kB (UM) 537*4096kB (EMR) = 2398960kB [22075.720846] 76009 total pagecache pages [22075.724687] 0 pages in swap cache [22075.728005] Swap cache stats: add 0, delete 0, find 0/0 [22075.733235] Free swap = 0kB [22075.736117] Total swap = 0kB [22075.875285] 2015232 pages RAM [22075.878256] 67000 pages reserved [22075.881487] 132517 pages shared [22075.884629] 445330 pages non-shared [22076.194326] KIC: Rebooting in 10 seconds... OCTEON eMMC stage 1 bootloader Success. Branching to stage 2 at: 0xFFFFFFFF81004000 U-Boot 2013.07 (Development build, svnversion: u-boot:531M, exec:)-svn531 (Build time: Jun 07 2017 - 18:06:27) Octeon unique ID: 004100008280a59e0256 Now loading stage3 u-boot... MMC read: dev # 0, block # 2048, count 8192 ... 8192 blocks read: OK Now booting stage3 u-boot of ZyXEL... U-Boot 2013.07 (Development build, svnversion: u-boot:531M, exec:)-svn531 (Build time: Jun 07 2017 - 22:45:05) Octeon unique ID: 004100008280a59e0256 BootModule Version: V1.19 | Jun 07 2017 22:45:05 DRAM: Size = 8192 Mbytes Press any key to enter debug mode within 3 seconds. ............................................................. Allocating memory for ELF segment: addr: 0xffffffff80800000 (adjusted to: 0x800000), size 0x521f420 Start to check file system... /dev/mmcblk0p9: 490/61440 files (1.0% non-contiguous), 118740/245760 blocks /dev/mmcblk0p10: 261/131072 files (3.1% non-contiguous), 32311/524288 blocks Done VALIDATE_CERTIFICATE_CHAIN:262 i=1, pkey=0x1088cbc0 Kernel Version: V3.10.87 | 2022-12-23 01:25:22 ZLD Version: V5.35(ABIQ.0) | 2022-12-23 02:29:57 INIT: version 2.86 booting Initializing Debug Account Authentication Seed (DAAS)... done. Setting the System Clock using the Hardware Clock as reference...System Clock set. Local time: Fri Apr 28 10:09:17 UTC 2023 INIT: Entering runlevel: 3 Insmod ZYKLOG Module. Starting zylog daemon: zylogd zylog starts. Starting syslog-ng secu-reporter. Starting ZLD Wrapper Daemon.... Starting uam daemon. Starting myzyxel daemon. Starting periodic command scheduler: cron. Start ZyWALL system daemon.... CDR Daemon Start! Load reputation filter exteral black list Starting link_updown daemon. Check idp signature package Check app signature package [ 66.204137] [zy_adp]load ZyADP Ver1.0.0 OK Check ip reputation signature package Check abn signature package Cloud Query Daemon Start! Check av signature package load av threat info ........... Got LINK_CHANGE Port [1] Copper is up --> Group [1] is up Got LINK_CHANGE Port [3] Copper is up --> Group [3] is up Got LINK_CHANGE Port [5] Copper is up --> Group [5] is up Got LINK_CHANGE Port [6] Copper is up --> Group [6] is up Got LINK_CHANGE Port [2] Copper is up --> Group [2] is up Got LINK_CHANGE Port [4] Copper is up --> Group [4] is up ...........................Applying system configuration file, please wait... . Got LINK_CHANGE Port [1] Copper is down --> Group [1] is down ...... Got LINK_CHANGE Port [2] Copper is down --> Group [2] is down ............ Got LINK_CHANGE Port [3] Copper is down --> Group [3] is down Got LINK_CHANGE Port [4] Copper is down --> Group [4] is down Got LINK_CHANGE Port [5] Copper is down --> Group [5] is down ....... Got LINK_CHANGE Port [6] Copper is down --> Group [6] is down .................................ZyWALL system is configured successfully with startup-config.conf Got LINK_CHANGE Port [1] Copper is up --> Group [1] is up Got LINK_CHANGE Port [2] Copper is up --> Group [2] is up Welcome to ATP800 Username: Got LINK_CHANGE Port [3] Copper is up --> Group [3] is up Got LINK_CHANGE Port [4] Copper is up --> Group [4] is up Got LINK_CHANGE Port [5] Copper is up --> Group [5] is up Got LINK_CHANGE Port [6] Copper is up --> Group [6] is up