Hi all, I'm attempting to use VMA for the first time with my application. I'm using a Mellanox ConnectX-3 (MCX311A-XCAT) card, The firmware-version is 2.31.5050 and I am using VMA version 6.6.4.0 which was installed on Ubuntu 14.04 via MLNX_OFED_LINUX-2.2-1.0.1-ubuntu14.04-x86_64. After running the application for a while (lets say 45 minutes), the application freezes and I see the kernel reporting soft locks on the CPUs. I do run the app with "chrt -f 70". Not sure if this may be part of the issue? I also recompiled the kernel to disable tickless and subsequently set the hz to 100. Not sure how this would affect VMA. I'd like to figure this out so I can use VMA if anyone has any ideas. Thanks. -- Bud
Here is the dmesg output:
[203642.915230] [sched_delayed] sched: RT throttling activated
[203663.082933] BUG: soft lockup - CPU#0 stuck for 23s! [SessionRx:27477]
[ 13.030306] waiting module removal not supported: please upgrade
[203663.084416] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[203663.084475] CPU: 0 PID: 27477 Comm: SessionRx Tainted: GF O 3.13.9-mykern #1
[203663.084476] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[203663.084478] task: ffff88026f0097f0 ti: ffff880075964000 task.ti: ffff880075964000
[203663.084479] RIP: 0033:[<00007f5d08dce6b6>] [<00007f5d08dce6b6>] 0x7f5d08dce6b6
[203663.084484] RSP: 002b:00007f5cc77fd9e0 EFLAGS: 00000202
[203663.084485] RAX: 0000000000000000 RBX: 000000000000fe2e RCX: 0000000000000001
[203663.084486] RDX: 0000000000000001 RSI: 00007f5cc77fda80 RDI: 00007f5d00050f10
[203663.084487] RBP: 0000000000006b55 R08: 00007f5cbc00a950 R09: 000000000000001d
[203663.084489] R10: 00000000ffffffff R11: 00007f5d00051590 R12: 00007f5d09514940
[203663.084490] R13: ffffffff81715e9d R14: ffff880075965f70 R15: 00007f5cbc00acc8
[203663.084491] FS: 00007f5cc77fe700(0000) GS:ffff88027fc00000(0000) knlGS:0000000000000000
[203663.084493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[203663.084494] CR2: 00007f39dfaa9130 CR3: 00000002718da000 CR4: 00000000001407f0
[203663.084495]
[203663.122967] BUG: soft lockup - CPU#4 stuck for 23s! [mcast_Incremen:27489]
[203663.124504] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[203663.124536] CPU: 4 PID: 27489 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[203663.124537] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[203663.124538] task: ffff88027421c7d0 ti: ffff8802289fa000 task.ti: ffff8802289fa000
[203663.124538] RIP: 0033:[<00007f5d08dce6b6>] [<00007f5d08dce6b6>] 0x7f5d08dce6b6
[203663.124541] RSP: 002b:00007f5c967f59f0 EFLAGS: 00000246
[203663.124542] RAX: 00007f5c880011d0 RBX: 00007f5d0818a970 RCX: 0000000000000001
[203663.124542] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f5d00050f10
[203663.124543] RBP: 0000000000006b61 R08: 00007f5c88000cb0 R09: 0000000000000000
[203663.124544] R10: 00000000ffffffff R11: 00007f5d00051590 R12: 00007f5c967f6700
[203663.124545] R13: 00007f5c88000910 R14: 00007f5d09515360 R15: 00007f5c967f5af0
[203663.124546] FS: 00007f5c967f6700(0000) GS:ffff88027fc80000(0000) knlGS:0000000000000000
[203663.124547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[203663.124547] CR2: 00007f5c85b68380 CR3: 00000002718da000 CR4: 00000000001407e0
[203663.124548]
[203663.152993] BUG: soft lockup - CPU#6 stuck for 23s! [mcast_Incremen:27487]
[203663.154529] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[203663.154576] CPU: 6 PID: 27487 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[203663.154577] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[203663.154579] task: ffff8800360f17f0 ti: ffff88007595a000 task.ti: ffff88007595a000
[203663.154580] RIP: 0033:[<00007f5d08dce8ed>] [<00007f5d08dce8ed>] 0x7f5d08dce8ed
[203663.154582] RSP: 002b:00007f5cb0ff29e0 EFLAGS: 00000297
[203663.154583] RAX: 0000000000000001 RBX: 00007f5d0818a970 RCX: 0000000000000001
[203663.154585] RDX: 0000000000000001 RSI: 00007f5cb0ff2af0 RDI: 00007f5d00050f10
[203663.154586] RBP: 0000000000006b5f R08: 00007f5c9c000cb0 R09: 0000000000000000
[203663.154587] R10: 00000000ffffffff R11: 00007f5d00051590 R12: 00007f5cb0ff3700
[203663.154588] R13: 00007f5c9c000910 R14: 00007f5d09515360 R15: 00007f5cb0ff2af0
[203663.154590] FS: 00007f5cb0ff3700(0000) GS:ffff88027fcc0000(0000) knlGS:0000000000000000
[203663.154591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[203663.154592] CR2: 00007f5cb03511c0 CR3: 00000002718da000 CR4: 00000000001407e0
[203663.154593]
[203783.188367] BUG: soft lockup - CPU#0 stuck for 22s! [mcast_Incremen:27951]
[203783.189906] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[203783.189945] CPU: 0 PID: 27951 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[203783.189946] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[203783.189947] task: ffff880270dadfc0 ti: ffff880216b50000 task.ti: ffff880216b50000
[203783.189948] RIP: 0033:[<00007f862523c8ed>] [<00007f862523c8ed>] 0x7f862523c8ed
[203783.189951] RSP: 002b:00007f85ccff29e0 EFLAGS: 00000297
[203783.189952] RAX: 0000000000000001 RBX: 000000000000fe2e RCX: 0000000000000001
[203783.189953] RDX: 0000000000000001 RSI: 00007f85ccff2af0 RDI: 00007f861c050f10
[203783.189954] RBP: 0000000000006d2f R08: 00007f85b8000cb0 R09: 0000000000000000
[203783.189955] R10: 00000000ffffffff R11: 00007f861c051590 R12: 0000000000006d2f
[203783.189955] R13: ffffffff81715e9d R14: ffff880216b51f70 R15: 00007f85ccff2af0
[203783.189956] FS: 00007f85ccff3700(0000) GS:ffff88027fc00000(0000) knlGS:0000000000000000
[203783.189957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[203783.189958] CR2: 00007f861811a0e0 CR3: 0000000263dde000 CR4: 00000000001407f0
[203783.189959]
[203783.228401] BUG: soft lockup - CPU#4 stuck for 22s! [mcast_Incremen:27953]
[203783.229938] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[203783.229970] CPU: 4 PID: 27953 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[203783.229971] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[203783.229972] task: ffff880228b08000 ti: ffff880216b54000 task.ti: ffff880216b54000
[203783.229973] RIP: 0033:[<00007f862523d7ee>] [<00007f862523d7ee>] 0x7f862523d7ee
[203783.229975] RSP: 002b:00007f85b27f5a38 EFLAGS: 00000246
[203783.229976] RAX: 0000000000006d31 RBX: 0000000000018699 RCX: 0000000000000001
[203783.229976] RDX: 00007f85a4000cb0 RSI: 0000000000000000 RDI: 00007f85a4000cb0
[203783.229977] RBP: 00007f85b27f5af0 R08: 0000000000000000 R09: 0000000000000000
[203783.229978] R10: 0000000000000000 R11: 0000000000000000 R12: 00007f8625983380
[203783.229979] R13: 00007f85a4000910 R14: 0000000000000001 R15: 00007f85b27f5af0
[203783.229980] FS: 00007f85b27f6700(0000) GS:ffff88027fc80000(0000) knlGS:0000000000000000
[203783.229981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[203783.229981] CR2: 00007f85b0364420 CR3: 0000000263dde000 CR4: 00000000001407e0
[203783.229982]
[203783.268437] BUG: soft lockup - CPU#7 stuck for 22s! [SessionRx:27941]
[203783.269916] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[203783.269946] CPU: 7 PID: 27941 Comm: SessionRx Tainted: GF O 3.13.9-mykern #1
[203783.269947] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[203783.269948] task: ffff88027230dfc0 ti: ffff8800368aa000 task.ti: ffff8800368aa000
[203783.269949] RIP: 0033:[<00007f862523c8ed>] [<00007f862523c8ed>] 0x7f862523c8ed
[203783.269951] RSP: 002b:00007f85e37fd9f0 EFLAGS: 00000297
[203783.269952] RAX: 0000000000000001 RBX: 000000000000fe2e RCX: 0000000000000001
[203783.269953] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f861c050f10
[203783.269953] RBP: 0000000000006d25 R08: 00007f85d800a950 R09: 000000000000001d
[203783.269954] R10: 00000000ffffffff R11: 00007f861c051590 R12: 0000000000006d25
[203783.269955] R13: ffffffff81715e9d R14: ffff8800368abf70 R15: 00007f85e37fda80
[203783.269956] FS: 00007f85e37fe700(0000) GS:ffff88027fce0000(0000) knlGS:0000000000000000
[203783.269957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[203783.269957] CR2: 00007f85e401c000 CR3: 0000000263dde000 CR4: 00000000001407e0
[203783.269958]
[214172.308441] BUG: soft lockup - CPU#0 stuck for 22s! [mcast_Snapshot:30334]
[214172.308443] BUG: soft lockup - CPU#1 stuck for 22s! [mcast_Incremen:30333]
[214172.308467] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214172.308483] CPU: 1 PID: 30333 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214172.308484] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214172.308485] task: ffff880273c5c7d0 ti: ffff880204226000 task.ti: ffff880204226000
[214172.308489] RIP: 0033:[<00007f9d85bff8ed>] [<00007f9d85bff8ed>] 0x7f9d85bff8ed
[214172.308490] RSP: 002b:00007f9d257f39e0 EFLAGS: 00000297
[214172.308490] RAX: 0000000000000001 RBX: 00007f9d84fbb970 RCX: 0000000000000001
[214172.308491] RDX: 0000000000000001 RSI: 00007f9d257f3af0 RDI: 00007f9d7c050f10
[214172.308491] RBP: 000000000000767d R08: 00007f9d10000cb0 R09: 0000000000000000
[214172.308492] R10: 00000000ffffffff R11: 00007f9d7c051590 R12: 00007f9d257f4700
[214172.308493] R13: 00007f9d10000910 R14: 00007f9d86346360 R15: 00007f9d257f3b20
[214172.308494] FS: 00007f9d257f4700(0000) GS:ffff88027fc20000(0000) knlGS:0000000000000000
[214172.308494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214172.308495] CR2: 00007f9d403602e0 CR3: 00000002042c0000 CR4: 00000000001407e0
[214172.308495]
[214172.311525] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214172.311575] CPU: 0 PID: 30334 Comm: mcast_Snapshot Tainted: GF O 3.13.9-mykern #1
[214172.311577] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214172.311578] task: ffff88026f758000 ti: ffff880204228000 task.ti: ffff880204228000
[214172.311579] RIP: 0033:[<00007f9d85bff8ed>] [<00007f9d85bff8ed>] 0x7f9d85bff8ed
[214172.311582] RSP: 002b:00007f9d24ff29e0 EFLAGS: 00000297
[214172.311584] RAX: 0000000000000001 RBX: 00007f9d84fbb970 RCX: 0000000000000001
[214172.311585] RDX: 0000000000000001 RSI: 00007f9d24ff2af0 RDI: 00007f9d7c050f10
[214172.311586] RBP: 000000000000767e R08: 00007f9d14000cb0 R09: 0000000000000000
[214172.311587] R10: 00000000ffffffff R11: 00007f9d7c051590 R12: 00007f9d24ff3700
[214172.311588] R13: 00007f9d14000910 R14: 00007f9d86346360 R15: 00007f9d24ff2b20
[214172.311590] FS: 00007f9d24ff3700(0000) GS:ffff88027fc00000(0000) knlGS:0000000000000000
[214172.311591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214172.311592] CR2: 00007f9d74002018 CR3: 00000002042c0000 CR4: 00000000001407f0
[214172.311593]
[214172.378502] BUG: soft lockup - CPU#6 stuck for 22s! [SessionRx:30323]
[214172.379981] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214172.380028] CPU: 6 PID: 30323 Comm: SessionRx Tainted: GF O 3.13.9-mykern #1
[214172.380029] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214172.380031] task: ffff88026be4c7d0 ti: ffff8802625d2000 task.ti: ffff8802625d2000
[214172.380032] RIP: 0033:[<00007f9d85bff8ed>] [<00007f9d85bff8ed>] 0x7f9d85bff8ed
[214172.380035] RSP: 002b:00007f9d43ffe9e0 EFLAGS: 00000297
[214172.380036] RAX: 0000000000000001 RBX: 00007f9d3800a5b0 RCX: 0000000000000001
[214172.380037] RDX: 0000000000000001 RSI: 00007f9d43ffea80 RDI: 00007f9d7c050f10
[214172.380038] RBP: 0000000000007673 R08: 00007f9d3800a950 R09: 000000000000001d
[214172.380039] R10: 00000000ffffffff R11: 00007f9d7c051590 R12: 00007f9d84fbb970
[214172.380040] R13: 00007f9d43fff700 R14: 00007f9d43ffeaa0 R15: 0000000000000001
[214172.380042] FS: 00007f9d43fff700(0000) GS:ffff88027fcc0000(0000) knlGS:0000000000000000
[214172.380043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214172.380044] CR2: 0000000000b3dd70 CR3: 00000002042c0000 CR4: 00000000001407e0
[214172.380045]
[214172.388511] BUG: soft lockup - CPU#7 stuck for 22s! [mcast_Incremen:30335]
[214172.390048] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214172.390097] CPU: 7 PID: 30335 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214172.390098] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214172.390100] task: ffff880273a3c7d0 ti: ffff88020422a000 task.ti: ffff88020422a000
[214172.390101] RIP: 0033:[<00007f9d85bff8ed>] [<00007f9d85bff8ed>] 0x7f9d85bff8ed
[214172.390104] RSP: 002b:00007f9d0eff69f0 EFLAGS: 00000297
[214172.390105] RAX: 0000000000000001 RBX: 0000000000018659 RCX: 0000000000000001
[214172.390106] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f9d7c050f10
[214172.390107] RBP: 000000000000767f R08: 00007f9d08000cb0 R09: 0000000000000000
[214172.390108] R10: 0000000000000000 R11: 0000000000000000 R12: 00007f9d86346380
[214172.390109] R13: 00007f9d08000910 R14: 0000000000000001 R15: 00007f9d0eff6af0
[214172.390111] FS: 00007f9d0eff7700(0000) GS:ffff88027fce0000(0000) knlGS:0000000000000000
[214172.390112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214172.390113] CR2: 00007f9d0c3613c0 CR3: 00000002042c0000 CR4: 00000000001407e0
[214172.390114]
[214224.354130] BUG: soft lockup - CPU#0 stuck for 22s! [mcast_Incremen:30615]
[214224.355668] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214224.355728] CPU: 0 PID: 30615 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214224.355729] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214224.355731] task: ffff88026f7397f0 ti: ffff88026f7ba000 task.ti: ffff88026f7ba000
[214224.355732] RIP: 0033:[<00007f22bfd998ed>] [<00007f22bfd998ed>] 0x7f22bfd998ed
[214224.355737] RSP: 002b:00007f2254fec9f0 EFLAGS: 00000297
[214224.355738] RAX: 0000000000000001 RBX: 000000000000fe2e RCX: 0000000000000001
[214224.355739] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f22b8050f10
[214224.355740] RBP: 0000000000007797 R08: 00007f223c000cb0 R09: 0000000000000000
[214224.355742] R10: 00000000ffffffff R11: 00007f22b8051590 R12: 0000000000000001
[214224.355743] R13: ffffffff81715e9d R14: ffff88026f7bbf70 R15: 0000000000000001
[214224.355744] FS: 00007f2254fed700(0000) GS:ffff88027fc00000(0000) knlGS:0000000000000000
[214224.355746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214224.355747] CR2: 00007f2254ff3000 CR3: 0000000076704000 CR4: 00000000001407f0
[214224.355748]
[214224.374147] BUG: soft lockup - CPU#2 stuck for 22s! [mcast_Incremen:30613]
[214224.375684] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214224.375716] CPU: 2 PID: 30613 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214224.375717] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214224.375718] task: ffff880273f8c7d0 ti: ffff880075ba0000 task.ti: ffff880075ba0000
[214224.375719] RIP: 0033:[<00007f22bfd9a7f4>] [<00007f22bfd9a7f4>] 0x7f22bfd9a7f4
[214224.375721] RSP: 002b:00007f2256ff6a38 EFLAGS: 00000246
[214224.375722] RAX: 0000000000007795 RBX: 000000000000fe2e RCX: 0000000000000001
[214224.375722] RDX: 00007f2244000cb0 RSI: 0000000000000000 RDI: 00007f2244000cb0
[214224.375723] RBP: 00007f22b8038de0 R08: 0000000000000000 R09: 0000000000000000
[214224.375724] R10: 00000000ffffffff R11: 00007f22b8051590 R12: 00007f22b8038de0
[214224.375725] R13: ffffffff81715e9d R14: ffff880075ba1f70 R15: 00007f2256ff6af0
[214224.375726] FS: 00007f2256ff7700(0000) GS:ffff88027fc40000(0000) knlGS:0000000000000000
[214224.375727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214224.375727] CR2: 00007f227c355180 CR3: 0000000076704000 CR4: 00000000001407e0
[214224.375728]
[214224.384155] BUG: soft lockup - CPU#3 stuck for 23s! [mcast_Snapshot:30616]
[214224.385692] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214224.385722] CPU: 3 PID: 30616 Comm: mcast_Snapshot Tainted: GF O 3.13.9-mykern #1
[214224.385723] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214224.385724] task: ffff88026f73dfc0 ti: ffff88026f7bc000 task.ti: ffff88026f7bc000
[214224.385725] RIP: 0033:[<00007f22bfd996b6>] [<00007f22bfd996b6>] 0x7f22bfd996b6
[214224.385727] RSP: 002b:00007f2243ffe9f0 EFLAGS: 00000246
[214224.385728] RAX: 00007f22380011d0 RBX: 000000000000fe2e RCX: 0000000000000001
[214224.385729] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f22b8050f10
[214224.385730] RBP: 0000000000007798 R08: 00007f2238000cb0 R09: 0000000000000000
[214224.385730] R10: 00000000ffffffff R11: 00007f22b8051590 R12: 00007f22b8038de0
[214224.385731] R13: ffffffff81715e9d R14: ffff88026f7bdf70 R15: 0000000000000001
[214224.385732] FS: 00007f2243fff700(0000) GS:ffff88027fc60000(0000) knlGS:0000000000000000
[214224.385733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214224.385734] CR2: 000000000071f8c0 CR3: 0000000076704000 CR4: 00000000001407e0
[214224.385735]
[214224.424190] BUG: soft lockup - CPU#6 stuck for 23s! [mcast_Snapshot:30614]
[214224.425726] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214224.425777] CPU: 6 PID: 30614 Comm: mcast_Snapshot Tainted: GF O 3.13.9-mykern #1
[214224.425778] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214224.425780] task: ffff88026f73afe0 ti: ffff88026f7b8000 task.ti: ffff88026f7b8000
[214224.425781] RIP: 0033:[<00007f22bfd996b6>] [<00007f22bfd996b6>] 0x7f22bfd996b6
[214224.425783] RSP: 002b:00007f22567f59e0 EFLAGS: 00000202
[214224.425784] RAX: 0000000000001000 RBX: 00007f22bf155970 RCX: 0000000000000001
[214224.425786] RDX: 0000000000000001 RSI: 00007f22567f5af0 RDI: 00007f22b8050f10
[214224.425787] RBP: 0000000000007796 R08: 00007f2248000cb0 R09: 0000000000000000
[214224.425788] R10: 00000000ffffffff R11: 00007f22b8051590 R12: 00007f22567f6700
[214224.425789] R13: 00007f2248000910 R14: 00007f22c04e0360 R15: 00007f22567f5b20
[214224.425790] FS: 00007f22567f6700(0000) GS:ffff88027fcc0000(0000) knlGS:0000000000000000
[214224.425792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214224.425793] CR2: 00000000007776b8 CR3: 0000000076704000 CR4: 00000000001407e0
[214224.425794]
[214472.612063] BUG: soft lockup - CPU#4 stuck for 23s! [mcast_Incremen:30719]
[214472.613600] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214472.613660] CPU: 4 PID: 30719 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214472.613662] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214472.613663] task: ffff8802037b47d0 ti: ffff8802037be000 task.ti: ffff8802037be000
[214472.613665] RIP: 0033:[<00007f732066f7ee>] [<00007f732066f7ee>] 0x7f732066f7ee
[214472.613669] RSP: 002b:00007f72b7ffea38 EFLAGS: 00000246
[214472.613670] RAX: 00000000000077ff RBX: 000000000000fe2e RCX: 0000000000000001
[214472.613672] RDX: 00007f72b0000cb0 RSI: 0000000000000000 RDI: 00007f72b0000cb0
[214472.613673] RBP: 00007f72b7ffeaf0 R08: 0000000000000000 R09: 0000000000000000
[214472.613674] R10: 00000000ffffffff R11: 00007f7318051780 R12: 00007f72b7ffeaf0
[214472.613675] R13: ffffffff81715e9d R14: ffff8802037bff70 R15: 0000000000000000
[214472.613677] FS: 00007f72b7fff700(0000) GS:ffff88027fc80000(0000) knlGS:0000000000000000
[214472.613678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214472.613679] CR2: 00007f225c41c198 CR3: 000000027147c000 CR4: 00000000001407e0
[214472.613680]
[214472.632080] BUG: soft lockup - CPU#5 stuck for 23s! [SessionRx:30430]
[214472.633559] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214472.633609] CPU: 5 PID: 30430 Comm: SessionRx Tainted: GF O 3.13.9-mykern #1
[214472.633611] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214472.633612] task: ffff88026fcf2fe0 ti: ffff8800775e6000 task.ti: ffff8800775e6000
[214472.633613] RIP: 0033:[<00007f89a41b06b6>] [<00007f89a41b06b6>] 0x7f89a41b06b6
[214472.633616] RSP: 002b:00007f89667fb9f0 EFLAGS: 00000246
[214472.633617] RAX: 0000000000000000 RBX: 00007f895c00a5b0 RCX: 0000000000000001
[214472.633619] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f899c051120
[214472.633620] RBP: 00000000000076de R08: 00007f895c00a950 R09: 000000000000001d
[214472.633621] R10: 00000000ffffffff R11: 00007f899c051780 R12: 00007f89a356c970
[214472.633622] R13: 00007f89667fc700 R14: 00007f89667fbaa0 R15: 0000000000000001
[214472.633624] FS: 00007f89667fc700(0000) GS:ffff88027fca0000(0000) knlGS:0000000000000000
[214472.633625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214472.633626] CR2: 00007f22bf1dc5e8 CR3: 0000000262729000 CR4: 00000000001407e0
[214472.633627]
[214472.652097] BUG: soft lockup - CPU#7 stuck for 23s! [mcast_Incremen:30440]
[214472.653634] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214472.653684] CPU: 7 PID: 30440 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214472.653685] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214472.653687] task: ffff88026f9997f0 ti: ffff8802627c0000 task.ti: ffff8802627c0000
[214472.653688] RIP: 0033:[<00007f89a41b08ed>] [<00007f89a41b08ed>] 0x7f89a41b08ed
[214472.653691] RSP: 002b:00007f8950ff89e0 EFLAGS: 00000297
[214472.653693] RAX: 0000000000000001 RBX: 00007f89a356c970 RCX: 0000000000000001
[214472.653694] RDX: 0000000000000001 RSI: 00007f8950ff8af0 RDI: 00007f899c051120
[214472.653695] RBP: 00000000000076e8 R08: 00007f8934000cb0 R09: 0000000000000000
[214472.653696] R10: 00000000ffffffff R11: 00007f899c051780 R12: 00007f8950ff9700
[214472.653697] R13: 00007f8934000910 R14: 00007f89a48f7360 R15: 00007f8950ff8b20
[214472.653699] FS: 00007f8950ff9700(0000) GS:ffff88027fce0000(0000) knlGS:0000000000000000
[214472.653700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214472.653701] CR2: 00007fd19791b170 CR3: 0000000262729000 CR4: 00000000001407e0
[214472.653702]
[214472.672115] BUG: soft lockup - CPU#8 stuck for 23s! [mcast_Incremen:30721]
[214472.673652] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214472.673682] CPU: 8 PID: 30721 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214472.673683] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214472.673684] task: ffff880046c08000 ti: ffff880046c10000 task.ti: ffff880046c10000
[214472.673685] RIP: 0033:[<00007f732066e8ed>] [<00007f732066e8ed>] 0x7f732066e8ed
[214472.673687] RSP: 002b:00007f72b5ff49f0 EFLAGS: 00000297
[214472.673688] RAX: 0000000000000001 RBX: 00007f731fa2a970 RCX: 0000000000000001
[214472.673689] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f7318051120
[214472.673690] RBP: 0000000000007801 R08: 00007f72ac000cb0 R09: 0000000000000000
[214472.673690] R10: 00000000ffffffff R11: 00007f7318051780 R12: 00007f72b5ff5700
[214472.673691] R13: 00007f72ac000910 R14: 00007f7320db5360 R15: 00007f72b5ff4b20
[214472.673692] FS: 00007f72b5ff5700(0000) GS:ffff88027fd00000(0000) knlGS:0000000000000000
[214472.673693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214472.673694] CR2: 00007f2270010038 CR3: 000000027147c000 CR4: 00000000001407e0
[214472.673694]
[214472.692132] BUG: soft lockup - CPU#10 stuck for 23s! [mcast_Incremen:30442]
[214472.693687] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214472.693736] CPU: 10 PID: 30442 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214472.693737] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214472.693739] task: ffff88026f75c7d0 ti: ffff8802627c4000 task.ti: ffff8802627c4000
[214472.693740] RIP: 0033:[<00007f89a41b04d4>] [<00007f89a41b04d4>] 0x7f89a41b04d4
[214472.693742] RSP: 002b:00007f89397eda30 EFLAGS: 00000246
[214472.693744] RAX: 0000000000000000 RBX: 00007f89a356c970 RCX: 0000000000000001
[214472.693745] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
[214472.693746] RBP: 00007f89397edaf0 R08: 00007f8930000cb0 R09: 0000000000000000
[214472.693747] R10: 00000000ffffffff R11: 00007f899c051780 R12: 00007f89397ee700
[214472.693748] R13: 00007f8930000910 R14: 00007f89a48f7360 R15: 00007f89397edb20
[214472.693750] FS: 00007f89397ee700(0000) GS:ffff88027fd40000(0000) knlGS:0000000000000000
[214472.693751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214472.693752] CR2: 00007f225c412000 CR3: 0000000262729000 CR4: 00000000001407e0
[214472.693753]
[214472.712150] BUG: soft lockup - CPU#11 stuck for 23s! [SessionRx:30709]
[214472.713647] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214472.713695] CPU: 11 PID: 30709 Comm: SessionRx Tainted: GF O 3.13.9-mykern #1
[214472.713696] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214472.713698] task: ffff8802040f2fe0 ti: ffff8802040fc000 task.ti: ffff8802040fc000
[214472.713699] RIP: 0033:[<00007f732066e8ed>] [<00007f732066e8ed>] 0x7f732066e8ed
[214472.713702] RSP: 002b:00007f72e2ffc9f0 EFLAGS: 00000297
[214472.713703] RAX: 0000000000000001 RBX: 000000000000fe2e RCX: 0000000000000001
[214472.713704] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007f7318051120
[214472.713705] RBP: 00000000000077f5 R08: 00007f72d800a950 R09: 000000000000001d
[214472.713706] R10: 00000000ffffffff R11: 00007f7318051780 R12: 00000000000077f5
[214472.713707] R13: ffffffff81715e9d R14: ffff8802040fdf70 R15: 0000d59300000002
[214472.713709] FS: 00007f72e2ffd700(0000) GS:ffff88027fd60000(0000) knlGS:0000000000000000
[214472.713710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214472.713711] CR2: 00007f225c40a000 CR3: 000000027147c000 CR4: 00000000001407e0
[214472.713712]
[214476.575542] BUG: soft lockup - CPU#0 stuck for 22s! [mcast_Incremen:30615]
[214476.577078] Modules linked in: ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) rdma_ucm(OF) ib_ucm(OF) rdma_cm(OF) iw_cm(OF) ib_ipoib(OF) ib_cm(OF) ib_uverbs(OF) ib_umad(OF) mlx5_ib(OF) mlx5_core(OF) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) mlx4_en(OF) rpcsec_gss_krb5(F) nfsv4(F) 8021q(F) garp(F) stp(F) mrp(F) llc(F) mlx4_core(OF) compat(OF) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) binfmt_misc(F) sunrpc(F) fscache(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) gpio_ich(F) crc32_pclmul(F) ghash_clmulni_intel(F) sb_edac(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) serio_raw(F) edac_core(F) joydev(F) lpc_ich(F) mei_me(F) mei(F) ioatdma(F) lp(F) parport(F) ipmi_si(F) wmi(F) mac_hid(F) vxlan(F) ip_tunnel(F) hid_generic(F) psmouse(F) isci(F) usbhid(F) hid(F) ahci(F) igb(F) i2c_algo_bit(F) libahci(F) dca(F) libsas(F) ptp(F) scsi_transport_sas(F) pps_core(F) [last unloaded: mlx4_core]
[214476.577108] CPU: 0 PID: 30615 Comm: mcast_Incremen Tainted: GF O 3.13.9-mykern #1
[214476.577109] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[214476.577110] task: ffff88026f7397f0 ti: ffff88026f7ba000 task.ti: ffff88026f7ba000
[214476.577111] RIP: 0033:[<00007f22bfd9a7f4>] [<00007f22bfd9a7f4>] 0x7f22bfd9a7f4
[214476.577113] RSP: 002b:00007f2254feca38 EFLAGS: 00000246
[214476.577114] RAX: 0000000000007797 RBX: 000000000000fe2e RCX: 0000000000000001
[214476.577115] RDX: 00007f223c000cb0 RSI: 0000000000000000 RDI: 00007f223c000cb0
[214476.577115] RBP: 00007f22b8038de0 R08: 0000000000000000 R09: 0000000000000000
[214476.577116] R10: 000000000300002a R11: 0000000000000000 R12: 0000000000007797
[214476.577117] R13: ffffffff81715e9d R14: ffff88026f7bbf70 R15: 0000000000000002
[214476.577118] FS: 00007f2254fed700(0000) GS:ffff88027fc00000(0000) knlGS:0000000000000000
[214476.577119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[214476.577120] CR2: 00007f731cef73c0 CR3: 0000000076704000 CR4: 00000000001407f0
[214476.577120]
[214629.299618] INFO: task khugepaged:107 blocked for more than 120 seconds.
[214629.299619] Tainted: GF O 3.13.9-mykern #1
[214629.299620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214629.299622] khugepaged D ffff88027fd343c0 0 107 2 0x00000000
[214629.299625] ffff88026f6fbd08 0000000000000002 ffff88026f6aafe0 ffff88026f6fbfd8
[214629.299627] 00000000000143c0 00000000000143c0 ffff88026f6aafe0 ffff88026f6aafe0
[214629.299630] ffff88026e009fe0 ffff88026e009fe8 0000000000000a01 0000000000000000
[214629.299631] Call Trace:
[214629.299636] [<ffffffff81715a09>] schedule+0x29/0x70
[214629.299638] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214629.299641] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214629.299642] [<ffffffff81718060>] ? down_read+0x20/0x30
[214629.299646] [<ffffffff811a47fb>] khugepaged+0x2bb/0x11b0
[214629.299649] [<ffffffff810a95f0>] ? prepare_to_wait_event+0x100/0x100
[214629.299651] [<ffffffff811a4540>] ? maybe_pmd_mkwrite+0x30/0x30
[214629.299653] [<ffffffff8108aa42>] kthread+0xd2/0xf0
[214629.299655] [<ffffffff8108a970>] ? kthread_create_on_node+0x1d0/0x1d0
[214629.299657] [<ffffffff81721efc>] ret_from_fork+0x7c/0xb0
[214629.299658] [<ffffffff8108a970>] ? kthread_create_on_node+0x1d0/0x1d0
[214629.299670] INFO: task Controller:30598 blocked for more than 120 seconds.
[214629.299671] Tainted: GF O 3.13.9-mykern #1
[214629.299671] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214629.299673] Controller D ffff88027fcb43c0 0 30598 30573 0x00000000
[214629.299675] ffff88026be83b88 0000000000000002 ffff88026f008000 ffff88026be83fd8
[214629.299676] 00000000000143c0 00000000000143c0 ffff88026f008000 ffff88026f008000
[214629.299678] ffff88026e009fe0 ffff88026e009fe8 ffffffff00000000 ffff88026e009ff0
[214629.299678] Call Trace:
[214629.299680] [<ffffffff81715a09>] schedule+0x29/0x70
[214629.299681] [<ffffffff81718605>] rwsem_down_write_failed+0x115/0x230
[214629.299683] [<ffffffff81368df3>] call_rwsem_down_write_failed+0x13/0x20
[214629.299686] [<ffffffff81076583>] ? __sigqueue_free.part.11+0x33/0x40
[214629.299687] [<ffffffff8171809d>] ? down_write+0x2d/0x30
[214629.299690] [<ffffffff81214207>] do_coredump+0x147/0xe90
[214629.299692] [<ffffffff81076b49>] ? __dequeue_signal+0x129/0x210
[214629.299693] [<ffffffff81079817>] get_signal_to_deliver+0x1c7/0x6f0
[214629.299696] [<ffffffff81013438>] do_signal+0x48/0x960
[214629.299698] [<ffffffff81013db9>] do_notify_resume+0x69/0xb0
[214629.299700] [<ffffffff8172226a>] int_signal+0x12/0x17
[214629.299701] INFO: task SessionTx:30604 blocked for more than 120 seconds.
[214629.299702] Tainted: GF O 3.13.9-mykern #1
[214629.299702] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214629.299703] SessionTx D ffff88027fd143c0 0 30604 30573 0x00000000
[214629.299705] ffff880228ad7d58 0000000000000002 ffff880228b0afe0 ffff880228ad7fd8
[214629.299707] 00000000000143c0 00000000000143c0 ffff880228b0afe0 ffff880228b0afe0
[214629.299708] ffff88026e009fe0 ffff88026e009fe8 00007f2270010038 ffff88026e009f80
[214629.299709] Call Trace:
[214629.299710] [<ffffffff81715a09>] schedule+0x29/0x70
[214629.299712] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214629.299713] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214629.299715] [<ffffffff81718060>] ? down_read+0x20/0x30
[214629.299716] [<ffffffff8171d577>] __do_page_fault+0x1d7/0x560
[214629.299718] [<ffffffff8110d97c>] ? acct_account_cputime+0x1c/0x20
[214629.299721] [<ffffffff8109c47b>] ? account_user_time+0x8b/0xa0
[214629.299722] [<ffffffff8109ca94>] ? vtime_account_user+0x54/0x60
[214629.299723] [<ffffffff8171d91a>] do_page_fault+0x1a/0x70
[214629.299725] [<ffffffff81719d88>] page_fault+0x28/0x30
[214629.299727] INFO: task mcast_Incremen:30613 blocked for more than 120 seconds.
[214629.299727] Tainted: GF O 3.13.9-mykern #1
[214629.299727] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214629.299729] mcast_Incremen D ffff88027fc543c0 0 30613 30573 0x00000000
[214629.299730] ffff880075ba1d58 0000000000000002 ffff880273f8c7d0 ffff880075ba1fd8
[214629.299732] 00000000000143c0 00000000000143c0 ffff880273f8c7d0 ffff880273f8c7d0
[214629.299734] ffff88026e009fe0 ffff88026e009fe8 00007f2256ffa000 ffff88026e009f80
[214629.299734] Call Trace:
[214629.299735] [<ffffffff81715a09>] schedule+0x29/0x70
[214629.299737] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214629.299738] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214629.299740] [<ffffffff81718060>] ? down_read+0x20/0x30
[214629.299741] [<ffffffff8171d577>] __do_page_fault+0x1d7/0x560
[214629.299743] [<ffffffff8110d97c>] ? acct_account_cputime+0x1c/0x20
[214629.299744] [<ffffffff8109c47b>] ? account_user_time+0x8b/0xa0
[214629.299746] [<ffffffff8109ca94>] ? vtime_account_user+0x54/0x60
[214629.299747] [<ffffffff8171d91a>] do_page_fault+0x1a/0x70
[214629.299748] [<ffffffff81719d88>] page_fault+0x28/0x30
[214629.299752] INFO: task ps:30787 blocked for more than 120 seconds.
[214629.299753] Tainted: GF O 3.13.9-mykern #1
[214629.299753] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214629.299754] ps D ffff88027fc543c0 0 30787 30786 0x00000000
[214629.299756] ffff8801bddc9cf8 0000000000000002 ffff8801bdd647d0 ffff8801bddc9fd8
[214629.299758] 00000000000143c0 00000000000143c0 ffff8801bdd647d0 ffff8801bdd647d0
[214629.299759] ffff88026e009fe0 ffff88026e009fe8 ffff88005c513000 0000000000000073
[214629.299760] Call Trace:
[214629.299761] [<ffffffff81715a09>] schedule+0x29/0x70
[214629.299763] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214629.299764] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214629.299766] [<ffffffff81718060>] ? down_read+0x20/0x30
[214629.299769] [<ffffffff81176302>] __access_remote_vm+0x42/0x1d0
[214629.299770] [<ffffffff811770e0>] access_process_vm+0x50/0x70
[214629.299772] [<ffffffff8121ecea>] proc_pid_cmdline+0x8a/0x120
[214629.299774] [<ffffffff8122019f>] proc_info_read+0x9f/0xf0
[214629.299776] [<ffffffff811b5eb5>] vfs_read+0x95/0x160
[214629.299778] [<ffffffff811b69c9>] SyS_read+0x49/0xa0
[214629.299780] [<ffffffff817221bf>] tracesys+0xe1/0xe6
[214749.405052] INFO: task khugepaged:107 blocked for more than 120 seconds.
[214749.405053] Tainted: GF O 3.13.9-mykern #1
[214749.405053] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214749.405056] khugepaged D ffff88027fd343c0 0 107 2 0x00000000
[214749.405058] ffff88026f6fbd08 0000000000000002 ffff88026f6aafe0 ffff88026f6fbfd8
[214749.405060] 00000000000143c0 00000000000143c0 ffff88026f6aafe0 ffff88026f6aafe0
[214749.405062] ffff88026e009fe0 ffff88026e009fe8 0000000000000a01 0000000000000000
[214749.405062] Call Trace:
[214749.405069] [<ffffffff81715a09>] schedule+0x29/0x70
[214749.405071] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214749.405074] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214749.405076] [<ffffffff81718060>] ? down_read+0x20/0x30
[214749.405079] [<ffffffff811a47fb>] khugepaged+0x2bb/0x11b0
[214749.405082] [<ffffffff810a95f0>] ? prepare_to_wait_event+0x100/0x100
[214749.405084] [<ffffffff811a4540>] ? maybe_pmd_mkwrite+0x30/0x30
[214749.405086] [<ffffffff8108aa42>] kthread+0xd2/0xf0
[214749.405088] [<ffffffff8108a970>] ? kthread_create_on_node+0x1d0/0x1d0
[214749.405090] [<ffffffff81721efc>] ret_from_fork+0x7c/0xb0
[214749.405091] [<ffffffff8108a970>] ? kthread_create_on_node+0x1d0/0x1d0
[214749.405100] INFO: task Controller:30598 blocked for more than 120 seconds.
[214749.405101] Tainted: GF O 3.13.9-mykern #1
[214749.405101] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214749.405103] Controller D ffff88027fcb43c0 0 30598 30573 0x00000000
[214749.405105] ffff88026be83b88 0000000000000002 ffff88026f008000 ffff88026be83fd8
[214749.405106] 00000000000143c0 00000000000143c0 ffff88026f008000 ffff88026f008000
[214749.405108] ffff88026e009fe0 ffff88026e009fe8 ffffffff00000000 ffff88026e009ff0
[214749.405108] Call Trace:
[214749.405110] [<ffffffff81715a09>] schedule+0x29/0x70
[214749.405111] [<ffffffff81718605>] rwsem_down_write_failed+0x115/0x230
[214749.405113] [<ffffffff81368df3>] call_rwsem_down_write_failed+0x13/0x20
[214749.405115] [<ffffffff81076583>] ? __sigqueue_free.part.11+0x33/0x40
[214749.405117] [<ffffffff8171809d>] ? down_write+0x2d/0x30
[214749.405119] [<ffffffff81214207>] do_coredump+0x147/0xe90
[214749.405121] [<ffffffff81076b49>] ? __dequeue_signal+0x129/0x210
[214749.405123] [<ffffffff81079817>] get_signal_to_deliver+0x1c7/0x6f0
[214749.405126] [<ffffffff81013438>] do_signal+0x48/0x960
[214749.405128] [<ffffffff81013db9>] do_notify_resume+0x69/0xb0
[214749.405129] [<ffffffff8172226a>] int_signal+0x12/0x17
[214749.405131] INFO: task SessionTx:30604 blocked for more than 120 seconds.
[214749.405132] Tainted: GF O 3.13.9-mykern #1
[214749.405132] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214749.405133] SessionTx D ffff88027fd143c0 0 30604 30573 0x00000000
[214749.405135] ffff880228ad7d58 0000000000000002 ffff880228b0afe0 ffff880228ad7fd8
[214749.405137] 00000000000143c0 00000000000143c0 ffff880228b0afe0 ffff880228b0afe0
[214749.405138] ffff88026e009fe0 ffff88026e009fe8 00007f2270010038 ffff88026e009f80
[214749.405139] Call Trace:
[214749.405140] [<ffffffff81715a09>] schedule+0x29/0x70
[214749.405142] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214749.405143] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214749.405145] [<ffffffff81718060>] ? down_read+0x20/0x30
[214749.405146] [<ffffffff8171d577>] __do_page_fault+0x1d7/0x560
[214749.405149] [<ffffffff8110d97c>] ? acct_account_cputime+0x1c/0x20
[214749.405151] [<ffffffff8109c47b>] ? account_user_time+0x8b/0xa0
[214749.405152] [<ffffffff8109ca94>] ? vtime_account_user+0x54/0x60
[214749.405154] [<ffffffff8171d91a>] do_page_fault+0x1a/0x70
[214749.405155] [<ffffffff81719d88>] page_fault+0x28/0x30
[214749.405157] INFO: task StatisticsNotif:30611 blocked for more than 120 seconds.
[214749.405157] Tainted: GF O 3.13.9-mykern #1
[214749.405158] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214749.405159] StatisticsNotif D ffff88027fd343c0 0 30611 30573 0x00000000
[214749.405161] ffff880263f15d58 0000000000000002 ffff880263e197f0 ffff880263f15fd8
[214749.405163] 00000000000143c0 00000000000143c0 ffff880263e197f0 ffff880263e197f0
[214749.405164] ffff88026e009fe0 ffff88026e009fe8 00007f225c41b000 ffff88026e009f80
[214749.405164] Call Trace:
[214749.405166] [<ffffffff81715a09>] schedule+0x29/0x70
[214749.405167] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214749.405169] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214749.405170] [<ffffffff81718060>] ? down_read+0x20/0x30
[214749.405172] [<ffffffff8171d577>] __do_page_fault+0x1d7/0x560
[214749.405173] [<ffffffff8110d97c>] ? acct_account_cputime+0x1c/0x20
[214749.405175] [<ffffffff8109c47b>] ? account_user_time+0x8b/0xa0
[214749.405176] [<ffffffff8109ca94>] ? vtime_account_user+0x54/0x60
[214749.405177] [<ffffffff8171d91a>] do_page_fault+0x1a/0x70
[214749.405179] [<ffffffff81719d88>] page_fault+0x28/0x30
[214749.405180] INFO: task mcast_Incremen:30613 blocked for more than 120 seconds.
[214749.405180] Tainted: GF O 3.13.9-mykern #1
[214749.405181] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214749.405182] mcast_Incremen D ffff88027fc543c0 0 30613 30573 0x00000000
[214749.405184] ffff880075ba1d58 0000000000000002 ffff880273f8c7d0 ffff880075ba1fd8
[214749.405186] 00000000000143c0 00000000000143c0 ffff880273f8c7d0 ffff880273f8c7d0
[214749.405187] ffff88026e009fe0 ffff88026e009fe8 00007f2256ffa000 ffff88026e009f80
[214749.405187] Call Trace:
[214749.405189] [<ffffffff81715a09>] schedule+0x29/0x70
[214749.405190] [<ffffffff81718815>] rwsem_down_read_failed+0xf5/0x150
[214749.405192] [<ffffffff81368dc4>] call_rwsem_down_read_failed+0x14/0x30
[214749.405193] [<ffffffff81718060>] ? down_read+0x20/0x30
[214749.405194] [<ffffffff8171d577>] __do_page_fault+0x1d7/0x560
[214749.405196] [<ffffffff8110d97c>] ? acct_account_cputime+0x1c/0x20
[214749.405197] [<ffffffff8109c47b>] ? account_user_time+0x8b/0xa0
[214749.405198] [<ffffffff8109ca94>] ? vtime_account_user+0x54/0x60
[214749.405200] [<ffffffff8171d91a>] do_page_fault+0x1a/0x70
[214749.405201] [<ffffffff81719d88>] page_fault+0x28/0x30