My Oracle Support Banner

OCFS2 Error "o2net_send_tcp_msg:962 ERROR: sendmsg returned -13 instead of 104" (Doc ID 2223726.1)

Last updated on AUGUST 04, 2018

Applies to:

Linux OS - Version Oracle Linux 7.1 with Unbreakable Enterprise Kerne [3.8.13] and later
Information in this document applies to any platform.

Symptoms

One node of the OCFS2 cluster gets hung intermittently with below log in the captured vmcore file:

[ 21.268347] virbr0: port 1(virbr0-nic) entered disabled state
[ 21.962581] cgroup: systemd (1) created nested cgroup for controller "devices" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
[ 22.062928] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[ 2736.733116] (updatedb,14952,3):o2net_send_tcp_msg:962 ERROR: sendmsg returned -13 instead of 104
[ 2736.733125] (updatedb,14952,3):dlm_send_remote_convert_request:395 ERROR: Error -13 when sending message 504 (key 0x72696d04) to node 1
[ 2736.733129] (updatedb,14952,3):dlmconvert_remote:325 ERROR: dlm status = DLM_BADARGS
[ 2736.733134] (updatedb,14952,3):__ocfs2_cluster_lock:1480 ERROR: DLM error -22 while calling ocfs2_dlm_lock on resource M000000000000000005e80500000000
[ 2736.733137] (updatedb,14952,3):ocfs2_inode_lock_full_nested:2332 ERROR: status = -22
[ 2736.733139] (updatedb,14952,3):ocfs2_inode_lock_atime:2458 ERROR: status = -22
[ 2736.733141] (updatedb,14952,3):ocfs2_readdir:2034 ERROR: status = -22
[ 2736.733179] (updatedb,14952,3):o2net_send_tcp_msg:962 ERROR: sendmsg returned -13 instead of 96
[ 2736.733182] (updatedb,14952,3):dlm_do_master_request:1379 ERROR: status = -13
[ 2736.733184] (updatedb,14952,3):dlm_do_master_request:1380 ERROR: unhandled error!
[ 2736.733223] ------------[ cut here ]------------
[ 2736.733226] kernel BUG at fs/ocfs2/dlm/dlmmaster.c:1381!
[ 2736.733228] invalid opcode: 0000 [#1] SMP
[ 2736.733231] Modules linked in: tun xt_CHECKSUM ipt_MASQUERADE ocfs2_dlmfs ip6t_rpfilter ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ocfs2_stack_o2cb ocfs2_dlm ocfs2 ocfs2_stackglue ocfs2_nodemanager configfs jbd2 ppdev ghash_clmulni_intel aesni_intel xts aes_x86_64 lrw gf128mul ablk_helper cryptd microcode sg pcspkr i2c_piix4 i2c_core parport_pc parport nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi xen_netfront xen_blkfront crc32c_intel floppy serio_raw
[ 2736.733290] ata_piix dm_mirror dm_region_hash dm_log dm_mod ipv6 autofs4
[ 2736.733297] CPU 3
[ 2736.733301] Pid: 14952, comm: updatedb Not tainted 3.8.13-118.14.1.el7uek.x86_64 #2 Xen HVM domU
[ 2736.733303] RIP: 0010:[<ffffffffa089e9e5>] [<ffffffffa089e9e5>] dlm_do_master_request.isra.18+0x4e5/0x730 [ocfs2_dlm]
[ 2736.733317] RSP: 0018:ffff8809c74f1728 EFLAGS: 00010282
[ 2736.733320] RAX: 0000000000000045 RBX: fffffffffffffff3 RCX: 000000000000d881
[ 2736.733321] RDX: 000000000000d881 RSI: 0000000000000046 RDI: 0000000000000246
[ 2736.733323] RBP: ffff8809c74f1800 R08: ffffffff81c16a02 R09: 000000000000026d
[ 2736.733325] R10: 0000000000000004 R11: ffff8809c74f1566 R12: ffff8809c7527e00
[ 2736.733326] R13: ffff8809c74f1788 R14: ffff880bc497d400 R15: 0000000000000001
[ 2736.733329] FS: 00007fc53f0ac740(0000) GS:ffff880c0fc60000(0000) knlGS:0000000000000000
[ 2736.733331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2736.733333] CR2: 00007fc53f0bf000 CR3: 00000009cdf2e000 CR4: 00000000001406e0
[ 2736.733338] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2736.733340] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2736.733342] Process updatedb (pid: 14952, threadinfo ffff8809c74f0000, task ffff8809c74ee480)
[ 2736.733343] Stack:
[ 2736.733345] fffffffffffffff3 ffff8809c74f1780 ffff880bca532940 ffff880bca5336c0
[ 2736.733350] ffff8809c74eeae0 ffff8809c74ee480 ffff8809cb190998 ffff8809cb190994
[ 2736.733354] ffff8809c74eeae0 ffff8809c74ee480 0000000000000001 00000000ca532960
[ 2736.733358] Call Trace:
[ 2736.733370] [<ffffffffa08a21fa>] dlm_get_lock_resource+0x98a/0x10a0 [ocfs2_dlm]
[ 2736.733378] [<ffffffff812830c2>] ? put_dec+0x72/0x90
[ 2736.733387] [<ffffffffa08ab01f>] ? dlm_new_lock+0x2f/0x140 [ocfs2_dlm]
[ 2736.733396] [<ffffffffa08ab682>] dlmlock+0x552/0x14c0 [ocfs2_dlm]
[ 2736.733428] [<ffffffffa063a7f2>] ? ocfs2_inode_cache_unlock+0x12/0x20 [ocfs2]
[ 2736.733461] [<ffffffffa0688ae9>] ? ocfs2_metadata_cache_unlock+0x19/0x20 [ocfs2]
[ 2736.733489] [<ffffffffa0688baa>] ? ocfs2_buffer_cached.isra.6+0x9a/0x1b0 [ocfs2]
[ 2736.733495] [<ffffffffa08cc020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]
[ 2736.733500] [<ffffffffa08cc000>] ? 0xffffffffa08cbfff
[ 2736.733523] [<ffffffffa063a852>] ? ocfs2_inode_cache_io_unlock+0x12/0x20 [ocfs2]
[ 2736.733550] [<ffffffffa0689159>] ? ocfs2_metadata_cache_io_unlock+0x19/0x20 [ocfs2]
[ 2736.733570] [<ffffffffa061c320>] ? ocfs2_read_blocks+0x350/0x6c0 [ocfs2]
[ 2736.733576] [<ffffffffa08cc2dc>] o2cb_dlm_lock+0x5c/0x80 [ocfs2_stack_o2cb]
[ 2736.733581] [<ffffffffa08cc000>] ? 0xffffffffa08cbfff
[ 2736.733586] [<ffffffffa08cc020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]
[ 2736.733593] [<ffffffffa05be392>] ocfs2_dlm_lock+0x42/0x50 [ocfs2_stackglue]
[ 2736.733613] [<ffffffffa062e98b>] __ocfs2_cluster_lock.isra.31+0x37b/0x830 [ocfs2]
[ 2736.733620] [<ffffffff81285921>] ? vsnprintf+0x411/0x670
[ 2736.733625] [<ffffffff81285c49>] ? snprintf+0x39/0x40
[ 2736.733646] [<ffffffffa062f0cb>] ocfs2_open_lock+0xfb/0x1a0 [ocfs2]
[ 2736.733669] [<ffffffffa063ddcc>] ocfs2_iget+0x45c/0x830 [ocfs2]
[ 2736.733693] [<ffffffffa0649430>] ocfs2_lookup+0xc0/0x300 [ocfs2]
[ 2736.733699] [<ffffffff8119fca8>] ? d_alloc+0x58/0x70
[ 2736.733705] [<ffffffff8119168d>] lookup_real+0x1d/0x50
[ 2736.733710] [<ffffffff81191c43>] __lookup_hash+0x33/0x40
[ 2736.733714] [<ffffffff81577f26>] lookup_slow+0x44/0xa9
[ 2736.733717] [<ffffffff81194bc9>] path_lookupat+0x799/0x7e0
[ 2736.733744] [<ffffffffa0688baa>] ? ocfs2_buffer_cached.isra.6+0x9a/0x1b0 [ocfs2]
[ 2736.733748] [<ffffffff8119327f>] ? getname_flags+0x4f/0x190
[ 2736.733752] [<ffffffff81194c3b>] filename_lookup+0x2b/0xc0
[ 2736.733755] [<ffffffff811980b4>] user_path_at_empty+0x54/0x90
[ 2736.733760] [<ffffffff81047049>] ? default_spin_lock_flags+0x9/0x10
[ 2736.733781] [<ffffffffa062f89b>] ? __ocfs2_cluster_unlock.isra.32+0x8b/0xe0 [ocfs2]
[ 2736.733785] [<ffffffff81198101>] user_path_at+0x11/0x20
[ 2736.733789] [<ffffffff8118caa0>] vfs_fstatat+0x50/0xb0
[ 2736.733794] [<ffffffff811a6a6e>] ? mntput_no_expire+0x3e/0x120
[ 2736.733799] [<ffffffff8118d077>] sys_newlstat+0x27/0x40
[ 2736.733806] [<ffffffff810df3b6>] ? __audit_syscall_exit+0x1f6/0x2a0
[ 2736.733810] [<ffffffff810df16c>] ? __audit_syscall_entry+0x9c/0xf0
[ 2736.733817] [<ffffffff81587a39>] system_call_fastpath+0x16/0x1b
[ 2736.733819] Code: fe ff ff 74 68 83 fb fc 74 63 83 fb e4 74 5e 48 b8 40 02 00 00 00 00 00 10 48 85 05 e6 21 cf ff 74 09 48 85 05 7d 3a cf ff 74 02 <0f> 0b 65 8b 0c 25 1c a0 00 00 65 48 8b 04 25 c0 b7 00 00 8b 90
[ 2736.733863] RIP [<ffffffffa089e9e5>] dlm_do_master_request.isra.18+0x4e5/0x730 [ocfs2_dlm]
[ 2736.733872] RSP <ffff8809c74f1728>
crash64>

 

Changes

 SELinux is in "Enforced" mode.

You can check this out by

# sestatus | grep -i mode
Current mode: enforcing
Mode from config file: permissive

 or

# getenforce

Enforcing

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.