Oracle Linux: Kernel Crashes with Call Stack "cdr_ops_rename+0x11/0x20 [cdr]" (Doc ID 2488829.1)

Last updated on MAY 14, 2020

Applies to:

Symptoms

Oracle Linux 6 server crashed and iLOM console shows below call stack:

Sat Nov 10 07:31:40 2018 | HOSTNAME login: [1969911.414856] BUG: unable to handle kernel NULL pointer dereference at (null)
Sat Nov 10 07:31:40 2018 | [1969911.422933] IP: [< (null)>] (null)
Sat Nov 10 07:31:40 2018 | [1969911.428213] PGD 8000001dcfeeb067 PUD 1da80ca067 PMD 0
Sat Nov 10 07:31:40 2018 | [1969911.433592] Oops: 0010 [#1] SMP
Sat Nov 10 07:31:40 2018 | [1969911.437033] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ib_sdp mpt3sas scsi_transport_sas raid_class cdr(POE) nfsd nfs_acl ipmi_poweroff ipmi_devintf rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs fscache lockd sunrpc grace bonding rds_rdma rds ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad mlx4_ib mlx4_core dm_multipath bnx2i cnic uio cxgb4i libcxgbi cxgb4 ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipv6 ext4 jbd2 mbcache2 fuse iTCO_wdt iTCO_vendor_support ipmi_ssif ipmi_si ipmi_msghandler acpi_cpufreq sb_edac edac_core i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ixgbe dca ptp pps_core vxlan udp_tunnel ip6_udp_tunnel mdio ext3 jbd mbcache sd_mod ahci libahci megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
Sat Nov 10 07:31:40 2018 | [1969911.510057] CPU: 9 PID: 24904 Comm: java Tainted: P OE 4.1.12-94.7.8.el6uek.x86_64 #2
Sat Nov 10 07:31:40 2018 | [1969911.519111] Hardware name: Oracle Corporation SUN SERVER X4-2 /ASSY,MB,X4-2, 1U , BIOS 25050100 09/19/2017
Sat Nov 10 07:31:40 2018 | [1969911.529908] task: ffff881e9c5a5400 ti: ffff881d39574000 task.ti: ffff881d39574000
Sat Nov 10 07:31:40 2018 | [1969911.537569] RIP: 0010:[] [< (null)>] (null)
Sat Nov 10 07:31:40 2018 | [1969911.545719] RSP: 0018:ffff881d395778e0 EFLAGS: 00010292
Sat Nov 10 07:31:40 2018 | [1969911.551440] RAX: ffff880cf74c1e00 RBX: ffff881da03226e0 RCX: ffff880e02182f00
Sat Nov 10 07:31:40 2018 | [1969911.559209] RDX: ffff881da03226e0 RSI: ffff880e0ed4b140 RDI: ffff881da03226e0
Sat Nov 10 07:31:40 2018 | [1969911.566992] RBP: ffff881d39577c08 R08: 0000000000000000 R09: 0000000000000000
Sat Nov 10 07:31:40 2018 | [1969911.574769] R10: 0000000000000000 R11: ffffffffa00dc328 R12: ffff8812f5f09800
Sat Nov 10 07:31:40 2018 | [1969911.582552] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880e0ed4b140
Sat Nov 10 07:31:40 2018 | [1969911.590282] FS: 00007f8e4d6d6700(0000) GS:ffff88407e840000(0000) knlGS:0000000000000000
Sat Nov 10 07:31:40 2018 | [1969911.598987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sat Nov 10 07:31:40 2018 | [1969911.605125] CR2: 0000000000000000 CR3: 0000001d9ce72000 CR4: 0000000000160670
Sat Nov 10 07:31:40 2018 | [1969911.612880] Stack:
Sat Nov 10 07:31:40 2018 | [1969911.615282] ffffffffa0787a00 ffff880fab4695b0 ffff881589f669c0 ffff880e1a159c30
Sat Nov 10 07:31:40 2018 | [1969911.623473] ffff88122952ebc8 ffff881da03226e0 ffff880e02182f00 ffff881d5fca1958
Sat Nov 10 07:31:40 2018 | [1969911.631593] ffff880f865a4ea0 ffff8811086e37b8 ffff880e1aff23a8 ffff8812297bc208
Sat Nov 10 07:31:40 2018 | [1969911.639706] Call Trace:
Sat Nov 10 07:31:40 2018 | [1969911.642598] [] ? cdr_ops_rename_flags+0x70/0xa20 [cdr]
Sat Nov 10 07:31:40 2018 | [1969911.649811] [] ? __find_get_block+0xfe/0x140
Sat Nov 10 07:31:40 2018 | [1969911.656148] [] ? __getblk_gfp+0x3a/0x80
Sat Nov 10 07:31:40 2018 | [1969911.662063] [] ? ext3_getblk+0xf0/0x270 [ext3]
Sat Nov 10 07:31:40 2018 | [1969911.668577] [] ? ext3fs_dirhash+0x120/0x270 [ext3]
Sat Nov 10 07:31:40 2018 | [1969911.675433] [] ? dx_release+0x28/0x50 [ext3]
Sat Nov 10 07:31:40 2018 | [1969911.681786] [] ? ext3_xattr_get+0x63/0xa0 [ext3]
Sat Nov 10 07:31:40 2018 | [1969911.688469] [] ? ext3_xattr_security_get+0x2f/0x40 [ext3]
Sat Nov 10 07:31:40 2018 | [1969911.696148] [] ? generic_getxattr+0x83/0x90
Sat Nov 10 07:31:40 2018 | [1969911.702377] [] ? get_vfs_caps_from_disk+0x62/0xe0
Sat Nov 10 07:31:40 2018 | [1969911.709172] [] ? ext3_find_entry+0x39b/0x490 [ext3]
Sat Nov 10 07:31:40 2018 | [1969911.716129] [] ? ext3_xattr_get+0x63/0xa0 [ext3]
Sat Nov 10 07:31:40 2018 | [1969911.722801] [] ? audit_copy_inode+0x5e/0x90
Sat Nov 10 07:31:40 2018 | [1969911.729030] [] cdr_ops_rename+0x11/0x20 [cdr]
Sat Nov 10 07:31:40 2018 | [1969911.735440] [] fsf_ops_rename+0x70/0x2c0 [cdr] <============== this is 3rd-party kernel module 'cdr'
Sat Nov 10 07:31:40 2018 | [1969911.741947] [] vfs_rename+0x5f0/0x680
Sat Nov 10 07:31:40 2018 | [1969911.747675] [] SYSC_renameat2+0x47c/0x500
Sat Nov 10 07:31:40 2018 | [1969911.753761] [] ? __call_rcu_nocb_enqueue+0xd7/0xe0
Sat Nov 10 07:31:40 2018 | [1969911.760624] [] ? fsf_ops_release+0x5e/0xa0 [cdr]
Sat Nov 10 07:31:40 2018 | [1969911.767291] [] ? __audit_syscall_entry+0xac/0x110
Sat Nov 10 07:31:40 2018 | [1969911.774066] [] ? do_audit_syscall_entry+0x6c/0x70
Sat Nov 10 07:31:41 2018 | [1969911.780821] [] ? syscall_trace_enter_phase1+0x153/0x180
Sat Nov 10 07:31:41 2018 | [1969911.788105] [] SyS_renameat2+0xe/0x10
Sat Nov 10 07:31:41 2018 | [1969911.793838] [] SyS_rename+0x1e/0x20
Sat Nov 10 07:31:41 2018 | [1969911.799362] [] system_call_fastpath+0x12/0xce
Sat Nov 10 07:31:41 2018 | [1969911.805774] Code: Bad RIP value.
Sat Nov 10 07:31:41 2018 | [1969911.809584] RIP [< (null)>] (null)
Sat Nov 10 07:31:41 2018 | [1969911.815142] RSP
Sat Nov 10 07:31:41 2018 | [1969911.819020] CR2: 0000000000000000

From system log, kernel is tainted by module 'cdr':

kernel: [ 258.660456] cdr: module license 'CommVault Systems' taints kernel.
kernel: [ 258.660460] Disabling lock debugging due to kernel taint
kernel: [ 258.660559] cdr: module verification failed: signature and/or required key missing - tainting kernel
kernel: [ 258.663656] 34940: fsf_init(): Initializing the driver.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.