My Oracle Support Banner

Oracle Linux Node Eviction during Updatedb File indexation" COMMAND: "cssdagent" SysRq : Trigger a crash " (Doc ID 2673083.1)

Last updated on MAY 26, 2020

Applies to:

Linux OS - Version Oracle Linux 6.0 and later
Information in this document applies to any platform.
Linux x86-64

Symptoms

Reported the node crash and System generated vmcore, vmcore shows on log the eviction
UPTIME: 219 days, 03:20:24
LOAD AVERAGE: 27.22, 24.33, 20.51
TASKS: 2468
NODENAME: <HOSTNAME>
RELEASE: 4.1.12-94.8.4.el6uek.x86_64
VERSION: #2 SMP Sat May 5 16:14:51 PDT 2018
MACHINE: x86_64 (2693 Mhz)
MEMORY: 256 GB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 159362
COMMAND: "cssdagent"
TASK: ffff881e22486200 [THREAD_INFO: ffff8808e2da8000]
CPU: 25
STATE: TASK_RUNNING (SYSRQ)
[18932925.875474] sysrq: SysRq : Trigger a crash
[18932925.879961] BUG: unable to handle kernel NULL pointer dereference at (null)
[18932925.888097] IP: [<ffffffff81415cc6>] sysrq_handle_crash+0x16/0x20
[18932925.894487] PGD 80000003d23da067 PUD 2b38a2067 PMD 0
[18932925.899882] Oops: 0002 [#1] SMP

[18932926.487105] megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: oracleoks]
[18932926.494701] CPU: 25 PID: 159362 Comm: cssdagent Tainted: P OEZ 4.1.12-94.8.4.el6uek.x86_64 #2
[18932926.504936] Hardware name: Oracle Corporation SUN SERVER X4-2 /ASSY,MB,X4-2, 1U , BIOS 25060400 05/28/2018
[18932926.516212] task: ffff881e22486200 ti: ffff8808e2da8000 task.ti: ffff8808e2da8000
[18932926.524360] RIP: 0010:[<ffffffff81415cc6>] [<ffffffff81415cc6>] sysrq_handle_crash+0x16/0x20
[18932926.533574] RSP: 0018:ffff8808e2dabdb8 EFLAGS: 00010292
[18932926.539360] RAX: ffffffff81415cb0 RBX: 0000000000000063 RCX:000000000000000f
[18932926.547163] RDX: ffff883f7de55138 RSI: ffff883f7de517c8 RDI:0000000000000063
[18932926.554973] RBP: ffff8808e2dabdb8 R08: 0000000000000dd0 R09:0000000000004238
[18932926.562782] R10: 0000000000000092 R11: 0000000000000002 R12:0000000000000004
[18932926.570591] R13: 0000000000000000 R14: ffffffff81b61060 R15:0000000000000000
[18932926.578399] FS: 00007f6fbab47700(0000) GS:ffff883f7de40000(0000) knlGS:0000000000000000
[18932926.587162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18932926.593381] CR2: 0000000000000000 CR3: 000000180d6b0000 CR4: 0000000000160670
[18932926.601188] Stack:
[18932926.603675] ffff8808e2dabde8 ffffffff814164db 0000000000000001 fffffffffffffffb
[18932926.611825] 00007f6fc424e194 0000000000000001 ffff8808e2dabe08 ffffffff81416983
[18932926.619978] 0000000000000000 ffff883f5bcc06c0 ffff8808e2dabe28 ffffffff812733cb
[18932926.628124] Call Trace:
[18932926.631054] [<ffffffff814164db>] __handle_sysrq+0x10b/0x170
[18932926.637193] [<ffffffff81416983>] write_sysrq_trigger+0x33/0x40
[18932926.643590] [<ffffffff812733cb>] proc_reg_write+0x4b/0x70
[18932926.649553] [<ffffffff81203dab>] __vfs_write+0x2b/0x110
[18932926.655341] [<ffffffff81206af9>] ? __sb_start_write+0x49/0x100
[18932926.661741] [<ffffffff812abb49>] ? security_file_permission+0x29/0xb0
[18932926.668751] [<ffffffff81204509>] vfs_write+0xa9/0x1b0
[18932926.674364] [<ffffffff816ade66>] ? mutex_lock+0x16/0x40
[18932926.680154] [<ffffffff81205316>] SyS_write+0x46/0xb0
[18932926.685683] [<ffffffff816b025f>] ? system_call_after_swapgs+0xe9/0x18c
[18932926.692774] [<ffffffff816b0258>] ? system_call_after_swapgs+0xe2/0x18c
[18932926.699872] [<ffffffff816b0251>] ? system_call_after_swapgs+0xdb/0x18c
[18932926.706963] [<ffffffff816b031a>] system_call_fastpath+0x18/0xd4
[18932926.713447] Code: ef e8 df f7 ff ff eb d8 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 55 48 89 e5 0f 1f 44 00 00 c7 05 d1 64 68 00 01 00 00 00 0f ae f8
<c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 53 48 83 ec 08 0f 1f
[18932926.734790] RIP [<ffffffff81415cc6>] sysrq_handle_crash+0x16/0x20
[18932926.741475] RSP <ffff8808e2dabdb8>
[18932926.745431] CR2: 0000000000000000

On the system log there is nothing on Messages other than the restart event:
Feb 25 03:07:09 <HOSTNAME> Oracle Audit[27682]: LENGTH : '425' ACTION :[174] 'select dbname,SUM(READ_TIME),SUM(WRITE_TIME),SUM(READS),SUM(WRITES),SUM(READ_ERRS),SU M(WRITE_ERRS),SUM(BYTES_READ),SUM(BYTES_W RITTEN) from gv$asm_disk_iostat group by dbname
Feb 25 03:07:09 <HOSTNAME> Oracle Audit[27682]: LENGTH : '443' ACTION :[192] 'select dbname,instname,SUM(READ_TIME),SUM(WRITE_TIME),SUM(READS),SUM(WRITES),SUM(READ _ERRS),SUM(WRITE_ERRS),SUM(BYTES_READ),SUM(BYTES_WRITTEN) from gv$asm_disk_iostat group by dbname,instname
Feb 25 03:07:15 <HOSTNAME> Oracle Audit[27682]: LENGTH : '425' ACTION :[174] 'select dbname,SUM(READ_TIME),SUM(WRITE_TIME),SUM(READS),SUM(WRITES),SUM(READ_ERRS),SU
Feb 25 03:16:54 <HOSTNAME> kernel: imklog 5.8.10, log source = /proc/kmsg started.
Feb 25 03:16:54 <HOSTNAME> rsyslogd: [origin software="rsyslogd"swVersion="5.8.10" x-pid="15490" x-info="http://www.rsyslog.com"] start
Feb 25 03:16:54 <HOSTNAME> kernel: [ 0.000000] Initializing cgroup subsys cpuset
Feb 25 03:16:54 <HOSTNAME> kernel: [ 0.000000] Initializing cgroup subsys cpu
Feb 25 03:16:54 <HOSTNAME> kernel: [ 0.000000] Initializing cgroup subsys cpuacct
Feb 25 03:16:54 <HOSTNAME> kernel: [ 0.000000] Linux version 4.1.12-94.8.4.el6uek.x86_64 (mockbuild@x86-ol6-builder-04) (gcc version 4.9.220150212 (Red Hat 4.9.2-6.2.0.3) (GCC) ) #2 SMP Sat May 5 16:1 4:51 PDT 2018
Feb 25 03:16:54 <HOSTNAME> kernel: [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.1.12-94.8.4.el6uek.x86_64 root=LABEL=DBSYS bootarea=dbsys bootfrom=BOOT ro loglevel=7 panic=60 debug pci=noaer log_buf
_len=1m nmi_watchdog=0 transparent_hugepage=never rd_NO_PLYMOUTH audit=1 console=tty1 console=ttyS0,115200n8 crashkernel=448M@128M numa=off
Feb 25 03:16:54 <HOSTNAME> kernel: [ 0.000000] e820: BIOS-provided physical RAM map:
Feb 25 03:16:54 <HOSTNAME> kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009abff] usable

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.