Instance Evicted After LMON to LMON IPC Send timeout Due to Storage Issue (Doc ID 2080029.1)

Last updated on MAY 03, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

Second RAC instance gets evicted:

alert_firsp11.log:

Wed Nov 11 00:49:37 2015
IPC Send timeout detected. Sender: ospid 35407 [oracle@loma1fpcms008 (LMON)]
Receiver: inst 2 binc 2136868166 ospid 28220
Communications reconfiguration: instance_number 2 by ospid 35407
Wed Nov 11 00:50:38 2015
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Wed Nov 11 00:50:57 2015
Remote instance kill is issued with system inc 5
Remote instance kill map (size 1) : 2
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x20000000
The instance eviction map is 2
Wed Nov 11 00:51:01 2015
Dumping diagnostic data in directory=[cdmp_20151111005101], requested by (instance=2, osid=28232 (LMS0)), summary=[abnormal instance termination].
Wed Nov 11 00:51:01 2015
Reconfiguration started (old inc 3, new inc 7)
List of instances (total 1) :
1
Dead instances (total 1) :
2
My inst 1

 

alert_firsp12.log

Wed Nov 11 00:51:01 2015
IPC Send timeout detected. Receiver ospid 28220 [oracle@loma1fpcms009 (LMON)]
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Wed Nov 11 00:51:01 2015
Please check instance 1 alert and LMON trace files for detail.
Please check instance 1 alert and LMON trace files for detail.
Wed Nov 11 00:50:57 2015
Instance Critical Process (pid: 5, ospid: 28182, GEN0) died unexpectedly
Wed Nov 11 00:51:01 2015
System state dump requested by (instance=2, osid=28232 (LMS0)), summary=[abnormal instance termination].
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Please check instance 1 alert and LMON trace files for detail.
Wed Nov 11 00:51:01 2015
LMS0 (ospid: 28232): terminating the instance due to error 481
Wed Nov 11 00:51:01 2015
System State dumped to trace file /fpcmsp02oma1/oracle/diag/rdbms/firsp1_oma/firsp12/trace/firsp12_diag_28196_20151111005101.trc
Wed Nov 11 00:51:03 2015
License high water mark = 74
Wed Nov 11 00:51:06 2015
Instance terminated by LMS0, pid = 28232
Wed Nov 11 00:51:07 2015
USER (ospid: 27395): terminating the instance
Wed Nov 11 00:51:07 2015
Instance terminated by USER, pid = 27395


LMON isn't talking to LMON on other node, systemstate shows LMON stuck for 6+ minutes before the instance was evicted:

*** 2015-11-11 00:51:01.589
PROCESS 14: LMON
----------------------------------------
SO: 0x25fc85f70, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x25fc85f70, name=process, file=ksu.h LINE:13949, pg=0 conuid=0
(process) Oracle pid:14, ser:1, calls cur/top: 0x25e50a340/0x25e50a340
....
0: waiting for 'control file sequential read'
file#=0x0, block#=0x23, blocks=0x1
disk number=0x0, au=0x14, offset=0xc000
wait_id=95164407 seq_num=9028 snap_id=1
wait times: snap=6 min 44 sec, exc=6 min 44 sec, total=6 min 44 sec ========>>>> 6+ minutes IO wait


Output from "ps" command confirms LMON stuck in IO:

zzz ***Wed Nov 11 00:46:54 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:47:54 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:48:55 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:49:56 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:50:57 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms