Instance Evicted After LMON to LMON IPC Send timeout Due to Storage Issue
(Doc ID 2080029.1)
Last updated on JUNE 18, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 12.1.0.2 and laterOracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
Second RAC instance gets evicted:
alert_<instance1>.log:
Wed Nov 11 00:49:37 2015
IPC Send timeout detected. Sender: ospid 35407 [oracle@<node1> (LMON)]
Receiver: inst 2 binc 2136868166 ospid 28220
Communications reconfiguration: instance_number 2 by ospid 35407
Wed Nov 11 00:50:38 2015
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Wed Nov 11 00:50:57 2015
Remote instance kill is issued with system inc 5
Remote instance kill map (size 1) : 2
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x20000000
The instance eviction map is 2
Wed Nov 11 00:51:01 2015
Dumping diagnostic data in directory=[cdmp_20151111005101], requested by (instance=2, osid=28232 (LMS0)), summary=[abnormal instance termination].
Wed Nov 11 00:51:01 2015
Reconfiguration started (old inc 3, new inc 7)
List of instances (total 1) :
1
Dead instances (total 1) :
2
My inst 1
IPC Send timeout detected. Sender: ospid 35407 [oracle@<node1> (LMON)]
Receiver: inst 2 binc 2136868166 ospid 28220
Communications reconfiguration: instance_number 2 by ospid 35407
Wed Nov 11 00:50:38 2015
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Wed Nov 11 00:50:57 2015
Remote instance kill is issued with system inc 5
Remote instance kill map (size 1) : 2
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x20000000
The instance eviction map is 2
Wed Nov 11 00:51:01 2015
Dumping diagnostic data in directory=[cdmp_20151111005101], requested by (instance=2, osid=28232 (LMS0)), summary=[abnormal instance termination].
Wed Nov 11 00:51:01 2015
Reconfiguration started (old inc 3, new inc 7)
List of instances (total 1) :
1
Dead instances (total 1) :
2
My inst 1
alert_<instance2>.log
Wed Nov 11 00:51:01 2015
IPC Send timeout detected. Receiver ospid 28220 [oracle@<node2> (LMON)]
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Wed Nov 11 00:51:01 2015
Please check instance 1 alert and LMON trace files for detail.
Please check instance 1 alert and LMON trace files for detail.
Wed Nov 11 00:50:57 2015
Instance Critical Process (pid: 5, ospid: 28182, GEN0) died unexpectedly
Wed Nov 11 00:51:01 2015
System state dump requested by (instance=2, osid=28232 (LMS0)), summary=[abnormal instance termination].
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Please check instance 1 alert and LMON trace files for detail.
Wed Nov 11 00:51:01 2015
LMS0 (ospid: 28232): terminating the instance due to error 481
Wed Nov 11 00:51:01 2015
System State dumped to trace file $TRACE/<instance2>_diag_28196_20151111005101.trc
Wed Nov 11 00:51:03 2015
License high water mark = 74
Wed Nov 11 00:51:06 2015
Instance terminated by LMS0, pid = 28232
Wed Nov 11 00:51:07 2015
USER (ospid: 27395): terminating the instance
Wed Nov 11 00:51:07 2015
Instance terminated by USER, pid = 27395
IPC Send timeout detected. Receiver ospid 28220 [oracle@<node2> (LMON)]
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Wed Nov 11 00:51:01 2015
Please check instance 1 alert and LMON trace files for detail.
Please check instance 1 alert and LMON trace files for detail.
Wed Nov 11 00:50:57 2015
Instance Critical Process (pid: 5, ospid: 28182, GEN0) died unexpectedly
Wed Nov 11 00:51:01 2015
System state dump requested by (instance=2, osid=28232 (LMS0)), summary=[abnormal instance termination].
Wed Nov 11 00:51:01 2015
Received an instance abort message from instance 1
Wed Nov 11 00:51:01 2015
Please check instance 1 alert and LMON trace files for detail.
Wed Nov 11 00:51:01 2015
LMS0 (ospid: 28232): terminating the instance due to error 481
Wed Nov 11 00:51:01 2015
System State dumped to trace file $TRACE/<instance2>_diag_28196_20151111005101.trc
Wed Nov 11 00:51:03 2015
License high water mark = 74
Wed Nov 11 00:51:06 2015
Instance terminated by LMS0, pid = 28232
Wed Nov 11 00:51:07 2015
USER (ospid: 27395): terminating the instance
Wed Nov 11 00:51:07 2015
Instance terminated by USER, pid = 27395
LMON isn't talking to LMON on other node, systemstate shows LMON stuck for 6+ minutes before the instance was evicted:
*** 2015-11-11 00:51:01.589
PROCESS 14: LMON
----------------------------------------
SO: 0x25fc85f70, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x25fc85f70, name=process, file=ksu.h LINE:13949, pg=0 conuid=0
(process) Oracle pid:14, ser:1, calls cur/top: 0x25e50a340/0x25e50a340
....
0: waiting for 'control file sequential read'
file#=0x0, block#=0x23, blocks=0x1
disk number=0x0, au=0x14, offset=0xc000
wait_id=95164407 seq_num=9028 snap_id=1
wait times: snap=6 min 44 sec, exc=6 min 44 sec, total=6 min 44 sec ========>>>> 6+ minutes IO wait
PROCESS 14: LMON
----------------------------------------
SO: 0x25fc85f70, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x25fc85f70, name=process, file=ksu.h LINE:13949, pg=0 conuid=0
(process) Oracle pid:14, ser:1, calls cur/top: 0x25e50a340/0x25e50a340
....
0: waiting for 'control file sequential read'
file#=0x0, block#=0x23, blocks=0x1
disk number=0x0, au=0x14, offset=0xc000
wait_id=95164407 seq_num=9028 snap_id=1
wait times: snap=6 min 44 sec, exc=6 min 44 sec, total=6 min 44 sec ========>>>> 6+ minutes IO wait
Output from "ps" command confirms LMON stuck in IO:
zzz ***Wed Nov 11 00:46:54 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:47:54 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:48:55 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:49:56 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:50:57 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:47:54 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:48:55 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:49:56 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
zzz ***Wed Nov 11 00:50:57 EST 2015
oracle 28220 1 19 0.1 0.0 8770620 81796 blockd D Sep 18 01:31:22 ora_lmon_firsp12
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |