Wrong Instance Evicted After Private Network Issue Brought Down HAIP
(Doc ID 2258000.1)
Last updated on APRIL 12, 2024
Applies to:
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.2.0.1 [Release 12.1 to 12.2]Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
12.1 RAC, HAIP failed on node1 but instances on other node got evicted:
ASM1 alert log:
Thu Feb 23 04:07:47 2017
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM1/trace/+ASM1_ora_31813.trc (incident=9918):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 169.xxx.xx.193 not found. Check output from ifconfig command
Thu Feb 23 04:07:47 2017
opiodr aborting process unknown ospid (31813) as a result of ORA-603
....
....
Thu Feb 23 04:09:51 2017
IPC Send timeout detected. Sender: ospid 11096 [oracle@node1 (PING)]
Receiver: inst 1 binc 429475473 ospid 11096
....
Thu Feb 23 04:10:25 2017
IPC Send timeout detected. Sender: ospid 11104 [oracle@node1 (LMS0)]
Receiver: inst 2 binc 435684360 ospid 3057
Thu Feb 23 04:10:30 2017
IPC Send timeout to 2.1 inc 8 for msg type 65518 from opid 13
Thu Feb 23 04:10:30 2017
LMON (ospid: 11100) drops the IMR request from LMS0 (ospid: 11104) because
IMR is in progress and inst 2 is marked bad.
....
Detected an inconsistent instance membership by instance 1
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Thu Feb 23 04:11:45 2017
opidrv aborting process M001 ospid (13004) as a result of ORA-603
Process m001 died, see its trace file
Thu Feb 23 04:11:49 2017
Reconfiguration started (old inc 8, new inc 12)
List of instances (total 1) :
1
Dead instances (total 1) :
2 <<<<<<<<<<<< second instance down
My inst 1
Global Resource Directory frozen
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM1/trace/+ASM1_ora_31813.trc (incident=9918):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 169.xxx.xx.193 not found. Check output from ifconfig command
Thu Feb 23 04:07:47 2017
opiodr aborting process unknown ospid (31813) as a result of ORA-603
....
....
Thu Feb 23 04:09:51 2017
IPC Send timeout detected. Sender: ospid 11096 [oracle@node1 (PING)]
Receiver: inst 1 binc 429475473 ospid 11096
....
Thu Feb 23 04:10:25 2017
IPC Send timeout detected. Sender: ospid 11104 [oracle@node1 (LMS0)]
Receiver: inst 2 binc 435684360 ospid 3057
Thu Feb 23 04:10:30 2017
IPC Send timeout to 2.1 inc 8 for msg type 65518 from opid 13
Thu Feb 23 04:10:30 2017
LMON (ospid: 11100) drops the IMR request from LMS0 (ospid: 11104) because
IMR is in progress and inst 2 is marked bad.
....
Detected an inconsistent instance membership by instance 1
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Thu Feb 23 04:11:45 2017
opidrv aborting process M001 ospid (13004) as a result of ORA-603
Process m001 died, see its trace file
Thu Feb 23 04:11:49 2017
Reconfiguration started (old inc 8, new inc 12)
List of instances (total 1) :
1
Dead instances (total 1) :
2 <<<<<<<<<<<< second instance down
My inst 1
Global Resource Directory frozen
ASM2 alert log:
Thu Feb 23 04:09:54 2017
IPC Send timeout detected. Sender: ospid 3047 [oracle@node1 (PING)] <<<<<<<<<<<<<<<<<
Receiver: inst 1 binc 429475473 ospid 11096
Thu Feb 23 04:10:44 2017
System state dump requested by (instance=2, osid=1865),
summary=[SYSTEMSTATE_GLOBAL: global system state dump request (kjdgdss_g)].
System State dumped to trace file
/u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_diag_3044_20170223041044.trc
Thu Feb 23 04:11:40 2017
Detected an inconsistent instance membership by instance 1
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_3052.trc (incident=3289):
ORA-29740: evicted by instance number 1, group incarnation 10
Incident details in:
/u01/app/grid/diag/diag/asm/+asm/+ASM2/incident/incdir_3289/+ASM2_lmon_3052_i3289.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Feb 23 04:11:45 2017
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_3052.trc:
ORA-29740: evicted by instance number 1, group incarnation 10
USER (ospid: 3052): terminating the instance due to error 29740
IPC Send timeout detected. Sender: ospid 3047 [oracle@node1 (PING)] <<<<<<<<<<<<<<<<<
Receiver: inst 1 binc 429475473 ospid 11096
Thu Feb 23 04:10:44 2017
System state dump requested by (instance=2, osid=1865),
summary=[SYSTEMSTATE_GLOBAL: global system state dump request (kjdgdss_g)].
System State dumped to trace file
/u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_diag_3044_20170223041044.trc
Thu Feb 23 04:11:40 2017
Detected an inconsistent instance membership by instance 1
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_3052.trc (incident=3289):
ORA-29740: evicted by instance number 1, group incarnation 10
Incident details in:
/u01/app/grid/diag/diag/asm/+asm/+ASM2/incident/incdir_3289/+ASM2_lmon_3052_i3289.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Feb 23 04:11:45 2017
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_3052.trc:
ORA-29740: evicted by instance number 1, group incarnation 10
USER (ospid: 3052): terminating the instance due to error 29740
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Cause |
Solution |
References |