Wrong Instance Evicted After Private Network Issue Brought Down HAIP (Doc ID 2258000.1)

Last updated on JULY 05, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.2.0.1 [Release 12.1 to 12.2]
Information in this document applies to any platform.

Symptoms

12.1 RAC, HAIP failed on node1 but instances on other node got evicted:

ASM1 alert log:

Thu Feb 23 04:07:47 2017
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM1/trace/+ASM1_ora_31813.trc (incident=9918):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 169.254.74.193 not found. Check output from ifconfig command
Thu Feb 23 04:07:47 2017
opiodr aborting process unknown ospid (31813) as a result of ORA-603
....
....
Thu Feb 23 04:09:51 2017
IPC Send timeout detected. Sender: ospid 11096 [oracle@usc1dolps21p (PING)]
Receiver: inst 1 binc 429475473 ospid 11096
....
Thu Feb 23 04:10:25 2017
IPC Send timeout detected. Sender: ospid 11104 [oracle@usc1dolps21p (LMS0)]
Receiver: inst 2 binc 435684360 ospid 3057
Thu Feb 23 04:10:30 2017
IPC Send timeout to 2.1 inc 8 for msg type 65518 from opid 13
Thu Feb 23 04:10:30 2017
LMON (ospid: 11100) drops the IMR request from LMS0 (ospid: 11104) because
IMR is in progress and inst 2 is marked bad.
....
Detected an inconsistent instance membership by instance 1
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Thu Feb 23 04:11:45 2017
opidrv aborting process M001 ospid (13004) as a result of ORA-603
Process m001 died, see its trace file
Thu Feb 23 04:11:49 2017
Reconfiguration started (old inc 8, new inc 12)
List of instances (total 1) :
1
Dead instances (total 1) :
2                                 <<<<<<<<<<<< second instance down
My inst 1
Global Resource Directory frozen

ASM2 alert log:

Thu Feb 23 04:09:54 2017
IPC Send timeout detected. Sender: ospid 3047 [oracle@usc1dolps22p (PING)]  <<<<<<<<<<<<<<<<<
Receiver: inst 1 binc 429475473 ospid 11096
Thu Feb 23 04:10:44 2017
System state dump requested by (instance=2, osid=1865),
summary=[SYSTEMSTATE_GLOBAL: global system state dump request (kjdgdss_g)].
System State dumped to trace file
/u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_diag_3044_20170223041044.trc
Thu Feb 23 04:11:40 2017
Detected an inconsistent instance membership by instance 1
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_3052.trc (incident=3289):
ORA-29740: evicted by instance number 1, group incarnation 10
Incident details in:
/u01/app/grid/diag/diag/asm/+asm/+ASM2/incident/incdir_3289/+ASM2_lmon_3052_i3289.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Feb 23 04:11:45 2017
Errors in file /u01/app/grid/diag/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_3052.trc:
ORA-29740: evicted by instance number 1, group incarnation 10
USER (ospid: 3052): terminating the instance due to error 29740

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms