Solaris: orarootagent: "empty ip address" and RAC Database/ASM Instances Evicts (Doc ID 1602821.1)

Last updated on MAY 06, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.2 to 11.2.0.4 [Release 11.2]
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)

Symptoms


11gR2 RAC on Solaris, ASM/Database instance evicts after the following is logged:

 

2013-06-18 07:20:06.795: [ CLSINET][16] (:CLSINE0015:)failed to determine IP list for mac '90-e2-ba-15-33-ec', ret 14
2013-06-18 07:20:07.090: [ GIPCNET][16] gipcmodNetworkAttrAddrOsd: failed to update name 'ipMask', addr 10223ee10 [0000000008b68f94] { gipcAddress : name '(invalid)', objFlags 0x0, addrFlags 0x3 }
2013-06-18 07:20:07.090: [ GIPCNET][16] gipcmodNetworkAttrAddrOsd: slos op : sgipcnRawAttributeFunc
2013-06-18 07:20:07.090: [ GIPCNET][16] gipcmodNetworkAttrAddrOsd: slos dep : Error 0 (0)
2013-06-18 07:20:07.090: [ GIPCNET][16] gipcmodNetworkAttrAddrOsd: slos loc : sgipcnRawFin
2013-06-18 07:20:07.090: [ GIPCNET][16] gipcmodNetworkAttrAddrOsd: slos info: empty ip address
2013-06-18 07:20:07.090: [GIPCXCPT][16] gipcInternalGetAttribute: failed during gipcInternalGetAttribute, ret gipcretFail (1)
2013-06-18 07:20:07.090: [GIPCXCPT][16] gipcGetAttributeStringF [clsinet_IntGetIpInformation : clsinet.c : 4936]: EXCEPTION[ ret gipcretFail (1) ] failure for obj 0000000008b68f94, name 'ipMask', val ffffffff7b9f9c88, len 1024, flags 0x4000
2013-06-18 07:20:07.090: [ CLSINET][16] (:CLSINE0014:)failed to get IP information for mac '90-e2-ba-15-33-ed', ip '1.1.1.2', ret 1
2013-06-18 07:20:07.109: [ USRTHRD][16] {0:0:2} HAIP: Updating member info HAIP1;1.1.1.0#0
2013-06-18 07:20:07.122: [ USRTHRD][16] {0:0:2} HAIP: Moving ip '169.254.41.84' from inf 'net2' to inf 'net4'

 

net2: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 4 
inet 1.1.1.2 netmask ffffff00 broadcast 1.1.1.255 
ether 90:e2:ba:15:33:ed                                                                                               ====>> 6-byte MAC address
net2:1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 4 
inet 169.254.53.210 netmask ffff8000 broadcast 169.254.127.255
....

 

2013-06-04 19:30:03.907: [ CLSINET][7] Returning NETDATA: 3 interfaces
2013-06-04 19:30:03.907: [ CLSINET][7] # 0 Interface 'net2',ip='1.1.1.2',mac='90-e2-ba-15-33-ed-00-00-00-00-00-00-00-00-00-00-00-2f-ff-ff',mask='255.255.255.0',net='1.1.1.0',use='cluster_interconnect'    ====>> 20-byte MAC

 

Private Interface 'net2:1' configured from GPnP for use as a private interconnect.
[name='net2:1', type=1, ip=169.254.53.210, mac=90-e2-ba-15-33-ed-00-00-00-00-00-00-00-00-00-00-00-2f-ff-ff, net=169.254.0.0/17, mask=255.255.128.0, use=haip:cluster_interconnect/62]    ====>> 20-byte MAC
....
Thu Sep 12 11:35:55 2013
IPC Send timeout detected. Receiver ospid 474 [oracle@bmblrac01 (LMS6)]
Thu Sep 12 11:35:55 2013
Errors in file /u01/app/oracle/diag/rdbms/db5/DB51/trace/DB51_lms6_474.trc:
Thu Sep 12 11:36:14 2013
IPC Send timeout detected. Sender: ospid 446 [oracle@bmblrac01 (LMON)]
Receiver: inst 2 binc 196006589 ospid 9046
Errors in file /u01/app/oracle/diag/rdbms/db5/DB51/trace/DB51_lmon_446.trc:
ORA-27508: IPC error sending a message
Detected an inconsistent instance membership by instance 2
Evicting instance 2 from cluster

 

 

 Other symptoms can include the following:

 

2013-03-18 12:44:45.973
[/oracle/app/11.2.0/grid/bin/orarootagent.bin(3134)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.0.0 / 255.255.128.0 / 169.254.27.7 / e1000g0:1

 

Mon Mar 18 12:51:51 2013
IPC Send timeout detected. Receiver ospid 28471 [oracle@sasmwrtrsdp03 (LMD0)]
Mon Mar 18 12:51:51 2013
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmd0_28471.trc:
Mon Mar 18 12:53:35 2013
Detected an inconsistent instance membership by instance 2
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Mon Mar 18 12:53:36 2013
Dumping diagnostic data in directory=[cdmp_20130318125336], requested by (instance=2, osid=3690 (LMD0)), summary=[abnormal instance termination].
Mon Mar 18 12:53:55 2013
Remote instance kill is issued with system inc 6
Remote instance kill map (size 1) : 2
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x20000000
The instance eviction map is 2
Mon Mar 18 12:54:05 2013
NOTE: client rtrprd1:rtrprd registered, osid 1472, mbr 0x2
Waiting for instances to leave: 2
Mon Mar 18 12:54:36 2013
Waiting for instances to leave: 2
Mon Mar 18 12:55:05 2013
Waiting for instances to leave: 2
Mon Mar 18 12:55:17 2013
Reconfiguration started (old inc 4, new inc 8)

  

Mon Mar 18 12:51:50 2013
IPC Send timeout detected. Sender: ospid 3690 [oracle@sasmwrtrsdp01 (LMD0)]
Receiver: inst 1 binc 56990 ospid 28471
IPC Send timeout to 1.0 inc 4 for msg type 65521 from opid 10
Mon Mar 18 12:51:52 2013
Communications reconfiguration: instance_number 1
Mon Mar 18 12:53:35 2013
Detected an inconsistent instance membership by instance 2
Mon Mar 18 12:53:35 2013
Received an instance abort message from instance 1
Please check instance 1 alert and LMON trace files for detail.
Mon Mar 18 12:53:35 2013
Received an instance abort message from instance 1
Please check instance 1 alert and LMON trace files for detail.
LMD0 (ospid: 3690): terminating the instance due to error 481
Mon Mar 18 12:53:36 2013
System state dump requested by (instance=2, osid=3690 (LMD0)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_3682.trc
Mon Mar 18 12:53:55 2013
ORA-1092 : opitsk aborting process
Mon Mar 18 12:53:55 2013
Termination issued to instance processes. Waiting for the processes to exit
Mon Mar 18 12:53:59 2013
License high water mark = 40
Instance terminated by LMD0, pid = 3690
USER (ospid: 28923): terminating the instance
Instance terminated by USER, pid = 28923

 

  

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms