Exadata: Instance eviction with IPC sendtimeout messages.

(Doc ID 1404576.1)

Last updated on MAY 24, 2012

Applies to:

Linux OS - Version Oracle Linux 5.5 to Oracle Linux 5.5 [Release OL5U5]
Oracle Exadata Storage Server Software - Version 11.2.2.4.0 to 11.2.2.4.2 [Release 11.2]
Information in this document applies to any platform.

Symptoms

Intermittent database/ASM instance evictions after the Exadata storage software version has been upgraded to 11.2.2.4.x

Fri Dec 23 05:58:29 2011
Errors in file /u01/app/oracle/diag/rdbms/ctra/ctra2/trace/ctra2_lmd0_1285.trc:
IPC Send timeout to 1.2 inc 16 for msg type 36 from opid 177
IPC Send timeout: Terminating pid 177 osid 30942
Fri Dec 23 06:02:45 2011
minact-scn master exiting with err:12751
Fri Dec 23 06:03:40 2011
LMS1 (ospid: 1291) has detected no messaging activity from instance 1
LMS1 (ospid: 1291) issues an IMR to resolve the situation
Please check LMS1 trace file for more detail.
Fri Dec 23 06:03:40 2011
Suppressed nested communications reconfiguration: instance_number 1
Detected an inconsistent instance membership by instance 1
Fri Dec 23 06:03:41 2011
Received an instance abort message from instance 1
Please check instance 1 alert and LMON trace files for detail.
LMS0 (ospid: 1287): terminating the instance due to error 481


The LMS/LMON/LMD trace files indicate that there were heartbeats  time outs across the instances and that resulted in the eviction.

*** 2011-12-23 06:02:12.550
===== Idle Connection Overview =====
Idle connections [0x307c3]: 2
IdleConn List: 2[r:0.23926378,t:0x307c3]
GSIPC:IKILL: ping to inst 2 start 198595 now 198878 icktm 280 psm 0
Timeout after 280 seconds
Reporting Communication error with instance 2


At the same time, the rds-ping statistics collected in the OS Watcher show that the RDS connectivity issues between some or all of the compute nodes.

[INFO:2011-12-23-05:52:35] To database hosts: rds-ping -c 1 -i 5 -I  192.168.10.8 192.168.10.9: <<<<

 

Changes

Exadata cell software upraded to 11.2.2.4.x which contain the OFA version 1.5.1-4.0.53.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms