Om crashes with error messages mentioning Rac failover

Last updated on MARCH 02, 2017

Applies to:

Symptoms

OM logs show following errors prior to crash:

(sqllog.cpp (671)) SBL-DAT-60171: End of the log stream is reached.ObjMgrLog Error 1
(sqllog.cpp (671)) SBL-DAT-60171: End of the log stream is reached.DBCLog DBCLogError 1

Oracle RAC is failing over...DBCLog DBCLogError 1
Failover type is SELECTDBCLog DBCLogError 1
Oracle RAC failover ended successfullyDBCLog DBCLogError 1
Oracle RAC is failing over...DBCLog DBCLogError 1
Failover type is SELECTDBCLog DBCLogError 1
Oracle RAC failover ended successfully


- crash.txt shows crash on Oracle client dlls when OM task tries to re login on the killed DB session.

- CALL STACK -
kernel32 RaiseException()
orageneric11 slcra()
orageneric11 skgdbgcra()
orageneric11 kpeDbgCrash()
OraClient11 xaolog()
OraClient11 kpufhndl()
OraClient11 OCIHandleFree()
OCI OCIHandleFree()
sscdo90 CSSOraSqlCursor::~CSSOraSqlCursor()
sscdo90 SSASqlObjLogon() +0x1c5
sscfdm CSSLockSqlCursor::~CSSLockSqlCursor()
sscfdm CSSDbConn::BeginTransaction()
sscfdm CSSLockSqlCursor::Delete()
sscfdm CSSDbConn::CacheSqlCursor()
sscfdm CSSDbConn::CachePendingCursors()
sscfdm CSSLockDbConn::Unlock()
sscfdm CSSDbLock::~CSSDbLock()
sscfdm CSSSqlCursor::NextRecord()
sscfdm CSSSqlObj::AddListRow()
sscfdm CSSSqlObj::NextRecord()
sscfom CSSBusComp::SqlNextRecord()
sscfom CSSBusComp::Home()
SSCASWBC CSSSWEFrame::ExecuteQuery(


- FDR shows the the thread crashes when trying to do "Sql Cursor Destruction"


FdrSub_FDR_CRASH ** CRASHING THREAD **
FdrSub_FDR_CRASH ** CRASHING THREAD **
FdrSub_DBC_CURSOR_DSTR Sql Cursor Destruction
FdrSub_OBJMGR_DATA_LST_DSTR Data List Destruction
FdrSub_DBC_CURSOR_DSTR Sql Cursor Destruction
FdrSub_OBJMGR_DATA_LST_DSTR Data List Destruction
FdrSub_DBC_CURSOR_DSTR Sql Cursor Destruction
FdrSub_OBJMGR_DATA_LST_DSTR Data List Destruction
FdrSub_DBC_CURSOR_DSTR Sql Cursor Destruction
FdrSub_OBJMGR_DATA_LST_DSTR Data List Destruction
FdrSub_OBJMGR_DATA_LST_DSTR Data List Destruction
FdrSub_FDR_CRASH ** CRASHING THREAD **

Changes

Customer developed new Script that kills session for long running SQL:

select 'REM SID: '||a.sid||' SERIAL#: '||a.serial#||' SQL ID: '||b.sql_id||chr(10)||' /*+ '||sql_fulltext||' */' from v\$session a, v\$sql b where b.address = a.sql_add
ress and program = 'siebmtshmw.exe' and command = 3 and status = 'ACTIVE' and last_call_et >= 300 and username = 'SADMIN';

select 'alter system kill session '||''''||sid||','||serial#||''''||';' from v\$session where program = 'siebmtshmw.exe' and command = 3 and status = 'ACTIVE' and last_c
all_et >= 300 and username = 'SADMIN';

Cause