How To Check if Linux Out Of Memory Killer Has Killed a TimesTen Process and Caused TimesTen Database to Get Invalidated (Doc ID 2074274.1)

Last updated on APRIL 16, 2017

Applies to:

Oracle TimesTen In-Memory Database - Version 11.2.2.2.0 and later
Generic Linux

Symptoms

TimesTen database invalidated unexpectedly:

From mydsn.inval file:
2015-10-27 12:12:40.988
Invalidated data store: /u01/app/oracle/product/TimesTen/tt1122/info/mydsn
Data store created: 2015/06/05 20:40:12
TimesTen Release 11.2.2.8.0 (Linux x86-64, 64-bit) (tt1122)
Source: Data store marked invalid by master daemon: unable to contact managing subdaemon during process recovery


From ttmesg.log, it indicated TimesTen invalidated cause it can't talk to subdaemon:
2015-10-27 12:12:40.98 Err :    : 14863: TT14000: TimesTen daemon internal error: Could not connect to subdaemon port 37880: invalidate [err 799 (Protocol error trying to talk to daemon at port 37880)]
2015-10-27 12:12:40.98 Info:    : 14863: Mark process #1000010 as down (0)
2015-10-27 12:12:40.98 Warn:    : 14863: Invalidating the data store (failcode -1, recovery for 13027)
2015-10-27 12:12:40.98 Info:    : 14863: daInvalidateDatabase(/u01/app/oracle/product/TimesTen/tt1122/info/mydsn); shmKey 1929412609, why=unable to contact managing subdaemon during process recovery

later in ttmesg.log we also see this:
2015-10-27 12:12:51.87 Warn:    : 14863: child process 16664 terminated with signal 9

signal 9 is KILL.

Looked for where it started up subdaemon, this time by searching for port 37880 , grep 37880 ttmes*:
ttmesg.log.1:2015-10-23 18:43:19.51 Info:    : 14863: mark process up: index 1000010, port 37880, pid 16664
ttmesg.log.1:2015-10-23 18:43:19.51 Info:    : 16664: subd: connectToDaemon(sub) port 37880 id 1000010

This all means that some external process killed the TimesTen subdaemon process, 16664.  

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms