Data Queue Threads Disappear After A Brief Database Connection Problem (Doc ID 1613595.1)

Last updated on MARCH 08, 2017

Applies to:

Oracle Transportation Management - Version 6.3.1 to 6.3.4 [Release 6.3]
Information in this document applies to any platform.

Symptoms

After the application server lost connection to the database server for a brief period, all threads in thread pool INTEGRATION IN disappeared. A thread dump using JRockit command "jrcmd print_threads" confirmed the observation. As a result, queue INTEGRATION IN was backed up. However, other parts of OTM survived the connection loss and continued to function correctly. The problem was eventually corrected by a server bounce.

Here is an excerpt from the error log showing how database connection errors caused a thread to terminate normally.

2013-11-15 04:55:17.436 0 Error Exception {call vpd.set_user_r(?,?,?,?)=null}; inParams=[DBA.ADMIN, DBA.AD
MIN, null, STATIC]
  java.sql.SQLRecoverableException: No more data to read from socket

java.sql.SQLRecoverableException: No more data to read from socket
  at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1157)
  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
  at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)
  at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1041)
  at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
  at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
  at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3685)
  at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4714)
  at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1376)
  at glog.util.jdbc.SqlCall.execute(SqlCall.java:70)
  at glog.util.jdbc.SqlCall.executeOut(SqlCall.java:126)
  at glog.util.jdbc.SqlCall.execute(SqlCall.java:114)
  at glog.util.jdbc.noserver.T2ConnUser.setToDB(T2ConnUser.java:63)
  at glog.util.jdbc.noserver.T2SharedConnection.setDatabaseUser(T2SharedConnection.java:245)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.setupVpdUser(T2SharedConnection.java:575)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.addRef(T2SharedConnection.java:449)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.addRef(T2SharedConnection.java:413)
  at glog.util.jdbc.noserver.T2SharedConnection.open(T2SharedConnection.java:66)
  at glog.util.jdbc.CheckForSQLExceptions.loadCustomErrorMessage(CheckForSQLExceptions.java:224)
  at glog.util.jdbc.CheckForSQLExceptions.mapSQLException(CheckForSQLExceptions.java:52)
  at glog.util.jdbc.CheckForSQLExceptions.map(CheckForSQLExceptions.java:26)
  at glog.util.exception.GLException.factory(GLException.java:440)
  at glog.util.exception.GLException.factory(GLException.java:426)
  at glog.util.exception.GLException.factory(GLException.java:422)
  at glog.util.exception.GLException.factory(GLException.java:402)
  at glog.util.datasource.GC3ConnectionEstablisher.connect(GC3ConnectionEstablisher.java:62)
  at glog.util.datasource.DataSource.reserve(DataSource.java:158)
  at glog.util.datasource.DataSource.reserve(DataSource.java:147)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.reopenIfDropped(T2SharedConnection.java:533)
  at glog.util.jdbc.noserver.T2SharedConnection.reopenIfDropped(T2SharedConnection.java:88)
  at glog.server.dataqueue.DataQueue.checkForDroppedConnection(DataQueue.java:82)
  at glog.server.dataqueue.DataQueueRunnable.run(DataQueueRunnable.java:107)
  at glog.util.event.EventThread.run(EventThread.java:86)
  at java.lang.Thread.run(Thread.java:662)
[INTEGRATION IN - 4]


2013-11-15 04:55:17.437 0 Error Exception java.sql.SQLRecoverableException: IO Error: Got minus one from a
read call

java.sql.SQLRecoverableException: IO Error: Got minus one from a read call
  at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
  at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:546)
  at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:236)
  at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)

It appears that the system had been idle right before thread INTEGRAION IN - 4 disappeared. I don't have the regular log to confirm this since the files had been overwritten by the time the problem was noticed. The timestamp was early morning local time (4:55 am) and there had been not entries in the exception log for 3 hours.

However it does not matter whether there was an XML input what the content of the XML was. A data queue thread pols the queue at regular intervals. The error happened during the polling. In the log you can see the following stack traces showing the sql statement and the error. The poller was try to get a lock on the data queue before a poll.

2013-11-15 04:55:17.323 0 Error Exception select 1 from data_queue_def where data_queue_def_gid=? for update; [INTEGRATION IN] [INTEGRATION IN - 4]
2013-11-15 04:55:17.324 0 Error Exception {call vpd.set_user_r(?,?,?,?)=null}; inParams=[DBA.ADMIN, DBA.ADMIN, null, STATIC]
  java.sql.SQLRecoverableException: No more data to read from socket

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms