My Oracle Support Banner

Data Queue Threads Disappear After A Brief Database Connection Problem (Doc ID 1613595.1)

Last updated on SEPTEMBER 04, 2018

Applies to:

Oracle Transportation Management - Version 6.3.1 to 6.3.4 [Release 6.3]
Information in this document applies to any platform.

Symptoms

After the application server lost connection to the database server for a brief period, all threads in thread pool INTEGRATION IN disappeared. A thread dump using JRockit command "jrcmd print_threads" confirmed the observation. As a result, queue INTEGRATION IN was backed up. However, other parts of OTM survived the connection loss and continued to function correctly. The problem was eventually corrected by a server bounce.

Here is an excerpt from the error log showing how database connection errors caused a thread to terminate normally.

2013-11-15 04:55:17.436 0 Error Exception {call vpd.set_user_r(?,?,?,?)=null}; inParams=[DBA.ADMIN, DBA.AD
MIN, null, STATIC]
  java.sql.SQLRecoverableException: No more data to read from socket

java.sql.SQLRecoverableException: No more data to read from socket
  at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1157)
  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
  at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)
  at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1041)
  at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
  at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
  at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3685)
  at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4714)
  at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1376)
  at glog.util.jdbc.SqlCall.execute(SqlCall.java:70)
  at glog.util.jdbc.SqlCall.executeOut(SqlCall.java:126)
  at glog.util.jdbc.SqlCall.execute(SqlCall.java:114)
  at glog.util.jdbc.noserver.T2ConnUser.setToDB(T2ConnUser.java:63)
  at glog.util.jdbc.noserver.T2SharedConnection.setDatabaseUser(T2SharedConnection.java:245)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.setupVpdUser(T2SharedConnection.java:575)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.addRef(T2SharedConnection.java:449)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.addRef(T2SharedConnection.java:413)
  at glog.util.jdbc.noserver.T2SharedConnection.open(T2SharedConnection.java:66)
  at glog.util.jdbc.CheckForSQLExceptions.loadCustomErrorMessage(CheckForSQLExceptions.java:224)
  at glog.util.jdbc.CheckForSQLExceptions.mapSQLException(CheckForSQLExceptions.java:52)
  at glog.util.jdbc.CheckForSQLExceptions.map(CheckForSQLExceptions.java:26)
  at glog.util.exception.GLException.factory(GLException.java:440)
  at glog.util.exception.GLException.factory(GLException.java:426)
  at glog.util.exception.GLException.factory(GLException.java:422)
  at glog.util.exception.GLException.factory(GLException.java:402)
  at glog.util.datasource.GC3ConnectionEstablisher.connect(GC3ConnectionEstablisher.java:62)
  at glog.util.datasource.DataSource.reserve(DataSource.java:158)
  at glog.util.datasource.DataSource.reserve(DataSource.java:147)
  at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.reopenIfDropped(T2SharedConnection.java:533)
  at glog.util.jdbc.noserver.T2SharedConnection.reopenIfDropped(T2SharedConnection.java:88)
  at glog.server.dataqueue.DataQueue.checkForDroppedConnection(DataQueue.java:82)
  at glog.server.dataqueue.DataQueueRunnable.run(DataQueueRunnable.java:107)
  at glog.util.event.EventThread.run(EventThread.java:86)
  at java.lang.Thread.run(Thread.java:662)
[INTEGRATION IN - 4]


2013-11-15 04:55:17.437 0 Error Exception java.sql.SQLRecoverableException: IO Error: Got minus one from a
read call

java.sql.SQLRecoverableException: IO Error: Got minus one from a read call
  at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
  at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:546)
  at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:236)
  at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)

It appears that the system had been idle right before thread INTEGRAION IN - 4 disappeared. I don't have the regular log to confirm this since the files had been overwritten by the time the problem was noticed. The timestamp was early morning local time (4:55 am) and there had been not entries in the exception log for 3 hours.

However it does not matter whether there was an XML input what the content of the XML was. A data queue thread pols the queue at regular intervals. The error happened during the polling. In the log you can see the following stack traces showing the sql statement and the error. The poller was try to get a lock on the data queue before a poll.

2013-11-15 04:55:17.323 0 Error Exception select 1 from data_queue_def where data_queue_def_gid=? for update; [INTEGRATION IN] [INTEGRATION IN - 4]
2013-11-15 04:55:17.324 0 Error Exception {call vpd.set_user_r(?,?,?,?)=null}; inParams=[DBA.ADMIN, DBA.ADMIN, null, STATIC]
  java.sql.SQLRecoverableException: No more data to read from socket

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process and therefore has not been subject to an independent technical review.
My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.