The DIP/ODSM Managed Server Dies After Some Time With Error org.quartz.impl.jdbcjobstore.JobStoreTX

(Doc ID 1226674.1)

Last updated on SEPTEMBER 06, 2016

Applies to:

Oracle Fusion Middleware - Version 11.1.1.2.0 and later
Oracle Weblogic Server - Version 10.3.2 and later
Oracle Solaris on SPARC (64-bit)
This article applies to Oracle Solaris 10 (64 bit)

Symptoms

Fusion Middleware (FMW) 11g is installed with Weblogic Server (WLS) 10.3.x and a certified version of the Sun JDK on Solaris 10 (64-bit).

After several days the WLS managed server for Directory Integration and Provisioning and Oracle Directory Services Manager (DIP /ODSM) dies. The corresponding server.out log shows:

<Error> <org.quartz.impl.jdbcjobstore.JobStoreTX> <BEA-000000> <ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'schedulerDS': java.sql.SQLException: Could not retrieve datasource via JNDI url 'jdbc/schedulerDS' weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: Socket read timed out
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'schedulerDS': java.sql.SQLException: Could not retrieve datasource via JNDI url 'jdbc/schedulerDS' weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: Socket read timed out [See nested exception: java.sql.SQLException: Could not retrieve datasource via JNDI url 'jdbc/schedulerDS' weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: Socket read timed out]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:636)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:72)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3070)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3713)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3749)

Caused By: java.sql.SQLException: Could not retrieve datasource via JNDI url 'jdbc/schedulerDS' weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: Socket read timed out
at org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:166)
at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:112)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:633)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:72)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3070)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3713)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3749)
>
<Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>
<Warning> <oracle.dip> <BEA-000000> <ERROR_CONTEXT_CLOSING>
<Warning> <oracle.dip> <BEA-000000> <obtaining LDAP connection failed in attempt number :1 -  Retrying>
:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xffffffff0f23a7a8, pid=22190, tid=5410
#
# JRE version: 6.0_19-b04
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.2-b04 mixed mode solaris-sparc )
# Problematic frame:
# C  [pkcs11_softtoken.so.1+0x3a7a8]
#
# An error report file with more information is saved as:
# <Middleware Home>/user_projects/domains/IDMDomain/hs_err_pid22190.log
#

The referenced hs_err_pid22190.log contains a java core output, indicating that the JVM has crashed:

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
=>0x0000000102150000 JavaThread "Thread-5338" daemon [_thread_in_native, id=5410, stack(0xffffffff09c00000,0xffffffff09c80000)]
0x000000010420f800 JavaThread "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=2737, stack(0xffffffff0d200000,0xffffffff0d280000)]
0x0000000104f39000 JavaThread "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=2522, stack(0xffffffff09b00000,0xffffffff09b80000)]
0x00000001032fb000 JavaThread "Thread-160" daemon [_thread_in_native, id=216, stack(0xffffffff09d00000,0xffffffff09d80000)]
0x0000000103ef4000 JavaThread "DynamicListenThread[Default]" daemon [_thread_in_native, id=92, stack(0xffffffff0d500000,0xffffffff0d580000)]
0x00000001045b1000 JavaThread "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=91, stack(0xffffffff0ad00000,0xffffffff0ad80000)]
0x0000000103b09000 JavaThread "weblogic.GCMonitor" daemon [_thread_blocked, id=90, stack(0xffffffff0a800000,0xffffffff0a880000)]
0x0000000103a15800 JavaThread "CacheCleaner" daemon [_thread_blocked, id=89, stack(0xffffffff0a900000,0xffffffff0a980000)]
0x0000000102299800 JavaThread "Worker-2" daemon [_thread_blocked, id=88, stack(0xffffffff0aa00000,0xffffffff0aa80000)]
0x000000010304e000 JavaThread "Worker-1" daemon [_thread_blocked, id=87, stack(0xffffffff0a700000,0xffffffff0a780000)]
0x0000000102a29800 JavaThread "Worker-0" daemon [_thread_blocked, id=86, stack(0xffffffff0a600000,0xffffffff0a680000)]
0x00000001032c1000 JavaThread "Thread-37" daemon [_thread_in_native, id=80, stack(0xffffffff0c100000,0xffffffff0c180000)]
0x0000000102bed000 JavaThread "Thread-36" daemon [_thread_in_native, id=79, stack(0xffffffff0ab00000,0xffffffff0ab80000)]
0x0000000104954000 JavaThread "Thread-35" daemon [_thread_in_native, id=78, stack(0xffffffff0ac00000,0xffffffff0ac80000)]
0x000000010292a000 JavaThread "QuartzScheduler_DefaultQuartzScheduler-host.domain1284456526920_MisfireHandler" [_thread_blocked, id=75, stack(0xffffffff0af00000,0xffffffff0af80000)]
0x00000001042c3000 JavaThread "QuartzScheduler_DefaultQuartzScheduler-host.domain1284456526920_ClusterManager" [_thread_blocked, id=74, stack(0xffffffff0b000000,0xffffffff0b080000)]
0x00000001032c1800 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=73, stack(0xffffffff0b100000,0xffffffff0b180000)]
0x000000010261e800 JavaThread "UpdateThread" [_thread_blocked, id=72, stack(0xffffffff0b200000,0xffffffff0b280000)]

The goal is to prevent the WLS managed server from dying unexpectedly.

Changes

This is a new installation of FMW Identity Management 11g.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms