OUD11g - The Replication Server Shuts Down Due to "IllegalStateException: Database closed while still referenced by other handles"
(Doc ID 2319721.1)
Last updated on MARCH 07, 2023
Applies to:
Oracle Unified Directory - Version 11.1.2.2.0 to 11.1.2.2.3 [Release 11g]Information in this document applies to any platform.
Symptoms
On : 11.1.2.2.0 version, configuration
Replication server suddenly stops. Seeing these messages in logs -
[10/Oct/2017:21:53:36 -0700] category=SYNC severity=SEVERE_ERROR msgID=-1 msg=Error during the replication server database trimming process. The replication ser
ver is going to shutdown(JE 5.0.86) Lock expired. Locker 1812336796 445944789_Replication DraftCN db _Txn: waited for lock on database=draftcndb LockAddr:179786
1199 LSN=0x168/0x8689e5 type=WRITE grant=WAIT_PROMOTION timeoutMillis=3600000 startTime=1507694016372 endTime=1507697616373
Owners: [<LockInfo locker="1143199497 -1_Worker Thread 15_ThreadLocker" type="READ"/>, <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type=
"READ"/>]
Waiters: []
Transaction 1812336796 445944789_Replication DraftCN db _Txn owns LockAddr:1797861199 <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type="
READ"/>
Transaction 1812336796 445944789_Replication DraftCN db _Txn waits for LockAddr:1797861199
(LockManager.java:664 LockManager.java:623 SyncedLockManager.java:97 LockManager.java:390 LockManager.java:276 Txn.java:510 Locker.java:443 CursorImpl.java:263
6 CursorImpl.java:1353 Cursor.java:2095 Cursor.java:2059 Cursor.java:696 DraftCNDB.java:731 DraftCNDbHandler.java:477 DraftCNDbHandler.java:383 DraftCNDbHandler
.java:337 Thread.java:744)
[10/Oct/2017:21:53:36 -0700] category=CORE severity=NOTICE msgID=458891 msg=The Directory Server has sent an alert notification generated by class org.opends.server.replication.server.ReplicationServer (alert type org.opends.server.replication.ReplicationServerDown, alert ID -1): Error during the replication server database trimming process. The replication server is going to shutdown(JE 5.0.86) Lock expired. Locker 1812336796 445944789_Replication DraftCN db _Txn: waited for lock on database=draftcndb LockAddr:1797861199 LSN=0x168/0x8689e5 type=WRITE grant=WAIT_PROMOTION timeoutMillis=3600000 startTime=1507694016372 endTime=1507697616373
Owners: [<LockInfo locker="1143199497 -1_Worker Thread 15_ThreadLocker" type="READ"/>, <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type="READ"/>]
Waiters: []
Transaction 1812336796 445944789_Replication DraftCN db _Txn owns LockAddr:1797861199 <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type="READ"/>
Transaction 1812336796 445944789_Replication DraftCN db _Txn waits for LockAddr:1797861199
(LockManager.java:664 LockManager.java:623 SyncedLockManager.java:97 LockManager.java:390 LockManager.java:276 Txn.java:510 Locker.java:443 CursorImpl.java:2636 CursorImpl.java:1353 Cursor.java:2095 Cursor.java:2059 Cursor.java:696 DraftCNDB.java:731 DraftCNDbHandler.java:477 DraftCNDbHandler.java:383 DraftCNDbHandler.java:337 Thread.java:744)
[10/Oct/2017:21:53:36 -0700] category=SYNC severity=NOTICE msgID=1513<SERVER_ID-3>9 msg=Replication server <host>/<ip>:<port> has properly disconnected for cn=schema in local server id <SERVER_ID-1>. Trying to reconnect to a suitable replication server
[10/Oct/2017:21:53:36 -0700] category=SYNC severity=NOTICE msgID=15138839 msg=Could not connect to any replication server on suffix cn=schema among the following RS candidates {}, retrying...
[10/Oct/2017:21:53:37 -0700] category=SYNC severity=NOTICE msgID=1513<SERVER_ID-3>9 msg=Replication server <host>/<ip>:<port> has properly disconnected for cn=admin data in local server id <SERVER_ID-2>. Trying to reconnect to a suitable replication server
[10/Oct/2017:21:53:37 -0700] category=SYNC severity=NOTICE msgID=15138839 msg=Could not connect to any replication server on suffix cn=admin data among the following RS candidates {}, retrying...
[10/Oct/2017:21:53:38 -0700] category=SYNC severity=NOTICE msgID=1513<SERVER_ID-3>9 msg=Replication server <host>/<ip>:<port> has properly disconnected for dc=<SUFFIX_DN> in local server id <SERVER_ID-3>. Trying to reconnect to a suitable replication server
[10/Oct/2017:21:53:38 -0700] category=SYNC severity=NOTICE msgID=15138839 msg=Could not connect to any replication server on suffix dc=<SUFFIX_DN> among the following RS candidates {}, retrying...
[10/Oct/2017:21:53:39 -0700] category=CORE severity=FATAL_ERROR msgID=327820 msg=An uncaught exception during processing for thread Replication DraftCN db has caused it to terminate abnormally. The stack trace for that exception is: java.lang.IllegalStateException: Unclosed Database: draftcnlimitsdb
Unclosed Database: draftcndb
While closing Database draftcndb encountered exception: java.lang.IllegalStateException: Database closed while still referenced by other handles. 1 open cursors.
at com.sleepycat.je.Database.closeInternalWork(Database.java:646)
at com.sleepycat.je.Database.closeInternal(Database.java:508)
at com.sleepycat.je.Database.close(Database.java:438)
at com.sleepycat.je.Environment.checkOpenDbs(Environment.java:518)
at com.sleepycat.je.Environment.checkForCloseErrors(Environment.java:414)
at com.sleepycat.je.Environment.close(Environment.java:375)
at org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
at org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
at org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
at java.lang.Thread.run(Thread.java:744)
Databases left open: 2
com.sleepycat.je.Environment.close(Environment.java:390)
org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
java.lang.Thread.run(Thread.java:744)
[10/Oct/2017:21:53:39 -0700] category=CORE severity=NOTICE msgID=458891 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID 327820): An uncaught exception during processing for thread Replication DraftCN db has caused it to terminate abnormally. The stack trace for that exception is: java.lang.IllegalStateException: Unclosed Database: draftcnlimitsdb
Unclosed Database: draftcndb
While closing Database draftcndb encountered exception: java.lang.IllegalStateException: Database closed while still referenced by other handles. 1 open cursors.
at com.sleepycat.je.Database.closeInternalWork(Database.java:646)
at com.sleepycat.je.Database.closeInternal(Database.java:508)
at com.sleepycat.je.Database.close(Database.java:438)
at com.sleepycat.je.Environment.checkOpenDbs(Environment.java:518)
at com.sleepycat.je.Environment.checkForCloseErrors(Environment.java:414)
at com.sleepycat.je.Environment.close(Environment.java:375)
at org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
at org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
at org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
at java.lang.Thread.run(Thread.java:744)
Databases left open: 2
com.sleepycat.je.Environment.close(Environment.java:390)
org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
java.lang.Thread.run(Thread.java:744)
ver is going to shutdown(JE 5.0.86) Lock expired. Locker 1812336796 445944789_Replication DraftCN db _Txn: waited for lock on database=draftcndb LockAddr:179786
1199 LSN=0x168/0x8689e5 type=WRITE grant=WAIT_PROMOTION timeoutMillis=3600000 startTime=1507694016372 endTime=1507697616373
Owners: [<LockInfo locker="1143199497 -1_Worker Thread 15_ThreadLocker" type="READ"/>, <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type=
"READ"/>]
Waiters: []
Transaction 1812336796 445944789_Replication DraftCN db _Txn owns LockAddr:1797861199 <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type="
READ"/>
Transaction 1812336796 445944789_Replication DraftCN db _Txn waits for LockAddr:1797861199
(LockManager.java:664 LockManager.java:623 SyncedLockManager.java:97 LockManager.java:390 LockManager.java:276 Txn.java:510 Locker.java:443 CursorImpl.java:263
6 CursorImpl.java:1353 Cursor.java:2095 Cursor.java:2059 Cursor.java:696 DraftCNDB.java:731 DraftCNDbHandler.java:477 DraftCNDbHandler.java:383 DraftCNDbHandler
.java:337 Thread.java:744)
[10/Oct/2017:21:53:36 -0700] category=CORE severity=NOTICE msgID=458891 msg=The Directory Server has sent an alert notification generated by class org.opends.server.replication.server.ReplicationServer (alert type org.opends.server.replication.ReplicationServerDown, alert ID -1): Error during the replication server database trimming process. The replication server is going to shutdown(JE 5.0.86) Lock expired. Locker 1812336796 445944789_Replication DraftCN db _Txn: waited for lock on database=draftcndb LockAddr:1797861199 LSN=0x168/0x8689e5 type=WRITE grant=WAIT_PROMOTION timeoutMillis=3600000 startTime=1507694016372 endTime=1507697616373
Owners: [<LockInfo locker="1143199497 -1_Worker Thread 15_ThreadLocker" type="READ"/>, <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type="READ"/>]
Waiters: []
Transaction 1812336796 445944789_Replication DraftCN db _Txn owns LockAddr:1797861199 <LockInfo locker="1812336796 445944789_Replication DraftCN db _Txn" type="READ"/>
Transaction 1812336796 445944789_Replication DraftCN db _Txn waits for LockAddr:1797861199
(LockManager.java:664 LockManager.java:623 SyncedLockManager.java:97 LockManager.java:390 LockManager.java:276 Txn.java:510 Locker.java:443 CursorImpl.java:2636 CursorImpl.java:1353 Cursor.java:2095 Cursor.java:2059 Cursor.java:696 DraftCNDB.java:731 DraftCNDbHandler.java:477 DraftCNDbHandler.java:383 DraftCNDbHandler.java:337 Thread.java:744)
[10/Oct/2017:21:53:36 -0700] category=SYNC severity=NOTICE msgID=1513<SERVER_ID-3>9 msg=Replication server <host>/<ip>:<port> has properly disconnected for cn=schema in local server id <SERVER_ID-1>. Trying to reconnect to a suitable replication server
[10/Oct/2017:21:53:36 -0700] category=SYNC severity=NOTICE msgID=15138839 msg=Could not connect to any replication server on suffix cn=schema among the following RS candidates {}, retrying...
[10/Oct/2017:21:53:37 -0700] category=SYNC severity=NOTICE msgID=1513<SERVER_ID-3>9 msg=Replication server <host>/<ip>:<port> has properly disconnected for cn=admin data in local server id <SERVER_ID-2>. Trying to reconnect to a suitable replication server
[10/Oct/2017:21:53:37 -0700] category=SYNC severity=NOTICE msgID=15138839 msg=Could not connect to any replication server on suffix cn=admin data among the following RS candidates {}, retrying...
[10/Oct/2017:21:53:38 -0700] category=SYNC severity=NOTICE msgID=1513<SERVER_ID-3>9 msg=Replication server <host>/<ip>:<port> has properly disconnected for dc=<SUFFIX_DN> in local server id <SERVER_ID-3>. Trying to reconnect to a suitable replication server
[10/Oct/2017:21:53:38 -0700] category=SYNC severity=NOTICE msgID=15138839 msg=Could not connect to any replication server on suffix dc=<SUFFIX_DN> among the following RS candidates {}, retrying...
[10/Oct/2017:21:53:39 -0700] category=CORE severity=FATAL_ERROR msgID=327820 msg=An uncaught exception during processing for thread Replication DraftCN db has caused it to terminate abnormally. The stack trace for that exception is: java.lang.IllegalStateException: Unclosed Database: draftcnlimitsdb
Unclosed Database: draftcndb
While closing Database draftcndb encountered exception: java.lang.IllegalStateException: Database closed while still referenced by other handles. 1 open cursors.
at com.sleepycat.je.Database.closeInternalWork(Database.java:646)
at com.sleepycat.je.Database.closeInternal(Database.java:508)
at com.sleepycat.je.Database.close(Database.java:438)
at com.sleepycat.je.Environment.checkOpenDbs(Environment.java:518)
at com.sleepycat.je.Environment.checkForCloseErrors(Environment.java:414)
at com.sleepycat.je.Environment.close(Environment.java:375)
at org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
at org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
at org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
at java.lang.Thread.run(Thread.java:744)
Databases left open: 2
com.sleepycat.je.Environment.close(Environment.java:390)
org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
java.lang.Thread.run(Thread.java:744)
[10/Oct/2017:21:53:39 -0700] category=CORE severity=NOTICE msgID=458891 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID 327820): An uncaught exception during processing for thread Replication DraftCN db has caused it to terminate abnormally. The stack trace for that exception is: java.lang.IllegalStateException: Unclosed Database: draftcnlimitsdb
Unclosed Database: draftcndb
While closing Database draftcndb encountered exception: java.lang.IllegalStateException: Database closed while still referenced by other handles. 1 open cursors.
at com.sleepycat.je.Database.closeInternalWork(Database.java:646)
at com.sleepycat.je.Database.closeInternal(Database.java:508)
at com.sleepycat.je.Database.close(Database.java:438)
at com.sleepycat.je.Environment.checkOpenDbs(Environment.java:518)
at com.sleepycat.je.Environment.checkForCloseErrors(Environment.java:414)
at com.sleepycat.je.Environment.close(Environment.java:375)
at org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
at org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
at org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
at java.lang.Thread.run(Thread.java:744)
Databases left open: 2
com.sleepycat.je.Environment.close(Environment.java:390)
org.opends.server.replication.server.ReplicationDbEnv.shutdown(ReplicationDbEnv.java:688)
org.opends.server.replication.server.ReplicationServer.shutdown(ReplicationServer.java:827)
org.opends.server.replication.server.DraftCNDbHandler.run(DraftCNDbHandler.java:358)
java.lang.Thread.run(Thread.java:744)
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |