The Coherence Cluster Is Not Functional: Cache Store Nodes Report ORA-1034 ORA-27101 (Doc ID 1445283.1)

Last updated on FEBRUARY 24, 2017

Applies to:

Oracle Coherence - Version 3.4.1 to 12.1.2.0.0
Information in this document applies to any platform.
***Checked for relevance on 06-Sep-2013***

Symptoms

Coherence Storage Nodes are reporting many errors after the database that is being used as a read-write-backing-map-scheme has been shut down.  Note that this has been configured to use write-behind:

2012-03-28 02:05:07.842 Coherence 3.7.1.1, Error, (thread=WriteBehindThread:CacheStoreWrapper(com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore):DistributedCache, member=2, role=CoherenceServer, location=site:,machine:mrs-dv-00122,process:7471294): Failed to store key="1-634684667297955814"
2012-03-28 02:05:07.842 Coherence 3.7.1.1, Error, (thread=WriteBehindThread:CacheStoreWrapper(com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore):DistributedCache, member=2, role=CoherenceServer, location=site:,machine:mrs-dv-00122,process:7471294): java.util.ConcurrentModificationException
at java.util.HashMap$AbstractMapIterator.checkConcurrentMod(HashMap.java:122)
at java.util.HashMap$AbstractMapIterator.makeNext(HashMap.java:127)
at java.util.HashMap$KeyIterator.next(HashMap.java:200)
at com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore.InsertFunctionalLog(FunctionalLogCacheStore.java:114)
at com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore.storeAll(FunctionalLogCacheStore.java:105)
at com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore.store(FunctionalLogCacheStore.java:93)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.storeInternal(ReadWriteBackingMap.java:5678)
at com.tangosol.net.cache.ReadWriteBackingMap$StoreWrapper.store(ReadWriteBackingMap.java:4764)
at com.tangosol.net.cache.ReadWriteBackingMap$WriteThread.run(ReadWriteBackingMap.java:4167)
at com.tangosol.util.Daemon$DaemonWorker.run(Daemon.java:781)
at java.lang.Thread.run(Thread.java:736)



SQLException
java.sql.SQLRecoverableException: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
IBM AIX RISC System/6000 Error: 2: No such file or directory

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)
at oracle.jdbc.driver.T4CTTIoauthenticate.processError(T4CTTIoauthenticate.java:441)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:404)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:385)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:207)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:157)
at com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore.openDbConnection(FunctionalLogCacheStore.java:40)
at com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore.InsertFunctionalLog(FunctionalLogCacheStore.java:120)
at com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore.storeAll(FunctionalLogCacheStore.java:105)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.storeAllInternal(ReadWriteBackingMap.java:5698)
at com.tangosol.net.cache.ReadWriteBackingMap$StoreWrapper.storeAll(ReadWriteBackingMap.java:4828)
at com.tangosol.net.cache.ReadWriteBackingMap$WriteThread.run(ReadWriteBackingMap.java:4172)
at com.tangosol.util.Daemon$DaemonWorker.run(Daemon.java:781)
at java.lang.Thread.run(Thread.java:736)

2012-03-28 02:05:11.069 Coherence 3.7.1.1, Error, (thread=WriteBehindThread:CacheStoreWrapper(com.ccs.coherence.laracis.writeBehind.FunctionalLogCacheStore):DistributedCache, member=n/a, role=, location=): (The exception will be ignored. The write-behind thread will continue.)
2012-03-28 02:05:14.100 Coherence 3.7.1.1, Error, (thread=main, member=n/a, role=, location=): Failed to restart services: com.tangosol.net.RequestTimeoutException: Timeout while waiting for cluster to stop.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms