Deadlock During PartitionedCache$Storage.reevaluateQueryResults (Fixed in 3.7.1.2) (Doc ID 1424363.1)

Last updated on NOVEMBER 03, 2016

Applies to:

Oracle Coherence - Version 3.5.0 to 3.7.1.1 [Release AS10g]
Information in this document applies to any platform.
***Checked for relevance on 06-Sep-2013***

Symptoms

There were a few occurrences of a deadlock on one of the caches in one of 5 storage nodes.

The Service Guardian had been disabled hence the threads are not terminated and the node has to be manually killed.

The thread was stuck in com.tangosol.net.internal.StorageVersion.waitForPendingUpdates() method.

As seen here:

"DistributedCacheWorker:2" daemon prio=10 tid=0x00000000024a4000 nid=0x3bf5 waiting on condition [0x00007f9d4e861000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:311)
at com.tangosol.net.internal.StorageVersion.waitForPendingUpdates(StorageVersion.java:210)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.reevaluateQueryResults(PartitionedCache.CDB:40)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.checkIndexConsistency(PartitionedCache.CDB:52)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.createQueryResult(PartitionedCache.CDB:77)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.query(PartitionedCache.CDB:74)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeFilterRequest(PartitionedCache.CDB:55)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeFilterRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:679)

"DistributedCacheWorker:0" daemon prio=10 tid=0x00000000024a1000 nid=0x3bf3 waiting on condition [0x00007f9d4ea63000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000000c01746a8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:840)
at com.tangosol.util.ThreadGateLite.acquireLock(ThreadGateLite.java:93)
at com.tangosol.util.ThreadGateLite.close(ThreadGateLite.java:27)
at com.tangosol.util.WrapperReentrantGate.close(WrapperReentrantGate.java:43)
at com.tangosol.util.SegmentedConcurrentMap.lock(SegmentedConcurrentMap.java:311)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ResourceCoordinator.lockAll(PartitionedCache.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.lockAll(PartitionedCache.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onIndexRequest(PartitionedCache.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$IndexRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:679)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms