My Oracle Support Banner

Unsynchronized HashMap Usage In GlassFish Server Connection Pool Causes High CPU Utilzation (Doc ID 1998861.1)

Last updated on JULY 31, 2018

Applies to:

Oracle GlassFish Server - Version 3.1.2 and later
Information in this document applies to any platform.

Symptoms

Running into situations where connection pool threads are spinning in infinite loops. It appears to be due to unsynchronized use of a HashMap. This causes corruption of the internal structure of the HashMap which then causes infinite loops.

Here is a sample prstat output from the GlassFish Server process taken when the problem is seen:

Dump 1
  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
25288 gfish 98 0.0 0.2 0.0 0.0 0.0 0.0 2.0 0 146 0 0 java/94880
25288 gfish 98 0.0 0.2 0.0 0.0 0.0 0.0 2.3 0 156 0 0 java/93544
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.6 0 224 0 0 java/94086
25288 gfish 97 0.0 0.3 0.0 0.0 0.0 0.0 2.8 0 256 0 0 java/93238
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.8 0 212 0 0 java/95163
25288 gfish 97 0.0 0.3 0.0 0.0 0.0 0.0 2.8 0 266 0 0 java/94308
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 3.0 0 214 0 0 java/117
25288 gfish 97 0.0 0.3 0.0 0.0 0.0 0.0 3.0 0 244 0 0 java/94725
25288 gfish 69 0.1 0.1 0.0 0.0 29 0.4 1.7 3 135 32 0 java/229
25288 gfish 0.2 0.1 0.0 0.0 0.0 100 0.0 0.1 100 0 100 0 java/37
25288 gfish 0.1 0.1 0.0 0.0 0.0 99 0.0 0.6 18 0 18 0 java/35
25288 gfish 0.1 0.0 0.0 0.0 0.0 100 0.0 0.1 2 3 6 0 java/46
25288 gfish 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 33 0 33 0 java/39
25288 gfish 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 9 0 java/51
25288 gfish 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 1 0 3 0 java/69
Total: 1 processes, 275 lwps, load averages: 30.75, 21.60, 20.70

Dump 2
  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
25288 gfish 98 0.0 0.2 0.0 0.0 0.0 0.0 1.9 0 153 0 0 java/93238
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.5 0 238 0 0 java/93544
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.6 0 195 0 0 java/94725
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.6 0 241 0 0 java/95163
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.7 0 241 0 0 java/94086
25288 gfish 97 0.0 0.3 0.0 0.0 0.0 0.0 2.8 0 255 0 0 java/94308
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.8 0 238 0 0 java/94880
25288 gfish 96 0.0 0.3 0.0 0.0 0.0 0.0 3.3 0 269 0 0 java/117
25288 gfish 0.5 1.2 0.0 0.0 0.0 98 0.1 0.0 4 0 202 0 java/41
25288 gfish 0.1 0.1 0.0 0.0 0.0 100 0.0 0.1 100 0 100 0 java/37
25288 gfish 0.1 0.0 0.0 0.0 0.0 100 0.0 0.0 2 1 6 0 java/46
25288 gfish 0.1 0.0 0.0 0.0 0.0 99 0.0 0.5 17 0 17 0 java/35
25288 gfish 0.1 0.0 0.0 0.0 0.0 100 0.0 0.0 1 0 2 0 java/42
25288 gfish 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 34 0 36 0 java/39
25288 gfish 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 1 0 1 0 java/83481
Total: 1 processes, 275 lwps, load averages: 28.38, 21.45, 20.66

Dump 3
  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
25288 gfish 98 0.0 0.2 0.0 0.0 0.0 0.0 2.0 0 160 0 0 java/95163
25288 gfish 98 0.0 0.2 0.0 0.0 0.0 0.0 2.0 0 155 0 0 java/94086
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.5 0 188 0 0 java/94308
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.5 0 209 0 0 java/94725
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 2.9 0 227 0 0 java/94880
25288 gfish 97 0.0 0.2 0.0 0.0 0.0 0.0 3.0 0 257 0 0 java/93238
25288 gfish 97 0.0 0.3 0.0 0.0 0.0 0.0 3.0 0 275 0 0 java/93544
25288 gfish 96 0.0 0.3 0.0 0.0 0.0 0.0 3.3 1 275 0 0 java/117
25288 gfish 0.2 0.1 0.0 0.0 0.0 100 0.0 0.1 101 1 100 0 java/37
25288 gfish 0.1 0.0 0.0 0.0 0.0 100 0.0 0.1 2 0 6 0 java/46
25288 gfish 0.1 0.1 0.0 0.0 0.0 99 0.0 0.6 17 0 17 0 java/35
25288 gfish 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 4 0 18 0 java/51
25288 gfish 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 33 0 33 0 java/39
25288 gfish 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 1 0 3 0 java/57
25288 gfish 0.0 0.0 0.0 0.0 0.0 100 0.0 0.1 1 0 1 0 java/83494
Total: 1 processes, 275 lwps, load averages: 26.37, 21.27, 20.61


The corresponding Java thread dumps shows that all are spinning on the connection pool hashmap.

For example:

"pool-167-thread-3093" prio=3 tid=0x0dc59400 nid=0x172a0 runnable [0x66ffd000]
  java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:465)
at java.util.HashMap.get(HashMap.java:417)
at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:313)
at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionUsed(ConnectionPoolEmitterImpl.java:234)
at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionUsed(PoolLifeCycleListenerRegistry.java:145)
at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:532)
at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:332)
at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
.......
.......
.......

"pool-167-thread-2897" prio=3 tid=0x0ba80c00 nid=0x16f86 runnable [0x6d2fd000]
  java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:465)
at java.util.HashMap.get(HashMap.java:417)
at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:313)
at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionUsed(ConnectionPoolEmitterImpl.java:234)
at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionUsed(PoolLifeCycleListenerRegistry.java:145)
at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:532)
at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:332)
at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
...........
...........
...........

"connector-timer-proxy" daemon prio=3 tid=0x03822400 nid=0x75 runnable [0x6f2ff000]
  java.lang.Thread.State: RUNNABLE
at java.util.HashMap.removeEntryForKey(HashMap.java:690)
at java.util.HashMap.remove(HashMap.java:656)
at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionDestroyed(ConnectionPoolEmitterImpl.java:184)
at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionDestroyed(PoolLifeCycleListenerRegistry.java:151)
at com.sun.enterprise.resource.pool.ConnectionPool.deleteResource(ConnectionPool.java:974)
at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.removeResource(RWLockDataStructure.java:153)
at com.sun.enterprise.resource.pool.resizer.Resizer.removeIdleAndInvalidResources(Resizer.java:208)
at com.sun.enterprise.resource.pool.resizer.Resizer.resizePool(Resizer.java:113)
at com.sun.enterprise.resource.pool.resizer.Resizer.run(Resizer.java:93)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.