OPSS - Response Time Gradually Climbing, getting slow responses (Doc ID 2152157.1)

Last updated on JULY 18, 2016

Applies to:

Oracle Platform Security for Java - Version 11.1.1.6.0 and later
Information in this document applies to any platform.

Symptoms

Using Oracle Communications Unified Inventory Management 7.2.2, seeing multiple outages

Many threads waiting on JPS layer, but seem to be moving slowly.

During the last occurrence, they were STUCK and HTTP requests coming to UIM were not fulfilled.

As per the thread dumps, All 3 Nodes experienced heavy synchronization in JPSS layer related to java.util.Collections$SynchronizedCollection.

The main pattern is noticed is that - thread executing JpsAbsSubjectResolver$SubjectWrapper.subjectCompare() operation is blocking all other execute threads from progressing due to synchronization.

See this pattern on all nodes -

For E.G Node 3 -

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002b57e52ba000 nid=0x1760 runnable [0x00002b580f0a2000]
  java.lang.Thread.State: RUNNABLE
  at java.util.Collections$SynchronizedCollection.size(Collections.java:1557)
  - locked (a java.util.Collections$SynchronizedSet)
  at java.util.HashSet.(HashSet.java:99)
  at oracle.security.jps.internal.jaas.JpsAbsSubjectResolver$SubjectWrapper.subjectCompare(JpsAbsSubjectResolver.java:826)
  - locked (a java.util.Collections$SynchronizedSet)
  at oracle.security.jps.internal.jaas.JpsAbsSubjectResolver$SubjectWrapper.equals(JpsAbsSubjectResolver.java:768)
  at oracle.security.jps.util.ReferenceMapFactory$SoftKeyHashMap.eq(ReferenceMapFactory.java:267)
  at oracle.security.jps.util.ReferenceMapFactory$SoftKeyHashMap.put(ReferenceMapFactory.java:414)
  at oracle.security.jps.internal.jaas.JpsAbsSubjectResolver.putValue(JpsAbsSubjectResolver.java:608)
  at oracle.security.jps.internal.jaas.JpsAbsSubjectResolver.resolveJpsSubject(JpsAbsSubjectResolver.java:273)
  at oracle.security.jps.internal.jaas.JpsAbsSubjectResolver.resolveSubject(JpsAbsSubjectResolver.java:214)
  at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
  at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
  at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
  at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:139)
  at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
  at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)
  at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
  at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
  at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
  at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
  at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
  at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)
  at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
  at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

  Locked ownable synchronizers:
  - (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)


Thread logic reports 32 threads waiting for 0x0000000509793a50.

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms