My Oracle Support Banner

Stuck Threads On The Platform And Servers Are Not Responsive (Doc ID 2801829.1)

Last updated on AUGUST 24, 2021

Applies to:

Oracle Access Manager - Version 12.2.1.3.0 and later
Information in this document applies to any platform.

Symptoms

We have outage on the platform where we have large number of Stuck threads on all servers, all the servers are in WARNING state and servers are not responding to the requests.

Post restart of the OAM managed servers the issue is mitigated. What is causing this?

<OAM_SERVER>.log has following error:

<DATE_TIME> <Info> <WorkManager> <HOST> <OAM_SERVER> <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002936> <maximum thread constraint MDCTermWMMaxThreads is reached>

<DATE_TIME> <Info> <WorkManager> <HOST> <OAM_SERVER> <[ACTIVE] ExecuteThread: '267' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002939> <The maximum thread constraint MDCTermWMMaxThreads has been reached 2 times for the last 924 seconds.>

<DATE_TIME> <Error> <WebLogicServer> <HOST> <OAM_SERVER> <[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000337> <[STUCK] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "625" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@11a64254[POST /oauth2/rest/token]
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
oracle.security.am.common.nap.util.pool.RoundRobinObjectPoolManager.getServerInfoForPrimaryServers(RoundRobinObjectPoolManager.java:379)
oracle.security.am.common.nap.util.pool.RoundRobinObjectPoolManager.acquireObject(RoundRobinObjectPoolManager.java:282)
oracle.security.am.common.nap.util.pool.ObjectPoolManager.acquireObject(ObjectPoolManager.java:118)
oracle.security.am.common.aaaclient.ObAAAServiceClient.sendMessageOnce(ObAAAServiceClient.java:2878)
oracle.security.am.common.aaaclient.ObAAAServiceClient.sendMessage(ObAAAServiceClient.java:2831)

...

 

Changes

 DCServerCapacity and AccessCapacity has been increased to a very large value in $DOMAIN_HOME/servers/<OAM_SERVER>/tmp/_WL_user/oam_server_11.1.2.0.0/88g74i/META-INF/weblogic-application.xml file based on your load test optimization -

<max-threads-constraint>
<name>MDCTermWMMaxThreads</name>
<count>10</count>
</max-threads-constraint>
<capacity>
<name>DCServerCapacity</name>
<count>500000</count>
</capacity>
<capacity>
<name>AccessCapacity</name>
<count>500000</count>
</capacity>

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.