Deployments Screen Loads in 2+Mins in a Large Domain in WebLogic 14c When One App is in Prepared State (Doc ID 2799257.1)

Last updated on MAY 01, 2023

Applies to:

Symptoms

Recently upgraded from WebLogic server 12.1.3 to 14.1.1 in a large domain with 23 clusters containing 45 managed servers. When clicking on the Deployments link to view the list of deployed applications, it takes several minutes for the Deployments page to load.

To calculate the status of the deployments, the AdminServer will use individual calls to check the status for every deployment in every target using multithreading. As a result, a high CPU usage will be seen in the first seconds of this call, when multiple threads are checking the state/health of deployments.

Using a 5 minutes JFR against the AdminServer, the issue was replicated twice.

Here are the findings when analyzing the JFR using JMC8 (https://www.oracle.com/java/technologies/javase/products-jmc8-downloads.html)

1- Under Servlet Invocation by URL, when ordering by duration, the /console/console.portal was the URL related to slowness.

The 2 minutes events are related to ExecuteThread 9 for the 1st replication and ExecuteThread 45 for the 2nd attempt.

2- Both attempts showed similar behavior, so concentrating on the 1st attempt, it was determined that activity beyond the first seconds after clicking on deployment page, was focused on 2 threads, ExecuteThread 9 and ExecuteThread 8:

In the graph, ExecuteThread 8 is represented by multiple small green/gray blocks representing multiple different calls. While, the wide red block for ExecuteThread 9, corresponding to the servlet executed during 2 minutes 27 seconds. Most of the time, waiting for ExecuteThread 8 ends its tasks.

When extracting these threads from the thread dump included in the same JFR file, it was determined that ExecuteThread 9 is the main thread orchestrating all the individual calls and finally waiting for all of them to finish. Actually, most of the threads finished during the first few seconds, except by Execute 8:

To know more about collecting JFR files use, <Note 2574782.1> Troubleshooting WLS Performance Issues With JFR.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.