12.x Demantra Worksheet Hangs- WebLogic Server in Hung State and Needs to be Bounced. Server Thread Dump Shows HashMap.get + Collaborator log may Show java.util.concurrent.ConcurrentHashMap.put (Doc ID 1919369.1)

Last updated on OCTOBER 21, 2024

Applies to:

Oracle Demantra Demand Management - Version 12.0 and later
Oracle Demantra Predictive Trade Planning - Version 12.0 and later
Information in this document applies to any platform.

Symptoms

Introduction

Demantra Customer, This note has several sections. Please see the 'Alternate Cause Investigation' to verify that you are not experiencing a worksheet slow down, weblogic hang or worksheet performance issue caused by typical scenarios.

This situation would most likely be exposed after an upgrade within the Demantra environment. This is not something that would 'suddenly' appear without environment update manipulation.

In this case, the Demantra user base reported:

They mentioned that they experience a condition where no worksheets will open in Demantra. You can sign in and out, and traverse the hierarchy on the main page, but when you try to open a worksheet it renders the headings but the worksheet cells and data do not appear. Instead the clock spins endlessly. Eventually they cannot work. The system goes in a hung state and they have to bounce the weblogic services to be able to access the system again.

Eventually, the same problem will resurface. This may take some time however it may not. PLEASE examine your logs and do verify that there is not a memory problem etc., as directed below.

Whenever there is a 'Hung Process' or 'Hang' happening on any server, the first step that you are advised to take is a Weblogic thread dump.

When you restart Weblogic, this will ensure that you clear cache / tmp folder and store folder under the specific weblogic server where your component is deployed.

Once you clear them you can restart them again.

If you need to verify Weblogic, Please Review <Document 1099996.1> How To Sart/Stop A Discoverer 11g Unix/Linux Environment Via Command-Line.

Here are a few symptoms while any App becomes Hung.

Specific resources are busy with other work resource or services like JMS, JDBC or any other services.
Causes due to lack enough Heap availability.
Also caused due to Garbage Collection (GC) not happening.
Happens due to your process being corrupt. However, restarting your personal work device, thus ensuring your own GC, and then attempting the process is always a good personal starting point.
Happens due to Network down or cluster communication drops or not able to access Admin Server URL.

Please proceed to the log investigation.

WebLogic Log

Error message in WebLogic server log is:
#### <> <> <> POST /demantra/RequestDispatcherServlet?action=task&tkn=-180860591116283 HTTP/1.1
Cookie: ORA_EBS_DEMANTRA_LOGIN_LANGUAGE=US; JSESSIONID=553bT1mMhPsb4c1bQ2n9lNrQHN0vXQvQxWXvGm1XVN2Lp2j870j8!648230417;
User-Agent: Jakarta Commons-HttpClient/3.0.1
Content-Length: 192
Content-Type: application/octet-stream

]", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
com.demantra.applicationServer.services.queryrun.DataProcessor.cancelRun(DataProcessor.java:158)
com.demantra.applicationServer.services.queryrun.DataProcessor.dispose(DataProcessor.java:120)
com.demantra.applicationServer.services.queryrun.BaseDataRunner.dispose(BaseDataRunner.java:265)
com.demantra.bridge.server.SoapBridge.dispose(SoapBridge.java:2772)
com.demantra.tunnel.server.RemoteFacadeRegistry.disposeFacade(RemoteFacadeRegistry.java:222)
com.demantra.tunnel.task.operations.InvokeTaskVisitor.visit(InvokeTaskVisitor.java:171)
com.demantra.tunnel.task.DisposeFacadeTask.acceptTaskVisitor(DisposeFacadeTask.java:37)
com.demantra.tunnel.task.operations.InvokeTaskVisitor.invoke(InvokeTaskVisitor.java:116)
com.

WebLogic Thread Dump

Follow these notes to produce proper thread dumps:

Different ways to take thread dumps in WebLogic Server (Doc ID 1098691.1)

The thread dump may contain messages such as:

"pool-1-thread-1" RUNNABLE

               java.util.HashMap.get(HashMap.java:303)

               com.demantra.applicationServer.services.queryrun.SubDataProcessorTask$NonTimeDependentSeriesValuesFiller.dataExistsForSeries(SubDataProcessorTask.java:1192)

               com.demantra.applicationServer.services.queryrun.SubDataProcessorTask.processValues(SubDataProcessorTask.java:635)

               com.demantra.applicationServer.services.queryrun.SubDataProcessorTask.processSubCombDataValues(SubDataProcessorTask.java:588)

               com.demantra.applicationServer.services.queryrun.SubDataProcessorTask.access$1000(SubDataProcessorTask.java:61)

               com.demantra.applicationServer.services.queryrun.SubDataProcessorTask$TransitionalQueryRecordCreatorCallable.call(SubDataProcessorTask.java:1463)

               com.demantra.applicationServer.services.queryrun.SubDataProcessorTask$TransitionalQueryRecordCreatorCallable.call(SubDataProcessorTask.java:1325)

               java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

               java.util.concurrent.FutureTask.run(FutureTask.java:138)

               java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

               java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

               java.lang.Thread.run(Thread.java:662)

            "tunneling-thread-_0" waiting for lock com.demantra.tunnel.server.RoundRobinQueue@63efcccc TIMED_WAITING

               java.lang.Object.wait(Native Method)

               com.demantra.tunnel.server.RoundRobinQueue.poll(RoundRobinQueue.java:233)

               java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)

               java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

               java.lang.Thread.run(Thread.java:662)

            "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.lang.Object@1bd51f02 WAITING

               java.lang.Object.wait(Native Method)

               java.lang.Object.wait(Object.java:485)

Collaborator Log

The collaborator logs do not have except following error message:

2014-06-04 13:22:33,231 EDT [[ACTIVE] ExecuteThread: '15' for queue:
'weblogic.kernel.Default (self-tuning)'] WARN tunnel.general: ODPM-10306:
Facade not found [77] 2014-06-04 13:23:49,344 EDT [[ACTIVE] ExecuteThread:
'23' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN tunnel.general:
ODPM-10306: Facade not found [3] 2014-06-04 13:24:09,172 EDT [[ACTIVE]
ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN
tunnel.general: ODPM-10306: Facade not found [305] 2014-06-04 13:24:47,642
EDT

ADDITIONALLY, the collaborator log could have something similar to this:

2013-09-04 14:45:49,607 CDT [DataProcessorManager_2] ERROR appserver.thread:
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.put(Unknown Source)
at
com.demantra.applicationServer.services.queryrun.SubDataProcessorTask$NonTimeD
ependentSeriesValuesFiller.<init>(SubDataProcessorTask.java:1080)

Java Process Stack Trace

<Document 1246024.1> - How to Obtain a Thread Dump (Stack Traces) From a Java Process or Core File of a Java Process on Windows

The stack trace heap dumps, show the following:
"pool-1-thread-18" prio=6 tid=0x000000000cf15000 nid=0x984 runnable
[0x000000001326f000]
    java.lang.Thread.State: RUNNABLE
          at java.util.HashMap.get(HashMap.java:303)
          at
com.demantra.applicationServer.services.queryrun.SubDataProcessorTask$NonTimeD
ependentSeriesValuesFiller.dataExistsForSeries(SubDataProcessorTask.java:1192)
.
          at
com.demantra.applicationServer.services.queryrun.SubDataProcessorTask.processV
alues(SubDataProcessorTask.java:635)
          at
com.demantra.applicationServer.services.queryrun.SubDataProcessorTask.processS
ubCombDataValues(SubDataProcessorTask.java:588)
          at
com.demantra.applicationServer.services.queryrun.SubDataProcessorTask.access$1
000(SubDataProcessorTask.java:61)
          at
com.demantra.applicationServer.services.queryrun.SubDataProcessorTask$Transiti
onalQueryRecordCreatorCallable.call(SubDataProcessorTask.java:1463)
          at
com.demantra.applicationServer.services.queryrun.SubDataProcessorTask$Transiti
onalQueryRecordCreatorCallable.call(SubDataProcessorTask.java:1325)
          at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java
:886)
          at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908
)
          at java.lang.Thread.run(Thread.java:662)

Alternate Cause Investigation

Perform JVM profiling.
For example: what is the max heap size you are running the application?
Check the CPU memory when the application server has just loaded and there are no users connected.
When the deadlock appears, what exactly is stuck?
If you will review the stack threads it may be rightly blocked because some other session has the data locked in a transaction, or it may be a driver/DBMS miscommunication where the DBMS thinks the client is idle, and the client (driver) has not gotten what it needs in a response.

In this case ask your DBA to find any sessions that seem blocked for minutes and what they are blocked on. You may be able to tell from the stack trace what

data/SQL the application is doing that is blocked. That info will help the DBA find who may be holding locks on that data.
Verify that the database can handle the desired jdbc connections.
Verify errors in the RDBMS alert.log
Open the log category appserver.sql through Logger Manager and continue to work. Once you see this message "Facade not found" again please review the collaborator.log file.

"Facade Not Found" message usually appears when there is bad communication between the client and the database.