My Oracle Support Banner

11.1.1.9.0 FMWC:Console Login And Navigation Is Slow In A Clustered BI Environment (Doc ID 2248591.1)

Last updated on SEPTEMBER 28, 2018

Applies to:

Enterprise Manager for Fusion Middleware - Version 11.1.1.9.0 to 11.1.1.9.0
Information in this document applies to any platform.

Symptoms

FMWC console is slow to login and performance is slow navigating pages.
It can take up to 30 minutes to login and navigating the different targets in the clustered environment can take similar time frames.

Analysis of the emoms.trc as well as the WebLogic Server AdminServer logs may reveal the following similar errors.

The $MW_HOME/user_projects/domains/base_domain/servers/AdminServer/sysman/log/emoms.trc shows this:


[2017-01-11T11:30:04.507-05:00] [AdminServer] [NOTIFICATION:28] [] [oracle.sysman.emas.sdk.model.mas.WLTransientTargetProvider] [tid: [ACTIVE].ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 2b709778113eb4dd:-66788c61:1598e5a5a54:-8000-0000000000000046,0] [APP: em] Discovery of Weblogic Targets in pre cache took :24906ms User=weblogic TargetsCount=160

...

[2017-01-11T12:06:44.805-05:00] [AdminServer] [NOTIFICATION:24] [] [oracle.sysman.core.app.menu.XMLMenuManager] [tid: [STUCK].ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 2b709778113eb4dd:-66788c61:1598e5a5a54:-8000-00000000000000a5,0] [APP: em] The Document for grid menu is not found.

 

The $MW_HOME/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log shows the following:


####<Jan 11, 2017 12:00:59 PM EST> <Error> <WebLogicServer> <hostname> <AdminServer> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <2b709778113eb4dd:-66788c61:1598e5a5a54:-8000-0000000000000164> <1484154059942> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,852" seconds working on the request "Workmanager: default, Version: 1, Scheduled=true, Started=true, Started time: 1852190 ms
[
GET /em/faces/as/as/wlFarmHome?target=Farm_bifoundation_domain&type=oracle_ias_farm&_afrLoop=9064242385306541 HTTP/1.1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: http://hostname:7001/em/faces/targetauth/emasLogin?target=Farm_bifoundation_domain&type=oracle_ias_farm&FromContextInitFilter=true&_afrLoop=9064187247005671&_afrWindowMode=0&_afrWindowId=null&_adf.ctrl-state=mqkbz368g_2
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,und;q=0.6
Cookie: JSESSIONID=pZ6OXXaUXYHD6HpAxGXdCUPdronQqyGVgjiT6HLzdwdpT41oGMZv!296536081; oracle.uix=0^^GMT-4:00^p

]", which is more than the configured time (StuckThreadMaxTime) of "1,800" seconds. Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:238)
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:434)
java.util.concurrent.FutureTask.get(FutureTask.java:211)
oracle.dms.collector.Retriever.waitForQuery(Retriever.java:469)
oracle.dms.collector.CollectorStorage._refreshCache(CollectorStorage.java:1250)
oracle.dms.collector.CollectorStorage.getTables(CollectorStorage.java:674)
oracle.dms.collector.CollectorTableSource.getTables(CollectorTableSource.java:147)
oracle.dms.aggregator.Storage.getTables(Storage.java:712)
oracle.dms.aggregator.Storage._getTables(Storage.java:540)
oracle.dms.aggregator.Storage.getTables(Storage.java:415)
oracle.dms.jmx.app.AggreMBeanImpl.getTableGroupsInOpenType(AggreMBeanImpl.java:249)
oracle.dms.jmx.app.DomainAggreMBeanImpl.getTableGroupsInOpenType(DomainAggreMBeanImpl.java:41)
oracle.dms.jmx.app.AggreMBeanWrapper.invokeOperation(AggreMBeanWrapper.java:218)
oracle.dms.jmx.app.AggreMBeanWrapper.invoke(AggreMBeanWrapper.java:182)
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:832)
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:814)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
java.security.AccessController.doPrivileged(AccessController.java:383)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
java.security.AccessController.doPrivileged(AccessController.java:383)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
weblogic.management.mbeanservers.domainruntime.internal.ManagedMBeanServerConnection.invoke(ManagedMBeanServerConnection.java:333)
weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:351)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)

...


  

The $MW_HOME/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer-diagnostic.log shows the following:

[2017-01-11T12:06:49.155-05:00] [AdminServer] [TRACE:32] [] [oracle.adf.view.rich.component.fragment.UIXRegion] [tid: [STUCK].ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 2b709778113eb4dd:-66788c61:1598e5a5a54:-8000-00000000000000a5,0] [APP: em] [DSID: 0000LaENNaAFg45TzQicKW1OT_n300000G] [SRC_CLASS: oracle.adf.view.rich.component.fragment.UIXRegion] [SRC_METHOD: _beginInterruptibleRegion] UIXRegion._beginInterruptibleRegion()
[2017-01-11T12:06:49.156-05:00] [AdminServer] [TRACE] [] [oracle.dms.collector] [tid: DmsThread-11] [userId: weblogic] [ecid: 2b709778113eb4dd:-66788c61:1598e5a5a54:-8000-00000000000000a5,1:16809] [APP: DMS Application#11.1.1.1.0] [SRC_CLASS: oracle.dms.collector.TbmlMetricFetcher] [SRC_METHOD: call] DMS-50985[[
oracle.dms.collector.DMSConnectException: DMS-50719: unable to connect to the server and retrieve metrics at "{0}"
Cause: Read timed out
Action: Review the exception chain in log files for the cause and make sure that the server starts up.
at oracle.dms.collector.OpmnConnector.connect(OpmnConnector.java:126)
at oracle.dms.collector.TbmlMetricFetcher.call(TbmlMetricFetcher.java:85)
at oracle.dms.collector.TbmlMetricFetcher.call(TbmlMetricFetcher.java:32)
at java.util.concurrent.FutureTask.run(FutureTask.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at oracle.dms.aggregator.DmsThreadFactory$1$1.run(DmsThreadFactory.java:56)
at java.security.AccessController.doPrivileged(AccessController.java:353)
at oracle.dms.aggregator.DmsThreadFactory$1.run(DmsThreadFactory.java:51)
at java.lang.Thread.run(Thread.java:801)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:164)
at java.net.SocketInputStream.read(SocketInputStream.java:134)
at com.ibm.jsse2.a.a(a.java:260)
at com.ibm.jsse2.a.a(a.java:204)
at com.ibm.jsse2.aq.a(aq.java:684)
at com.ibm.jsse2.aq.a(aq.java:432)
at com.ibm.jsse2.e.read(e.java:20)
at com.ibm.jsse2.e.read(e.java:38)
at oracle.dms.collector.OpmnConnector._skipHeader(OpmnConnector.java:220

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.