MetaSolv Solution Application Server Out of Memory; java.lang.OutOfMemoryError: JVMCI015:OutOfMemoryError, cannot create anymore threads due to memory or resource constraints. (Doc ID 1068965.1)

Last updated on JULY 05, 2017

Applies to:

Oracle Communications MetaSolv Solution - Version 6.0.10 to 6.0.16 [Release 6.0.10 to 6.0.16]
IBM AIX on POWER Systems (32-bit)
Reviewed for currency on February 20th, 2013.

Symptoms

Users are unable to connect to MetaSolv Solution application because the application server experienced an "out of memory" condition.  The {SERVER}.mss.log log has several "java.lang.OutOfMemoryError" messages and a heap dump was also generated.  Below are the errors captured in the ${SERVER}.mss.log.

<Feb 5, 2010 10:03:46 AM GMT-04:30> <Info> <EJB> <BEA-010213> <Message-Driven EJB: ProcessTrackingEventListener's transaction was rolledback. The transaction details are: Xid=BEA1-0C718F99429B1FC33BEA(1217763283),Status=Prepared,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=1,seconds left=119,XAServerResourceInfo[JMS_cgJMSStore]=(ServerResourceInfo[JMS_cgJMSStore]=(state=prepared,assigned=MSSD01),xar=JMS_cgJMSStore,re-Registered = false),XAServerResourceInfo[cgPool]=(ServerResourceInfo[cgPool]=(state=prepared,assigned=MSSD01),xar=cgPool,re-Registered = false),SCInfo[m6domain+MSSD01]=(state=prepared),properties=({ISOLATION LEVEL=2}),local properties=({weblogic.jdbc.jta.cgPool=weblogic.jdbc.wrapper.TxInfo@304900f}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=MSSD01+XXX.XXX.XXX.XX:PORT+m6domain+t3+, XAResources={JMS_FileStore, cgPool, MSLVPool, JMS_plugin_NotDurable, JMS_cgJMSStore, MSLVwliPool, bpmArchPool},NonXAResources={})],CoordinatorURL=MSSD01+XXX.XXX.XXX.XX:PORT+m6domain+t3+).>
<Feb 5, 2010 10:03:46 AM GMT-04:30> <Warning> <EJB> <BEA-010065> <MessageDrivenBean threw an Exception in onMessage(). The exception was:
java.lang.OutOfMemoryError: JVMCI015:OutOfMemoryError, cannot create anymore threads due to memory or resource constraints.
java.lang.OutOfMemoryError: JVMCI015:OutOfMemoryError, cannot create anymore threads due to memory or resource constraints
at java.lang.Thread.start(Native Method)
at weblogic.kernel.ExecuteThreadManager.startThreads(ExecuteThreadManager.java(Compiled Code))
at weblogic.kernel.ExecuteThreadManager.setThreadCount(ExecuteThreadManager.java(Compiled Code))
at weblogic.kernel.ExecuteThreadManager.expandThreadPool(ExecuteThreadManager.java(Compiled Code))
at weblogic.kernel.ExecuteThreadManager.execute(ExecuteThreadManager.java(Compiled Code))
at weblogic.kernel.Kernel.execute(Kernel.java(Compiled Code))
at weblogic.kernel.Kernel.execute(Kernel.java(Inlined Compiled Code))
at weblogic.kernel.Kernel.execute(Kernel.java(Inlined Compiled Code))
at weblogic.transaction.internal.TransactionLoggerImpl.store(TransactionLoggerImpl.java(Inlined Compiled Code))
at weblogic.transaction.internal.ServerTransactionImpl.log(ServerTransactionImpl.java(Compiled Code))
at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java(Compiled Code))
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java(Compiled Code))
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java(Compiled Code))
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:430)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:333)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:298)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2686)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2598)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java(Compiled Code))
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

 

<Feb 5, 2010 10:03:48 AM GMT-04:30> <Error> <WLW> <000000> <Exception processing com.company.mslv.custom.jpd.order.QueryOrderHistory
java.lang.OutOfMemoryError: JAVA002:OutOfMemoryError, malloc for body failed
at java.lang.ClassLoader.defineClass0(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java(Compiled Code))
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java(Compiled Code))
at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java(Compiled Code))
at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java(Compiled Code))
at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
at java.lang.ClassLoader.loadClass(ClassLoader.java(Inlined Compiled Code))
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java(Compiled Code))
at com.bea.wlw.runtime.core.dispatcher.DispUnit.<init>(DispUnit.java:110)
at com.bea.wlw.runtime.core.dispatcher.DispCache.ensureDispUnit(DispCache.java:578)
at com.bea.wlw.runtime.core.dispatcher.DispCache.ensureDispUnitForURI(DispCache.java:967)
at com.bea.wlw.runtime.core.dispatcher.DispCache.ensureDispUnitForURI(DispCache.java:889)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.jcs:415)
at com.bea.wlw.runtime.core.control.ServiceControlImpl.invoke(ServiceControlImpl.jcs:1155)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.jcs:378)
at com.bea.wlw.runtime.core.dispatcher.DispMethod.invoke(DispMethod.java:377)
at com.bea.wlw.runtime.core.container.Invocable.invoke(Invocable.java:433)
at com.bea.wlw.runtime.core.container.Invocable.invoke(Invocable.java:406)
at com.bea.wlw.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:388)
at $Proxy45.doQueryOrderHistory(Unknown Source)
at com.company.mslv.custom.jpd.customProcessFeasibilityChangePlan.queryOrderHistoryPControlDoQueryOrderHistory(customProcessFeasibilityChangePlan.jpd:194)
at com.company.mslv.custom.jpd.customProcessFeasibilityChangePlan_wf$ImplControlSend3.invoke(customProcessFeasibilityChangePlan_wf.java:50)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:32)
at com.bea.wli.bpm.runtime.Receive.messageDelivery(Receive.java:91)
at com.bea.wli.bpm.runtime.Receive.messageDeliveryAction(Receive.java:71)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:176)
at com.company.mslv.custom.jpd.customProcessFeasibilityChangePlan_wf$_ProcessState.processMessage(customProcessFeasibilityChangePlan_wf.java:339)
at com.company.mslv.custom.jpd.customProcessFeasibilityChangePlan_wf.clientRequest(customProcessFeasibilityChangePlan_wf.java:319)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.bea.wlw.runtime.core.dispatcher.DispMethod.invoke(DispMethod.java:371)
at com.bea.wlw.runtime.core.container.Invocable.invoke(Invocable.java:433)
at com.bea.wlw.runtime.core.container.Invocable.invoke(Invocable.java:406)
at com.bea.wlw.runtime.core.container.Invocable.invoke(Invocable.java:249)
at com.bea.wlw.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wlw.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:109)
at com.bea.wlwgen.customProcessFeasibilityChangePlanSLSBContImpl.clientRequest(customProcessFeasibilityChangePlanSLSBContImpl.java:25)
at com.bea.wlwgen.StatelessContainer_a13nud_ELOImpl.clientRequest(StatelessContainer_a13nud_ELOImpl.java:99)
at com.bea.wlwgen.customProcessFeasibilityChangePlanSLSBContAdpt.invokeOnBean(customProcessFeasibilityChangePlanSLSBContAdpt.java:58)
at com.bea.wlw.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:153)
at com.bea.wlw.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wlw.runtime.core.bean.AsyncDispatcherBean.onMessage(AsyncDispatcherBean.java:248)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:400)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:333)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:298)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2686)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2598)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java(Compiled Code))
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
>

 

<[ServletContext(id=783388677,name=main,context-path=/main)] action: MSLVActionServlet: Permission: 4 granting access.>
JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMDG315: JVM Requesting Heap dump file
..........................................................................................................................JVMDG318: Heap dump file written to /opt1/mslv_home/m6domain/heapdump184482.1265810434.phd
JVMDG303: JVM Requesting Java core file



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms