Slow Performance in Network Provisioning Assistant (STUCK THREAD and TimedOutException) after Oracle Database Upgrade to 12cR2 (Doc ID 2441270.1)

Last updated on JULY 26, 2023

Applies to:

Oracle Communications MetaSolv Solution - Version 6.3.0 and later
Information in this document applies to any platform.

Symptoms

Problem Statement

The Network Provisioning Assistant causes a STUCK thread and eventually throws a transaction "TimedOutException" while searching for the Working Path between an Entry and Exit Network Elements.

Error observed in the GUI

javax.ejb.EJBException: Transaction Rollback.; nested exception is: weblogic.transaction.internal.TimedOutException: Transaction timed out after 900 seconds BEA1-044886E7B156DBC4FDF2

Error observed in the ${SERVER_NAME}.mss.log file

<[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "612" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@7b80dcf3[POST /main/paSelectWorkingPath.do]
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
oracle.net.ns.Packet.receive(Packet.java:311)
oracle.net.ns.DataPacket.receive(DataPacket.java:105)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:957)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1111)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4845)
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1501)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:144)
com.metasolv.jdbc.MSVPreparedStatement.executeQuery(MSVPreparedStatement.java:77)
MetaSolv.persistent.jdbc.JDSC.issueSelect(JDSC.java:937)
com.mslv.core.api.internal.NetProv.ConnDesign.query.AvailableChannelQuery.getFirstRateCodeMatch(AvailableChannelQuery.java:356)
com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.PathFinderBaseBusObj.findAvailableChannel(PathFinderBaseBusObj.java:1953)
com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.PathFinderBaseBusObj.findAvailableChannel(PathFinderBaseBusObj.java:1852)
com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.PathFinderBaseBusObj.verifyPath(PathFinderBaseBusObj.java:1696)
com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.WorkingPathFinderBusObj.verifyWorkingPathWithSecondary(WorkingPathFinderBusObj.java:470)
com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.WorkingPathFinderBusObj.findWorkingPathsWithProtectPath(WorkingPathFinderBusObj.java:419)
com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.WorkingPathFinderBusObj.getPaths(WorkingPathFinderBusObj.java:109)
com.mslv.ejb.netprov.query.connDesign.PathFinderQuery.performQueryArrayList(PathFinderQuery.java:60)
com.mslv.ejb.BaseQueryArrayList.getResults(BaseQueryArrayList.java:59)
com.mslv.ejb.MSLVEJB3SessionBean.getResults(MSLVEJB3SessionBean.java:478)
com.mslv.ejb.netprov.ConnectionDesignBean.getPaths(ConnectionDesignBean.java:219)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)
com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:101)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
sun.reflect.GeneratedMethodAccessor431.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
com.sun.proxy.$Proxy331.getPaths(Unknown Source)
com.mslv.ejb.netprov.ConnectionDesign_rx851k_IConnectionDesignImpl.__WL_invoke(Unknown Source)
weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invokeInternal(SessionRemoteMethodInvoker.java:54)
weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21)
com.mslv.ejb.netprov.ConnectionDesign_rx851k_IConnectionDesignImpl.getPaths(Unknown Source)
com.mslv.ejb.netprov.ConnectionDesign_rx851k_IConnectionDesignImpl_CBV.getPaths(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:86)
com.sun.proxy.$Proxy228.getPaths(Unknown Source)
com.mslv.ejb.netprov.ConnectionDesignHandler.getPaths(ConnectionDesignHandler.java:216)
com.mslv.webapp.netprov.data.PaSelectPathData.getWorkingPaths(PaSelectPathData.java:70)
com.mslv.webapp.netprov.action.PaSelectWorkingPathAction.execute(PaSelectWorkingPathAction.java:311)
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
com.mslv.webapp.filter.ParamFilter.doFilter(ParamFilter.java:39)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:626)
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:282)
org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1069)
org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:274)
org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:455)
org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:320)
com.mslv.webapp.context.action.MSLVRequestProcessor.processForwardConfig(MSLVRequestProcessor.java:340)
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279)
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
.....
.....
weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)
weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
>

javax.ejb.EJBException: Transaction Rolledback.; nested exception is:
weblogic.transaction.internal.TimedOutException: Transaction timed out after 900 seconds
BEA1-044886E7B156DBC4FDF2
at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.unwrapRemoteException(RemoteBusinessIntfProxy.java:127)
at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:99)
at com.sun.proxy.$Proxy228.getPaths(Unknown Source)
at com.mslv.ejb.netprov.ConnectionDesignHandler.getPaths(ConnectionDesignHandler.java:216)
at com.mslv.webapp.netprov.data.PaSelectPathData.getWorkingPaths(PaSelectPathData.java:70)
at com.mslv.webapp.netprov.action.PaSelectWorkingPathAction.execute(PaSelectWorkingPathAction.java:311)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.mslv.webapp.filter.ParamFilter.doFilter(ParamFilter.java:39)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:626)
at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:282)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1069)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:274)
at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:455)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:320)
at com.mslv.webapp.context.action.MSLVRequestProcessor.processForwardConfig(MSLVRequestProcessor.java:340)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
.....
.....
at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
Caused by: weblogic.transaction.internal.TimedOutException: Transaction timed out after 900 seconds
BEA1-044886E7B156DBC4FDF2
at weblogic.transaction.internal.ServerTransactionImpl.wakeUp(ServerTransactionImpl.java:2044)
at weblogic.transaction.internal.ServerTransactionManagerImpl.processTimedOutTransactions(ServerTransactionManagerImpl.java:1959)
at weblogic.transaction.internal.TransactionManagerImpl.wakeUp(TransactionManagerImpl.java:2447)
at weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTransactionManagerImpl.java:1871)
at weblogic.transaction.internal.WLSTimer.timerExpired(WLSTimer.java:19)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:290)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:666)
... 7 more

Error observed in the appserverlog.xml file

<log4j:event logger="cmm.MSLVSessionBean" timestamp="1524485842987" level="ERROR" thread="[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" dateTime="Mon Apr 23 08:17:22 EDT 2018" debugCode="0" moduleName="cmm.MSLVSessionBean" messageID="10119" appServerName="DEF_APPSERVER" className="DEF_CLASS" userName="<USER>" machineName="<MACHINENAME>" productName="nur">
<log4j:message><![CDATA[The server encountered the following Exception: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

   at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
   at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
   at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059)
   at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)
   at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
   at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
   at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
   at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
   at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
   at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:957)
   at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1111)
   at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798)
   at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4845)
   at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1501)
   at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:144)
   at com.metasolv.jdbc.MSVPreparedStatement.executeQuery(MSVPreparedStatement.java:77)
   at MetaSolv.persistent.jdbc.JDSC.issueSelect(JDSC.java:937)
   at com.mslv.core.api.internal.NetProv.ConnDesign.query.AvailableChannelQuery.getFirstRateCodeMatch(AvailableChannelQuery.java:356)
   at com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.PathFinderBaseBusObj.findAvailableChannel(PathFinderBaseBusObj.java:1953)
   at com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.PathFinderBaseBusObj.findAvailableChannel(PathFinderBaseBusObj.java:1852)
   at com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.PathFinderBaseBusObj.verifyPath(PathFinderBaseBusObj.java:1696)
   at com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.WorkingPathFinderBusObj.verifyWorkingPathWithSecondary(WorkingPathFinderBusObj.java:470)
   at com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.WorkingPathFinderBusObj.findWorkingPathsWithProtectPath(WorkingPathFinderBusObj.java:419)
   at com.mslv.core.api.internal.NetProv.ConnDesign.provisioning.busObj.WorkingPathFinderBusObj.getPaths(WorkingPathFinderBusObj.java:109)
   at com.mslv.ejb.netprov.query.connDesign.PathFinderQuery.performQueryArrayList(PathFinderQuery.java:60)
   at com.mslv.ejb.BaseQueryArrayList.getResults(BaseQueryArrayList.java:59)
   at com.mslv.ejb.MSLVEJB3SessionBean.getResults(MSLVEJB3SessionBean.java:478)
   at com.mslv.ejb.netprov.ConnectionDesignBean.getPaths(ConnectionDesignBean.java:219)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
   at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
   at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
   at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)
   at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:101)
   at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)
   at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
   at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
   at sun.reflect.GeneratedMethodAccessor431.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94)
   at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
   at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
   at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
   at com.sun.proxy.$Proxy331.getPaths(Unknown Source)
   at com.mslv.ejb.netprov.ConnectionDesign_rx851k_IConnectionDesignImpl.__WL_invoke(Unknown Source)
   at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invokeInternal(SessionRemoteMethodInvoker.java:54)
   at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21)
   at com.mslv.ejb.netprov.ConnectionDesign_rx851k_IConnectionDesignImpl.getPaths(Unknown Source)
   at com.mslv.ejb.netprov.ConnectionDesign_rx851k_IConnectionDesignImpl_CBV.getPaths(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:86)
   at com.sun.proxy.$Proxy228.getPaths(Unknown Source)
   at com.mslv.ejb.netprov.ConnectionDesignHandler.getPaths(ConnectionDesignHandler.java:216)
   at com.mslv.webapp.netprov.data.PaSelectPathData.getWorkingPaths(PaSelectPathData.java:70)
   at com.mslv.webapp.netprov.action.PaSelectWorkingPathAction.execute(PaSelectWorkingPathAction.java:311)
   at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
   at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
   at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
   at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
    .....
   .....
   at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)
   at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
]]></log4j:message>
</log4j:event>

Recreation Steps

1. Navigate to Connection Design window and search for a Special/Facility circuit

2. Open a Special circuit from the search result window.

3. Go to the CLR/DLR Design page and click the Network Action link

4. Provide the Network Name in the Entry Network Element tab and then click Next button on the left corner.

5. Select any one of the Entry Network Element and then click Next button.

6. Select an Exit Network Element. Enter the Entry and Exit Network Element Limits. Select the 'Use Rapid Path Validation'/any path (as required) checkbox.
Select Next option or Auto Assign option.

Limit the path search to [10] paths of no more than [7]hops
Use Rapid Path Validation. Custom Tuning: Fastest results, Fewest Paths
Sort the results by (*) hops

7. The Provisioning Assistant window ends with below timed out error.

javax.ejb.EJBException: Transaction Rollback.; nested exception is: weblogic.transaction.internal.TimedOutException: Transaction timed out after 900 seconds BEA1-044886E7B156DBC4FDF2

* NOTE: The Same network with same circuit can be searchable in existing 12cR1 environments without any issues. The search results retrieved in 4-5 seconds approximately in production.