REMOTEJVMCONNECTION TIMEOUT WHEN ATTEMPTING TO START MULTIPLE INSTANCES OF CCB In Managed Server Cluster (Doc ID 2168478.1)

Last updated on AUGUST 18, 2016

Applies to:

Oracle Utilities Customer Care and Billing - Version 2.3.1 and later
Information in this document applies to any platform.

Symptoms

In a managed server cluster environment, when attempting to start CCB on multiple servers at once, the following errors may be encountered, and no applications are started.:

- 2016-06-22 14:16:23,866 [JVM 5 Ping] ERROR (cobol.host.SocketStrategy) Unable to establish connection on port 11437 after waiting 20 seconds.
java.net.ConnectException: Cannot connect to socket '/ccbsmanagedserver/SPTX/runtime/splSock11437' caused by: 146 Connection refused


- 2016-06-07 16:21:28,536 [Parent Reader:Thread-26] ERROR (cobol.host.OptimizedRemoteExecuterSkel) An exception occurred executing a remote command.
java.util.NoSuchElementException

- 2016-06-07 16:21:39,266 [JVM 1 ERROR logger] ERROR (cobol.host.ProcessLogger) java.lang.RuntimeException: No command runner was registered with this remote JVM after waiting 10000ms
- 2016-06-07 16:21:39,376 [JVM 1 ERROR logger] ERROR (cobol.host.ProcessLogger) at com.splwg.base.support.cobol.host.RemoteJVM.waitForServerToRegisterRunner(RemoteJVM.java:195)
- 2016-06-07 16:21:39,486 [JVM 1 ERROR logger] ERROR (cobol.host.ProcessLogger) at com.splwg.base.support.cobol.host.RemoteJVM.main(RemoteJVM.java:153)
- 2016-06-07 16:21:49,338 [pool-2-thread-1] ERROR (cobol.host.SocketStrategy) Unable to establish connection on port 11440 after waiting 20 seconds.
java.net.ConnectException: Cannot connect to socket '/ccbsmanagedserver/SPTX/runtime/splSock11440' caused by: 2 No such file or directory
  at com.splwg.base.support.cobol.host.sockets.UnixDomainSocketNative.connectDomainSocket(Native Method)
  at com.splwg.base.support.cobol.host.sockets.UnixDomainSocket.connectDomainSocket(UnixDomainSocket.java:87)
  at com.splwg.base.support.cobol.host.sockets.UnixDomainSocketStrategy.openSocket(UnixDomainSocketStrategy.java:40)
  at com.splwg.base.support.cobol.host.SocketStrategy.unsafeOpen(SocketStrategy.java:119)
  at com.splwg.base.support.cobol.host.SocketStrategy.open(SocketStrategy.java:104)
  at com.splwg.base.support.cobol.host.OptimizedRemoteExecuterStub.createNewHolder(OptimizedRemoteExecuterStub

 

- 2016-06-22 14:16:23,866 [JVM 5 Ping] ERROR (cobol.host.SocketStrategy) Unable to establish connection on port 11437 after waiting 20 seconds.
java.net.ConnectException: Cannot connect to socket '/ccbsmanagedserver/SPTX/runtime/splSock11437' caused by: 146 Connection refused

at com.splwg.base.support.cobol.host.sockets.UnixDomainSocketNative.connectDomainSocket(Native Method)
at com.splwg.base.support.cobol.host.sockets.UnixDomainSocket.connectDomainSocket(UnixDomainSocket.java:87)
at com.splwg.base.support.cobol.host.sockets.UnixDomainSocketStrategy.openSocket(UnixDomainSocketStrategy.java:40)
at com.splwg.base.support.cobol.host.SocketStrategy.unsafeOpen(SocketStrategy.java:119)
at com.splwg.base.support.cobol.host.SocketStrategy.open(SocketStrategy.java:104)
at com.splwg.base.support.cobol.host.OptimizedRemoteExecuterStub.createNewHolder(OptimizedRemoteExecuterStub.java:169)
at com.splwg.base.support.cobol.host.OptimizedRemoteExecuterStub.getStreamHolder(OptimizedRemoteExecuterStub.java:132)
at com.splwg.base.support.cobol.host.OptimizedRemoteExecuterStub.invoke(OptimizedRemoteExecuterStub.java:52)
at com.splwg.base.support.cobol.host.RemoteRunnerImpl.invoke(RemoteRunnerImpl.java:111)
at com.splwg.base.support.cobol.host.RemoteJVMConnectionImpl$PingChildRunnable.performPingLoop(RemoteJVMConnectionImpl.java:386)
at com.splwg.base.support.cobol.host.RemoteJVMConnectionImpl$PingChildRunnable.run(RemoteJVMConnectionImpl.java:374)
at java.lang.Thread.run(Thread.java:662)
- 2016-06-22 14:16:23,869 [JVM 5 Ping] ERROR (cobol.host.SocketStrategy) Unable to create socket for inter JVM communication on port 11437
- 2016-06-22 14:16:23,869 [JVM 5 Ping] INFO (cobol.host.RemoteJVMConnectionImpl) Connection to JVM 5 being shunned
- 2016-06-22 14:16:23,869 [JVM 5 Ping] ERROR (cobol.host.RemoteJVMConnectionImpl) An exception has occurred calling the remote JVM
- 2016-06-22 14:16:23,869 [JVM 5 Ping] ERROR (cobol.host.RemoteJVMConnectionImpl) Error encountered while pinging child JVM 5. Will 'shun' that JVM.
com.splwg.shared.common.LoggedException:

 

 

####<May 31, 2016 10:31:03 AM EDT> <Warning> <HTTP> <m5000-omf-15.ceg.corp.net> <CCBSPT_AGY_SERVER1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1464705063506> <BEA-101304> <Webapp: ServletContext@849139661[app:SPLWebAgy module:/spl path:/spl spec-version:2.5], the role: cisusers defined in web.xml has not been mapped to principals in security-role-assignment in weblogic.xml. Will use the rolename itself as the principal-name.>

com.splwg.base.web.common.WebInstallationDataHelper.outputInstallationData(WebInstallationDataHelper.java:64): Error retrieving Installation Data
####<May 31, 2016 10:31:00 AM EDT> <Warning> <Munger> <m5000-omf-15.ceg.corp.net> <CCBSPT_AGY_SERVER1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1464705060020> <BEA-2156203> <A version attribute was not found in element application in the deployment descriptor in /apps/ouapps/managed_server/spt/weblogic/user_projects/domains/CCBSPT_DOMAIN/servers/CCBSPT_AGY_SERVER1/tmp/_WL_user/SPLServiceAgy/2y0h1i/META-INF/application.xml. A version attribute is required, but this version of the Weblogic Server will assume that the JEE5 is used. Future versions of the Weblogic Server will reject descriptors that do not specify the JEE version.>

####<May 31, 2016 10:36:25 AM EDT> <Warning> <HTTP> <m5000-omf-22.ceg.corp.net> <CCBSPT_AGY_SERVER2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1464705385796> <BEA-101162> <User defined listener com.splwg.base.web.startup.SPLWebStartup failed: com.splwg.shared.common.LoggedException:
The following stacked messages were reported as the LoggedException was rethrown:
com.splwg.base.api.service.ServiceDispatcher$13.execute(ServiceDispatcher.java:569): Caught exception from SessionExecutable.execute()
com.splwg.base.support.service.JavaPageReadService$1.execute(JavaPageReadService.java:26): Rollback: Read failed
com.splwg.base.web.common.WebInstallationDataHelper.outputInstallationData(WebInstallationDataHelper.java:64): Error retrieving Installation Data
com.splwg.base.support.service.cobol.CobolPageRereadHelper.read(CobolPageRereadHelper.java:34): Encountered exception, session marked for rollback

The root LoggedException was: Exceeded maximum timeout of 60 secs while attempting to obtain a RemoteJVMConnection.
com.splwg.shared.common.LoggedException:
The following stacked messages were reported as the LoggedException was rethrown:
com.splwg.base.api.service.ServiceDispatcher$13.execute(ServiceDispatcher.java:569): Caught exception from SessionExecutable.execute()
com.splwg.base.support.service.JavaPageReadService$1.execute(JavaPageReadService.java:26): Rollback: Read failed
com.splwg.base.web.common.WebInstallationDataHelper.outputInstallationData(WebInstallationDataHelper.java:64): Error retrieving Installation Data
com.splwg.base.support.service.cobol.CobolPageRereadHelper.read(CobolPageRereadHelper.java:34): Encountered exception, session marked for rollback

The root LoggedException was: Exceeded maximum timeout of 60 secs while attempting to obtain a RemoteJVMConnection
at com.splwg.shared.common.LoggedException.raised(LoggedException.java:65)
at com.splwg.base.support.cobol.host.RotatingCommandRunnerProvider.getLeastLoadedRunner(RotatingCommandRunnerProvider.java:234)
at com.splwg.base.support.cobol.host.RotatingCommandRunnerProvider.getLeastLoadedRunner(RotatingCommandRunnerProvider.java:228)

 

 

It should be noted that the above does not occur if each application is started one at a time on each server.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms