Gateway Tomcat & Zookeeper is Not Responding After Some Time of Load Test & Srvrmgr Is Taking Huge Time to Connect During This Time (Doc ID 2619872.1)

Last updated on JANUARY 16, 2023

Applies to:

Siebel CRM - Version 19.4 and later
Information in this document applies to any platform.

Symptoms

While running load test with One Siebel Server, after few mins Gateway Tomcat & Zookeeper are not responding and also srvrmgr is taking huge time to connect,

Steps:
---------

Install Gateway, Siebel Server & AI on separate machine
Configure Siebel
Run load test for 500 users
After few mins Gateway tomcat CPU is 100% and zookeeper is not responding with zkCli command
Srvrmgr also taking nearly 30 mins to connect

Below Logs are verified:

CommonLogger.log

[DEBUG] 2019-11-27 14:30:08.688 [http-nio-8000-exec-2] CommonLogger - com.siebel.swsm.util.CGUtil:initializeServiceDiscovery Unable to discover cloud gateway service.Check system configuration, network connection,node availability and zookeeper configuration
com.siebel.opcgw.utils.common.exception.GatewayRegistryGenericException: null
at com.siebel.opcgw.utils.servicediscovery.ServiceDiscoveryInternal.discover(ServiceDiscoveryInternal.java:103) ~[Siebel.jar:?]
at com.siebel.opcgw.utils.servicediscovery.ServiceDiscovery.discover(ServiceDiscovery.java:95) ~[Siebel.jar:?]
at com.siebel.swsm.util.CGUtil.initializeServiceDiscovery(CGUtil.java:201) [ai.jar:?]
at com.siebel.swsm.util.CGUtil.cgGet(CGUtil.java:37) [ai.jar:?]
at com.siebel.swsm.store.ConfigStore.getConnString(ConfigStore.java:685) [ai.jar:?]
at com.siebel.swsm.store.ConfigStore.getAppConnString(ConfigStore.java:667) [ai.jar:?]
at com.siebel.swsm.store.ConfigStore.getProperty(ConfigStore.java:265) [ai.jar:?]
at com.siebel.swsm.connmgr.SessionConnectionMgr.createNewSession(SessionConnectionMgr.java:69) [ai.jar:?]
at com.siebel.swsm.connmgr.SessionConnectionMgr.createNewSession(SessionConnectionMgr.java:57) [ai.jar:?]
at com.siebel.swsm.sessmgr.UIConnectionModule.createNewSession(UIConnectionModule.java:304) [ai.jar:?]
at com.siebel.swsm.sessmgr.UIConnectionModule.getSession(UIConnectionModule.java:373) [ai.jar:?]
at com.siebel.swsm.sessmgr.UIConnectionModule.processModule(UIConnectionModule.java:52) [ai.jar:?]
at com.siebel.swsm.handlers.UIHandler.processRequest(UIHandler.java:60) [ai.jar:?]

[FATAL] 2019-11-27 14:34:18.315 [http-nio-8000-exec-10] CommonLogger - com.siebel.swsm.util.CGUtil:initializeServiceDiscovery Unable to discover cloud gateway service.Check system configuration, network connection,node availability and zookeeper configuration
[WARN ] 2019-11-27 14:34:18.315 [http-nio-8000-exec-10] CommonLogger - com.siebel.swsm.util.CGUtil:cgGet Unable to fetch CGHost URI from Service Discovery. Using the default CGHost from config file : https://GTWYHOST:9443/siebel/v1.0/cloudgateway/discovery/services/emediaobjmgr_enu/connectstring
[INFO ] 2019-11-27 14:34:18.328 [http-nio-8000-exec-10] CommonLogger - com.siebel.swsm.store.ConfigStore:getConnString Connect String JSON:{
"ConnectString" : "siebel.TCPIP.NONE.NONE://GTWYHOST:2321/PSBL_ENT/eMediaObjMgr_enu"
}

localhost_access*log from CGW

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/bootstrapCG HTTP/1.1" 200 265

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/discovery/services/emediaobjmgr_enu/connectstring HTTP/1.1" 200 93

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/bootstrapCG HTTP/1.1" 200 265

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/discovery/services/emediaobjmgr_enu/connectstring HTTP/1.1" 200 93

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/bootstrapCG HTTP/1.1" 200 265

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/discovery/services/emediaobjmgr_enu/connectstring HTTP/1.1" 200 93

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/bootstrapCG HTTP/1.1" 200 265

10.1.119.203 - - [27/Nov/2019:13:33:37 +0530] "GET /siebel/v1.0/cloudgateway/discovery/services/emediaobjmgr_enu/connectstring HTTP/1.1" 200 93

Around 13000 bootstrap requests in 15 mins which is huge load on zookeeper.

At the same gatewaframework.log reported certificate errors

Line 13797: 49700 write(159, "2019-11-27 14:30:08,614 INFO GtwySvcFrmwrkLog : Discovery: Authentication to framework successfully done\n", 106 <unfinished ...>
Line 13811: 49668 write(10, "27-Nov-2019 14:30:08.614 INFO [http-nio-8000-exec-2] org.apache.catalina.core.ApplicationContext.log Discovery: Authentication to framework successfully done\n\n", 159 <unfinished ...>
Line 13849: 49700 write(159, "2019-11-27 14:30:08,616 INFO GtwySvcFrmwrkLog : Discovery: CGmetafile created successfully:: /app/siebel/XXXX/ai/applicationcontainer/webapps/cgclientstore.dat\n", 168 <unfinished ...>
Line 13853: 49668 write(10, "27-Nov-2019 14:30:08.616 INFO [http-nio-8000-exec-2] org.apache.catalina.core.ApplicationContext.log Discovery: CGmetafile created successfully:: /app/siebel/PSSBLWEB203/ai/applicationcontainer/webapps/cgclientstore.dat\n\n", 221 <unfinished ...>
Line 14680: 49700 write(159, "2019-11-27 14:30:08,687 ERROR GtwySvcFrmwrkLog : Discovery: Connection to Registry Failed\n", 90 <unfinished ...>
Line 14685: 49668 write(10, "27-Nov-2019 14:30:08.687 INFO [http-nio-8000-exec-2] org.apache.catalina.core.ApplicationContext.log Discovery: Connection to Registry Failed\n\n", 143) = 143
Line 14701: 49700 write(159, "2019-11-27 14:30:08,687 ERROR GtwySvcFrmwrkLog : Discovery: Exception Occurred:: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\n", 308 <unfinished ...>
Line 14713: 49668 write(10, "27-Nov-2019 14:30:08.687 INFO [http-nio-8000-exec-2] org.apache.catalina.core.ApplicationContext.log Discovery: Exception Occurred:: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException

Changes

Cause

	To view full details, sign in with your My Oracle Support account.
	Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.

Gateway Tomcat & Zookeeper is Not Responding After Some Time of Load Test & Srvrmgr Is Taking Huge Time to Connect During This Time (Doc ID 2619872.1)

Applies to:

Symptoms

Changes

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!