Managing MQ/TIBCO Connections And JMS Connection Leaks
(Doc ID 983912.1)
Last updated on JANUARY 25, 2024
Applies to:
Siebel CRM - Version 7.7.1 [18306] and laterInformation in this document applies to any platform.
Symptoms
This documentation addresses:
1. Connections leaks and what are the recommended actions to be performed when this happens.
Connections should not grow indefinitely.
2. Configuration steps required to make Siebel JMS Receiver to close and re-connect to JMS Queue atter a message receipt.
This approach can be used to efficiently manage connection distributed Queues of Oracle WebLogicServer or to re-set connection pool issues of encountered by other Messaging systems (such as IBM WebSphere MQ, TIBCO EMS etc ).
The original issue (Siebel 7.8) could be replicated with the following steps:
Steps | Number of anonymous connections in TIBCO EMS Server with zero sessions |
---|---|
Log in to application | 0 |
Run EAI JMS Transport in BS Simulator | 1 |
Log off | 1 (leak) |
Log in to application | 1 |
Run EAI JMS Transport in BS Simulator | 2 |
Log off | 2 (leak) |
Log in to application | 2 |
3 | |
Log off | 3 |
Log in to application | 3 |
Start a repeating component request that runs 20 SEQUENTIAL calls to workflow batch manager so that EAI JMS Transport is run in a batch component. (no parallel calls at this time). | 3 |
Log off | 3(no leak yet due to batch components. 1 connection was created for the batch component and re-used by each of the 20 sequential calls). |
Log in to application | 3 |
Start 10 parallel repeating component requests. Each will run several calls to to workflow batch manager so that EAI JMS Transport is run in a batch component. (10 parallel requests at this time) | 12 (no leak yet due to batch components. 1 batch connection was re-used. 9 new ones were created for batch components totaling 10 paralell requests). |
Log off | 12 (no leak yet as batch components only clear connections on component shutdown). |
Shutdown component WfProcBatchMgr | 12 (LEAK. component was shutdown and connections were not cleared). |
Stop Server | 0 |
At the table above it is possible to see :
- The first connection is not closed when exiting the session after running the simulator and when starting a second session with the simulator a second session is created and it will not be closed either - That was the original problem. These 2 connections are leaked. They have not been closed but are not reused for anything else. And if you run new sessions and new BS simulator tests again and again the number of connections will increase.
- When testing the repeating job, performing a logoff has no effect which is normal since the connection are now created from the process which runs the workflow so the WfProcBatchMgr
- The connection created for the first job is reused for the second job.This one is not leaked since it is reused for the second test.
- When starting the 10 jobs this connection is reused and 9 (not 10) new connections created to accommodate the number of concurrent jobs.
- When starting the 10 last jobs the 10 connections created by the WfProcBatchMgr are reused. So again these 10 connections are not leaked but cached and reused.
- When shutting down the WfProcBatchMgr these connections are not closed.
- At that time they are leaked since they are not closed but will not be reused.
Every time a log off / log in happens, plus running the transport, a new connection is created and never terminated.
This means that connections are leaking.
Sometimes, this can lead to 'EAI JMS Transport' opening an excessive number of connections and error "java.lang.OutOfMemoryError: Java heap space".
The expected behavior should be:
Steps | Number of anonymous connections in TIBCO EMS Server with zero sessions |
---|---|
Log in to application | 0 |
Run EAI JMS Transport in BS Simulator | 1 |
Log off | 0 (correct behavior. no leak anymore). |
Log in to application | 0 |
Run EAI JMS Transport in BS Simulator | 1 |
Log off | 0 (correct behavior. no leak anymore). |
Log in to application | 0 |
Start a repeating component request that runs 20 SEQUENTIAL calls to workflow batch manager so that EAI JMS Transport is run in a batch component. (no paralell calls at this time). | 1 (no leak. one connection was created by the batch component. It was reused by each sequential call to the batch component). |
Log off | 1 (no leak. batch components only close at component shutdown). |
Log in to application | 1 |
Start 10 paralell repeating component requests. Each will run several calls to to workflow batch manager so that EAI JMS Transport is run in a batch component. (10 parallel requests at this time) | 10 (no leak. one connection was re-used. 9 new ones were created totaling 10 parallel batch component calls). |
Log off | 10 (no leak. batch components only close at component shutdown). |
Shutdown component WfProcBatchMgr | 0 (correct behavior. Connections cleared at component shutdown. no leak anymore). |
Stop Server | 0 (no need to stop the whole server anymore. Connections are now closed upon component stop in previous step). |
In the BS Simulator test, the connection is closed when exiting the session.
This is also true of the 10 connections of the WfProcBatchMgr process when we shutdown the component.
These are cached connections in the WfProcBatchMgr process which are reused for other requests.
These connections should not be closed but cached and reused by later requests.
The number of connections should not increase indefinitely.
There is an upper limit of connections to accommodate the maximum number of concurrent requests (here 10) which remain available for re-use.
List of possible errors:
Unexpected exception: java.rmi.UnexpectedException: Marshalling: ; nested exception is:
java.rmi.ConnectException: This RJVM has already been shutdown 2419568587898972232S:dcprbpmsut01:[8005,8005,-1,-1,-1,-1,-1]:soa_domain_prd:WLS_OSB1
at weblogic.common.internal.BootServicesStub.authenticate(BootServicesStub.java:80)
at weblogic.security.acl.internal.RemoteAuthenticate.authenticate(RemoteAuthenticate.java:85)
at weblogic.jndi.WLInitialContextFactoryDelegate.authenticateRemotely(WLInitialContextFactoryDelegate.java:927)
at weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:857)
ErrorCode = JNDI_ERROR
ErrorMessage = java.lang.OutOfMemoryError: Java heap space
[SIEBEL ERROR] Thread[http-nio-9001-exec-9,5,main] [5:08:45.780 PM Wed May 15: ] Error:JNdi File
[SIEBEL ERROR] Thread[http-nio-9001-exec-23,5,main] [5:08:45.780 PM Wed May 15: ] Error:JNdi File
[SIEBEL ERROR] Thread[http-nio-9001-exec-8,5,main] [5:08:45.780 PM Wed May 15: ] Error:JNdi File
[SIEBEL ERROR] Thread[http-nio-9001-exec-8,5,main] [5:08:45.781 PM Wed May 15: ] Errorjava.lang.NullPointerException
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2024/01/12 22:01:42 - please wait.
Exception in thread "TIBCO EMS TCPLink Reader (Server-697266268)" java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
at java.lang.Thread.startImpl(Native Method)
at java.lang.Thread.start(Thread.java:948)
at com.tibco.tibjms.TibjmsConnection._startConnectTimeoutThread(TibjmsConnection.java:1109)
at com.tibco.tibjms.TibjmsConnection._reconnect(TibjmsConnection.java:1900)
at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventReconnect(TibjmsConnection.java:369)
Changes
N/A
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |