WebLogic Server Experiences Transaction Rollbacks When Multiple FileStores Are Located On Remote Disks (Doc ID 1102473.1)

Last updated on MAY 24, 2017

Applies to:

Oracle Weblogic Server - Version 8.1 to 10.3.3
Information in this document applies to any platform.

Symptoms

Under the circumstance where there are a number of many JMS destinations involved within a transaction and the JMS destinations are backed by File Stores, which are being held on remote disks, the transactions can take a long time to complete. This can result in a situation where the transaction exceeds the JTATimeout are the transactions are rolled back

####<17-Dec-2009 12:06:35 o'clock GMT> <Info> <EJB> <bys148a001> <managed11_blwls009> <ExecuteThread: '45' for queue: 'default'> <> <> <BEA-010213> <Message-Driven EJB: OrderIncomingQueueSubscriber's transaction was rolledback. The transaction details are: Xid=BEA1-335F6F48E36149CA073A(519438),Status=Rolled back. [Reason=weblogic.transaction.internal.AppSetRollbackOnlyException],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=29,seconds left=49,XAServerResourceInfo[JMS_managed11_blwls009JMSStore]=(ServerResourceInfo[JMS_managed11_blwls009JMSStore]=(state=rolledback,assigned=managed11_blwls009),xar=JMS_managed11_blwls009JMSStore,re-Registered = false),SCInfo[blwls009+managed11_blwls009]=(state=rolledback),properties=({START_AND_END_THREAD_EQUAL=false}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=managed11_blwls009+xx.xx.xxx.xxx:50303+blwls009+t3+, XAResources={weblogic.ejb20.JMSConnectionPoller.MQ_aServiceOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_bOrderResponseInputQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_cServiceOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_dResponseIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_eServiceOrderIncomingQueueSubscriber, OFSMultiPool, weblogic.ejb20.JMSConnectionPoller.MQ_fSalesOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_gServiceOrderResponseInputSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_hSalesOrderIncomingQueueSubscriber, JMS_managed11_blwls009JMSStore, weblogic.ejb20.JMSConnectionPoller.MQ_iOrderResponseIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_jResponseIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_kBillingOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_lSettlementOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_mWorkOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_nOrderResponseIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_oOrderResponseInputQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_pResponseInputIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_qGoodsOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_rServiceOrderIncomingQueueSubscriber, weblogic.ejb20.JMSConnectionPoller.MQ_sSalesOrderIncomingQueueSubscriber},NonXAResources={})],CoordinatorURL=managed11_blwls009+xx.xx.xxx.xxx:50303+blwls009+t3+).>


These exceptions can be sporadic and simply increasing the transaction timeout can be insufficient. At the time an exception occurs, a thread dump shows that they are may thread waiting to access the files store, usually the same one, with one thread waiting for an IO completion.

"ExecuteThread: '8' for queue: 'default'" daemon prio=5 tid=0x00dbc900 nid=0xb2 in Object.wait() [0x4e7ff000..0x4e7ffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x7ff87a60> (a weblogic.jms.store.StoreRequest)
at java.lang.Object.wait(Object.java:429)
at weblogic.jms.store.StoreRequest.waitForIOComplete(StoreRequest.java:467)
- locked <0x7ff87a60> (a weblogic.jms.store.StoreRequest)
at weblogic.jms.backend.BEXATranEntryReceive.finishCommit(BEXATranEntryReceive.java:151)
at weblogic.jms.backend.BEXAResource.commit(BEXAResource.java:1164)
at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:1343)
at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:563)
at weblogic.transaction.internal.ServerSCInfo.startCommit(ServerSCInfo.java:414)
at weblogic.transaction.internal.ServerTransactionImpl.localCommit(ServerTransactionImpl.java:1902)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryCommit(ServerTransactionImpl.java:2498)
at weblogic.transaction.internal.ServerTransactionImpl.globalCommit(ServerTransactionImpl.java:2429)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:262)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:228)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:430)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:333)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:298)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2698)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2610)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

"ExecuteThread: '18' for queue: 'FOC_FOCOut_EQ'" daemon prio=5 tid=0x006c8578 nid=0x83 in Object.wait() [0x55d7e000..0x55d7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x8128f6d0> (a weblogic.jms.store.StoreRequest)
at java.lang.Object.wait(Object.java:429)
at weblogic.jms.store.StoreRequest.waitForIOComplete(StoreRequest.java:467)
- locked <0x8128f6d0> (a weblogic.jms.store.StoreRequest)
at weblogic.jms.backend.BEXATranEntryReceive.finishPrepare(BEXATranEntryReceive.java:115)
at weblogic.jms.backend.BEXAResource.prepare(BEXAResource.java:943)
at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:1286)
at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:485)
at weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:289)
at weblogic.transaction.internal.ServerTransactionImpl.localPrepare(ServerTransactionImpl.java:2365)
at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2113)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:259)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:228)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:430)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:333)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:298)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2698)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2610)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

"ExecuteThread: '29' for queue: 'FOC_FOCIn_EQ'" daemon prio=5 tid=0x00afc208 nid=0x70 in Object.wait() [0x58cfe000..0x58cffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x8128b3f8> (a weblogic.jms.store.StoreRequest)
at java.lang.Object.wait(Object.java:429)
at weblogic.jms.store.StoreRequest.waitForIOComplete(StoreRequest.java:467)
- locked <0x8128b3f8> (a weblogic.jms.store.StoreRequest)
at weblogic.jms.backend.BEXATranEntryReceive.finishPrepare(BEXATranEntryReceive.java:115)
at weblogic.jms.backend.BEXAResource.prepare(BEXAResource.java:943)
at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:1286)
at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:485)
at weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:289)
at weblogic.transaction.internal.ServerTransactionImpl.localPrepare(ServerTransactionImpl.java:2365)
at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2113)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:259)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:228)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:430)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:333)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:298)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2698)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2610)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

"ExecuteThread: '35' for queue: 'FOC_FOCIn_EQ'" daemon prio=5 tid=0x00afb610 nid=0x6f in Object.wait() [0x58f7f000..0x58f7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x80b12430> (a weblogic.jms.store.StoreRequest)
at java.lang.Object.wait(Object.java:429)
at weblogic.jms.store.StoreRequest.waitForIOComplete(StoreRequest.java:467)
- locked <0x80b12430> (a weblogic.jms.store.StoreRequest)
at weblogic.jms.backend.BEXATranEntryReceive.finishCommit(BEXATranEntryReceive.java:151)
at weblogic.jms.backend.BEXAResource.commit(BEXAResource.java:1164)
at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:1343)
at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:563)
at weblogic.transaction.internal.ServerSCInfo.startCommit(ServerSCInfo.java:414)
at weblogic.transaction.internal.ServerTransactionImpl.localCommit(ServerTransactionImpl.java:1902)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryCommit(ServerTransactionImpl.java:2498)
at weblogic.transaction.internal.ServerTransactionImpl.globalCommit(ServerTransactionImpl.java:2429)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:262)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:228)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:430)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:333)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:298)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2698)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2610)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms