CCB 2.4 - Thread Aborts With Maximum Number Of Replays (1) Have Been Exceeded At Time Of Commit

(Doc ID 2164804.1)

Last updated on AUGUST 01, 2016

Applies to:

Oracle Utilities Customer Care and Billing - Version 2.4.0.2.0 and later
Information in this document applies to any platform.

Symptoms

In instances with long running cm jobs, only a hand full of threads may complete and the rest will fail.  The job may run for 5-8 hours before it times out with the following seen in the logs:


15:39:14,069 [DEFAULTWorker:1] INFO (com.splwg.shared.environ.ApplicationProperties) Milliseconds to suspend thread before attempting to replay database transaction is not specified via property ouaf.batch.worker.errorRecovery.transactionReplayDelayMs, defaulting to 100
I also saw the following just before this error:
15:39:13,898 [DEFAULTWorker:1] WARN (org.hibernate.engine.jdbc.spi.SqlExceptionHelper) SQL Error: 22922, SQLState: 99999
15:39:13,898 [DEFAULTWorker:1] ERROR (org.hibernate.engine.jdbc.spi.SqlExceptionHelper) ORA-22922: nonexistent LOB value

15:39:13,979 [DEFAULTWorker:1] ERROR (com.splwg.base.support.context.FrameworkSession) Failed to flush session
org.hibernate.exception.GenericJDBCException: ORA-22922: nonexistent LOB value

  at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
  at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
  at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
  at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:129)
  at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
  at com.sun.proxy.$Proxy7.executeUpdate(Unknown Source)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3017)
  at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2915)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3244)
  at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:113)
  at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:362)
  at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:354)
  at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:276)
  at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:326)
  at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)
  at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1127)
  at com.splwg.base.support.context.FrameworkSession.flush(FrameworkSession.java:1769)
  at com.splwg.base.support.context.FrameworkSession.flush(FrameworkSession.java:1744)
  at com.splwg.base.api.service.ServiceExecutionPolicy.requestCommitAndInitialize(ServiceExecutionPolicy.java:76)
  at com.splwg.base.api.service.ServiceExecutionPolicy.requestCommit(ServiceExecutionPolicy.java:63)
  at com.splwg.base.support.service.PageChangeService.privateExecuteWithoutRead(PageChangeService.java:61)
  at com.splwg.base.support.service.PageChangeService.executeWithoutRead(PageChangeService.java:39)
  at com.splwg.base.api.service.ServiceDispatcher.changeItemWithoutReadInSession(ServiceDispatcher.java:319)
  at com.splwg.base.api.businessObject.BusinessObjectProcessor.updateBody(BusinessObjectProcessor.java:572)
  at com.splwg.base.api.businessObject.BusinessObjectProcessor.updateBody(BusinessObjectProcessor.java:74)
  at com.splwg.base.support.schema.AbstractSchemaBasedDispatcher.updateViaPageService(AbstractSchemaBasedDispatcher.java:94)
  at com.splwg.base.api.businessObject.BusinessObjectProcessor.privateUpdate(BusinessObjectProcessor.java:270)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher$3.run(BusinessObjectDispatcher.java:641)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher$JumpToFWRunner.run(BusinessObjectDispatcher.java:854)
  at com.splwg.base.support.interception.SessionInterceptionManager.notifyGenericCallStackJump(SessionInterceptionManager.java:294)
  at com.splwg.base.support.context.FrameworkSession.notifyGenericCallStackJump(FrameworkSession.java:2202)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher.executeCallStackJump(BusinessObjectDispatcher.java:792)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher.instanceUpdate(BusinessObjectDispatcher.java:647)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher.execute(BusinessObjectDispatcher.java:336)
  at com.splwg.base.support.scripting.actions.InvokeBO.privateInvokeViaDom4J(InvokeBO.java:55)
  at com.splwg.base.support.scripting.actions.AbstractInvoke.privateInvokeViaDom(AbstractInvoke.java:91)
  at com.splwg.base.support.scripting.actions.AbstractInvoke.privateInvoke(AbstractInvoke.java:83)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.actions.StructuredAction.executeStatementBlock(StructuredAction.java:34)
  at com.splwg.base.support.scripting.actions.EditData.privateInvoke(EditData.java:22)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:22)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.InvocableScript.invoke(InvocableScript.java:308)
  at com.splwg.base.support.scripting.plugin.PluginScriptInvocationCallback.algorithmInvoked(PluginScriptInvocationCallback.java:47)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler$1.run(AlgorithmProxyInvocationHandler.java:58)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler$JumpToFWRunner.run(AlgorithmProxyInvocationHandler.java:125)
  at com.splwg.base.support.interception.SessionInterceptionManager.notifyGenericCallStackJump(SessionInterceptionManager.java:294)
  at com.splwg.base.support.context.FrameworkSession.notifyGenericCallStackJump(FrameworkSession.java:2202)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler.executeCallStackJump(AlgorithmProxyInvocationHandler.java:111)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler.invoke(AlgorithmProxyInvocationHandler.java:61)
  at com.sun.proxy.$Proxy20.invoke(Unknown Source)
  at com.splwg.base.support.schema.BusinessObjectStateInfo.enterStatus(BusinessObjectStateInfo.java:319)
  at com.splwg.base.support.schema.BusinessObjectInfo.enterStatus(BusinessObjectInfo.java:957)
  at com.splwg.base.support.schema.rules.BusinessObjectStateTransitioner.enterStatus(BusinessObjectStateTransitioner.java:468)
  at com.splwg.base.support.schema.rules.BusinessObjectStateTransitioner.singleTransition(BusinessObjectStateTransitioner.java:300)
  at com.splwg.base.support.schema.rules.BusinessObjectStateTransitioner.transition(BusinessObjectStateTransitioner.java:245)
  at com.splwg.base.support.schema.rules.BusinessObjectStateTransitioner.transitionState(BusinessObjectStateTransitioner.java:116)
  at com.splwg.base.support.schema.rules.AbstractModifyingBusinessObjectRuleProcessor.transitionState(AbstractModifyingBusinessObjectRuleProcessor.java:215)
  at com.splwg.base.support.schema.rules.AbstractModifyingBusinessObjectRuleProcessor.postProcess(AbstractModifyingBusinessObjectRuleProcessor.java:91)
  at com.splwg.base.support.service.PageAddService.privateAdd(PageAddService.java:118)
  at com.splwg.base.support.service.PageAddService.privateExecuteWithoutRead(PageAddService.java:64)
  at com.splwg.base.support.service.PageAddService.executeWithoutRead(PageAddService.java:44)
  at com.splwg.base.api.service.ServiceDispatcher.addItemWithoutReadInSession(ServiceDispatcher.java:189)
  at com.splwg.base.api.businessObject.BusinessObjectProcessor.addBody(BusinessObjectProcessor.java:516)
  at com.splwg.base.api.businessObject.BusinessObjectProcessor.addBody(BusinessObjectProcessor.java:74)
  at com.splwg.base.support.schema.AbstractSchemaBasedDispatcher.addViaPageService(AbstractSchemaBasedDispatcher.java:55)
  at com.splwg.base.api.businessObject.BusinessObjectProcessor.privateAdd(BusinessObjectProcessor.java:184)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher$5.run(BusinessObjectDispatcher.java:669)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher$JumpToFWRunner.run(BusinessObjectDispatcher.java:854)
  at com.splwg.base.support.interception.SessionInterceptionManager.notifyGenericCallStackJump(SessionInterceptionManager.java:294)
  at com.splwg.base.support.context.FrameworkSession.notifyGenericCallStackJump(FrameworkSession.java:2202)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher.executeCallStackJump(BusinessObjectDispatcher.java:792)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher.instanceAdd(BusinessObjectDispatcher.java:675)
  at com.splwg.base.api.businessObject.BusinessObjectDispatcher.execute(BusinessObjectDispatcher.java:330)
  at com.splwg.base.support.scripting.actions.InvokeBO.privateInvokeViaDom4J(InvokeBO.java:55)
  at com.splwg.base.support.scripting.actions.AbstractInvoke.privateInvokeViaDom(AbstractInvoke.java:91)
  at com.splwg.base.support.scripting.actions.AbstractInvoke.privateInvoke(AbstractInvoke.java:83)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.actions.StructuredAction.executeStatementBlock(StructuredAction.java:34)
  at com.splwg.base.support.scripting.actions.StructuredIf.privateInvoke(StructuredIf.java:42)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.actions.StructuredAction.executeStatementBlock(StructuredAction.java:34)
  at com.splwg.base.support.scripting.actions.StructuredIf.privateInvoke(StructuredIf.java:42)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.actions.StructuredAction.executeStatementBlock(StructuredAction.java:34)
  at com.splwg.base.support.scripting.actions.For.performOneIteration(For.java:53)
  at com.splwg.base.support.scripting.actions.For.privateInvoke(For.java:40)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.actions.StructuredAction.executeStatementBlock(StructuredAction.java:34)
  at com.splwg.base.support.scripting.actions.For.performOneIteration(For.java:53)
  at com.splwg.base.support.scripting.actions.For.privateInvoke(For.java:40)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.actions.StructuredAction.executeStatementBlock(StructuredAction.java:34)
  at com.splwg.base.support.scripting.actions.EditData.privateInvoke(EditData.java:22)
  at com.splwg.base.support.scripting.actions.Action.invoke(Action.java:54)
  at com.splwg.base.support.scripting.InvocationContext.invokeAction(InvocationContext.java:71)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:27)
  at com.splwg.base.support.scripting.ScriptTreeWalker.evaluate(ScriptTreeWalker.java:22)
  at com.splwg.base.support.scripting.InvocationContext.executeAST(InvocationContext.java:167)
  at com.splwg.base.support.scripting.InvocableScript.invoke(InvocableScript.java:308)
  at com.splwg.base.support.scripting.plugin.PluginScriptInvocationCallback.algorithmInvoked(PluginScriptInvocationCallback.java:47)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler$1.run(AlgorithmProxyInvocationHandler.java:58)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler$JumpToFWRunner.run(AlgorithmProxyInvocationHandler.java:125)
  at com.splwg.base.support.interception.SessionInterceptionManager.notifyGenericCallStackJump(SessionInterceptionManager.java:294)
  at com.splwg.base.support.context.FrameworkSession.notifyGenericCallStackJump(FrameworkSession.java:2202)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler.executeCallStackJump(AlgorithmProxyInvocationHandler.java:111)
  at com.splwg.base.support.scripting.plugin.helpers.AlgorithmProxyInvocationHandler.invoke(AlgorithmProxyInvocationHandler.java:61)
  at com.sun.proxy.$Proxy17.invoke(Unknown Source)
  at com.splwg.base.support.schema.MaintenanceObjectInfo.notifyOfChange(MaintenanceObjectInfo.java:540)
  at com.splwg.base.support.context.FrameworkSession.callMOAudits(FrameworkSession.java:2927)
  at com.splwg.base.support.context.FrameworkSession.beforeCommit(FrameworkSession.java:696)
  at com.splwg.base.support.context.FrameworkSession.basicCommit(FrameworkSession.java:664)
  at com.splwg.base.support.context.FrameworkSession.commit(FrameworkSession.java:660)
  at com.splwg.base.support.grid.node.MultiSessionStrategy.doCommit(MultiSessionStrategy.java:74)
  at com.splwg.base.support.cluster.ClusteredExecutionContext.commit(ClusteredExecutionContext.java:44)
  at com.splwg.base.api.batch.AbstractExecutionStrategy.updateRestartDataAndCommit(AbstractExecutionStrategy.java:114)
  at com.splwg.base.api.batch.AbstractCommitStrategy.saveRestartDataAndCommit(AbstractCommitStrategy.java:259)
  at com.splwg.base.api.batch.AbstractCommitStrategy.doCommit(AbstractCommitStrategy.java:191)
  at com.splwg.base.api.batch.AbstractCommitStrategy.concreteExecuteWork(AbstractCommitStrategy.java:75)
  at com.splwg.base.api.batch.AbstractExecutionStrategy.executeWork(AbstractExecutionStrategy.java:243)
  at com.splwg.base.support.batch.JavaBatchWork.executeThreadWork(JavaBatchWork.java:109)
  at com.splwg.base.support.batch.JavaBatchWork.access$300(JavaBatchWork.java:29)
  at com.splwg.base.support.batch.JavaBatchWork$DoExecuteWorkInSession.doBatchWorkInSession(JavaBatchWork.java:92)
  at com.splwg.base.support.batch.BatchWorkInSessionExecutable.run(BatchWorkInSessionExecutable.java:62)
  at com.splwg.base.support.batch.JavaBatchWork.doExecuteWork(JavaBatchWork.java:51)
  at com.splwg.base.support.grid.AbstractGridWork.executeWork(AbstractGridWork.java:72)
  at com.splwg.base.support.cluster.ThreadWork.processBatchWork(ThreadWork.java:54)
  at com.splwg.base.support.cluster.BatchWorkManager.processWork(BatchWorkManager.java:78)
  at com.tangosol.coherence.commonj.WorkManager$ScheduleWork.run(WorkManager.java:2266)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.InvocationService.onInvocationRequest(InvocationService.CDB:12)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.InvocationService$InvocationRequest.run(InvocationService.CDB:1)
  at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
  at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
  at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:66)
  at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
  at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.SQLException: ORA-22922: nonexistent LOB value

  at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
  at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
  at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
  at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
  at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044)
  at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
  at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
  at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665)
  at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352)
  at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at oracle.ucp.jdbc.proxy.StatementProxyFactory.invoke(StatementProxyFactory.java:230)
  at oracle.ucp.jdbc.proxy.PreparedStatementProxyFactory.invoke(PreparedStatementProxyFactory.java:124)
  at com.sun.proxy.$Proxy6.executeUpdate(Unknown Source)
  at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122)
  ... 153 more
15:39:14,044 [DEFAULTWorker:1] ERROR (com.splwg.base.api.service.ServiceExecutionPolicy) Encountered exception, session marked for rollback
15:39:14,047 [DEFAULTWorker:1] ERROR (com.splwg.base.support.scripting.actions.Action) Failed invoking step : invokeBO $syncRequestBo using "syncRequestBo" for update;
15:39:14,048 [DEFAULTWorker:1] ERROR (com.splwg.base.support.scripting.actions.Action) Failed invoking step : 40: edit data
  move "C1-ConMDM2SP/output" to "syncRequestBo/syncRequestDetails/finalSnapshot";
  move "C1-ConMDM2SP/output" to "syncRequestBo/syncRequestDetails/finalSnapshot/formattedElements";
  invokeBO $syncRequestBo using "syncRequestBo" for update;
  //exit the algorithm
  goto 60;
end-edit;
15:39:14,049 [DEFAULTWorker:1] ERROR (com.splwg.base.support.schema.rules.BusinessObjectStateTransitioner) BO C1-MDM2SPSyncRequest SyncRequest_Id(16088948850332) had status transition error transitioning from BusinessObjectStatusCode='PENDING' to BusinessObjectStatusCode='DETSYNCNEED'
com.splwg.shared.common.LoggedException:
The following stacked messages were reported as the LoggedException was rethrown:
com.splwg.base.support.scripting.actions.EditData.privateInvoke(EditData.java:22): Failed invoking step : 40: edit data
  move "C1-ConMDM2SP/output" to "syncRequestBo/syncRequestDetails/finalSnapshot";
  move "C1-ConMDM2SP/output" to "syncRequestBo/syncRequestDetails/finalSnapshot/formattedElements";
  invokeBO $syncRequestBo using "syncRequestBo" for update;
  //exit the algorithm
  goto 60;
end-edit;
com.splwg.base.support.scripting.actions.AbstractInvoke.privateInvoke(AbstractInvoke.java:83): Failed invoking step : invokeBO $syncRequestBo using "syncRequestBo" for update;
com.splwg.base.api.service.ServiceExecutionPolicy.requestCommitAndInitialize(ServiceExecutionPolicy.java:76): Encountered exception, session marked for rollback

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms