My Oracle Support Banner

Managed Server Failed With Error "Critical Subsystem JTAMigratableGroup has failed" (Doc ID 1668865.1)

Last updated on OCTOBER 10, 2018

Applies to:

Oracle WebLogic Server - Version 10.3 and later
Information in this document applies to any platform.

Symptoms

The WebLogic Server log shows the following messages:

<BEA-310006> <Critical Subsystem JTAMigratableGroup has failed. Setting server state to FAILED. Reason: JTA migratable target lost a lease>
...
<Critical> <WebLogicServer> <unixs1080> <pgmprpc1cN2> <[STANDBY] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1397136070079> <BEA-000385> <Server health failed. Reason: health of critical service 'PersistentStore._WLS_pgmprpc1cN4' failed>
####<10-Apr-2014 9:21:10 o'clock AM EDT> <Warning> <JTA> <unixs1080> <pgmprpc1cN2> <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1397136070090> <BEA-110213> <The activation of Transaction Recovery Service for server [pgmprpc1cN4] failed.
weblogic.store.PersistentStoreException: weblogic.management.DeploymentException:
at weblogic.transaction.internal.PlatformHelperImpl.getStore(PlatformHelperImpl.java:355)
at weblogic.transaction.internal.StoreTransactionLoggerImpl.<init>(StoreTransactionLoggerImpl.java:173)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.<init>(ServerTransactionManagerImpl.java:1092)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.getOrCreate(ServerTransactionManagerImpl.java:1135)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.access$1400(ServerTransactionManagerImpl.java:1072)
at weblogic.transaction.internal.ServerTransactionManagerImpl.recover(ServerTransactionManagerImpl.java:1329)
at weblogic.transaction.internal.TransactionRecoveryService.migratableActivate(TransactionRecoveryService.java:91)
at weblogic.cluster.migration.MigratableGroup.activateTarget(MigratableGroup.java:393)
at weblogic.cluster.migration.MigratableGroup.activateAllMigratables(MigratableGroup.java:356)
...
<Error> <Store> <unixs1080> <pgmprpc1cN2> <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1397136070056> <BEA-280061> <The persistent store "_WLS_pgmprpc1cN4" could not be deployed: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_pgmprpc1cN4" cannot open file _WLS_PGMPRPC1CN4000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_pgmprpc1cN4" cannot open file _WLS_PGMPRPC1CN4000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
at weblogic.store.admin.FileAdminHandler.createMigratedDefaultStore(FileAdminHandler.java:201)
at weblogic.transaction.internal.PlatformHelperImpl.getStore(PlatformHelperImpl.java:353)
at weblogic.transaction.internal.StoreTransactionLoggerImpl.<init>(StoreTransactionLoggerImpl.java:173)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.<init>(ServerTransactionManagerImpl.java:1092)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.getOrCreate(ServerTransactionManagerImpl.java:1135)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.access$1400(ServerTransactionManagerImpl.java:1072)
at weblogic.transaction.internal.ServerTransactionManagerImpl.recover(ServerTransactionManagerImpl.java:1329)
at weblogic.transaction.internal.TransactionRecoveryService.migratableActivate(TransactionRecoveryService.java:91)
at weblogic.cluster.migration.MigratableGroup.activateTarget(MigratableGroup.java:393)
at weblogic.cluster.migration.MigratableGroup.activateAllMigratables(MigratableGroup.java:356)
at weblogic.cluster.migration.MigratableGroup.activate(MigratableGroup.java:337)
at weblogic.cluster.singleton.SingletonServicesManager.activateService(SingletonServicesManager.java:195)
at weblogic.cluster.singleton.SingletonServicesManager_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:132)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
at weblogic.store.io.file.Heap.open(Heap.java:325)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
at weblogic.store.admin.FileAdminHandler.createMigratedDefaultStore(FileAdminHandler.java:201)
at weblogic.transaction.internal.PlatformHelperImpl.getStore(PlatformHelperImpl.java:353)
at weblogic.transaction.internal.StoreTransactionLoggerImpl.<init>(StoreTransactionLoggerImpl.java:173)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.<init>(ServerTransactionManagerImpl.java:1092)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.getOrCreate(ServerTransactionManagerImpl.java:1135)
at weblogic.transaction.internal.ServerTransactionManagerImpl$MigratedTLog.access$1400(ServerTransactionManagerImpl.java:1072)
at weblogic.transaction.internal.ServerTransactionManagerImpl.recover(ServerTransactionManagerImpl.java:1329)
at weblogic.transaction.internal.TransactionRecoveryService.migratableActivate(TransactionRecoveryService.java:91)
at weblogic.cluster.migration.MigratableGroup.activateTarget(MigratableGroup.java:393)
at weblogic.cluster.migration.MigratableGroup.activateAllMigratables(MigratableGroup.java:356)
at weblogic.cluster.migration.MigratableGroup.activate(MigratableGroup.java:337)
at weblogic.cluster.singleton.SingletonServicesManager.activateService(SingletonServicesManager.java:195)
at weblogic.cluster.singleton.SingletonServicesManager_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.