NFS Error: Failed To Recover Lost Lock NFS4ERR_NO_GRACE

(Doc ID 2372191.1)

Last updated on MARCH 15, 2018

Applies to:

Oracle WebLogic Server - Version 12.1.2.0.0 and later
Information in this document applies to any platform.

Symptoms

After reconfiguring and restarting NFS the following error occurs:

Oct 18 12:16:49 Server-01 kernel: nfs: [ID 269213 kern.info] NOTICE: [NFS4][Server: XXXXXXXXX][Mntpt: /system/zones/Server-01/root/bea/share]Process 12007 lost its locks on file ./fstores/syspfstore_mw01/SYSPFILESTORE_1000000.DAT (rnode_pt: 3040162658d00) due to NFS recovery error (NFS4ERR_NO_GRACE) on server XXXXXXXXX.
Oct 18 12:16:49 Server-01 kernel: nfs: [ID 257235 kern.info] NOTICE: [NFS4][Server: XXXXXXXXX][Mntpt: /system/zones/Server-01/root/bea/share]Couldn't reclaim lock for pid 12007 for file ./fstores/appfstore_mw01/APPFILESTORE_1000000.DAT (rnode_pt 0x30401623188b0) on (server XXXXXXXXX): error 10033
Oct 18 12:16:49 Server-01 kernel: nfs: [ID 286389 kern.info] NOTICE: [NFS4][Server: XXXXXXXXX][Mntpt: /system/zones/Server-01/root/bea/share]File ./fstores/appfstore_mw01/APPFILESTORE_1000000.DAT (rnode_pt: 30401623188b0) was closed due to NFS recovery error on server XXXXXXXXX(Failed to recover lost lock NFS4ERR_NO_GRACE)
Oct 18 12:16:49 Server-01 kernel: nfs: [ID 269213 kern.info] NOTICE: [NFS4][Server: XXXXXXXXX][Mntpt: /system/zones/Server-01/root/bea/share]Process 12007 lost its locks on file ./fstores/appfstore_mw01/APPFILESTORE_1000000.DAT (rnode_pt: 30401623188b0) due to NFS recovery error (NFS4ERR_NO_GRACE) on server XXXXXXXXX.
Oct 18 12:19:53 Server-01 kernel: nfs: [ID 273629 kern.info] NOTICE: [NFS4][Server: XXXXXXXXX][Mntpt: /system/zones/Server-01/root/bea/share]NFS Recovery done for mount /system/zones/Server-01/root/bea/share (mi 0x304011a6ee980) on server XXXXXXXXX, rnode_pt1 ./fstores/appfstore_mw01/APPFILESTORE_1000000.DAT (0x30401623188b0), rnode_pt2 (0x0)
Oct 18 12:19:53 Server-01 kernel: nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: XXXXXXXXX][Mntpt: /system/zones/Server-01/root/bea/share]NFS op OP_GETATTR got error NFS4ERR_STALE causing recovery action NR_STALE.
Oct 18 12:19:53 Server-01 kernel: nfs: [ID 286389 kern.info] NOTICE: [NFS4][Server: XXXXXXXXX][Mntpt: /system/zones/Server-01/root/bea/share]File ./wily/wily97/logs/AutoProbe_${weblogic.Name}.WebLogic_Agent.log (rnode_pt: 30401626584f0) was closed due to NFS recovery error on server XXXXXXXXX(failed to recover from NFS4ERR_STALE NFS4ERR_STALE)
Oct 18 12:19:53 Server-01 kernel: nfs: [ID 941083 kern.info] NOTICE: NFS4 FACT SHEET:
Action: NR_STALE
NFS4 error: NFS4ERR_STALE

This underlying NFS failure causes lock issue's with WebLogic Server which are using NFS for their JMS File Stores. As a result, WebLogic Sever crashes as it can no longer access the store.

<Feb 9, 2018, 12:14:51,512 PM CET> <Emergency> <Store> <BEA-280060> <The store "AppFileStore_2" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage.

weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage.

at weblogic.store.io.file.BaseStoreIO.flush(BaseStoreIO.java:745)

at weblogic.store.internal.PersistentStoreImpl.synchronousFlush(PersistentStoreImpl.java:1170)

at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:1140)

at java.lang.Thread.run(Thread.java:748)

Caused By: java.io.IOException: Error writing to file, I/O error, errno=5

at weblogic.store.io.file.direct.DirectIONative.write(Native Method)

at weblogic.store.io.file.direct.DirectIONativeImpl.write(DirectIONativeImpl.java:131)

at weblogic.store.io.file.direct.DirectFileChannel.write(DirectFileChannel.java:277)

at weblogic.store.io.file.StoreFile.write(StoreFile.java:268)

at weblogic.store.io.file.Heap$StoreFileCursor.finalWrite(Heap.java:2840)

Truncated. see log file for complete stacktrace

>

<Feb 9, 2018, 12:14:51,521 PM CET> <Error> <Store> <BEA-280074> <The store "AppFileStore_2" encountered an unresolvable failure while processing transaction "BEA1-37ADF602DAD3DF2D7992". Shut down and restart to resolve this transaction. weblogic.store.gxa.GXAException: weblogic.store.PersistentStoreException: weblogic.store.PersistentStoreFatalException: [Store:280032]The store suffered a fatal error and it must be reopened.

weblogic.store.gxa.GXAException: weblogic.store.PersistentStoreException: weblogic.store.PersistentStoreFatalException: [Store:280032]The store suffered a fatal error and it must be reopened.

at weblogic.store.gxa.internal.GXAResourceImpl.registerFailedTransaction(GXAResourceImpl.java:626)

at weblogic.store.gxa.internal.GXATransactionImpl.doOperationCallbacks(GXATransactionImpl.java:243)

at weblogic.store.gxa.internal.GXAResourceImpl.commit(GXAResourceImpl.java:1477)

at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:1558)

at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:625)

Truncated. see log file for complete stacktrace

Caused By: weblogic.store.PersistentStoreException: weblogic.store.PersistentStoreFatalException: [Store:280032]The store suffered a fatal error and it must be reopened.

at weblogic.store.gxa.internal.GXATransactionImpl.commitStoreIO(GXATransactionImpl.java:103)

at weblogic.store.gxa.internal.GXATransactionImpl.doOperationCallbacks(GXATransactionImpl.java:215)

at weblogic.store.gxa.internal.GXAResourceImpl.commit(GXAResourceImpl.java:1477)

at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:1558)

at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:625)

Truncated. see log file for complete stacktrace

Caused By: weblogic.store.PersistentStoreFatalException: [Store:280032]The store suffered a fatal error and it must be reopened.

at weblogic.store.io.file.BaseStoreIO.checkOpened(BaseStoreIO.java:1187)

at weblogic.store.io.file.BaseStoreIO.delete(BaseStoreIO.java:448)

at weblogic.store.internal.DeleteRequest.run(DeleteRequest.java:36)

at weblogic.store.internal.StoreRequest.doTheIO(StoreRequest.java:100)

at weblogic.store.internal.PersistentStoreImpl.synchronousFlush(PersistentStoreImpl.java:1156)

Truncated. see log file for complete stacktrace

>

<Feb 9, 2018, 12:14:51,536 PM CET> <Critical> <Health> <BEA-310006> <Critical subsystem PersistentStore.AppFileStore_2 in partition DOMAIN has failed. Setting server state to FAILED.

Reason: weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage.>

<Feb 9, 2018, 12:14:51,536 PM CET> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: [Health:310006]Critical subsystem PersistentStore.AppFileStore_2 in partition DOMAIN has failed. Setting server state to FAILED.

Reason: weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage.>

<Feb 9, 2018, 12:14:51,539 PM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED.>

 

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms