My Oracle Support Banner

DB Instances Not Starting After OS Patching - runaway NFS READ activity on Oracle binary NFS mount (Doc ID 2403709.1)

Last updated on AUGUST 27, 2018

Applies to:

Solaris Operating System - Version 10 1/13 U11 to 11.3 [Release 10.0 to 11.0]
Information in this document applies to any platform.

Symptoms

DB instances, with all DB data, and binaries mounted over NFS, failed to start after patching to or beyond Solaris 10 kernel patch 150400-52 (SPARC) or 150401-52 (x64). 

The database instance attempted to be started never came up - sqlplus connection to the database received no response - the database appears to have hung at startup. 

The DB binary mountpoint was observed to be very aggressively issuing NFS READS - mostly small reads, at a very high rate - up to 15,000/sec, enough to saturate the 1Gbps link. This would go on apparently forever, at least for 30-60 minutes.

Further research indicated that most - a few dozen at least - DB processes were constantly issuing NFS READS via page faults - on mmap'ed files. 

Threadlists indicated just about every Oracle DB daemon was constantly issuing NFS READs due to page faulted on mmap-ed files.  

Here is one example thread:

PC: _resume_from_idle+0xfb CMD: ora_dbw1_PYH
stack pointer for thread ffffffffdabfd0a0: fffffe80020843b0
[ fffffe80020843b0 _resume_from_idle+0xfb() ]
swtch+0x141()
cv_timedwait_hires+0xc6()
cv_timedwait+0x34()
clnt_cots_kcallit+0x69e()
rfscall+0x361()
rfs3call+0x8c()
nfs3read+0x30b()
nfs3_bio+0x154()
nfs3_getapage+0x230()
nfs3_getpage+0xdd()
fop_getpage+0x47()
segvn_fault+0x8b0()
as_fault+0x205()
pagefault+0x8b()
trap+0x3d7()
_cmntrap+0x140()

Just about any DB daemon was in this state - constantly. 

Changes

 Customer had patched, from prior to 150401-48  to 150491-59.    Rolling back the kernel patch - to anything before 150401-52 (x64) or 150400-52 (SPARC) - restored the customer's databases  to normal operation. 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.