NFSv3 over UDP WRITE requests receive NFS3ERR_JUKEBOX from Solaris 11.4 NFS server; File Writes Cannot Complete. (Doc ID 2721737.1)

Last updated on AUGUST 29, 2023

Applies to:

Symptoms

An NFS client utilizing NFSv3 with UDP - which is writing to multple files quite often (50-100 times/sec overall) - after about 15 minutes starts seeing NFS3ERR_JUKEBOX responses to NFSv3/UDP WRITE requests.  

These NFS3ERR_JUKEBOX responses are the NFS server's way of saying "I can't complete your request right now, please try again later".  This is so called due to NFS servers "back in the day" having data stored on CD-ROM jukebox devices, which required a few seconds to access.   In this case files on the NFS server are located in a normal ZFS dataset, which are immediately available (and almost certainly cached in the ZFS ARC).

Trying again later is what the NFS client does - it waits a few seconds and tries again -  which often ends up with another NFS3ERR_JUKEBOX response.

The NFS3ERR_JUKEBOX responses are not consistent.  Not every NFSv3/UDP WRITE requests receives the NFS3ERR_JUKEBOX response - just enough to cause a serious problem for the NFS client.  Some files can't be written to for quite a long time.  

While this is happening, other NFS operations, READS or GETATTR requests in particular -  always receive a response.  This is true of any operations that do not alter data on the NFS server (including READDIRPLUS,etc). 

During the issue, NFSv3 badcalls stats increment, as does the  "full" duplicate request cache counter. 

This kstat command illustrates these counters:

 The badcalls and rpcmod:0:drc_0_clts:full counter will increment each time a NFS3ERR_JUKEBOX response is generated. 

Changes

 The customer upgraded from Solaris 11.3 to Soiaris 11.4

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.