TCP Sends To STANDBY Can Hang and Cause ORA-16146 (Doc ID 844117.1)

Last updated on MARCH 08, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 9.2.0.1 to 11.1.0.7 [Release 9.2 to 11.1]
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
Sun Solaris SPARC (64-bit)Sun Solaris x86-64 (64-bit)
Sun Solaris 9 SPARC (64-bit)
Sun Solaris 10 SPARC (64-bit)
Sun Solaris 10 x86_64 (64-bit)

Symptoms

When doing TCP sends to the remote standby database an Oracle shadow process might hang due to the
Solaris kernel thread being stuck.
If the Oracle process was previously holding the CF enqueue then another process trying to acquire
the CF enqueue like ARC<n> would  timeout after 120 sec and signal ORA-16146:

16146, 00000, "standby destination control file enqueue unavailable"
// *Cause: The target standby destination control file is currently
// unavailable to the Remote File Server (RFS) process.
// This indicates that the target destination is the primary
// database itself.
// *Action: Check for and eliminate the standby destination archive log
// parameter in question.

This problem has been reported only in a dataguard configuration but due to its nature it can happen with
any Solaris TCP socket sends.

Another symptom of the problem is that pstack on the Oracle process holding the CF enqueue does hang too
for some hundred of seconds until the hang clears up but does not show the process call stack at hanging time (this is a typical symptom of a kernel thread being hung)

Changes

The stuck kernel thread would look like:

genunix:cv_wait_sig+0x114(, 0x60039be6d30)
genunix:str_cv_wait+0x28(0x600322db60a, 0x60039be6d30, 0xffffffffffffffff, 0x0, , 0x4)
genunix:strwaitq+0x238(0x60039be6cb0, 0x5, , 0x3, 0xffffffffffffffff, 0x2a1060b175c)
genunix:strwrite_common+0x278(, 0x2a1060b1aa0, 0x30013a280c8, 0x4)
sockfs:sostream_direct+0x210(0x600352b7768, 0x2a1060b1aa0, 0x0, 0x30013a280c8?)
sockfs:sotpi_sendmsg+0x4e8(0x600352b7768, 0x2a1060b1a70, 0x2a1060b1aa0)
sockfs:sendit+0x134(, 0x2a1060b1a70, 0x2a1060b1aa0, 0x8)
sockfs:send+0x60(, 0xffffffff72e1e000)
unix:syscall_trap+0xac()

and having one of the TCP stream queues with flag QFULL:

CSolarisCAT(./vmcore.0/10U)> stream -l 0x60039be6cb0

write queue: 0x3002321a3a0 mi_idnum: 0x13f1 mi_idname: tcp
IPv4 src 192.9.200.73 port: 47536 , dst 10.100.20.103 port: 13724 vnetd
state:ESTABLISHED

q_qinfo: 0x70195610 q_next: 0x0
q_first: 0x0 q_last: 0x0
q_link: 0x0 q_ptr: 0x3007703e900 q_count: 0
q_minpsz: 0x0 q_maxpsz: 0x10430
q_hiwat: 0x7f q_lowat: 0x10
q_bandp: 0x0 q_stream: 0x60039be6cb0
q_syncq: 0x3002321a498 q_nband: 0x0
q_nfsrv: 0x3002321a3a0 q_nbsrv: 0x600322db588
q_sqhead: 0x0 q_sqtail: 0x0
QWANTR - Someone wants to read Q
QWANTW - Someone wants to write Q
QFULL - Q is considered full
QUSE - This queue in use (allocation)
QMTSAFE - stream module is MT-safe
QEND - last queue in stream
QISDRV - the Queue is attached to a driver
messages: 0 mblks: 0 alloc: 0
syncq messages:0 mblks:0 alloc:0

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms