SuperCluster: 12.1.0.2 ASM XDMG process causes hang in PR_P_LOCK on ORA-15311: process terminated due to fenced I/O

Last updated on JULY 28, 2016

Applies to:

Symptoms

Solaris zones and or Logical Domains can hang . A core file obtained will show

==== user (LWP_SYS) thread: 0xc4010c3c1700 PID: 29194 ====
 cmd: asm_xdmg_+ASM6
 fmri: lrc:/etc/rc3_d/S96ohasd
 t_wchan: 0x2a10b7bf5d8 sobj: condition var (from
 genunix:vmtask_run_xjob+0x1f0)
 t_procp: 0xc4013eb1e010
 p_as: 0xc400be67f2f0 size: 351559680 RSS: 326246400
 a_hat: 0xc400be265000
 cnum: CPU48:4/46021
 cpusran:
 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,2
 9,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,5
 5,56,57,58,59,60,61,62,63
 p_zone: 0x106affd8 (global)
 t_stk: 0x2a10b7bfad0 sp: 0x2a10b7bec51 t_stkbase: 0x2a10b7b8000
 t_pri: 60 (TS) pctcpu: 0.000000
 t_transience: 0 t_wkld_flags: 0
 t_lwp: 0xc4010466b2a0 t_tid: 1
 machpcb: 0x2a10b7bfad0
 lwp_ap: 0x2a10b7bfbc0
 t_mstate: LMS_SLEEP ms_prev: LMS_SYSTEM
 ms_state_start: 16 hours 8 minutes 30.525219929 seconds earlier
 ms_start: 23 hours 23 minutes 6.660387809 seconds earlier
 t_cpupart: 0x1041fa20(0) last CPU: 62
 idle: 58110525212482 nsec (16h8m30.525212482s)
 start: Sat Mar 5 11:43:25 2016
 age: 84191 seconds (23 hours 23 minutes 11 seconds)
 t_state: TS_SLEEP
 t_flag: 0x1000 (T_LWPREUSE)
 t_proc_flag: 0x104 (TP_TWAIT|TP_MSACCT)
 t_schedflag: 0x8003 (TS_LOAD|TS_DONT_SWAP|TS_WKLD_PERM)
 t_acflag: 3 (TA_NO_PROCESS_LOCK|TA_BATCH_TICKS)
 p_flag: 0x4a004000 (SEXECED|SMSACCT|SAUTOLPG|SMSFORK)

 pc: genunix:cv_wait+0x3c: call unix:swtch

 void genunix:cv_wait+0x3c((kcondvar_t *), (kmutex_t *)0x2a10b7bf5d0)
 int genunix:vmtask_run_xjob+0x1f0((ulong_t), (ulong_t), (vmtask_func_t),
 (void *), (vmtask_undo_t *), (vmtask_ops_t *))
 genunix:vmtask_run_job((ulong_t), (ulong_t)1, (vmtask_func_t)0x129c608, (void
 *)0x2a10b7bf6c8, (vmtask_undo_t *)0) - frame recycled
 void genunix:anon_free_pages+0x44((struct anon_hdr *)0xc4010bf4a6f0,
 (ulong_t)0, (size_t)0x70000, (uint_t)1)
 void genunix:segvn_free+0x90((struct seg *)0xc40134bd1b38)
 void genunix:seg_free+0x28((struct seg *)0xc40134bd1b38)
 int genunix:segvn_unmap+0x334((struct seg *)0xc40134bd1b38,
 (caddr_t)0xffffffff7cb70000, (size_t)0x70000)
 int genunix:as_unmap+0x1e4((struct as *)0xc400be67f2f0,
 (caddr_t)0xffffffff7cb70000, (size_t)0x80000)
 int genunix:munmap+0x50((caddr_t), (size_t)0x80000)
 unix:_syscall_no_proc_exit+0x58()
 -- switch to user thread's user stack --

ASM XDMG Trace files will show a termination due to fencing

Reconnect: Attempts: 1 Last TS: 23486775047 Last Use TS: 22936303758 ctime:
22936303759 is_idle: 0 has_open_disks: Yes
2016-03-05 20:50:06.925218 : Exadata operations failed at oss_open failed  with err: I/O request fenced (221)
error 15311 detected in background process
OPIRIP: Uncaught error 447. Error stack:
ORA-447: fatal error in background process
ORA-15311: process terminated due to fenced I/O

Changes

None but impacts only 12.1.0.2

Cause