Instance crashes during DRM with ORA-481 after dumped MMAN for many times (Doc ID 1996388.1)

Last updated on APRIL 09, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

11.2.0.4 RAC database instance fails during DRM after dumping MMAN process for many times:

 

alert_<ORACLE_SID>.log 

Mon Mar 09 05:20:02 2015
LMS3 (ospid: 6078): terminating the instance due to error 481
Mon Mar 09 05:20:02 2015
System state dump requested by (instance=3, osid=6078 (LMS3)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/geocentral/GC3/trace/GC3_diag_6042_20150309052002.trc
Instance terminated by LMS3, pid = 6078

 

LMS3 trace

*** 2015-03-09 05:18:02.280
Process diagnostic dump for oracle@grid03.prod.quova.com (MMAN), OS id=6086,
pid: 19, proc_ser: 1, sid: 629, sess_ser: 1
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 04:18:01 ]
NOTE: scheduling delay has not been sampled for 0.452235 secs 0.000000 secs from [ 04:17:57 - 04:18:02 ], 5 sec avg
0.000000 secs from [ 04:17:02 - 04:18:02 ], 1 min avg
0.000000 secs from [ 04:13:02 - 04:18:02 ], 5 min avg
loadavg : 1.51 1.48 1.24
Memory (Avail / Total) = 21358.01M / 193821.82M
Swap (Avail / Total) = 32767.99M / 32767.99M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S oracle 6086 1 0 80 0 - 65947 semtim Feb27 ? 00:00:31 ora_mman_GC3
0 S oracle 6100 1 0 80 0 - 60865 semtim Feb27 ? 00:00:05 ora_reco_GC3
Short stack dump:
ksedsts()+465<-ksdxfstk()+32<-ksdxcb()+1927<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+2022<-kslwaitctx()+163<-kslwait()+141<-ksarcv()+207<-ksbabs()+330<-ksbrdp()+1045<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+250<-ssthrdmain()+265<-main()+201<-__libc_start_main()+253
....

*** 2015-03-09 05:18:02.007
Process diagnostic dump for oracle@grid03.prod.quova.com (MMAN), OS id=6086,
....
*** 2015-03-09 05:18:02.145
Process diagnostic dump for oracle@grid03.prod.quova.com (MMAN), OS id=6086,
....
*** 2015-03-09 05:18:02.280
Process diagnostic dump for oracle@grid03.prod.quova.com (MMAN), OS id=6086,
....
*** 2015-03-09 05:18:02.414
Process diagnostic dump for oracle@grid03.prod.quova.com (MMAN), OS id=6086,
....
*** 2015-03-09 05:18:02.549
Process diagnostic dump for oracle@grid03.prod.quova.com (MMAN), OS id=6086,
....=====>>>> every second mman process got dumped for quite a few times all the way until LMS crashed the instance

Terminating instance due to drm freeze timeout
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+465<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+63<-ksuitm()+5570<-kjfzpdrmfrz()+1799<-kjmdrmchk()+774<-kjmsm()+2593<-ksbrdp()+1045<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+250<-ssthrdmain()+265<-main()+201<-__libc_start_main()+253

----- End of Abridged Call Stack Trace -----

*** 2015-03-09 05:20:02.088
LMS3 (ospid: 6078): terminating the instance due to error 481

 

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms