Frequent ORA-00240 and KSV master wait due to frequent EM agent query

(Doc ID 2184379.1)

Last updated on SEPTEMBER 20, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

Frequent ORA-00240 errors and IPC send time out due to slow response from ASM instance.

Error in the alert log file:
Tue Aug 23 19:50:32 2016
Errors in file /u01/app/oracle/diag/rdbms/a00wumdb/a00wumdb1/trace/a00wumdb1_arc3_32168.trc (inc
65577):
ORA-00240: control file enqueue held for more than 120 seconds
Incident details in: /u01/app/oracle/diag/rdbms/a00wumdb/a00wumdb1/incident/incdir_465577/a00wumd
_32168_i465577.trc
Tue Aug 23 19:50:33 2016
Dumping diagnostic data in directory=[cdmp_20160823195033], requested by (instance=1, osid=32168
, summary=[incident=465577].
Tue Aug 23 19:52:41 2016
IPC Send timeout detected. Receiver ospid 27163 [oracle@wdc06-s173.ium.internal (PZ99)]
Tue Aug 23 19:52:41 2016
Errors in file /u01/app/oracle/diag/rdbms/a00wumdb/a00wumdb1/trace/a00wumdb1_pz99_27163.trc:
Tue Aug 23 19:53:26 2016
IPC Send timeout detected. Receiver ospid 42493 [oracle@wdc06-s173.ium.internal (LMS1)]
Tue Aug 23 19:53:26 2016
Errors in file /u01/app/oracle/diag/rdbms/a00wumdb/a00wumdb1/trace/a00wumdb1_lms1_42493.trc:
Tue Aug 23 19:53:53 2016
IPC Send timeout detected. Receiver ospid 77958 [oracle@wdc06-s173.ium.internal]

 

RDBMS ARC process is waiting on 'KSV master wait'' for long time and shows the repeating the following call stacks==>

*** 2016-08-23 20:00:30.394 <<== HERE
The current process holds the control file enqueue for too long
Dump of the process state:
ksedsts()+461<-kcc_tac_callback()+148<-ksu_dispatch_tac()+1591<-ksvcheckwait()+948<-ksvsend()+1241<-kfncSlaveSendReal()+923<-kfncSlaveSubmitSlv()+1054<-kfncFileIdentify()+2023<-kfioIdentify()+1706<-ksfd_osmopn()+1120<-ksfdopn1()+2535<-ksfdopn()+217<-kcropn()+693
<-kcroio()+12<-krsk_rlh_get_info()+398<-krse_arc_source_init()+190<-krse_arc_driver_core()+870<-krse_arc_driver()+528<-kcrrwkx()+4299<-kcrrwk()+283<-ksbabs()+771<-krsv_abs()+54<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252
<-main()+201<-__libc_start_main()+253<-_start()+36Current Wait Stack:
1: waiting for 'KSV master wait'
=0x0, =0x0, =0x0
wait_id=484833 seq_num=34869 snap_id=2
wait times: snap=9 min 58 sec, exc=12 min 1 sec, total=12 min 1 sec
wait times: max=infinite, heur=12 min 1 sec
wait counts: calls=240 os=240
in_wait=1 iflags=0x1520
0: waiting for 'Disk file operations I/O'
FileOperation=0x2, fileno=0x1, filetype=0x3
wait_id=484832 seq_num=34866 snap_id=1
wait times: snap=0.000000 sec, exc=0.000064 sec, total=12 min 1 sec
wait times: max=infinite, heur=12 min 1 sec
wait counts: calls=0 os=0
in_wait=1 iflags=0x15a0
There are 101 sessions blocked by this session.
Dumping one waiter:
inst: 1, sid: 4005, ser: 1
wait event: 'enq: CF - contention'
p1: 'name|mode'=0x43460005
p2: '0'=0x0
p3: 'operation'=0x0

 

The ASM client session shows the EM agent process runs the fixed table query==>

+ASM1_ora_xxxx.trc:

*** 2016-08-23 19:58:53.870
*** SESSION ID:(3163.43) 2016-08-23 19:58:53.870
*** CLIENT ID:() 2016-08-23 19:58:53.870
*** SERVICE NAME:() 2016-08-23 19:58:53.870
*** MODULE NAME:(emagent_SQL_osm_cluster) 2016-08-23 19:58:53.870 <<== HERE
*** ACTION NAME:(ACFS_State) 2016-08-23 19:58:53.870

Received ORADEBUG command (#1) 'dump errorstack 1' from process 'Unix process pid: 2696, image: '

*** 2016-08-23 19:58:53.870
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=1, mask=0x0)
----- Error Stack Dump -----
----- Current SQL Statement for this session (sql_id=gshxwdk92h1vn) -----
SELECT v.volume_device, v.mountpath, DECODE(fs.fs_name,null, 'DISMOUNTED','MOUNTED'),
fs.state, fs.available_time, v.inst_id
FROM gv$asm_volume v, gv$asm_acfsvolumes a, gv$asm_filesystem fs
WHERE v.usage = 'ACFS'
AND v.volume_device = a.vol_device(+)
AND v.inst_id=a.inst_id(+)
AND a.primary_vol(+) = 'TRUE'
AND v.mountpath = a.fs_name(+)
AND v.mountpath = fs.fs_name(+)
AND v.inst_id = fs.inst_id(+)
UNION ALL
SELECT 'none','none','none','none',SYSDATE,0 FROM dual
WHERE NOT EXISTS (SELECT 1
FROM gv$asm_volume where usage = 'ACFS')
ORDER BY 1

5381 : | 14 | FIXED TABLE FULL | | | | | |:Q2000| PCWP | | <<== HERE
5401 : | 34 | FIXED TABLE FULL | X$KFFIL | | | | |:Q1000| PCWP | |

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms