"ORA-29701 raised in ASM I/O path" in Oracle Restart After the GI Standalone Brought up by root user (Doc ID 1364558.1)

Last updated on OCTOBER 30, 2014

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.2 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.

Symptoms


Intermittent ORA-29701 errors in both ASM and DB alert logs after the Grid infrastructure restart.

Tue Sep 13 00:37:56 2011
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 15261
Tue Sep 13 00:39:28 2011
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 14121


The trace file of the failed process indicate that it had problems in communicating with the ocssd.bin running in the system.

2011-09-13 00:37:56.822: [ CSSCLNT]clssscConnect: gipcWait failed with 16 (0x12)
2011-09-13 00:37:56.833: [ CSSCLNT]
clsssInitNative: connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)) failed, rc 16
kgxgncin: CLSS init failed with status 3
kgxgncin: return status 3 (1311719766 SKGXN not av) from CLSS
NOTE: kfmsInit: ASM failed to initialize group services

Error ORA-29701 signaled at ksedsts()+461<-ksf_short_stack()+77<-kge_snap_callstack()+63<-kge_sigtrace_dump()+69
<-kgepop()+712<-kgeselv()+290<-ksesecl0()+162<-kfmsInit()+175<-kfmsSlvReg()+1142<-kfmdSlvOpPriv()+2152
<-kfmdWriteSubmitted()+649<-kfk_process_an_ioq()+210<-kfk_submit_io()+74<-kfk_io1()+1089<-kfkRequest()+14
<-kfk_transitIO()+1696<-kfioSubmitIO()+5272<-kfioRequestPriv()+199<-kfioRequest()+706<-ksfd_kfioRequest()
+649<-ksfd_osmgo()+256<-ksfdgo()+861<-ksfdaio()+1680<-kcflbi()+908<-kcbldio()+3042<-kcblio()+318
<-kcblnb_dscn()+1129<-kdlwdb()+1240<-kdlwfb()+574<-kdl_write1()+5750<-kdl_copy()+1439<-kdlf_copy()+356
<-kokliclo()+399<-koklcre()+672<-kokleva()+1006<-evaopn2()+812<-qesltcEvalOutofLineCols()+221
<-qesltcBeforeRowProcessing()+885<-__PGOSF794_qerltcNoKdtBufferedInsRowCBK()+223<-qerltcSingleRowLoad()
+279<-qerltcFetch()+379<-insexe()+682<-opiexe()+5531<-opipls()+2073<-opiodr()+2797<-__PGOSF175_rpidrus(
)+211<-skgmstack()+148<-rpiswu2()+638<-rpidrv()+1384<-psddr0()+473<-psdnal()+457<-pevm_EXECC()+505
<-pfrinstr_EXECC()+80<-pfrrun_n

ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 15261


Every database process wants to perform an IO operation with ASM, need to register itself with the ocssd.bin. And  CSSD log file around the time of the issue, reports that cssd has reached the max open file limit at the OS level, and can't open any more files to authenticate the incoming client connection requests.

2011-09-13 00:37:56.269: [GIPCXCPT][1097873728] gipcmodClsaAuthStart: failuring during clsaauthmsg ret clsaretOSD (8), endp 0x2aaab052e980 [000000000004227a] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_fxgliws_)(GIPCID=01201d45-8e19035d-8599))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_fxgliws_)(GIPCID=8e19035d-01201d45-14986))', numPend 5, numReady 1, numDone 3, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 14986, flags 0x603710, usrFlags 0x14000 }

2011-09-13 00:37:56.269: [GIPCXCPT][1097873728] gipcmodClsaAuthStart: slos op : getpwuid_scls
2011-09-13 00:37:56.269: [GIPCXCPT][1097873728] gipcmodClsaAuthStart: slos dep : Too many open files (24)
2011-09-13 00:37:56.269: [GIPCXCPT][1097873728] gipcmodClsaAuthStart: slos loc : authrespchk7
2011-09-13 00:37:56.269: [GIPCXCPT][1097873728] gipcmodClsaAuthStart: slos info: failed for uid 500


2011-09-13 00:37:56.269: [GIPCXCPT][1097873728] gipcmodClsaAuthStart: failuring during clsaauthmsg ret clsaretPROTERR (4), endp 0x2aaab052e980 [000000000004227a] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_fxgliws_)(GIPCID=01201d45-8e19035d-8599))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_fxgliws_)(GIPCID=8e19035d-01201d45-14986))', numPend 4, numReady 1, numDone 4, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 14986, flags 0x603710, usrFlags 0x14000 }


Changes

None.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms