Clusterware CRSD daemon fail to start with ORA-12547 (Doc ID 1991424.1)

Last updated on AUGUST 25, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 to 12.1.0.2 [Release 11.2 to 12.1]
Information in this document applies to any platform.

Symptoms

This issue can occur during a clusteware restart, CRSD daemon, which provide the high availability features fail to start due to ASM login issues.

[[ crsd.log ]]

2015-03-12 10:22:31.389: [ CRSMAIN][1173026544] Initializing OCR [ CLWAL][1173026544]clsw_Initialize: OLR initlevel [70000]

2015-03-12 10:22:32.197: [ OCRASM][1173026544]proprasmo: Error in open/create file in dg [DBFS_DG] [ OCRASM][1173026544]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=12547, loc=kgfokge

2015-03-12 10:22:32.197: [ OCRASM][1173026544]ASM Error Stack : ORA-12547: TNS:lost contact

 Looking at the CRSD client fog file, it appears that, ASM connection from the "root" user fails with TNS-12547 

[[ log_file_client.log ]]

Fatal NI connect error 12547, connecting to:

(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.4/grid/bin/oracle)(ARGV0=oracle+ASM1_ocr)(ENVS='ORACLE_HOME=/u01/app/11.2.0.4/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(PRIVS=(USER=grid)(GROUP=oinstall)))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM crsd.bin)(HOST=myhostname)(USER=root))))
.
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Unix Domain Socket IPC NT Protocol Adaptor for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
Time: 12-MAR-2015 19:50:16
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 517
TNS-00517: Lost contact
nt secondary err code: 32

The issue is even reproducible with the tnsping and sqlplus logins  from root to ASM.

[root@racdb01]# tnsping x
.
TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 12-MAR-2015 15:55:33
.
Copyright (c) 1997, 2013, Oracle. All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.4/grid/bin/oracle)(ARGV0=oracle+ASM1_ocr)(ENVS='ORACLE_HOMM
E=/u01/app/11.2.0.4/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(PRIVS=(USER=grid)(( GROUP=oinstall)))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=crsd.bin)(HOST=myhostname)(USER=root))))

TNS-12547: TNS:lost contact

[root@racdb01]# sqlplus /@x as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Thu Mar 12 22:02:16 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.

ERROR:
ORA-12547: TNS:lost contact

The strace of the tnsping actually indicate the process actually core dumps and signal ORA-00600: [dbkc_init_bs_ctx-10] when initiated the ASM connection.  

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms