Exadata Rac Node Instance Crash with kjzdattdlm: Can not attach to DLM (Doc ID 1386843.1)

Last updated on OCTOBER 25, 2013

Applies to:

Oracle Exadata Hardware - Version 11.2.1.2.1 and later
Information in this document applies to any platform.
***Checked for relevance on 25-Oct-2013***

Symptoms

All Database instances are crashed on Node 2 of a 2 Node RAC
with Exadata Quarter Rack.
ASM is up and running .

when trying to start instances by srvctl
[oracle@wdw1db02 trace]$ srvctl start instance -d maximod -i maximod2
PRCR-1013 : Failed to start resource ora.maximod.db
PRCR-1064 : Failed to start resource ora.maximod.db on node wdw1db02
CRS-5017: The resource action "ora.maximod.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0


CRS-2674: Start of 'ora.maximod.db' on 'wdw1db02' failed

and same error when trying to startup by sqlplus
ORA-03113: end-of-file on communication channel


Found below messages in alert log of database edwqa:
====================================================
lmon registered with NM - instance number 2 (internal mem no 1)
Sat Nov 19 11:04:28 2011
NOTE: initiating MARK startup
Starting background process MARK
Sat Nov 19 11:04:28 2011
MARK started with pid=32, OS id=19148
NOTE: MARK has subscribed
Sat Nov 19 11:06:06 2011
System state dump requested by (instance=2, osid=18227 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/edwqa/edwqa2/trace/edwqa2_diag_18289.trc
Sat Nov 19 11:06:06 2011
PMON (ospid: 18227): terminating the instance due to error 481
Sat Nov 19 11:06:06 2011
ORA-1092 : opitsk aborting process
Sat Nov 19 11:06:06 2011
License high water mark = 2
Dumping diagnostic data in directory=[cdmp_20111119110606], requested by (instance=2, osid=18227 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 18227
USER (ospid: 21470): terminating the instance
Instance terminated by USER, pid = 21470
====================================================

On trace extt2_diag_19298.trc found below messages :
====================================================
*** 2011-11-19 11:04:28.969
Reconfiguration starts [incarn=0]

*** 2011-11-19 11:04:28.969
I'm the voting node
Group reconfiguration cleanup
kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

====================================================

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms