SRVCTL : Disabled DB Instance Status Changed to Online when ASM Restarted (Doc ID 2299664.1)

1) Stopping the database with srvctl.

$ srvctl stop database -d <dbname> -o immediate
[2013-12-03 14:54]

2) Disable the database instance 2

$ srvctl disable instance -d <dbname> -i <instancename2>

3) Cheking the status of database

$ srvctl status instance -d indy -i <instancename1>,<instancename2> -f -v
Instance <instancename1> is not running on node <nodename1>
Instance <instancename2> is disabled on node <nodename2>                                    <========= Disabled
Instance <instancename2> is not running on node <nodename2>

$ crsctl status resource ora.indy.db -t
-----------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
-----------------------------------------------------------------------------
Cluster Resources
-----------------------------------------------------------------------------
ora.<dbname>.db
1 OFFLINE OFFLINE Instance Shutdown
2 OFFLINE OFFLINE Instance Shutdown

4) Starting the database with srvctl.
[2013-12-03 14:58]

$ srvctl start database -d <dbname>

$ crsctl status resource ora.<dbname>.db -t
-----------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
-----------------------------------------------------------------------------
Cluster Resources
-----------------------------------------------------------------------------
ora.<dbname>.db
1 ONLINE ONLINE <instancename1> Open
2 ONLINE OFFLINE Instance Shutdown                                           ---> Does not start the DB instance because instance 2 is disabled

 

5) Kill the PMON process of ASM instance in Node2 (To simulate the crash, and restart the ASM instance)

$ ps -ef| grep pmon
root 4512 1 0 Nov26 ? 00:00:12 /usr/bin/perl -w /opt/oracle. cellos/compmon/exadata_mon_hw_asr.pl -server

oracle 6132 27150 0 15:05 pts/0 00:00:00 grep pmon
oracle 16667 1 0 Nov26 ? 00:01:37 asm_pmon_+ASM2

$ kill -9 16667

6) Cheking the status of database

$ date; ps -ef| grep pmon

$ ps -ef| grep pmon
root 4512 1 0 Nov26 ? 00:00:12 /usr/bin/perl -w /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -server

oracle 7246 27150 0 15:06 pts/0 00:00:00 grep pmon

$ crsctl status resource ora.indy.db -t
-----------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
-----------------------------------------------------------------------------
Cluster Resources
-----------------------------------------------------------------------------
ora.<dbname>.db
1 ONLINE ONLINE <nodename1> Open
2 ONLINE OFFLINE Instance Shutdown,STARTING

re-cheking..
$ ps -ef| grep pmon                                                                                                ---> re-checking                                                      

 

$ ps -ef| grep pmon
root 4512 1 0 Nov26 ? 00:00:12 /usr/bin/perl -w /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -server

oracle 7364 1 0 15:06 ? 00:00:00 asm_pmon_+ASM2
oracle 7684 1 0 15:06 ? 00:00:00 ora_pmon_<instancename2>
oracle 8863 27150 0 15:07 pts/0 00:00:00 grep pmon

$ crsctl status resource ora.<dbname>.db -t
-----------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
-----------------------------------------------------------------------------
Cluster Resources
-----------------------------------------------------------------------------
ora.<dbname>.db
1 ONLINE ONLINE <nodename1> Open
2 ONLINE ONLINE <nodename2> Open                                                                            ---> DB instance is changed available.

 

crsd.log (node 1, PE master, 2013-12-03)
---------------------------------------------

15:06:39.968: [CRSPE]CRS-2676: Start of 'ora.DATA.dg' on '<nodename2>' succeeded
15:06:39.968: [CRSPE]Resource ora.<dbname>.db is disabled on <nodename2>                                  <----(*) crsd knows this is disabled resource

:
15:06:39.969: [CRSPE]CRS-2676: Start of 'ora.DBFS_DG.dg' on '<nodename2>' succeeded
15:06:39.969: [CRSPE]Resource ora.indy.db is disabled on <nodename2>                                      <----(*) crsd knows this is disabled resource

:

15:06:40.324: [CRSPE]ora.<dbname>.db 2 1: uptime exceeds uptime threshold ,resetting restart count
15:06:40.324: [CRSPE]Scheduled retry of local recovery for[ ora.<dbname>.db 2 1]                              <----(*) retry of local recovery for indy2 occurs

:
15:06:40.326: [CRSPE]RI [ora.<dbname>.db 2 1] new internal state: [STARTING] old value: [STABLE]
15:06:40.326: [CRSPE]Sending message to agfw: id = 292879
15:06:40.326: [CRSPE]CRS-2672: Attempting to start 'ora.<dbname>.db' on '<nodename2>'                      <----(*) sends start message to node 2