Srvctl Can Not Start/Stop Resource For Remote Node ORA-216/ORA-215 (Doc ID 551726.1)

Last updated on JANUARY 04, 2012

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.3 to 11.1.0.7 - Release: 10.2 to 11.1
Information in this document applies to any platform.

Symptoms

Can not use srvctl to control remote resources. For example can not use srvctl stop/start nodeapps for remote node.

crsstat (per <>) show:
HA Resource                                   Target     State
-----------                                   ------     -----
ora.C3D1C.C3D1C1.inst                         ONLINE     ONLINE on node11
ora.C3D1C.C3D1C2.inst                         ONLINE     ONLINE on node12
ora.C3D1C.db                                  ONLINE     ONLINE on node11
ora.node11.ASM1.asm                           ONLINE     ONLINE on node11
ora.node11.LISTENER_NODE11.lsnr               ONLINE     ONLINE on node11
ora.node11.gsd                                ONLINE     ONLINE on node11
ora.node11.ons                                ONLINE     ONLINE on node11
ora.node11.vip                                ONLINE     ONLINE on node11
ora.node12.ASM2.asm                           ONLINE     ONLINE on node12
ora.node12.LISTENER_NODE12.lsnr               ONLINE     ONLINE on node12
ora.node12.gsd                                ONLINE     ONLINE on node12
ora.node12.ons                                ONLINE     ONLINE on node12
ora.node12.vip                                ONLINE     ONLINE on node12


Try "srvctl stop nodeapps -n node12" on node11, it errors out with CRS-216
$ srvctl stop nodeapps -n node12
CRS-0216: Could not stop resource 'ora.node12.LISTENER_NODE12.lsnr'.
CRS-0216: Could not stop resource 'ora.node12.ons'.
CRS-0216: Could not stop resource 'ora.node12.vip'.
CRS-0216: Could not stop resource 'ora.node12.gsd'.

Try "srvctl stop nodeapps -n node12" on node 12, it succeeds.
$ srvctl stop nodeapps -n node12
$                                

Then try "srvctl start nodeapps -n node12" on node 11, it starts VIP for node12 on node 11, but gets CRS-215 for other resources:
$ srvctl start nodeapps -n node12
CRS-0215: Could not start resource 'ora.onde12.gsd'.
CRS-0215: Could not start resource 'ora.node12.ons'.
CRS-0215: Could not start resource 'ora.node12.LISTENER_NODE12.lsnr'.

Try "srvctl start nodeapps -n node12" on node 12, it starts ons and gsd, but could not start listener due to VIP is ONLINE on node11

$ srvctl start nodeapps -n node12
CRS-0215: Could not start resource 'ora.node12.LISTENER_NODE12.lsnr'.

crsstat now show:
HA Resource                                   Target     State
-----------                                   ------     -----
ora.C3D1C.C3D1C1.inst                         ONLINE     ONLINE on node11
ora.C3D1C.C3D1C2.inst                         ONLINE     ONLINE on node12
ora.C3D1C.db                                  ONLINE     ONLINE on node11
ora.node11.ASM1.asm                           ONLINE     ONLINE on node11
ora.node11.LISTENER_NODE11.lsnr               ONLINE     ONLINE on node11
ora.node11.gsd                                ONLINE     ONLINE on node11
ora.node11.ons                                ONLINE     ONLINE on node11
ora.node11.vip                                ONLINE     ONLINE on node11
ora.node12.ASM2.asm                           ONLINE     ONLINE on node12
ora.node12.LISTENER_NODE12.lsnr               ONLINE     OFFLINE
ora.node12.gsd                                ONLINE     ONLINE on node12
ora.node12.ons                                ONLINE     ONLINE on node12
ora.node12.vip                                ONLINE     ONLINE on node11


At the same time, crsd.log report:
2008-04-07 15:09:41.774: [  CRSRES][1474455904]0ora.node12.vip target set to OFFLINE before stop action
2008-04-07 15:09:41.774: [  CRSRES][1474455904]0StopResource: setting CLI values
2008-04-07 15:09:41.814: [ COMMCRS][1510144352]clsc_send_msg: (0xce9e40) NS err(12571, 12560), transport (530, 111, 0)

2008-04-07 15:09:41.814: [ CRSCOMM][1474455904]0CLSC connect failed tonode12ret = 9
2008-04-07 15:09:41.814: [  CRSEVT][1474455904]0invokepeer ret 200
2008-04-07 15:09:41.814: [  CRSRES][1474455904]0Remote stop failed to execute: X_E2E_NotSent : Failed to connect to node: node12
(File: caa_CmdRTI.cpp, line: 504

2008-04-07 15:09:41.814: [  CRSRES][1474455904][ALERT]0Remote stop for `ora.node12.vip` failed on member `node12`
2008-04-07 15:09:41.865: [  CRSRES][1474455904]0Resource recovery not purged:ora.node12.gsd
2008-04-07 15:09:41.877: [  CRSRES][1474455904]0ora.node12.gsd target set to OFFLINE before stop action
2008-04-07 15:09:41.877: [  CRSRES][1474455904]0StopResource: setting CLI values
2008-04-07 15:09:41.924: [ COMMCRS][1510144352]clsc_send_msg: (0xce9e40) NS err(12571, 12560), transport (530, 111, 0)

2008-04-07 15:09:41.924: [ CRSCOMM][1474455904]0CLSC connect failed tonode12ret = 9
2008-04-07 15:09:41.924: [  CRSEVT][1474455904]0invokepeer ret 200
2008-04-07 15:09:41.924: [  CRSRES][1474455904]0Remote stop failed to execute: X_E2E_NotSent : Failed to connect to node: node12
(File: caa_CmdRTI.cpp, line: 504

2008-04-07 15:09:41.925: [  CRSRES][1474455904][ALERT]0Remote stop for `ora.node12.gsd` failed on member `node12`
......

Changes

resource stop/start operation has been performed on two nodes.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms