HAVIP Started Up Failed With Error: CRS-2632

(Doc ID 2327953.1)

Last updated on NOVEMBER 15, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

1. HAVIP can not be started up with following error:

root@node01:/u01/app/12.1.0/grid/bin# ./srvctl start havip -id nfsexports
PRCR-1079 : Failed to start resource ora.nfsexports.havip
CRS-2674: Start of 'ora.nfsexports.havip' on 'node01' failed
CRS-2674: Start of 'ora.nfsexports.havip' on 'node02' failed
CRS-2632: There are no more servers to try to place resource 'ora.nfsexports.havip' on that would satisfy its placement policy

2. And the exportfs resource is already online:

root@node01:/u01/app/12.1.0/grid/bin# oracle@node01:/u01/app/12.1.0/grid/log/node01 % crsctl stat res -t
ora.nfsexport.export
1 ONLINE ONLINE node01 STABLE  <<<<<<<<<<<< Online
ora.nfsexports.havip
1 ONLINE OFFLINE STABLE <<<<<<<<<<< Offline

3. We could find following error in crsd_orarootagent_root.trc:

2017-10-26 17:41:50.681511 : AGFW:9: {1:42423:53172} Agent received the message: RESOURCE_START[ora.sharenfs.havip 1 1] ID 4098:372530 <<<<<< Startup HAVIP
2017-10-26 17:41:50.681587 : AGFW:9: {1:42423:53172} Preparing START command for: ora.sharenfs.havip 1 1
2017-10-26 17:41:50.681604 : AGFW:9: {1:42423:53172} ora.sharenfs.havip 1 1 state changed from: OFFLINE to: STARTING
2017-10-26 17:41:50.682351 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (:CLSN00107:) clsn_agent::start {
2017-10-26 17:41:50.710213 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] /usr/sbin/ifconfig -a | /usr/bin/grep groupname returned non existent IPMP
2017-10-26 17:41:50.791106 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] InterfaceName = net1
2017-10-26 17:41:50.791131 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] InterfaceNameList = net1
2017-10-26 17:41:50.791150 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] NetworkMask = 255.255.255.0
2017-10-26 17:41:50.791169 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] NetworkAuto = static
2017-10-26 17:41:50.791188 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] AddressType = IPV4
2017-10-26 17:41:50.791206 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] SubNet Number = 172.18.2.0
2017-10-26 17:41:50.791246 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Checking if net1 Interface is fine, flag=0
2017-10-26 17:41:50.791649 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] ifname=net1
2017-10-26 17:41:50.791668 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] subnetmask=255.255.255.0
2017-10-26 17:41:50.791686 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] subnetnumber=172.18.2.0
2017-10-26 17:41:50.791713 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] InterfaceName = net1
2017-10-26 17:41:50.791789 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] HostName 172.18.2.235 translated to 172.18.2.235
2017-10-26 17:41:50.791849 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Interface Name = net1
2017-10-26 17:41:50.791867 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Ip Address = 172.18.2.235
2017-10-26 17:41:50.792103 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] VipActions::startIp {
2017-10-26 17:41:51.292965 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:41:51.292993 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other:
2017-10-26 17:41:51.320652 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] /usr/sbin/ifconfig -a | /usr/bin/grep groupname returned non existent IPMP
2017-10-26 17:41:51.421630 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] VendorType=0
2017-10-26 17:41:51.451568 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] /usr/sbin/ifconfig -a | /usr/bin/grep groupname returned non existent IPMP
2017-10-26 17:41:51.561630 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] VendorType=0
2017-10-26 17:41:51.590228 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] /usr/sbin/ifconfig -a | /usr/bin/grep groupname returned non existent IPMP
2017-10-26 17:41:51.702347 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] VendorType=0
2017-10-26 17:41:51.730825 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] /usr/sbin/ifconfig -a | /usr/bin/grep groupname returned non existent IPMP
2017-10-26 17:41:51.833519 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] VendorType=0
2017-10-26 17:41:51.858750 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] /usr/sbin/ifconfig -a | /usr/bin/grep groupname returned non existent IPMP
2017-10-26 17:41:51.958921 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] VendorType=0
2017-10-26 17:41:51.959038 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] InetAtonException occured while getting subnet for ip :: with subnetmask ::. exception dump fallows
2017-10-26 17:41:51.959070 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: -1, operation: failed system call, loc: inet_aton, OS error: 2, other:<<<<<<
2017-10-26 17:41:51.959189 : default:21: ICMP Ping from 172.18.2.245 to 172.18.2.235
2017-10-26 17:41:52.068797 : default:21: gipcWait failed.
2017-10-26 17:41:52.169065 : default:21: gipcWait failed.
2017-10-26 17:41:52.269314 : default:21: gipcWait failed.
2017-10-26 17:41:52.369529 : default:21: gipcWait failed.
2017-10-26 17:41:52.469788 : default:21: gipcWait failed.
2017-10-26 17:41:52.570113 : default:21: gipcWait failed.
2017-10-26 17:41:52.670452 : default:21: gipcWait failed.
2017-10-26 17:41:52.770732 : default:21: gipcWait failed.
2017-10-26 17:41:52.773219 : default:21: ICMP Ping from 172.18.2.245 to 172.18.2.235
2017-10-26 17:41:52.874152 : default:21: gipcWait failed.
2017-10-26 17:41:52.974439 : default:21: gipcWait failed.
2017-10-26 17:41:53.074723 : default:21: gipcWait failed.
2017-10-26 17:41:53.174979 : default:21: gipcWait failed.
2017-10-26 17:41:53.275273 : default:21: gipcWait failed.
2017-10-26 17:41:53.378710 : default:21: gipcWait failed.
2017-10-26 17:41:53.488768 : default:21: gipcWait failed.
2017-10-26 17:41:53.589132 : default:21: gipcWait failed.
2017-10-26 17:41:54.591637 : default:21: ICMP Ping from 172.18.2.245 to 172.18.2.235
2017-10-26 17:41:54.692776 : default:21: gipcWait failed.
2017-10-26 17:41:54.792991 : default:21: gipcWait failed.
2017-10-26 17:41:54.898700 : default:21: gipcWait failed.
2017-10-26 17:41:54.998923 : default:21: gipcWait failed.
2017-10-26 17:41:55.099265 : default:21: gipcWait failed.
2017-10-26 17:41:55.199506 : default:21: gipcWait failed.
2017-10-26 17:41:55.299750 : default:21: gipcWait failed.
2017-10-26 17:41:55.400149 : default:21: gipcWait failed.
2017-10-26 17:41:56.403075 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Adding 172.18.2.235 on net1  <<<<<<  172.18.2.235 is the IP address for the HAVIP with ID "nfsexports"
2017-10-26 17:41:56.429699 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] VipActions::startIp }
2017-10-26 17:41:56.939319 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:41:56.939346 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other:
2017-10-26 17:41:58.440375 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:41:58.440405 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other:
2017-10-26 17:41:59.941392 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:41:59.941422 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other:
......
......
......
2017-10-26 17:43:46.644621 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:43:46.644646 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other:
2017-10-26 17:43:48.145638 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:43:48.145668 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other:
2017-10-26 17:43:49.646888 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:43:49.646943 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other: <<<<<< Repeated check failed.
2017-10-26 17:43:50.688171 : AGENT:22: {1:42423:53172} {1:42423:53172} Created alert : (:CRSAGF00113:) : Aborting the command: start for resource: ora.sharenfs.havip 1 1
2017-10-26 17:43:50.688205 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] (:CLSN00110:) clsn_agent::abort {
2017-10-26 17:43:50.688259 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] abort {
2017-10-26 17:43:50.688280 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] Agent::abort last call uninititalized
2017-10-26 17:43:50.688301 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] abort command: start
2017-10-26 17:43:50.688321 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] tryActionLock {
2017-10-26 17:43:51.154361 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Failed to check 172.18.2.235 on net1
2017-10-26 17:43:51.154386 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (null) category: 0, operation: , loc: , OS error: 0, other:
2017-10-26 17:43:52.154473 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] start action aborted
2017-10-26 17:43:52.154575 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Action for VIP aborted Exception
2017-10-26 17:43:52.174371 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] Agent::commonStart Exception ActionAbortedException
2017-10-26 17:43:52.176003 :CLSDYNAM:21: [ora.sharenfs.havip]{1:42423:53172} [start] (:CLSN00107:) clsn_agent::start }
2017-10-26 17:43:54.688629 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] got lock
2017-10-26 17:43:54.688676 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] tryActionLock }
2017-10-26 17:43:54.688711 :CLSDYNAM:22: [ora.sharenfs.havip]{1:42423:53172} [start] abort }

 4. The normal VIP could be started up without problem:

ora.node01.vip
1 ONLINE ONLINE node01 STABLE
ora.scan1.vip
1 ONLINE ONLINE node01 STABLE

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms