GI rootupgrade.sh fails on last node: ORA-15164: cluster rolling downgrade incomplete (Doc ID 1613556.1)

Last updated on AUGUST 08, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

Upgrading from 11.2.0.2 to 11.2.0.4 Grid Infrastructure, rootupgrade.sh fails on last node:

 

2013-12-13 13:50:33: /oracle/grid/product/11.2.0.4/bin/crsctl set crs activeversion ... passed
2013-12-13 13:51:33: Rolling upgrade is set to 1
2013-12-13 13:51:33: End ASM rolling upgrade
2013-12-13 13:51:33: Executing as oracle: /oracle/grid/product/11.2.0.4/bin/asmca -silent -upgradeLocalASM -lastNode /oracle/grid/product/11.2.0
2013-12-13 13:51:33: Running as user oracle: /oracle/grid/product/11.2.0.4/bin/asmca -silent -upgradeLocalASM -lastNode /oracle/grid/product/11.2.0
2013-12-13 13:51:33: Invoking "/oracle/grid/product/11.2.0.4/bin/asmca -silent -upgradeLocalASM -lastNode /oracle/grid/product/11.2.0" as user "oracle"
2013-12-13 13:51:33: Executing /bin/su oracle -c "/oracle/grid/product/11.2.0.4/bin/asmca -silent -upgradeLocalASM -lastNode /oracle/grid/product/11.2.0"
2013-12-13 13:51:33: Executing cmd: /bin/su oracle -c "/oracle/grid/product/11.2.0.4/bin/asmca -silent -upgradeLocalASM -lastNode /oracle/grid/product/11.2.0"
2013-12-13 13:51:41: Command output:
>
> ORA-15164: cluster rolling downgrade incomplete
>
>
>End Command output
2013-12-13 13:51:41: ASM rolling upgrade action failed, see logs for details

 

[main] [ 2013-12-13 13:51:41.191 CET ] [USMInstanceRac.upgradeLocalASMLast:1228] sql to be executed:=select SYS_CONTEXT('sys_cluster_properties', 'cluster_state') from dual
[main] [ 2013-12-13 13:51:41.206 CET ] [USMInstanceRac.upgradeLocalASMLast:1231] sql to be executed:=ALTER SYSTEM STOP ROLLING MIGRATION
[main] [ 2013-12-13 13:51:41.209 CET ] [SQLEngine.done:2189] Done called
[main] [ 2013-12-13 13:51:41.209 CET ] [USMInstanceRac.upgradeLocalASMLast:1293] oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1658)
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeSql(SQLEngine.java:1925)
oracle.sysman.assistants.usmca.backend.USMInstanceRac.upgradeLocalASMLast(USMInstanceRac.java:1232)
oracle.sysman.assistants.usmca.service.UsmcaService.upgradeOnLastNode(UsmcaService.java:1100)
oracle.sysman.assistants.usmca.model.UsmcaModel.performUpgradeOnLastNode(UsmcaModel.java:884)
oracle.sysman.assistants.usmca.model.UsmcaModel.performOperation(UsmcaModel.java:806)
oracle.sysman.assistants.usmca.model.UsmcaModelRac.performOperation(UsmcaModelRac.java:303)
oracle.sysman.assistants.usmca.Usmca.execute(Usmca.java:174)
oracle.sysman.assistants.usmca.Usmca.main(Usmca.java:369)
[main] [ 2013-12-13 13:51:41.209 CET ] [UsmcaLogger.logException:173] SEVERE:method oracle.sysman.assistants.usmca.backend.USMInstanceRac:upgradeLocalASMLast
[main] [ 2013-12-13 13:51:41.210 CET ] [UsmcaLogger.logException:174] ORA-15164: cluster rolling downgrade incomplete

 

Fri Dec 13 13:37:14 2013
NOTE: No asm libraries found in the system
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'bond0:1' configured from GPnP for use as a private interconnect.
[name='bond0:1', type=1, ip=169.254.152.208, mac=f4-ce-46-84-c8-a2, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=10.233.42.84, mac=f4-ce-46-84-c8-a0, net=10.233.42.0/24, mask=255.255.255.0, use=public/1]
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/grid/product/11.2.0/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
NOTE: Volume support enabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production               ==>>>> last ASM restart was from 11.2.0.2 home

 

2013-12-13 13:34:45: check old crs stack
2013-12-13 13:34:45: 11gR2 check routine
2013-12-13 13:34:45: Executing cmd: /oracle/grid/product/11.2.0/bin/crsctl check crs
2013-12-13 13:34:45: Command output:
> CRS-4638: Oracle High Availability Services is online
>End Command output
2013-12-13 13:34:45: Earlier version Oracle Grid Infrastructure is not running
....
2013-12-13 13:35:22: Checking to see if Oracle CRS stack is already configured
2013-12-13 13:35:22: Oracle CRS stack is already configured and will be running under init (1M)
....
2013-12-13 13:36:29: Executing cmd: /oracle/grid/product/11.2.0.4/bin/crsctl start resource ora.cssd -init
2013-12-13 13:36:56: Command output:

==>>>> 11.2.0.4 root script started all daemons (including ASM) in 11.2.0.2 home

 

2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0
2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = GIPCNM, LogLevel = 2, TraceLevel = 0
2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = GIPCGM, LogLevel = 2, TraceLevel = 0
2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = GIPCCM, LogLevel = 2, TraceLevel = 0
2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = CLSF, LogLevel = 0, TraceLevel = 0
2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = SKGFD, LogLevel = 0, TraceLevel = 0
2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = GPNP, LogLevel = 1, TraceLevel = 0
2013-12-13 13:36:32.041: [ CSSD][1234937600]clsu_load_ENV_levels: Module = OLR, LogLevel = 0, TraceLevel = 0
[ CSSD][1234937600]clsugetconf : Configuration type [4].
2013-12-13 13:36:32.041: [ CSSD][1234937600]clssscmain: Starting CSS daemon, version 11.2.0.2.0, in (clustered) mode with uniqueness value 1386938192

==>>>> 11.2.0.2 CSS started by 11.2.0.4 root script

 

2013-12-12 11:50:03: /oracle/grid/product/11.2.0/bin/crsctl stop crs -f
2013-12-12 11:50:03: Executing cmd: /oracle/grid/product/11.2.0.4/bin/crsctl check cluster -n racnode1
....
2013-12-12 11:50:12: Executing cmd: /usr/lib/lsb/install_initd /etc/init.d/ohasd
2013-12-12 11:50:13: Command output:
....
> insserv: exiting now without changing boot order!
>End Command output
2013-12-12 11:50:13: Failed to install ohasd startup script, error: Inappropriate ioctl for device
2013-12-12 11:50:13: ###### Begin DIE Stack Trace ######
2013-12-12 11:50:13: Package File Line Calling
2013-12-12 11:50:13: --------------- -------------------- ---- ----------
2013-12-12 11:50:13: 1: main rootcrs.pl 393 crsconfig_lib::dietrap
2013-12-12 11:50:13: 2: crsconfig_lib crsconfig_lib.pm 5427 main::__ANON__
2013-12-12 11:50:13: 3: crsconfig_lib crsconfig_lib.pm 5401 crsconfig_lib::register_service
2013-12-12 11:50:13: 4: main rootcrs.pl 772 crsconfig_lib::perform_register_service
2013-12-12 11:50:13: ####### End DIE Stack Trace #######

==>>>> as root script failed to install service ohasd to the OS, a workaround was implemented and node rebooted.

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms