Consolidation: GI upgrade fails with CRS-1112 due to ORA-00020 in ASM (Doc ID 2112429.1)

Last updated on APRIL 06, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

GI Upgrade from 11.2.0.4 to 12.1.0.2 fails:

root script log:

2016/01/16 18:48:37 CLSRSC-482: Running command: '/gmioracle/rac_home/product/12.1.0.2/bin/crsctl set crs activeversion'

Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the OCR.
CRS-1111: Error upgrading the Oracle Clusterware. Details [Upgrade Coordinator failed].
CRS-1112: Failed to set the Oracle Clusterware operating version 12.1.0.2.0
CRS-4000: Command Set failed, or completed with errors.
2016/01/16 18:48:41 CLSRSC-180: An error occurred while executing the command '/gmioracle/rac_home/product/12.1.0.2/bin/crsctl set crs activeversion'(error code 1)

2016/01/16 18:48:44 CLSRSC-281: Failed to set active version of Oracle Clusterware

Died at /gmioracle/rac_home/product/12.1.0.2/crs/install/crsupgrade.pm line 4149.
The command '/gmioracle/rac_home/product/12.1.0.2/perl/bin/perl-I /gmioracle/rac_home/product/12.1.0.2/perl/lib -I/gmioracle/rac_home/product/12.1.0.2/crs/install
/gmioracle/rac_home/product/12.1.0.2/crs/install/rootcrs.pl  -upgrade' execution failed

 

crsd.trc on node3

2016-01-16 18:48:41.036504 :GIPCHDEM:3627759360:  gipchaDaemonProcessHAInvalidate: completed ha name invalidate for node 0x7fcfc81f63b0 { host 'xmgpa1', haName '906d-e11f-e905-1b11', srcLuid ad3fdf23-be434c75, dstLuid 17795e05-76e5d27f numInf 1, sentRegister 1, localMonitor 0, baseStream 0x7fcfc81fb2c0 type gipchaNodeType12001 (20), nodeIncarnation 3e89ce31-c6495efe incarnation 2 flags 0x82c}
2016-01-16 18:48:41.290340 :  CRSCCL:3237918464: Processing reconfig: CSS inc 16, mbr count 3
2016-01-16 18:48:41.290353 :  CRSCCL:3237918464: cclGetMemberData called, current mbrcount=3
2016-01-16 18:48:41.290362 :  CRSCCL:3237918464: Current map: 0x3650e58: : 2, 3, 4  <<<<<<<<<<<<<<<crsd on master node1 crashed....
2016-01-16 18:48:41.290369 :  CRSCCL:3237918464: Previous map: 0x7fcfc0fe6ef8: : 1, 2, 3, 4
2016-01-16 18:48:41.290378 :  CRSCCL:3237918464: Member (2, 3325713794, xmgpa2:12.1.0.2.0) @  found.
2016-01-16 18:48:41.290382 :  CRSCCL:3237918464: Member (3, 548829048, xmgpa3:12.1.0.2.0) @  found.
2016-01-16 18:48:41.290386 :  CRSCCL:3237918464: Member (4, 7110484, xmgpa4:12.1.0.2.0) @  found.
2016-01-16 18:48:41.290398 :  CRSCCL:3237918464: clsCclGipcDisconnect: called with pEndp->endp= 0x949
2016-01-16 18:48:41.290531 :  CRSCCL:3237918464: clsCclGipcDisconnect: calling gipcDisconnect with pEndp->endp= 0x949
2016-01-16 18:48:41.290573 :  CRSCCL:3237918464: clsCclgipcDisconnect: Disconnection successful, going to destroy pEndp->endp= 0x949
2016-01-16 18:48:41.290627 :  CRSCCL:3237918464: clsCclMembershipChangeCallback:  Calling mem callback fn.with ctx.
2016-01-16 18:48:41.290703 :CLSFRAME:3237918464:  CCL MEMBER LEFT:1:1:CRSD:xmgpa1  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-01-16 18:48:41.290781 :CLSFRAME:3237918464:  Disconnected from CRSD:xmgpa1 process: {Absolute|Node:1|Process:-968166842|Type:1}
2016-01-16 18:48:41.291003 :    AGFW:2550134528: {3:38964:924} Agfw Proxy Server received process disconnected notification, count=1
2016-01-16 18:48:41.291014 :UiServer:2541729536: {3:38964:927} Master PE disconnected during upgrade
2016-01-16 18:48:41.291068 :UiServer:2541729536: {3:38964:927} Rolling upgrade of CRS has finished. Unblocking all UI commands.
2016-01-16 18:48:41.291149 :  CRSAPP:2541729536: {3:38964:927} Enabled Application layer
2016-01-16 18:48:41.291222 :  CRSCCL:3237918464: Reconfig handled

 

alert_+ASM1.log (OCR master node)

Sat Jan 16 18:48:10 2016
ORA-00020: maximum number of processes (200) exceeded
ORA-20 errors will not be written to the alert log for
the next minute. Please look at trace files to see all
the ORA-20 errors.
Sat Jan 16 18:48:10 2016
ERROR: paging ASM fault extent map failed gn=10 fn=255 extent=60
Sat Jan 16 18:48:10 2016
Errors in file /gmioracle/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_ora_6250.trc:
ORA-15055: unable to connect to ASM instance
ORA-00020: maximum number of processes (200) exceeded

 

 

 

Changes

GI upgrade from 11.2.0.4 to 12.1.0.2

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms