Cluster Upgrade to 19c Fails with PROC-60 During Backup of Oracle Cluster Registry (Doc ID 2608239.1)

Last updated on JUNE 10, 2020

Applies to:

Symptoms

Upgrade of 18c Grid Infrastructure Cluster to 19c fails on Exadata OVM Environment in oedacli as follows:

oedacli> deploy actions
Deploying Action ID : 3 UPGRADE CLUSTER GIVERSION=19.5.0.0.191015 GIHOMELOC=/u01/app/19.0.0.0/grid WHERE CLUSTERNAME=prod-cluster-x01 STEPNAME=RUN_ROOTSCRIPT
Deploying UPGRADE CLUSTER
Upgrading Cluster
Running Cluster Verification Utility for upgrade readiness..
Pre-upgrade Validation completed successfully.
Running rootupgrade.sh on node exa01vm01.com
Checking file root_exa01vm01.com_2019-11-05_16-55-14-355831276.log on node exa01vm01.com
Error: Error running root scripts, please investigate...
Collecting diagnostics...
Errors occurred. Send /EXAVMIMAGES/joc/linux-x64/WorkDir/Diag-191105_170057.zip to Oracle to receive assistance.

ERROR: Err Create Step RUN_ROOTSCRIPT exception : Error running root scripts, please investigate...
ERROR : Err Create Step RUN_ROOTSCRIPT exception : Error running root scripts, please investigate...
Done
oedacli>

Logfile "/u01/app/19.0.0.0/grid/install/root_exa01vm01.com_2019-11-05_16-55-14-355831276.log" shows:

The log of current session can be found at:
/u01/app/oracle/crsdata/exa01vm01/crsconfig/rootcrs_exa01vm01_2019-11-05_04-55-28PM.log
2019/11/05 16:56:01 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.
2019/11/05 16:56:01 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2019/11/05 16:56:01 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.
2019/11/05 16:56:04 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.
2019/11/05 16:56:04 CLSRSC-464: Starting retrieval of the cluster configuration data
2019/11/05 16:56:41 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2019/11/05 16:58:16 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.
2019/11/05 17:00:05 CLSRSC-693: CRS entities validation completed successfully.
2019/11/05 17:00:09 CLSRSC-515: Starting OCR manual backup.
PROT-23: failed to back up Oracle Cluster Registry
PROC-60: The Oracle Cluster Registry backup file '+DATAC1/prod-cluster-x01/OCRBACKUP/prod-cluster-x01_backup18.0.0.0.0.ocr.258.1023555611' is corrupt.
2019/11/05 17:00:19 CLSRSC-183: OCR manual backup operation failed (error code 60)
Died at /u01/app/19.0.0.0/grid/crs/install/oraocr.pm line 1541.

CRS alert.log:

2019-11-06 12:02:29.756 [CRSCTL(192495)]CRS-2962: Consistency problem in registry while processing server pool 'ora.PROD'.

crsctl_192495.trc:

2019-11-06 12:02:29.733 : CRSSEC:2058815232: {4294967280:65505:2} Exception: PrimaryGroupEntry constructor failed to validate group name with error: 0 groupId: 0x7fae54274560 acl_string: pgrp:oper:rwx
2019-11-06 12:02:29.756 : CRSSEC:2058815232: {4294967280:65505:2} Exception: ACL entry creation failed for: pgrp:oper:rwx
2019-11-06 12:02:29.832 : CRSPE:2058815232: {4294967280:65505:2} {4294967280:65505:2} Created alert : (:CRSPE00239:) : ServerPool not loaded: ora.PROD Error Stack: CRS-2918: Authorization failure: operating system group 'oper' does not exist on server 'exa01vm01'

Changes

Customer added database manually to OCR with srvctl and added group "oper" to "/etc/group", because they got the CRS-2918: Authorization failure: operating system group 'oper' does not exist on server 'exa01vm01'

Later they removed the group "oper" again from /etc/group file.

Cause