ERROR: Unable to start stack, see /var/log/cellos/dbnodeupdate.log for more info. Re-run dbnodeupdate.sh -c after resolving the issue (Doc ID 2772578.1)

Last updated on NOVEMBER 16, 2022

Applies to:

Symptoms

On : 20.1.6.0.0 version, Exadata Machine Patching

patchmgr release: 21.210314 There was an error during the completion step on <hostname>

Hello,
during an domU patching, we receive the following error :
  SUMMARY OF ERRORS FOR <hostname>

2021-04-28 19:38:28 +0200 :ERROR : There was an error during the completion step on frepcou-lqora01.
2021-04-28 19:38:28 +0200 :ERROR : Please correct the error and run "/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh -c" on <hostname> to complete the update.
2021-04-28 19:38:28 +0200 :ERROR : The dbnodeupdate.log and diag files can help to find the root cause.
2021-04-28 19:38:28 +0200 :ERROR : Initiate completion step from dbnodeupdate.sh on frepcou-lqora01
2021-04-28 19:39:00 +0200 :INFO : SUMMARY FOR ALL NODES:
2021-04-28 19:38:50 +0200 :ERROR : <hostname> has state: COMPLETE STEP FAILED
2021-04-28 19:39:01 +0200 :FAILED : Initiate update on node(s).
2021-04-28 19:39:02 +0200 :ERROR : FAILED run of command:./patchmgr -dbnodes /tmp/d06domu_group -upgrade -log_dir auto -iso_repo /EXAVMIMAGES/Patches_Jan2021/DomU/p32248102_201000_Linux-x86-64.zip -target_version 20.1.6.0.0.210113 -allow_active_network_mounts
2021-04-28 19:39:02 +0200 :INFO : Upgrade attempted on nodes in file /tmp/d06domu_group: [frepcou-lqora01]
2021-04-28 19:39:03 +0200 :INFO : Current image version on dbnode(s) is:
2021-04-28 19:39:03 +0200 :INFO : <hostname>: 20.1.6.0.0.210113
2021-04-28 19:39:03 +0200 :INFO : For details, check the following files in /EXAVMIMAGES/Patches_Jan2021/Tools/Dbnu/dbserver_patch_20.210314/log/
2021-04-28 19:39:03 +0200 :INFO : - <dbnode_name>_dbnodeupdate.log
2021-04-28 19:39:03 +0200 :INFO : - patchmgr.log
2021-04-28 19:39:03 +0200 :INFO : - patchmgr.trc
2021-04-28 19:39:03 +0200 :INFO : Exit status:1
2021-04-28 19:39:03 +0200 :INFO : Exiting.
 

ERROR
-----------------------
error during the completion step

Relinking /u01/app/oracle/product/12.2.0.1/dbhome_2 as oracle (with rds option)
(*) 2021-04-28 22:33:56: Locking and starting Grid Infrastructure (/u01/app/12.2.0.1/grid)
(*) 2021-04-28 22:38:54: Sleeping another 60 seconds while stack is starting (1/15)
(*) 2021-04-28 22:39:55: Sleeping another 60 seconds while stack is starting (2/15)
(*) 2021-04-28 22:40:55: Sleeping another 60 seconds while stack is starting (3/15)
(*) 2021-04-28 22:41:55: Sleeping another 60 seconds while stack is starting (4/15)
(*) 2021-04-28 22:42:55: Sleeping another 60 seconds while stack is starting (5/15)
(*) 2021-04-28 22:43:55: Sleeping another 60 seconds while stack is starting (6/15)
(*) 2021-04-28 22:44:55: Sleeping another 60 seconds while stack is starting (7/15)
(*) 2021-04-28 22:45:55: Sleeping another 60 seconds while stack is starting (8/15)
(*) 2021-04-28 22:46:56: Sleeping another 60 seconds while stack is starting (9/15)
(*) 2021-04-28 22:47:56: Sleeping another 60 seconds while stack is starting (10/15)
(*) 2021-04-28 22:47:56: CRS not up after 10 tries, restarting...
(*) 2021-04-28 22:48:56: Sleeping another 60 seconds while stack is starting (11/15)
(*) 2021-04-28 22:49:56: Sleeping another 60 seconds while stack is starting (12/15)
(*) 2021-04-28 22:50:56: Sleeping another 60 seconds while stack is starting (13/15)
(*) 2021-04-28 22:51:56: Sleeping another 60 seconds while stack is starting (14/15)
(*) 2021-04-28 22:52:57: Sleeping another 60 seconds while stack is starting (15/15)

ERROR: Unable to start stack, see /var/log/cellos/dbnodeupdate.log for more info. Re-run dbnodeupdate.sh -c after resolving the issue. If you wish to skip relinking append an extra '-i' flag. Exiting...

/var/log/cellos/dbnodeupdate.log
=========================

[1619641931][2021-04-28 22:33:56 +0200][INFO][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][PrintMsg][] (ACTION:) Locking and starting Grid Infrastructure (/u01/app/12.2.0.1/grid)
[1619641931][2021-04-28 22:33:56 +0200][INFO][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] Locking and starting Grid Infrastructure (/u01/app/12.2.0.1/grid)
[1619641931][2021-04-28 22:33:56 +0200][INFO][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][DiaryEntry][] Entering CopySystemdHas
[1619641931][2021-04-28 22:33:56 +0200][CMD][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][CopySystemdHas][] [CMD: systemctl stop ohasd.service] [CMD_STATUS: 0]
[1619641931][2021-04-28 22:34:10 +0200][WARNING][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][CopySystemdHas][] [CMD: systemctl disable ohasd.service] [CMD_STATUS: 0]
----- START STDERR -----
ohasd.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig ohasd off
----- END STDERR -----
[1619641931][2021-04-28 22:34:10 +0200][CMD][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][CopySystemdHas][] [CMD: rm -f /etc/init.d/ohasd] [CMD_STATUS: 0]
[1619641931][2021-04-28 22:34:10 +0200][INFO][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][DiaryEntry][] Leaving CopySystemdHas
[1619641931][2021-04-28 22:34:11 +0200][INFO][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] locking as root: /u01/app/12.2.0.1/grid/crs/install/rootcrs.sh -lock
[1619641931][2021-04-28 22:34:11 +0200][CMD][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] [CMD: /u01/app/12.2.0.1/grid/crs/install/rootcrs.sh -lock] [CMD_STATUS: 0]
----- START STDOUT -----
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/<hostname>/crsconfig/crslock_ <hostname>_2021-04-28_10-34-11PM.log
2021/04/28 22:34:13 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
----- END STDOUT -----
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/<hostname>/crsconfig/crslock_frepcou-lqora01_2021-04-28_10-34-11PM.log
2021/04/28 22:34:13 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
[1619641931][2021-04-28 22:34:14 +0200][CMD][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] [CMD: sleep 30] [CMD_STATUS: 0]
[1619641931][2021-04-28 22:34:44 +0200][INFO][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] sleeping 30 seconds..before starting stack
[1619641931][2021-04-28 22:34:44 +0200][CMD][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] [CMD: /u01/app/12.2.0.1/grid/bin/crsctl start crs] [CMD_STATUS: 1]
----- START STDOUT -----
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.
----- END STDOUT -----
[1619641931][2021-04-28 22:34:44 +0200][WARNING][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] stack start failed retrying
[1619641931][2021-04-28 22:34:44 +0200][CMD][/u01/dbnodeupdate.patchmgr/dbnodeupdate.sh][RelinkOHomes][] [CMD: /u01/app/12.2.0.1/grid/bin/crsctl stop crs -f] [CMD_STATUS: 1]
----- START STDOUT -----
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.