How to recover from a failed Linux Exadata DB Server dbnodeupdate or rollback (Doc ID 1952372.1)

Last updated on JUNE 28, 2023

Applies to:

Goal

When updates or rollbacks (possibly via dbnodeupdate.sh) fail, an Exadata database server may become unbootable. It also happens a database server does boot but fails to load the right modules or libraries after a broken or interrupted update, effectively making the system unusable. As a result it can happens users and administrators are unable to login to the system, even via the console. This document describes how to recover from a failing update or rollback on an Exadata DB server. This document only applies to X2 and later DB servers where logical volume management (lvm) is in use and where a backup was made to the inactive system partition prior the update. 

If the database server was configured with logical volume management and a backup was made prior to the update, there is a way to 'rollback' to that backup, by 'switching' system partitions. When switching system partitions, the active logical volume becomes inactive and the inactive (the backup) becomes active. In addition to the switching of active and inactive system partitions, the /boot directory belonging to the system partition being made the new active system partition needs to be restored. Also the Grub bootloader needs to be re-installed (for Grub1 only).

On systems where administrators can login and run dbnodeupdate.sh, rollbacks should always be performed by dbnodeupdate.sh On system where you cannot login, the process of switching system partitions, restoring /boot and bootloader reinstallation can only be done when booting from the diagnostic iso first. This note will guide the reader through the process of booting from diagnostic iso and restoring the relevant components.

Situations where the steps in this document can be used for:

• A failing update to Exadata 12.1.2.1.0 (Oracle Linux 6 or Linux 7)
• Failing rollbacks

Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.