How to recover from a failed Linux Exadata DB Server dbnodeupdate or rollback

(Doc ID 1952372.1)

Last updated on JUNE 16, 2016

Applies to:

Oracle Exadata Storage Server Software - Version 11.2.2.4.2 and later
Linux x86-64

Goal

When updates or rollbacks (possibly via dbnodeupdate.sh) fail, an Exadata database server may become unbootable. It can also happen a database server does boot but fails to load the right modules or libraries after an broken or interrupted update, effectively making the system unusable. As a result it can happens users and administrators are unable to login to the system, even via the console. This document describes how to recover from a failing update or rollback on an Exadata DB server. This document only applies to X2 and later DB servers where logical volume management (lvm) is in use and where a backup was made to the inactive system partition prior the update. 

If the database server was configured with logical volume management and a backup was made prior to the update, there is a way to 'rollback' to that backup, by 'switching' system partitions. When switching system partitions, the active logical volume becomes inactive and the inactive (the backup) becomes active. In addition to the switching of active and inactive system partitions, the /boot directory belonging to the system partition being made the new active system partition needs to be restored. Also the Grub bootloader needs to be re-installed.

On systems where can login and run dbnodeupdate.sh, rollbacks should always be performed by dbnodeupdate.sh On system where you cannot login, the process of switching system partitions, restoring /boot and bootloader reinstallation can only be done when booting from the diagnostic iso first. This note will guide the reader through the process of booting from diagnostic iso and restoring the relevant components.

Situations where the steps in this document can be used for:

Note: Failing updates to Exadata 12.1.2.1.0 (Oracle Linux 6) should in most cases rollback automatically. 

Solution

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms