How to recover from a failed Linux Exadata DB Server dbnodeupdate or rollback
Last updated on JANUARY 23, 2018
Applies to:Oracle Exadata Storage Server Software - Version 126.96.36.199.2 and later
When updates or rollbacks (possibly via dbnodeupdate.sh) fail, an Exadata database server may become unbootable. It can also happen a database server does boot but fails to load the right modules or libraries after an broken or interrupted update, effectively making the system unusable. As a result it can happens users and administrators are unable to login to the system, even via the console. This document describes how to recover from a failing update or rollback on an Exadata DB server. This document only applies to X2 and later DB servers where logical volume management (lvm) is in use and where a backup was made to the inactive system partition prior the update.
If the database server was configured with logical volume management and a backup was made prior to the update, there is a way to 'rollback' to that backup, by 'switching' system partitions. When switching system partitions, the active logical volume becomes inactive and the inactive (the backup) becomes active. In addition to the switching of active and inactive system partitions, the /boot directory belonging to the system partition being made the new active system partition needs to be restored. Also the Grub bootloader needs to be re-installed.
On systems where can login and run dbnodeupdate.sh, rollbacks should always be performed by dbnodeupdate.sh On system where you cannot login, the process of switching system partitions, restoring /boot and bootloader reinstallation can only be done when booting from the diagnostic iso first. This note will guide the reader through the process of booting from diagnostic iso and restoring the relevant components.
Situations where the steps in this document can be used for:
- A failing update to Exadata 188.8.131.52.0 (Oracle Linux 6)
- Failing rollbacks
Sign In with your My Oracle Support account
Don't have a My Oracle Support account? Click to get started
My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms