How to recover from a failed Linux Exadata DB Server dbnodeupdate or rollback
(Doc ID 1952372.1)
Last updated on JANUARY 25, 2020
Oracle Exadata Storage Server Software - Version 188.8.131.52.2 and later Linux x86-64
When updates or rollbacks (possibly via dbnodeupdate.sh) fail, an Exadata database server may become unbootable. It also happens a database server does boot but fails to load the right modules or libraries after a broken or interrupted update, effectively making the system unusable. As a result it can happens users and administrators are unable to login to the system, even via the console. This document describes how to recover from a failing update or rollback on an Exadata DB server. This document only applies to X2 and later DB servers where logical volume management (lvm) is in use and where a backup was made to the inactive system partition prior the update.
If the database server was configured with logical volume management and a backup was made prior to the update, there is a way to 'rollback' to that backup, by 'switching' system partitions. When switching system partitions, the active logical volume becomes inactive and the inactive (the backup) becomes active. In addition to the switching of active and inactive system partitions, the /boot directory belonging to the system partition being made the new active system partition needs to be restored. Also the Grub bootloader needs to be re-installed (for Grub1 only).
On systems where administrators can login and run dbnodeupdate.sh, rollbacks should always be performed by dbnodeupdate.sh On system where you cannot login, the process of switching system partitions, restoring /boot and bootloader reinstallation can only be done when booting from the diagnostic iso first. This note will guide the reader through the process of booting from diagnostic iso and restoring the relevant components.
Situations where the steps in this document can be used for:
A failing update to Exadata 184.108.40.206.0 (Oracle Linux 6 or Linux 7)
Note: Failing updates to Exadata 220.127.116.11.0 (Oracle Linux 6) or Oracle Linux 7 should in most cases rollback automatically.
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!