[ PCA 2.3.1+] Procedure and Scripts to Re-install OVM Manager and Re-build OVM Manager Database On a PCA System
(Doc ID 2657826.1)
Last updated on JANUARY 02, 2024
Applies to:
Private Cloud Appliance X5-2 Hardware - Version All Versions and later Private Cloud Appliance - Version 2.3.1 and later Private Cloud Appliance X8-2 - Version All Versions and later Linux x86-64
Goal
This document provides the procedures required to rebuild the OVM Manager database on a PCA system.
Private Cloud Applicance includes the Oracle VM (OVM) product. The Oracle VM Manager core application is a Java-based Oracle WebLogic application. The Oracle VM Manager database stores configuration information and event data. The actual database management server is a MySQL Enterprise database that is bundled with the Oracle VM Manager. Objects are managed using the Oracle Distributed Object Fabric (ODOF) layer.
Some circumstances can cause a logical disconnection between ODOF and the MySQL database resulting in loss of service to some or all OVM Manager functions. Extensive root cause analysis of such errors has identified different causes, including but not limited to:
OVM manager server crash (e.g. MySQL process unable to complete a write operation)
Using third-party software to access the database.
Manual manipulation of data directory objects.
Execution of commands directly on Compute Nodes (e.g. using "xm") which do not get reflected to the OVM Manager.
Loss of communication between the ovs-agent on a Compute Node and the OVM Manager, e.g. if the OVM Manager has an outage and tasks are carried out on a Compute Node
Network issues between compute nodes
Use of development-only tools, even under supervision of Oracle personnel, to attempt temporary database corrections to allow operations to continue. For example, loss of a storage device / filesystem can result in being unable to delete references to objects it contained using the OVM Manager UI or CLI.
Examples of symptoms of a logical corruption of an OVM Manager database are, but not limited to:
"ObjectNotFound" exceptions in the OVM Manager AdminServer.log
'<BEA-000627> <Reached maximum capacity of pool "ovm-odof-ds"' exceptions in the OVM Manager AdminServer.log
Automatic backups of the MySQL database not occurring
such backups are written to /u01/app/oracle/mysql/dbbackup on the active OVM Management node
The OVM Manager AdminServer.log (located in /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/AdminServer/logs/) records successful and unsuccessful backups.
e.g. ####<2021-05-07T10:54:33.993+0000> <Info> <com.oracle.ovm.mgr.api.util.exporter.ModelDataExporter> <ovcamn05r1> <AdminServer> <Odof Tcp Client Thread: /127.0.0.1:54321/1207> <> <> <8067b1eb-6d22-4df6-be76-095e342cb9df-00000004> <1620384873993> <BEA-000000> <Exporting OVM model data to file: /u01/app/oracle/mysql/dbbackup/OVMModelExport-20210507_105433.xml>
if the automated database backup procedure detects the database is corrupt it will not back it and record the error in the AdminServer.log.
e.g. "<BEA-000000> <Backup task: skipping backup- inconsistencies have been detected in the database>"
The Solution section below documents the required steps to correct the issue. The entire procedure should be reviewed before commencing.
The length of time it will take depends on the size of the installation - e.g. number of compute nodes, number of storage servers and number of their underlying resources.
The procedure only affects the OVM Manager. provided the PCA system is at PCA 2.3.4 or later. Compute nodes and the virtual machines they run on PCA 2.3.4 or later systems are not impacted. However, they cannot be managed using the OVM Manager until the step "Restore the Simple Names for PCA and Oracle VM" is complete.
Note that PCA systems at PCA 2.3.3 or earlier should take heed of the Note box in section "8g) Refresh the physical disks". This affects systems that have had Fibre-Channel storage systems added.
This document applies only to PCA 2.3.1 and later, including all patch levels. For earlier releases refer to "[ PCA <= 2.2.2 ] How to Rebuild an Oracle VM Manager Database ( including User-Friendly Names) On a Private Cloud Appliance (Doc ID 2224820.1)"
Solution
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!
