EM 12c: Oracle Site Guard Switchover Operation Plan Failed

Last updated on JULY 17, 2017

Applies to:

Symptoms

Oracle Site Guard switchover fails with this error stack:

Oracle Site Guard Switchover Operation Plan Failed:
 
Run Prechecks -> Run Precheck Databases in Parallel -> mycompany.mydomain.com -> Run Precheck Database
 
with this error

==============================================================================================================
Verifying sysdba credentials
==============================================================================================================
echo '@logon_check_1472157907174_561090.sql' | /u01/app/oracle/product/11.2.0.4/db_1/bin/sqlplus sys/******** as sysdba

oraclekronosp1: symbol lookup error: /u01/app/oracle/product/EMagent/core/12.1.0.4.0/instantclient/libnnz11.so: undefined symbol: nzdacvalue
Use of uninitialized value $value in substitution (s///) at /tmp/JOB_3A84574ADC2727BBE053C412790AA433/preCheckDatabase.pl line 354, line 6.
Use of uninitialized value $value in string eq at /tmp/JOB_3A84574ADC2727BBE053C412790AA433/preCheckDatabase.pl line 361, line 6.

SQL*Plus: Release 11.2.0.4.0 Production on Thu Aug 25 16:45:07 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

ERROR:
ORA-12547: TNS:lost contact


Enter user-name: SP2-0306: Invalid option.
Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM}] [edition=value]]
where ::= [/][@<connect_identifier>]
   ::= [][/][@<connect_identifier>]
SP2-0306: Invalid option.
Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM}] [edition=value]]
where ::= [/][@<connect_identifier>]
   ::= [][/][@<connect_identifier>]
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
 

ERROR >>> ERROR: Could not assert the SYSDBA credentials


The switchover was attempted using Site Guard via the emcli command:

emcli submit_operation_plan -name=${OPERATION_NAME} -database_lag_checks="true" -run_prechecks="true"

The switchover failed in the precheck steps, please reference uploaded job log doc "Failed - JobStepsLog_Run Precheck Database_kronosp_mil.txt" where you can see that the precheck attempted to execute dgmgrl and sqlplus commands and received the following error:

oraclekronosp1: symbol lookup error: /u01/app/oracle/product/EMagent/core/12.1.0.4.0/instantclient/libnnz11.so: undefined symbol: nzdacvalue

This same switchover operation (using Site Guard / emcli) has been performed on many other databases successfully, this is the first one where we have encountered this error. We have since had it occur on another database and have had success with others.

BTW, the path /u01/app/oracle/product/EMagent/core/12.1.0.4.0/instantclient is not the path to the database's ORACLE_HOME.

Changes

Cause