EM 12c,13c: Plugin Upgrade On OMS Fails With Error 'java.sql.SQLRecoverableException: IO Error: Connection reset'

(Doc ID 2318012.1)

Last updated on OCTOBER 24, 2017

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

Upgrading the database plug-in to 13.2.2.0.0 on the Enterprise Manager (EM) 13.2 Cloud Control OMS fails with following hang indication during the 'create or replace view' step:

Destination          : Management Server - <host name>:4889_Management_Service
Plug-in Name         : Oracle Database
Version              : 13.2.2.0.0
ID                   : oracle.sysman.db
Content              : Plug-in
Action               : Deployment
Status               : Deploying
Steps Info:

From <OMS_HOME>/cfgtoollogs/pluginca/configplugin_all_2017_0927_15_30-25.log:

2017-09-27 15:42:49,288 [main] WARN pluginca.PluginCADriver - Error while committing conn java.sql.SQLRecoverableException: Closed Connection

2017-09-27 15:42:49.284 [ PluginCAUtil - Emergency log ]
oracle.sysman.core.common.extensibility.pluginca.PluginCAException: EM-04026: Server-side plug-in deployment failed. For details, check the log file /u01/app/oracle/middleware/cfgtoollogs/pluginca/plugin_deployment_137/configplugin_all_2017-09-27_15-30-25.log on the host omsnode1-pp.crb.apmoller.net.

Caused by: oracle.sysman.core.common.extensibility.pluginca.PluginCAException: EM-04016: Server-side plug-in deployment failed. Error in schema configuration: ORA-17002: IO Error: Connection resetFile:/u01/app/oracle/middleware/plugins/oracle.sysman.db.oms.plugin_13.2.2.0.0/sql/db/latest/rac/rac_views.sql

2017-09-27 15:42:49.280 [ PluginCAUtil - Emergency log ]
oracle.sysman.core.common.extensibility.pluginca.PluginCAException: EM-04001: Server-side plug-in deployment failed with the message Error while restoring the system to previous state: EM-04001: Server-side plug-in deployment failed with the message Error while retrieving back up information: Closed Connection. Contact Oracle Support.. Contact Oracle Support.

 

From <OMS_HOME>/sysman/log/schemamanager/m_<Timestamp>/rcu.log:

Wed Sep 27 15:42:06.801 UTC 2017 ERROR rcu: oracle.sysman.assistants.rcu.backend.action.AbstractAction::handleNonIgnorableError: Received Non-Ignorable Error: ORA-17002: IO Error: Connection resetFile:/u01/app/oracle/middleware/plugins/oracle.sysman.db.oms.plugin_13.2.2.0.0/sql/db/latest/rac/rac_views.sql
Statement:CREATE OR REPLACE VIEW MGMT$RAC_MEMBERSHIPS
AS
SELECT ct.target_name COMPOSITE_TARGET_NAME,
ct.target_type COMPOSITE_TARGET_TYPE,
ct.target_guid COMPOSITE_TARGET_GUID,
mt.target_name MEMBER_TARGET_NAME,
mt.target_type MEMBER_TARGET_TYPE,
mt.target_guid MEMBER_TARGET_GUID,
' ' ASSOCIATION
FROM gc_assoc_instances assoc,
mgmt_targets ct,
mgmt_targets mt
WHERE assoc.assoc_type = 'cluster_contains'
AND assoc.source_me_guid = ct.target_guid
AND ct.target_type = 'cluster'
AND assoc.dest_me_guid = mt.target_guid
AND mt.target_type = 'host'
UNION
SELECT ct.target_name COMPOSITE_TARGET_NAME,
ct.target_type COMPOSITE_TARGET_TYPE,
ct.target_guid COMPOSITE_TARGET_GUID,
mt.target_name MEMBER_TARGET_NAME,
mt.target_type MEMBER_TARGET_TYPE,
mt.target_guid MEMBER_TARGET_GUID,
' ' ASSOCIATION
FROM gc_assoc_instances assoc,
mgmt_targets ct,
mgmt_targets mt
WHERE assoc.assoc_type = 'cluster_contains'
AND assoc.source_me_guid = ct.target_guid
AND ct.target_type = 'rac_database'
AND assoc.dest_me_guid = mt.target_guid
AND mt.target_type = 'oracle_database'
UNION
SELECT ct.target_name COMPOSITE_TARGET_NAME,
ct.target_type COMPOSITE_TARGET_TYPE,
ct.target_guid COMPOSITE_TARGET_GUID,
mt.target_name MEMBER_TARGET_NAME,
mt.target_type MEMBER_TARGET_TYPE,
mt.target_guid MEMBER_TARGET_GUID,
' ' ASSOCIATION
FROM gc_assoc_instances assoc,
mgmt_targets ct,
mgmt_targets mt
WHERE assoc.assoc_type = 'cluster_contains'
AND assoc.source_me_guid = ct.target_guid
AND ct.target_type = 'osm_cluster'
AND assoc.dest_me_guid = mt.target_guid
AND mt.target_type = 'osm_instance'

Wed Sep 27 15:42:06.801 UTC 2017 TRACE rcu: oracle.sysman.assistants.rcu.backend.action.AbstractAction::handleNonIgnorableError: Delegating to caller ErrorListener
Wed Sep 27 15:42:06.802 UTC 2017 ERROR rcu: oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine::onException: SQLException: IO Error: Connection reset
java.sql.SQLRecoverableException: IO Error: Connection reset

 

 

From the repository database trace file:

2017-10-03 08:57:43.378453 : ntt2err:entry
2017-10-03 08:57:43.378593 : ntt2err:soc 18 error - operation=5, ntresnt[0]=517, ntresnt[1]=104, ntresnt[2]=0
2017-10-03 08:57:43.378608 : ntt2err:exit
2017-10-03 08:57:43.378623 : nserror:entry
2017-10-03 08:57:43.378636 : nserror:nsres: id=0, op=68, ns=12547, ns2=12560; nt[0]=517, nt[1]=104, nt[2]=0; ora[0]=0, ora[1]=0, ora[2]=0
2017-10-03 08:57:43.378655 : nioqper: error from nioqrc
2017-10-03 08:57:43.378666 : nioqper: ns main err code: 12547
2017-10-03 08:57:43.378691 : nioqper: ns (2) err code: 12560
2017-10-03 08:57:43.378703 : nioqper: nt main err code: 517
2017-10-03 08:57:43.378713 : nioqper: nt (2) err code: 104
2017-10-03 08:57:43.378723 : nioqper: nt OS err code: 0
2017-10-03 08:57:43.378740 : nioqer:entry
2017-10-03 08:57:43.378753 : nioqer: incoming err = 12151
2017-10-03 08:57:43.378763 : nioqce:entry
2017-10-03 08:57:43.378774 : nioqce:exit
2017-10-03 08:57:43.378784 : nioqer: returning err = 3135
2017-10-03 08:57:43.378794 : nioqer:exit
2017-10-03 08:57:43.378851 : nioqds:entry
2017-10-03 08:57:43.378868 : nioqds: disconnecting...

Problem seems to be related to either a FIREWALL setting (being too aggressive for certain packets such as "Authentication" packets) or a Switch / setting that may be disallowing certain SIZES of packets through.

 

 

Note:

This issue can happen for any plugin upgrade. 

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms