EM 13c: Enterprise Manager 13.2 Cloud Control system Generates Numerous Incidents: java.lang.NullPointerException:oracle.sysman.gcagent.state.StateMgr:1096 (Doc ID 2226172.1)

Last updated on JUNE 16, 2017

Applies to:

Enterprise Manager Base Platform - Version 13.2.0.0.0 and later
Enterprise Manager for Oracle Database - Version 13.2.1.0.0 and later
Information in this document applies to any platform.

Symptoms

An upgraded Enterprise Manager (EM) 13.2 Cloud Control system generates numerous incidents from the EM 13.2 Agents with this error message:

java.lang.NullPointerException:oracle.sysman.gcagent.state.StateMgr:1096


The incident readme.txt file has this typical error stack:

Context Values
--------------
TARGET_ID : has.has_myrac1
threadName : GC.Executor.113 (has:has_myrac1:CRSAlert11gR2_dup)
METRIC_NAME : clusterware_alerts

Because of the multitude of the incidents, the EM agent may also experience performance issues where tasks are being executed more slowly and a backlog may appear:

emagent@myrac1:~$ emctl status agent scheduler -summary
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Number of currently running scheduled items :: 24
Number of currently ready scheduled items :: 15
Number of scheduled items :: 482
Schedule Attempts Delay Percentage :: 6.452%

 

 

From <agent base directory>/agent_inst/sysman/log/gcagent.log:

2016-11-03 11:08:13,775 [26283:GC.Executor.163 (has:has_exapdb05.akbank.com.tr:CRSAlert11gR2_dup)] DEBUG - Executing cleanup action End session [145782] for task has:has_<target_hostname>:CRSAlert11gR2_dup
2016-11-03 11:08:13,775 [28:IncCreHandler] DEBUG - Adding diag fact [INVOKER_STACK] to thread 28: java.lang.Exception: Critical error
at oracle.sysman.gcagent.tmmain.diag.DiagManager.handleException(DiagManager.java:1019)
at oracle.sysman.gcagent.tmmain.diag.DiagManager.handleException(DiagManager.java:935)
at oracle.sysman.gcagent.tmmain.diag.DiagManager.handleException(DiagManager.java:707)
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:736)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)

 

From <agent base directory>/agent_inst/sysman/log/emagent.nohup file

Nov 03, 2016 11:08:14 AM oracle.sysman.diag.EMDiagImpl createIncident.648
INFO: incident 565 created with problem key java.lang.NullPointerException:oracle.sysman.gcagent.state.StateMgr:1096, in directory <Agent_Inst_Home>/diag/ofm/emagent/emagent/incident/incdir_565
Nov 03, 2016 11:38:13 AM oracle.sysman.diag.EMDiagImpl captureDiagData.478
SEVERE: Critical error: <no message>
java.lang.NullPointerException

 

 

Changes

Upgraded to 13.2 Cloud Control.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms