My Oracle Support Banner

OUD 12c EM Grid Control Plugin 13.4.1 Cloud Control EMCC Agent "Too many open files" (Doc ID 2777553.1)

Last updated on MAY 24, 2021

Applies to:

Oracle Unified Directory - Version 12.2.1.4.0 and later
Enterprise Manager for Fusion Middleware - Version 13.4.1.0.0 and later
Information in this document applies to any platform.

Symptoms

Oracle Enterprise Manager (OEM / EM) 13.4 with EM Grid Control Plugin 13.4.1.0.0 for Oracle Unified Directory (OUD) 12c.

EM Cloud Control (EMCC / EM CC) agent (ccagent) runs out of open files.

gcagent.log shows:

...
2021-04-18 16:56:35,056 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Agent Key Performance Indicators
2021-04-18 16:56:35,056 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (2) on Target Interaction Manager
2021-04-18 16:56:35,057 [1:835979FD] WARN - Failed to persist mce entries
java.io.FileNotFoundException: <agent_instance>/sysman/emd/state/mcePersist/mceEntries (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
at java.io.FileWriter.<init>(FileWriter.java:63)
at oracle.sysman.gcagent.target.interaction.execution.Threshold.persistMCEState(Threshold.java:5357)
at oracle.sysman.gcagent.target.interaction.execution.Threshold.shutdown(Threshold.java:5632)
at oracle.sysman.gcagent.target.interaction.execution.TargetInteractionMgr.tmNotifier(TargetInteractionMgr.java:198)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeNotifier(TMComponentSvc.java:1052)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeCleanupStep(TMComponentSvc.java:1098)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.doCleanupStep(TMComponentSvc.java:952)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.notifierDriver(TMComponentSvc.java:867)
at oracle.sysman.gcagent.tmmain.TMMain.abort(TMMain.java:484)
at oracle.sysman.gcagent.tmmain.TMMain.waitForSignal(TMMain.java:773)
at oracle.sysman.gcagent.tmmain.TMMain.agentMain(TMMain.java:648)
at oracle.sysman.gcagent.tmmain.TMMain.main(TMMain.java:595)
2021-04-18 16:56:35,058 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (2) on Target Interaction Manager
2021-04-18 16:56:35,058 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on Target Interaction Manager
2021-04-18 16:56:35,058 [1:835979FD] WARN - Failed to persist mce entries
java.io.FileNotFoundException: <agent_instance>/sysman/emd/state/mcePersist/mceEntries (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
at java.io.FileWriter.<init>(FileWriter.java:63)
at oracle.sysman.gcagent.target.interaction.execution.Threshold.persistMCEState(Threshold.java:5357)
at oracle.sysman.gcagent.target.interaction.execution.Threshold.shutdown(Threshold.java:5632)
at oracle.sysman.gcagent.target.interaction.execution.TargetInteractionMgr.tmNotifier(TargetInteractionMgr.java:198)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeNotifier(TMComponentSvc.java:1052)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeCleanupStep(TMComponentSvc.java:1098)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.doCleanupStep(TMComponentSvc.java:952)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.notifierDriver(TMComponentSvc.java:867)
at oracle.sysman.gcagent.tmmain.TMMain.abort(TMMain.java:484)
at oracle.sysman.gcagent.tmmain.TMMain.waitForSignal(TMMain.java:773)
at oracle.sysman.gcagent.tmmain.TMMain.agentMain(TMMain.java:648)
at oracle.sysman.gcagent.tmmain.TMMain.main(TMMain.java:595)
2021-04-18 16:56:35,068 [130:A772DC06:GC.SysExecutor.4 (Ping OMS)] INFO - attempting another heartbeat
2021-04-18 16:56:35,068 [130:GC.SysExecutor.4 (Ping OMS)] ERROR - File not found exception
java.io.FileNotFoundException: <agent_instance>/sysman/emd/state/statemgmt/oracle_emd/agent/UPLOADMGR/OLDESTTS.tmp (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at oracle.sysman.gcagent.state.StateMgr.upsertStateEntry(StateMgr.java:1164)
at oracle.sysman.gcagent.state.StateMgr.upsertStateEntry(StateMgr.java:1131)
at oracle.sysman.gcagent.upload.UploadManager.getOldestTSPendingUpload(UploadManager.java:1908)
at oracle.sysman.gcagent.ping.PingManager.pingOMS(PingManager.java:1024)
at oracle.sysman.gcagent.ping.PingManager.pingOms(PingManager.java:878)
at oracle.sysman.gcagent.ping.PingManager$PingTask.call(PingManager.java:510)
at oracle.sysman.gcagent.ping.PingManager$PingTask.call(PingManager.java:468)
at oracle.sysman.gcagent.task.scheduler.DispatchingTaskScheduler$ReschedulingHelper$ReschedulingTask.call(DispatchingTaskScheduler.java:470)
at oracle.sysman.gcagent.task.scheduler.DispatchingTaskScheduler$ReschedulingHelper$ReschedulingTask.call(DispatchingTaskScheduler.java:432)
at oracle.sysman.gcagent.task.executor.DiagWrappedTask.call(DiagWrappedTask.java:60)
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:600)
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:644)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:373)
at oracle.sysman.gcagent.task.TaskFutureImpl.run(TaskFutureImpl.java:330)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at oracle.sysman.gcagent.task.executor.TrackThreadFactory$1.run(TrackThreadFactory.java:54)
at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:198)
at java.lang.Thread.run(Thread.java:748)
2021-04-18 16:56:35,069 [130:A772DC06] WARN - Ping communication error
o.s.emSDK.agent.comm.exception.ConnectException [Failure connecting to https://<hostname>:<port>/empbs/upload , err <hostname>: System error]
                 java.net.UnknownHostException [qa-em.sherwin.com: System error]
2021-04-18 16:56:36,058 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on Target Interaction Manager
2021-04-18 16:56:36,058 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on State Manager
2021-04-18 16:56:36,058 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on State Manager
2021-04-18 16:56:36,058 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Forwarder Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Forwarder Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Persistence Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Persistence Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Target Type Component Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Target Type Component Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT on Agent Up & Down Status
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT on Agent Up & Down Status
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (2) on Target Metadata Loader
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (2) on Target Metadata Loader
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Agent Event Receiver
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Agent Event Receiver
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT on Deadlock Detector
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT on Deadlock Detector
2021-04-18 16:56:36,059 [1:835979FD:main (@ 2021-04-17 20:15:30 EDT)] INFO - Invoking SHUTDOWN_ABORT (0) on Collection Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Collection Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on Metric Extension Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on Metric Extension Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Resource Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Resource Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on Target Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on Target Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT on PlugIn Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT on PlugIn Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Remote Agent Component
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Remote Agent Component
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on Target Metadata Loader
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on Target Metadata Loader
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT on Target Event Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT on Target Event Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on CollectionItemTracker
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on CollectionItemTracker
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (2) on Upload Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (2) on Upload Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on Upload Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on Upload Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT on Health Monitor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT on Health Monitor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on Task Scheduler
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on Task Scheduler
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (5) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (5) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (2) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (2) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (1) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (1) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Task Executor
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on Ping Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on Ping Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT on Registry Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT on Registry Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on State Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Done: SHUTDOWN_ABORT (0) on State Manager
2021-04-18 16:56:36,059 [1:835979FD] INFO - Invoking SHUTDOWN_ABORT (0) on HTTP Listener
2021-04-18 16:56:52,182 [40:E7F13351:HTTP Listener-40-acceptor-0@2421cc4-regularConnector@4e0ae11f{SSL,[ssl, http/1.1]}{0.0.0.0:1830}] WARN - *jetty*:
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:198)
at java.lang.Thread.run(Thread.java:748)
...

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.