My Oracle Support Banner

EMCLI Commands Are Starting To Hang. OMS Restart is Required (Doc ID 2643354.1)

Last updated on AUGUST 10, 2020

Applies to:

Enterprise Manager Base Platform - Version 13.2.0.0.0 and later
Information in this document applies to any platform.

Symptoms

EMCLI is hanging after some time. Upon checking, there are a lot of inactive sessions left in the OMS from the emcli sessions. After restarting the OMS, the sessions are cleared, but after intensively using emcli, the sessions are again hanging.

Taking a java thread dump on the OMS process (jstack <OMS PID>) is showing a lot of blocked threads similar to:

"emcli-work-manager:emcli-request" daemon prio=10 tid=0x00007f0c04b04800 nid=0x2f2b runnable [0x00007f0be2dea000]
  java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:153)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
- locked (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:903)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
- locked (a sun.security.ssl.AppInputStream)
at HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:206)
at HTTPClient.BufferedInputStream.read(BufferedInputStream.java:126)
at HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:361)
- locked (a HTTPClient.StreamDemultiplexor)
at HTTPClient.RespInputStream.read(RespInputStream.java:151)
- locked (a HTTPClient.StreamDemultiplexor)
at HTTPClient.RespInputStream.read(RespInputStream.java:112)
at HTTPClient.Response.readResponseHeaders(Response.java:1169)
at HTTPClient.Response.getHeaders(Response.java:892)
- locked (a HTTPClient.StreamDemultiplexor)
at HTTPClient.Response.getStatusCode(Response.java:341)
at HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:94)
at HTTPClient.HTTPResponse.handleResponseImpl(HTTPResponse.java:872)
at HTTPClient.HTTPResponse.access$000(HTTPResponse.java:62)
at HTTPClient.HTTPResponse$2.run(HTTPResponse.java:839)
at HTTPClient.HTTPResponse$2.run(HTTPResponse.java:837)
at HTTPClient.HttpClientConfiguration.doAction(HttpClientConfiguration.java:1083)
at HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:837)
- locked (a HTTPClient.HTTPResponse)
at HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:242)
at oracle.sysman.gcagent.comm.http.ClientConnection$Response.getStatusCode(ClientConnection.java:168)
at oracle.sysman.gcagent.comm.oms.http.HTTPClientTerminus.receiveReply(HTTPClientTerminus.java:924)
at oracle.sysman.gcagent.comm.oms.http.HTTPClientTerminus.getReplyFromMessage(HTTPClientTerminus.java:671)
at oracle.sysman.gcagent.oms.TMRemoteClient.makeSingleRequest(TMRemoteClient.java:6485)
at oracle.sysman.gcagent.oms.TMRemoteClient.makeSingleRequest(TMRemoteClient.java:6418)
at oracle.sysman.gcagent.oms.TMRemoteClient.performOperationInternal(TMRemoteClient.java:5878)
at oracle.sysman.gcagent.oms.TMRemoteClient.performOperation(TMRemoteClient.java:3039)
at oracle.sysman.gcagent.oms.TMRemoteClient.performOperation(TMRemoteClient.java:3011)
at oracle.sysman.emSDK.agent.client.TMClientFactory$Wrapper.performOperation(TMClientFactory.java:3953)
at oracle.sysman.emSDK.emd.comm.EMDClient$Translator.remoteOperation(EMDClient.java:11522)
at oracle.sysman.emSDK.core.sec.credstore.RemoteCtrl.remoteOperation(RemoteCtrl.java:162)
at oracle.sysman.emSDK.core.sec.credstore.RemoteCtrl.remoteOperation(RemoteCtrl.java:135)
at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:164)
at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:244)
at oracle.sysman.eml.creds.emcli.CredUtil.testCredential(CredUtil.java:137)
at oracle.sysman.eml.creds.emcli.CredUtil.testCredential(CredUtil.java:121)
at oracle.sysman.eml.creds.emcli.CredUtil.testCredential(CredUtil.java:100)
at oracle.sysman.eml.creds.emcli.TestCredentialVerb.execute(TestCredentialVerb.java:267)
at oracle.sysman.emSDK.emCLI.CLIPageHandler.executeCommand(CLIPageHandler.java:2152)
at oracle.sysman.emSDK.emCLI.CLIPageHandler.access$2000(CLIPageHandler.java:168)
at oracle.sysman.emSDK.emCLI.CLIPageHandler$CLIExecutor.executeRequest(CLIPageHandler.java:1461)
at oracle.sysman.emSDK.emCLI.CLIPageHandler$CLIExecutor.doWork(CLIPageHandler.java:1438)
at oracle.sysman.core.common.workmanager.Work.call(Work.java:274)
at oracle.sysman.core.common.workmanager.Work.call(Work.java:50)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:745)

All the java calls are coming from the same method:

at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:164)
at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:244)

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.