EMCLI Commands Are Starting To Hang. OMS Restart is Required (Doc ID 2643354.1)

Last updated on NOVEMBER 16, 2022

Applies to:

Symptoms

EMCLI is hanging after some time. Upon checking, there are a lot of inactive sessions left in the OMS from the emcli sessions. After restarting the OMS, the sessions are cleared, but after intensively using emcli, the sessions are again hanging.

Taking a java thread dump on the OMS process (jstack <OMS PID>) is showing a lot of blocked threads similar to:

"emcli-work-manager:emcli-request" daemon prio=10 tid=0x00007f0c04b04800 nid=0x2f2b runnable [0x00007f0be2dea000]
  java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:153)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
- locked (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:903)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
- locked (a sun.security.ssl.AppInputStream)
at HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:206)
at HTTPClient.BufferedInputStream.read(BufferedInputStream.java:126)
at HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:361)
- locked (a HTTPClient.StreamDemultiplexor)
at HTTPClient.RespInputStream.read(RespInputStream.java:151)
- locked (a HTTPClient.StreamDemultiplexor)
at HTTPClient.RespInputStream.read(RespInputStream.java:112)
at HTTPClient.Response.readResponseHeaders(Response.java:1169)
at HTTPClient.Response.getHeaders(Response.java:892)
- locked (a HTTPClient.StreamDemultiplexor)
at HTTPClient.Response.getStatusCode(Response.java:341)
at HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:94)
at HTTPClient.HTTPResponse.handleResponseImpl(HTTPResponse.java:872)
at HTTPClient.HTTPResponse.access$000(HTTPResponse.java:62)
at HTTPClient.HTTPResponse$2.run(HTTPResponse.java:839)
at HTTPClient.HTTPResponse$2.run(HTTPResponse.java:837)
at HTTPClient.HttpClientConfiguration.doAction(HttpClientConfiguration.java:1083)
at HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:837)
- locked (a HTTPClient.HTTPResponse)
at HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:242)
at oracle.sysman.gcagent.comm.http.ClientConnection$Response.getStatusCode(ClientConnection.java:168)
at oracle.sysman.gcagent.comm.oms.http.HTTPClientTerminus.receiveReply(HTTPClientTerminus.java:924)
at oracle.sysman.gcagent.comm.oms.http.HTTPClientTerminus.getReplyFromMessage(HTTPClientTerminus.java:671)
at oracle.sysman.gcagent.oms.TMRemoteClient.makeSingleRequest(TMRemoteClient.java:6485)
at oracle.sysman.gcagent.oms.TMRemoteClient.makeSingleRequest(TMRemoteClient.java:6418)
at oracle.sysman.gcagent.oms.TMRemoteClient.performOperationInternal(TMRemoteClient.java:5878)
at oracle.sysman.gcagent.oms.TMRemoteClient.performOperation(TMRemoteClient.java:3039)
at oracle.sysman.gcagent.oms.TMRemoteClient.performOperation(TMRemoteClient.java:3011)
at oracle.sysman.emSDK.agent.client.TMClientFactory$Wrapper.performOperation(TMClientFactory.java:3953)
at oracle.sysman.emSDK.emd.comm.EMDClient$Translator.remoteOperation(EMDClient.java:11522)
at oracle.sysman.emSDK.core.sec.credstore.RemoteCtrl.remoteOperation(RemoteCtrl.java:162)
at oracle.sysman.emSDK.core.sec.credstore.RemoteCtrl.remoteOperation(RemoteCtrl.java:135)
at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:164)
at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:244)
at oracle.sysman.eml.creds.emcli.CredUtil.testCredential(CredUtil.java:137)
at oracle.sysman.eml.creds.emcli.CredUtil.testCredential(CredUtil.java:121)
at oracle.sysman.eml.creds.emcli.CredUtil.testCredential(CredUtil.java:100)
at oracle.sysman.eml.creds.emcli.TestCredentialVerb.execute(TestCredentialVerb.java:267)
at oracle.sysman.emSDK.emCLI.CLIPageHandler.executeCommand(CLIPageHandler.java:2152)
at oracle.sysman.emSDK.emCLI.CLIPageHandler.access$2000(CLIPageHandler.java:168)
at oracle.sysman.emSDK.emCLI.CLIPageHandler$CLIExecutor.executeRequest(CLIPageHandler.java:1461)
at oracle.sysman.emSDK.emCLI.CLIPageHandler$CLIExecutor.doWork(CLIPageHandler.java:1438)
at oracle.sysman.core.common.workmanager.Work.call(Work.java:274)
at oracle.sysman.core.common.workmanager.Work.call(Work.java:50)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:745)

All the java calls are coming from the same method:

at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:164)
at oracle.sysman.emSDK.core.sec.credstore.TestHostCredential.testCredential(TestHostCredential.java:244)

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.