EM12c: Cloud Control Agent causing hangs for different FMW products ( OPMN / Forms / OSB / SOA ) on Windows/Unix
(Doc ID 1981009.1)
Last updated on SEPTEMBER 10, 2021
Applies to:
Enterprise Manager for Fusion Middleware - Version 12.1.0.3.0 to 12.1.0.7.0Information in this document applies to any platform.
Symptoms
Cloud Control 12c agent is hanging the Oracle Process Manager OPMN process .The hung effected immediate, approximately after 48 hours running.
Oracle Process Manager service shows running, command line opmnctl status return Not running. Currently we kill opmn process tree using proexp and reboot server.
By looking at the different logs for the hang duration we see following messages:
gcagent.log
=========
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.(Socket.java:375)
at java.net.Socket.(Socket.java:218)
at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:212)
at HTTPClient.HTTPConnection$EstablishConnection$2.run(HTTPConnection.java:4427)
at HTTPClient.HttpClientConfiguration.doAction(HttpClientConfiguration.java:771)
at HTTPClient.HTTPConnection.doAction(HTTPConnection.java:5538)
at HTTPClient.HTTPConnection$EstablishConnection.run(HTTPConnection.java:4413)
2015-01-27 13:36:42,707 [57:E4B433C6:GC.SysExecutor.0 (AgentSystemMonitorTask)] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Warning* +}
2015-01-27 13:36:48,838 [83:980F9148] INFO - attempting another heartbeat
2015-01-27 13:36:48,854 [83:980F9148] WARN - Ping communication error
o.s.emSDK.agent.comm.exception.ConnectException [Failure connecting to https://<HOST>:<PORT>/empbs/upload , err No buffer space available (maximum connections reached?): connect]
java.net.SocketException [No buffer space available (maximum connections reached?): connect]
2015-01-27 13:37:18,868 [80:905DEBE9] INFO - attempting another heartbeat
2015-01-27 13:37:18,884 [80:905DEBE9] WARN - Ping communication error
o.s.emSDK.agent.comm.exception.ConnectException [Failure connecting to https://<HOSTNAME>.<DOMAIN_NAME>:4903/empbs/upload , err No buffer space available (maximum connections reached?): connect]
java.net.SocketException [No buffer space available (maximum connections reached?): connect]
2015-01-27 13:37:30,834 [73:C09F1BB3] INFO - attempting another heartbeat
2015-01-27 13:37:42,705 [157:F5581FDD] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Warning* +}
2015-01-27 13:37:48,899 [149:BE87CC23] INFO - attempting another heartbeat
2015-01-27 13:37:49,273 [73:C09F1BB3] WARN - Communication error with repository URL = https://<HOST>:<PORT>/empbs/upload
=========
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.(Socket.java:375)
at java.net.Socket.(Socket.java:218)
at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:212)
at HTTPClient.HTTPConnection$EstablishConnection$2.run(HTTPConnection.java:4427)
at HTTPClient.HttpClientConfiguration.doAction(HttpClientConfiguration.java:771)
at HTTPClient.HTTPConnection.doAction(HTTPConnection.java:5538)
at HTTPClient.HTTPConnection$EstablishConnection.run(HTTPConnection.java:4413)
2015-01-27 13:36:42,707 [57:E4B433C6:GC.SysExecutor.0 (AgentSystemMonitorTask)] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Warning* +}
2015-01-27 13:36:48,838 [83:980F9148] INFO - attempting another heartbeat
2015-01-27 13:36:48,854 [83:980F9148] WARN - Ping communication error
o.s.emSDK.agent.comm.exception.ConnectException [Failure connecting to https://<HOST>:<PORT>/empbs/upload , err No buffer space available (maximum connections reached?): connect]
java.net.SocketException [No buffer space available (maximum connections reached?): connect]
2015-01-27 13:37:18,868 [80:905DEBE9] INFO - attempting another heartbeat
2015-01-27 13:37:18,884 [80:905DEBE9] WARN - Ping communication error
o.s.emSDK.agent.comm.exception.ConnectException [Failure connecting to https://<HOSTNAME>.<DOMAIN_NAME>:4903/empbs/upload , err No buffer space available (maximum connections reached?): connect]
java.net.SocketException [No buffer space available (maximum connections reached?): connect]
2015-01-27 13:37:30,834 [73:C09F1BB3] INFO - attempting another heartbeat
2015-01-27 13:37:42,705 [157:F5581FDD] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Warning* +}
2015-01-27 13:37:48,899 [149:BE87CC23] INFO - attempting another heartbeat
2015-01-27 13:37:49,273 [73:C09F1BB3] WARN - Communication error with repository URL = https://<HOST>:<PORT>/empbs/upload
opmn.log
=======
[2015-01-27T13:58:17-06:00] [opmn] [WARNING:1] [660] [pm-process] Process Crashed: emagent_asinst_1~EMAGENT~EMAGENT~1 (1679495578:2508) - Restarting
[2015-01-27T13:58:17-06:00] [opmn] [NOTIFICATION:1] [662] [pm-process] Starting Process: emagent_asinst_1~EMAGENT~EMAGENT~1 (1679495637:0)
[2015-01-27T13:58:30-06:00] [opmn] [NOTIFICATION:1] [662] [pm-process] Starting Process: ohs1~OHS~OHS~1 (1679495636:0)
[2015-01-27T13:58:30-06:00] [opmn] [NOTIFICATION:1] [662] [pm-process] Starting Process: emagent_asinst_1~EMAGENT~EMAGENT~1 (1679495637:0)
[2015-01-27T13:58:31-06:00] [opmn] [NOTIFICATION:1] [676] [pm-internal] OPMN server stopped. Request handling disabled
[2015-01-27T13:58:31-06:00] [opmn] [NOTIFICATION:1] [667] [pm-requests] Request 10721 Started. Command: /shutdown
=======
[2015-01-27T13:58:17-06:00] [opmn] [WARNING:1] [660] [pm-process] Process Crashed: emagent_asinst_1~EMAGENT~EMAGENT~1 (1679495578:2508) - Restarting
[2015-01-27T13:58:17-06:00] [opmn] [NOTIFICATION:1] [662] [pm-process] Starting Process: emagent_asinst_1~EMAGENT~EMAGENT~1 (1679495637:0)
[2015-01-27T13:58:30-06:00] [opmn] [NOTIFICATION:1] [662] [pm-process] Starting Process: ohs1~OHS~OHS~1 (1679495636:0)
[2015-01-27T13:58:30-06:00] [opmn] [NOTIFICATION:1] [662] [pm-process] Starting Process: emagent_asinst_1~EMAGENT~EMAGENT~1 (1679495637:0)
[2015-01-27T13:58:31-06:00] [opmn] [NOTIFICATION:1] [676] [pm-internal] OPMN server stopped. Request handling disabled
[2015-01-27T13:58:31-06:00] [opmn] [NOTIFICATION:1] [667] [pm-requests] Request 10721 Started. Command: /shutdown
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| Solution for Windows |
| Solution for Unix |