OFSAA Services not responding to requests
(Doc ID 2513529.1)
Last updated on OCTOBER 18, 2019
Applies to:
Oracle Financial Services Analytical Applications Infrastructure - Version 7.3.5.1.0 to 8.0.7 [Release 7 to 8]Information in this document applies to any platform.
Symptoms
On OFSAAI 7.x, 8.x, Batch Execution requests are hung. ICC, AM, OFSAAI services are not responding to execution requests with below errors when all the processes are running as per the "PS" command output.
=========================
ICC server log has below Socket errors:
=========================
[17:29:32,761] - [DEBUG LOG]Reading from native socket.
[17:29:32,761] - [DEBUG LOG]Header Length= 8
[17:29:32,761] - [DEBUG LOG]hdrBytes = 'Z####JUNK####CHAR####'
[17:29:32,761] - Error writing to client.
java.net.SocketException: Socket is closed
at java.net.Socket.shutdownOutput(Socket.java:1407)
at com.iflex.fic.icc.iccal.iccserver.server.natives.NativeSocket.write(NativeSocket.java:206)
at com.iflex.fic.icc.iccal.iccserver.server.natives.NativeSocket.write(NativeSocket.java:167)
at com.iflex.fic.icc.iccal.iccserver.services.NativeServiceProvider.run(NativeServiceProvider.java:349)
[17:29:32,761] - Error writing to client.
java.lang.NumberFormatException: For input string: "Z####JUNK####CHAR####'
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:449)
at java.lang.Integer.parseInt(Integer.java:499)
at com.iflex.fic.icc.iccal.iccserver.server.natives.NativeSocket.read(NativeSocket.java:109)
at com.iflex.fic.icc.iccal.iccserver.services.NativeServiceProvider.<init>(NativeServiceProvider.java:242)
at com.iflex.fic.icc.iccal.iccserver.server.NativeServer.startServer(NativeServer.java:130)
at com.iflex.fic.icc.iccal.iccserver.server.NativeServer.run(NativeServer.java:85)
[17:29:32,762] - Unable to service request. Illegal request string: null
java.lang.NullPointerException
at com.iflex.fic.icc.iccal.iccserver.services.NativeServiceProvider.run(NativeServiceProvider.java:269)
[17:29:32,762] - Writing to native socket: ##ICC##null##-1##-3##Illegal request string.## with padding
[17:29:32,762] - Error writing to client.
java.net.SocketException: Socket is closed
at java.net.Socket.shutdownOutput(Socket.java:1407)
at com.iflex.fic.icc.iccal.iccserver.server.natives.NativeSocket.write(NativeSocket.java:206)
at com.iflex.fic.icc.iccal.iccserver.server.natives.NativeSocket.write(NativeSocket.java:167)
at com.iflex.fic.icc.iccal.iccserver.services.NativeServiceProvider.run(NativeServiceProvider.java:349)
[17:29:32,763] - Connection from d131035a002.svr.emea.jpmchase.net
[17:29:32,763] - Creating new NativeSocket.
=========================
RevAppServer.log has below errors:
=========================
[REVELEUSLOG] May 19,16 03:30:38 : [JAVASOCKETCONNECTION]Unable to setup I/O Streamsjava.io.StreamCorruptedException: invalid stream header: 04010000
[REVELEUSLOG] May 19,16 03:30:38 : [JAVASOCKETCONNECTION]
java.io.StreamCorruptedException: invalid stream header: 04010000
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279)
at com.iflex.fic.ficserver.JavaSocketConnection.initialize(JavaSocketConnection.java:129)
at com.iflex.fic.ficserver.ServerConnectionManager.processJavaConnection(ServerConnectionManager.java:339)
at com.iflex.fic.ficserver.ServerConnectionManager.run(ServerConnectionManager.java:245)
=========================
am_log_file.log has below Socket errors:
=========================
Log Message here 1: Thu May 19 02:18:38 2016||SEVERE||SocketServerImpl::Get_Error : ERRNO = 107 Error Message = Transport endpoint is not connected
Log Message here 1: Thu May 19 02:18:38 2016||SEVERE||SocketServerImpl::Get_Error : Default case
Log Message here 1: Thu May 19 02:19:07 2016||WARN||SocketServerImpl::Read : Socket Read Failed
Log Message here 1: Thu May 19 02:19:07 2016||WARN||Thread 1 : Read from socket failed
Log Message here 1: Thu May 19 02:19:07 2016||WARN||SocketServerImpl::Close_Conn : Call to Shutdown-RDWR Failed
Log Message here 1: Thu May 19 02:19:07 2016||SEVERE||SocketServerImpl::Get_Error : ERRNO = 107 Error Message = Transport endpoint is not connected
Log Message here 1: Thu May 19 02:19:07 2016||SEVERE||SocketServerImpl::Get_Error : Default case
Log Message here 1: Thu May 19 02:19:30 2016||WARN||SocketServerImpl::Read : Socket Read Failed
Log Message here 1: Thu May 19 02:19:30 2016||WARN||Thread 1 : Read from socket failed
Log Message here 1: Thu May 19 02:19:30 2016||WARN||SocketServerImpl::Close_Conn : Call to Shutdown-RDWR Failed
Changes
Port Scanning/Monitoring tool was deployed to monitor the Ports configured in OFSAAI
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |