ORA-3135 or ORA-3113 Connecting From App Servers To Database (Caused By IPS Filter) (Doc ID 1091364.1)

Suddenly unable to connect to multiple 10g R2 Database servers on particular host from 2 PeopleSoft Application Servers.

Errors reported:

ERROR:
ORA-03135: connection lost contact

or

ORA-03113: end-of-file on communications channel

Problem reproduces consistently using Sqlplus from these 2 App Servers. However, connections from other remote clients to the same database servers are working fine.

Matching Oracle Net Client & Server Traces show the client (App Server) sends an 1144 byte packet (approx) to TCP/IP, but this 'NSPTDA packet' never is seen arriving in the Oracle Net Server trace on the database side. Connection then terminates with the error stack below once the Inbound Connect Timeout (IBCT) expires, (which default is 60 seconds in 10.2 DB and higher):

Excerpt from Oracle Net Client trace showing the 60 second wait after sending a NSPTDA packet and before the connection terminates:

=============
[28-APR-2010 12:39:08:133] nsdo: switching to application buffer
[28-APR-2010 12:39:08:133] nsrdr: entry
[28-APR-2010 12:39:08:133] nsrdr: recving a packet
[28-APR-2010 12:39:08:134] nsprecv: entry
[28-APR-2010 12:39:08:134] nsprecv: reading from transport...
[28-APR-2010 12:39:08:134] nttrd: entry
[28-APR-2010 12:40:08:064] ntt2err: entry
[28-APR-2010 12:40:08:065] ntt2err: soc 9 error - operation=5, ntresnt[0]=517, ntresnt[1]=131, ntresnt[2]=0
[28-APR-2010 12:40:08:065] ntt2err: exit
[28-APR-2010 12:40:08:065] nttrd: exit
[28-APR-2010 12:40:08:065] nsprecv: error exit
[28-APR-2010 12:40:08:065] nserror: entry
[28-APR-2010 12:40:08:065] nserror: nsres: id=0, op=68, ns=12547, ns2=12560; nt[0]=517, nt[1]=131, nt[2]=0; ora[0]=0, ora[1]=0, ora[2]=0
[28-APR-2010 12:40:08:065] nsrdr: error exit
[28-APR-2010 12:40:08:065] nsdo: nsctxrnk=0
[28-APR-2010 12:40:08:065] nsdo: error exit
[28-APR-2010 12:40:08:065] nioqrc: wanted 1 got 0, type 0
[28-APR-2010 12:40:08:065] nioqper: error from nioqrc
[28-APR-2010 12:40:08:066] nioqper: ns main err code: 12547
[28-APR-2010 12:40:08:066] nioqper: ns (2) err code: 12560
[28-APR-2010 12:40:08:066] nioqper: nt main err code: 517
[28-APR-2010 12:40:08:066] nioqper: nt (2) err code: 131
[28-APR-2010 12:40:08:066] nioqper: nt OS err code: 0
[28-APR-2010 12:40:08:066] nioqer: entry
[28-APR-2010 12:40:08:066] nioqer: incoming err = 12151
[28-APR-2010 12:40:08:066] nioqce: entry
[28-APR-2010 12:40:08:066] nioqce: exit
[28-APR-2010 12:40:08:066] nioqer: returning err = 3135
[28-APR-2010 12:40:08:066] nioqer: exit
[28-APR-2010 12:40:08:066] nioqrc: exit
[28-APR-2010 12:40:08:066] nioqds: entry
[28-APR-2010 12:40:08:067] nioqds: disconnecting...
[28-APR-2010 12:40:08:067] nsclose: entry
=============

Changes

Recent security scan for Intrusion Prevention System (IPS) added an Oracle Net Authentication filter to an intermediate IPS device blocking the Oracle Net communications from the App Servers to the DB server.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.