WEBLOGIC LEAKS IDLE SOCKETS WHEN AN HTTP REQUEST GETS A NETWORK RESET (Doc ID 2077320.1)

Last updated on NOVEMBER 02, 2017

Applies to:

Oracle WebLogic Server - Version 10.3.6 and later
Information in this document applies to any platform.

Symptoms

The problem doesn't cause a loss of functionality, but you can expect to see memory growth in the WebLogic JVM along with the following symptoms.  A Unix netstat -an command shows leaked idle sockets: 

TCP: IPv4
   Local Address        Remote Address    Swind Send-Q Rwind Recv-Q    State
-------------------- -------------------- ----- ------ ----- ------ -----------

      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE
      *.*                  *.*                0      0 49152      0 IDLE

The WebLogic Admin console shows leaked connections that stay there until the server gets rebooted: 


Channel Statistics

Name                                            Connections     Messages Received     Messages Sent     Bytes Received     Bytes Sent    Accept Count
Default[http://10.23.456.789:10360]    16        189                              172                       172613                  1001432        52

The following exception can be seen in the WebLogic standard output log.  The exception can be frequent, but is intermittent.  The error is related to the HTTP GET call, but it doesn't always happen.

#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <[ServletContext@251449240[app:Service module:/Service/ServiceServer path:/Service/ServiceServer spec-version:2.5]] Root cause of ServletException.
java.io.IOException: Broken pipe
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:268)
at com.sun.xml.ws.transport.http.HttpAdapter.publishWSDL(HttpAdapter.java:610)
at weblogic.wsee.jaxws.WLSServletAdapter.publishWSDL(WLSServletAdapter.java:252)
at weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:166)
at weblogic.wsee.jaxws.HttpServletAdapter.get(HttpServletAdapter.java:193)
at weblogic.wsee.jaxws.JAXWSServlet.doRequest(JAXWSServlet.java:97)
at weblogic.servlet.http.AbstractAsyncServlet.service(AbstractAsyncServlet.java:99)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3750)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: com.ctc.wstx.exc.WstxIOException: Broken pipe
at com.ctc.wstx.sw.BaseNsStreamWriter.doWriteStartTag(BaseNsStreamWriter.java:617)
at com.ctc.wstx.sw.SimpleNsStreamWriter.writeStartOrEmpty(SimpleNsStreamWriter.java:265)
at com.ctc.wstx.sw.BaseNsStreamWriter.writeStartElement(BaseNsStreamWriter.java:313)
at com.sun.xml.ws.util.xml.XMLStreamWriterFilter.writeStartElement(XMLStreamWriterFilter.java:217)
at weblogic.wsee.jaxws.framework.policy.PatchFilter.writeStartElement(WSDLPatchFilter.java:264)
at com.sun.xml.ws.util.xml.XMLStreamReaderToXMLStreamWriter.handleStartElement(XMLStreamReaderToXMLStreamWriter.java:166)
at com.sun.xml.ws.server.WSDLPatcher.handleStartElement(WSDLPatcher.java:205)
at com.sun.xml.ws.util.xml.XMLStreamReaderToXMLStreamWriter.bridge(XMLStreamReaderToXMLStreamWriter.java:105)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:291)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:265)
at com.sun.xml.ws.transport.http.HttpAdapter.publishWSDL(HttpAdapter.java:610)
at weblogic.wsee.jaxws.WLSServletAdapter.publishWSDL(WLSServletAdapter.java:252)
at weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:166)
at weblogic.wsee.jaxws.HttpServletAdapter.get(HttpServletAdapter.java:193)
at weblogic.wsee.jaxws.JAXWSServlet.doRequest(JAXWSServlet.java:97)
at weblogic.servlet.http.AbstractAsyncServlet.service(AbstractAsyncServlet.java:99)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
....
Caused By: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at weblogic.servlet.internal.ChunkOutput.writeChunkTransfer(ChunkOutput.java:568)
at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:539)
at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:427)
at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:298)
at weblogic.servlet.internal.ChunkOutput$2.checkForFlush(ChunkOutput.java:648)
at weblogic.servlet.internal.CharsetChunkOutput.write(CharsetChunkOutput.java:200)
at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:148)
at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:148)
at weblogic.wsee.jaxws.HttpServletAdapter$RequestResponseWrapper$OutputStreamWrapper.write(HttpServletAdapter.java:506)
at com.ctc.wstx.io.UTF8Writer.write(UTF8Writer.java:143)
at com.ctc.wstx.sw.BufferingXmlWriter.flushBuffer(BufferingXmlWriter.java:1358)
at com.ctc.wstx.sw.BufferingXmlWriter.fastWriteRaw(BufferingXmlWriter.java:1404)
at com.ctc.wstx.sw.BufferingXmlWriter.writeStartTagStart(BufferingXmlWriter.java:750)
at com.ctc.wstx.sw.BaseNsStreamWriter.doWriteStartTag(BaseNsStreamWriter.java:612)
at com.ctc.wstx.sw.SimpleNsStreamWriter.writeStartOrEmpty(SimpleNsStreamWriter.java:265)
at com.ctc.wstx.sw.BaseNsStreamWriter.writeStartElement(BaseNsStreamWriter.java:313)
at com.sun.xml.ws.util.xml.XMLStreamWriterFilter.writeStartElement(XMLStreamWriterFilter.java:217)
at weblogic.wsee.jaxws.framework.policy.PatchFilter.writeStartElement(WSDLPatchFilter.java:264)
at com.sun.xml.ws.util.xml.XMLStreamReaderToXMLStreamWriter.handleStartElement(XMLStreamReaderToXMLStreamWriter.java:166)
at com.sun.xml.ws.server.WSDLPatcher.handleStartElement(WSDLPatcher.java:205)
at com.sun.xml.ws.util.xml.XMLStreamReaderToXMLStreamWriter.bridge(XMLStreamReaderToXMLStreamWriter.java:105)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:291)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:265)
at com.sun.xml.ws.transport.http.HttpAdapter.publishWSDL(HttpAdapter.java:610)
at weblogic.wsee.jaxws.WLSServletAdapter.publishWSDL(WLSServletAdapter.java:252)
at weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:166)
at weblogic.wsee.jaxws.HttpServletAdapter.get(HttpServletAdapter.java:193)
at weblogic.wsee.jaxws.JAXWSServlet.doRequest(JAXWSServlet.java:97)
at weblogic.servlet.http.AbstractAsyncServlet.service(AbstractAsyncServlet.java:99)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
...

A heap dump contains a leak suspect with this object:                 122,753,656 bytes (15.1 %) of Java heap is used by  726 instances of weblogic/utils/collections/WeakConcurrentHashMap$Entry

A heap dump shows a leak of weblogic.socket.DevPollSocketInfo objects: 

 class Name                                             | Objects | Shallow Heap | Retained Heap
 --------------------------------------------------------------------------
 weblogic.socket.DevPollSocketInfo      |   1,189  |          47,592  |   47,592
 --------------------------------------------------------------------------



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms