My Oracle Support Banner

Doing a Full Document Crawl with Elasticsearch in WebCenter Portal is Taking a Long Time and Failing with: "java.io.IOException: Broken pipe" and "java.lang.OutOfMemoryError: GC overhead limit exceeded" (Doc ID 2763562.1)

Last updated on SEPTEMBER 30, 2022

Applies to:

Oracle WebCenter Portal - Version 12.2.1.4.0 and later
Information in this document applies to any platform.

Symptoms

Doing a full document crawl with Elasticsearch in WebCenter Portal won't complete the crawl when having many documents.  The operation takes a long time and fails with "java.io.IOException: Broken pipe" and "java.lang.OutOfMemoryError: GC overhead limit exceeded".

The Portal managed servers become unresponsive and need to be restarted.
The portal logs shows the following errors:

ERROR 

####<DATE-TIME> <Info> <com.oracle.coherence> <Portal> <WC_Portal> <Logger@9219882 12.2.1.4.5> <> <> <ECID> <> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <DATE-TIME> Oracle Coherence GE 12.2.1.4.5 <Info> (thread=Cluster, member=16): Service guardian is 32268ms late, indicating that this JVM may be running slowly or experienced a long GC>

####<DATE-TIME> <Error> <RJVM> <Portal> <WC_Portal> <[STANDBY] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <ECID> <> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000580> <Failed to send JVMMessage from: '-<Number>S:<IP-ADDRESS>:wc_domain:WC_Portal1' to: '-<Number>S:<IP-ADDRESS>:[<Port>,-1,-1,-1,-1,-1,-1]:wc_domain:AdminServer' cmd: 'CMD_RESPONSE', QOS: '101', responseId: '0', invokableId: '0', flags: 'JVMIDs Not Sent, TX Context Not Sent, 0x8', abbrev offset: '0'.
java.io.IOException: Broken pipe
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
    at weblogic.socket.NIOOutputStream$BlockingWriter.write(NIOOutputStream.java:326)
...
>

####<DATE-TIME> <Common> <Portal> <WC_Portal> <[ACTIVE] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <ECID> <> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000628> <Created "1" resources for pool "mds-owsm", out of which "1" are available and "0" are unavailable.>

####<DATE-TIME> <Error> <com.oracle.coherence> <Portal> <WC_Portal> <Logger@9219882 12.2.1.4.5> <> <> <ECID> <> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <DATE-TIME Oracle Coherence GE 12.2.1.4.5 <Error> (thread=PacketListener1, member=16): Stopping cluster due to unhandled exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
>

####<DATE-TIME> <Info> <com.oracle.coherence> <Portal> <WC_Portal> <Logger@9219882 12.2.1.4.5> <> <> <ECID> <> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <DATE-TIME Oracle Coherence GE 12.2.1.4.5 <Info> (thread=Invocation:Management, member=n/a): Service Management left the cluster>

####<DATE-TIME> <Info> <com.oracle.coherence> <Portal> <WC_Portal> <Logger@9219882 12.2.1.4.5> <> <> <ECID> <> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <DATE-TIMEOracle Coherence GE 12.2.1.4.5 <Info> (thread=Cluster, member=n/a): Service guardian is 410677ms late, indicating that this JVM may be running slowly or experienced a long GC>

####<DATE-TIME> <Error> <RJVM> <Portal> <WC_Portal> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <ECID> <> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000580> <Failed to send JVMMessage from: '-<Number>S:<IP-ADDRESS>:wc_domain:WC_Portal' to: '-<Number>S:<IP-ADDRESS>:[<Port>,<Port>,-1,-1,-1,-1,-1]:wc_domain:WC_Portal' cmd: 'CMD_ONE_WAY', QOS: '101', responseId: '0', invokableId: '0', flags: 'JVMIDs Not Sent, TX Context Not Sent, 0x0', abbrev offset: '0'.
java.io.IOException: Broken pipe
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
...

 
STEPS
The issue can be reproduced at will with the following steps: 

  1. Upload many documents to WebCenter portal (e.g.: ~ 500k+ documents).
     
  2. Go to Administration > Settings > Tools and Services > Search.
     
  3. Start a full documents crawl.
     
  4. You will see crawling is taking days to complete.
    Eventually the portal managed servers must be restarted.
     
  5. The WebCenter Portal log will show the "java.lang.OutOfMemoryError: GC overhead limit exceeded" error.
     
  6. Setting the following parameters when starting the portal manager server solves the "java.lang.OutOfMemoryError: GC overhead limit exceeded" error but the crawls is still very slow and consuming CPU:

    -Xms12g -Xmx12g -XX:-UseGCOverheadLimit -XX:ParallelGCThreads=4 -XX:ReservedCodeCacheSize=512M

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.