Large Weblogic cluster shows excessive garbage collection (Doc ID 1513226.1)

Last updated on NOVEMBER 03, 2016

Applies to:

Oracle WebCenter Sites - Version 11.1.1.6.1 and later
Information in this document applies to any platform.

Symptoms

A large cluster of WebCenter Sites nodes (~60 of them) exhibits a memory leak (slow heap growth over time, until the heap is exhausted, and garbage collection does not free any heap). Also there are errors showing Unicast errors with members going in and out of the cluster. There is no traffic at all to the cluster and no user activity. The WLS cluster is configured using unicast.

WLS logs show:

####<09-Dec-2012 23:59:41 o'clock GMT> <Info> <Cluster> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1355097581380> <BEA-000111> <Adding ms3-cbpcsd01 with ID -2167371751009697640S:ms3-cbpcsd01.customer.com:[61003,61003,-1,-1,-1,-1,-1]:cbpcsd01:ms3-cbpcsd01 to cluster: cluster_cbpcsd01 view.>
####<09-Dec-2012 23:59:41 o'clock GMT> <Error> <Kernel> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1355097581381> <BEA-000802> <ExecuteRequest failed
java.lang.NullPointerException.
java.lang.NullPointerException
at weblogic.cluster.RemoteMemberInfo.add(RemoteMemberInfo.java:400)
at weblogic.cluster.RemoteMemberInfo.updateRuntimeState(RemoteMemberInfo.java:358)
at weblogic.cluster.MemberManager.updateMemberRuntimeState(MemberManager.java:577)
at weblogic.cluster.RuntimeStateMessage.execute(RuntimeStateMessage.java:33)
at weblogic.cluster.HTTPExecuteRequest$1.run(HTTPExecuteRequest.java:96)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.cluster.HTTPExecuteRequest.run(HTTPExecuteRequest.java:93)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)


####<10-Dec-2012 00:12:43 o'clock GMT> <Info> <Cluster> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1355098363609> <BEA-000128> <Updating -3363555987484002118S:ms2-cbpcsd01.customer.com:[61002,61002,-1,-1,-1,-1,-1]:cbpcsd01:ms2-cbpcsd01 in the cluster.>
####<10-Dec-2012 00:12:43 o'clock GMT> <Error> <Kernel> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1355098363610> <BEA-000802> <ExecuteRequest failed
java.lang.NullPointerException.
java.lang.NullPointerException
at weblogic.cluster.UpgradeUtils.clusterMembersChanged(UpgradeUtils.java:84)
at weblogic.cluster.MemberManager$3.run(MemberManager.java:523)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

####<10-Dec-2012 05:54:11 o'clock GMT> <Error> <Kernel> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1355118851424> <BEA-000802> <ExecuteRequest failed
java.lang.NullPointerException.
java.lang.NullPointerException
at weblogic.iiop.ClusterServices.clusterMembersChanged(ClusterServices.java:77)
at weblogic.cluster.MemberManager$3.run(MemberManager.java:523)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms