Large Weblogic cluster shows excessive garbage collection (Doc ID 1513226.1)

Last updated on JUNE 20, 2023

Applies to:

Symptoms

A large cluster of WebCenter Sites nodes (~60 of them) exhibits a memory leak (slow heap growth over time, until the heap is exhausted, and garbage collection does not free any heap). Also there are errors showing Unicast errors with members going in and out of the cluster. There is no traffic at all to the cluster and no user activity. The WLS cluster is configured using unicast.

WLS logs show:

####<09-Dec-2012 23:59:41 o'clock GMT> <Info> <Cluster> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1355097581380> <BEA-000111> <Adding ms3-cbpcsd01 with ID -2167371751009697640S:ms3-cbpcsd01.customer.com:[61003,61003,-1,-1,-1,-1,-1]:cbpcsd01:ms3-cbpcsd01 to cluster: cluster_cbpcsd01 view.>
####<09-Dec-2012 23:59:41 o'clock GMT> <Error> <Kernel> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1355097581381> <BEA-000802> <ExecuteRequest failed
java.lang.NullPointerException.
java.lang.NullPointerException
at weblogic.cluster.RemoteMemberInfo.add(RemoteMemberInfo.java:400)
at weblogic.cluster.RemoteMemberInfo.updateRuntimeState(RemoteMemberInfo.java:358)
at weblogic.cluster.MemberManager.updateMemberRuntimeState(MemberManager.java:577)
at weblogic.cluster.RuntimeStateMessage.execute(RuntimeStateMessage.java:33)
at weblogic.cluster.HTTPExecuteRequest$1.run(HTTPExecuteRequest.java:96)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.cluster.HTTPExecuteRequest.run(HTTPExecuteRequest.java:93)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)


####<10-Dec-2012 00:12:43 o'clock GMT> <Info> <Cluster> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1355098363609> <BEA-000128> <Updating -3363555987484002118S:ms2-cbpcsd01.customer.com:[61002,61002,-1,-1,-1,-1,-1]:cbpcsd01:ms2-cbpcsd01 in the cluster.>
####<10-Dec-2012 00:12:43 o'clock GMT> <Error> <Kernel> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1355098363610> <BEA-000802> <ExecuteRequest failed
java.lang.NullPointerException.
java.lang.NullPointerException
at weblogic.cluster.UpgradeUtils.clusterMembersChanged(UpgradeUtils.java:84)
at weblogic.cluster.MemberManager$3.run(MemberManager.java:523)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

####<10-Dec-2012 05:54:11 o'clock GMT> <Error> <Kernel> <cbl12074app08> <ms41-cbpcsd01> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1355118851424> <BEA-000802> <ExecuteRequest failed
java.lang.NullPointerException.
java.lang.NullPointerException
at weblogic.iiop.ClusterServices.clusterMembersChanged(ClusterServices.java:77)
at weblogic.cluster.MemberManager$3.run(MemberManager.java:523)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.