How to Identify a Multicast Communication Problem in a GlassFish Cluster?

(Doc ID 1313330.1)

Last updated on NOVEMBER 05, 2016

Applies to:

Oracle GlassFish Server - Version 2.1 to 2.1.1 [Release 2.1]
Information in this document applies to any platform.
***Checked for relevance on 4-Sep-2014***

Symptoms

In this article, the following setup is used:

A cluster "MyCluster" has been configured using two physical machines. 

Machine A consists of instance1 and instance2.
Machine B consists of instance 3 and instance 4.

The following command "list-instances" indicates that the cluster instances are running:

# asadmin list-instances MyCluster
instance1 running
instance2 running
instance3 running
instance4 running
Command list-instances executed successfully.


However, if there is a multicast communication issue in a GlassFish cluster, some of the following symptoms might be observed:

1. The "asadmin get-health" Command Indicates an Ambiguous Status for Some Instances

When you run the get-health command it indicates that some of the instances are not started:

# asadmin get-health MyCluster
Instance instance2 Started since Mar 3, 2011 6:18:37 AM
Instance instance3 not Started
Instance instance4 not Started

Instance instance1 Started since Mar 3, 2011 6:10:57 AM
Command get-health executed successfully.


According to the description of get-health command, the output seems to suggest that the GMS is not enabled in the cluster setup:

"The get-health command gets information about the health of the cluster.
Note that if GMS is not enabled in Enterprise Server, the basic information about whether the server instances in this cluster are running or not running is returned"

2. Missing "Member id" of the GMS Cluster Group in the server.log

In the cluster configuration mentioned above, you can verify the multicast communication problem between the cluster instances by looking at the server.log, to see if one instance's server.log does not report all the other instances that are part of the cluster.

Instance2 server.log extract:
[#|2011-03-08T06:36:34.595-0500|INFO|sun-appserver2.1|ShoalLogger|_ThreadID=14;_ThreadName=ViewWindowThread:MyCluster;|GMS View Change Received for group MyCluster : Members in view for ADD_EVENT(before change analysis) are :
1: MemberId: server, MemberType: SPECTATOR, Address: urn:jxta:uuid-59616261646162614A787461503250332DE658F932AB436995B78E0CB3E080DA03
2: MemberId: instance2, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A787461503250338190A99DE4454A3CB1C2DFAD78C842B803
3: MemberId: instance1, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A78746150325033D2514084BA4743008335F7FF8BF4D78803
|#]
Instance4 server.log extract:
[#|2011-03-08T06:24:24.405-0500|INFO|sun-appserver2.1|ShoalLogger|_ThreadID=14;_ThreadName=ViewWindowThread:MyCluster;|GMS View Change Received for group MyCluster : Members in view for ADD_EVENT(before change analysis) are :
1: MemberId: instance3, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A78746150325033BC1C68764FF047268846F42DB478399803
2: MemberId: instance4, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A78746150325033F21F8E859EB54C9CAA7EEFEE670B95C503
|#]


In the log file extracts above, the member ids of instance3 and 4 are missing from the server.log of instance2.  Similarly, in instance4 's server.log,  the membership ids of instance1 and instance2 are also missing.

Ideally, if the multicast communication is working fine, you should see all the cluster instances' membership ids listed in the GMS View as shown below in the server.log of all participating instances:

[#|2011-03-25T08:34:58.669-0700|INFO|sun-appserver2.1.1|ShoalLogger|_ThreadID=13;_ThreadName=ViewWindowThread:MyCluster;|GMS View Change Received for group MyCluster : Members in view for JOINED_AND_READY_EVENT(before change analysis) are :
1: MemberId: server, MemberType: SPECTATOR, Address: urn:jxta:uuid-59616261646162614A787461503250332DE658F932AB436995B78E0CB3E080DA03
2: MemberId: instance1, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A787461503250333A1A7E2BB03647E5ABFDEEC3B08FE61903
3: MemberId: instance3, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A787461503250335E20871EC1E4467EB86892F88D90267D03
4: MemberId: instance2, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A78746150325033CAF6C027B1BA45F4AA44B1328ACC574103
5: MemberId: instance4, MemberType: CORE, Address: urn:jxta:uuid-59616261646162614A78746150325033E773183BBE5D476F928FFA7042E070A903
|#]

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms