"WARNING [B2190]: HALTING BROKER BECAUSE OF [B3203]: Another broker in the cluster has taken over this broker's persistent store (split-brain situation), restarting to recover" message in GlassFish MQ (Doc ID 1347820.1)

Last updated on NOVEMBER 05, 2016

Applies to:

Oracle GlassFish Server - Version: 2.1 and later   [Release: 2.1 and later ]
Sun Java System Message Queue - Version: 4.3 to 4.4u2   [Release: 4.3 to 4.4]
Information in this document applies to any platform.

Symptoms

Broker 1 log shows the split-brain situation with the [B3203] message:

[03/Aug/2011:22:06:26 EDT] ERROR [B4117]: Unexpected Broker Exception: [(brokerID=xxxxxxxxxxxx, URL=mq://x.x.x.x:32000/, version=40, state=9 [BrokerState[FAILOVER_PENDING]], sessionID=7018631656318815488, heartbeatTS=1312423586556 [Wed Aug 03 22:06:26 EDT 2011], takeoverBrokerID=xxxxxxxxxxxxxxx)]:
com.sun.messaging.jmq.jmsserver.util.StoreBeingTakenOverException: [B3223]: Database operation failed. Persistent store could possibly being taken over by another broker.
[03/Aug/2011:22:06:26 EDT] ERROR [B3203]: Another broker in the cluster has taken over this broker's persistent store (split-brain situation), restarting to recover:
com.sun.messaging.jmq.jmsserver.util.StoreBeingTakenOverException: [B3223]: Database operation failed. Persistent store could possibly being taken over by another broker.

at com.sun.messaging.jmq.jmsserver.persist.jdbc.ConsumerStateDAOImpl.clearTransaction(ConsumerStateDAOImpl.java:683)
at com.sun.messaging.jmq.jmsserver.persist.jdbc.JDBCStore.removeTransaction(JDBCStore.java:1573)
at com.sun.messaging.jmq.jmsserver.data.TransactionList.removeTransactionID(TransactionList.java:630)
at com.sun.messaging.jmq.jmsserver.data.TransactionList.reapTransactionID(TransactionList.java:592)
at com.sun.messaging.jmq.jmsserver.data.TransactionReaper.run(TransactionList.java:3338)
at com.sun.messaging.jmq.util.timer.WakeupableTimer.run(WakeupableTimer.java:118)
at java.lang.Thread.run(Thread.java:595)
[03/Aug/2011:22:06:26 EDT] WARNING [B2190]: HALTING BROKER BECAUSE OF [B3203]: Another broker in the cluster has taken over this broker's persistent store (split-brain situation), restarting to recover
[03/Aug/2011:22:06:28 EDT]


Broker 2 log shows the following messages for the same time frame:

[03/Aug/2011:22:06:26 EDT] [B1210]: HA Monitor is watching 1 indoubt brokers
[03/Aug/2011:22:06:26 EDT] [B1214]: Broker xxxxxxxxxx is unresponsive, attempting to takeover
[03/Aug/2011:22:06:26 EDT] [B1183]: Unsuspect xxxxxxxxxxxx
[03/Aug/2011:22:06:26 EDT] [B1184]: Unsuspected /x.x.x.x:32000 [brokerID=xxxxxxxxxxxxx, brokerSession=2579491188336708864] (seq#=0, ts=1312423586538, interval=2, len=228) sender=/x.x.x.x:38096
[03/Aug/2011:22:06:26 EDT] [B1186]: Stopping heartbeat to xxxxxxxxxxx[BrokerSession:2579491188336708864]
[03/Aug/2011:22:06:26 EDT] [B1182]: Removed heartbeat endpoint /x.x.x.x:32000 [brokerID=xxxxxxxxxxxx, brokerSession=2579491188336708864] (seq#=0, ts=1312423586538, interval=2, len=228) sender=/x.x.x.x:38096
[03/Aug/2011:22:06:26 EDT] WARNING [B2127]: Close broker link mq://x.x.x.x:32000/?instName=xxxxxxxxxx&brokerID=xxxxxxxxxxxxxxx&brokerSessionUID=2579491188336708864&ha=true&storeSessionUID=7018631656318815488 because of notified down by HA monitor
[03/Aug/2011:22:06:26 EDT] [B1072]: Closed cluster connection to broker mq://x.x.x.x:32000/?instName=xxxxxxxxxxxx&brokerID=xxxxxxxxxxxxx&brokerSessionUID=2579491188336708864&ha=true&storeSessionUID=7018631656318815488
[03/Aug/2011:22:06:26 EDT] [B1186]: Stopping heartbeat to xxxxxxxxxxxxx[BrokerSession:2579491188336708864]
[03/Aug/2011:22:06:26 EDT] WARNING [B2121]: Unable to remove heartbeat endpoint for broker [brokerID=xxxxxxxxxxxx, brokerSession=2579491188336708864] (seq#=0, ts=0, interval=0, len=0) because of not found
[03/Aug/2011:22:06:26 EDT] [B1180]: Deactivated broker
Address = mq://x.x.x.x:32000/?instName=xxxxxxxxxxxxx&brokerID=xxxxxxxxxxxxxx&brokerSessionUID=2579491188336708864&ha=true&storeSessionUID=7018631656318815488
StartTime = 1312409128950
ProtocolVersion = 410
HeartbeatHost = x.x.x.x
HeartbeatPort = 32000
[03/Aug/2011:22:06:26 EDT] [B1216]: Attempting to takeover the store for Broker xxxxxxxxxxxxx
[03/Aug/2011:22:06:26 EDT] [B1175]: This broker is about to TAKEOVER broker [brokerID=xxxxxxxxxxxxx, storeSession=7018631656318815488]
[03/Aug/2011:22:06:26 EDT] [B1168]: Takeover lock has been acquired for failed broker xxxxxxxxxxxxx (timestamp: 1312423586619)

[03/Aug/2011:22:06:26 EDT] Taking over 0 local destinations from broker xxxxxxxxxxxxxx: []
[03/Aug/2011:22:06:26 EDT] Taking over 1 messages from broker xxxxxxxxxxxxxx
[03/Aug/2011:22:06:27 EDT] Taking over 861 transactions from broker xxxxxxxxxxxxxxx
[03/Aug/2011:22:06:27 EDT] Taking over 0 remote transactions from broker xxxxxxxxxxxxxx
[03/Aug/2011:22:06:27 EDT] Taking over 1 store sessions from broker xxxxxxxxxxxxxxxx: [7018631656318815488]
[03/Aug/2011:22:06:27 EDT] [B1217]: Takeover of broker xxxxxxxxxxxxxxx was successful, processing data and state information owned by that broker
[03/Aug/2011:22:06:27 EDT] [B1218]: Processing 861 transactions from broker xxxxxxxxxxxxxxxxxxx
[03/Aug/2011:22:06:27 EDT] [B1297]: Processing 0 remote transactions from broker xxxxxxxxxxxxxxxx

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms