OVM: Adding Server to Server Pool Fails With Error "Cannot configure server for cluster in membership state Heartbeat_Configured" (Doc ID 2376256.1)

Last updated on FEBRUARY 09, 2024

Applies to:

Symptoms

One of the Oracle VM server nodes is not joining to the Pool.

In AdminServer.log, found following error message:

####<2018-03-08T17:55:00.717+0100> <Error> <com.oracle.ovm.mgr.api.job.InternalJob> <xxxxxxx> <AdminServer> <Odof Tcp Client Thread: /127.0.0.1:54321/475110> <> <> <e94e5500-5ae5-43ae-80a3-750d92d79ed7-00000004> <1520528100717> <BEA-000000> <Job: Add Server: xxxxxx to Server Pool: xxxxx, Time: 1520528100060, Internal Error (Operation) OVMAPI_4010E Attempt to send command: configure_server_for_cluster to server: xxxxxx failed. OVMAPI_4004E Sync command failed on server: xx.xx.xx.xx. Command: configure_server_for_cluster, Server error: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:Cannot configure server for cluster in membership state Heartbeat_Configured [Thu Mar 08 17:55:00 CET 2018]
com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: configure_server_for_cluster to server: xxxxxx failed. OVMAPI_4004E Sync command failed on server: xx.xx.xx.xx. Command: configure_server_for_cluster,

Server error: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:Cannot configure server for cluster in membership state Heartbeat_Configured

[Thu Mar 08 17:55:00 CET 2018]
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:559)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:454)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:418)
at com.oracle.ovm.mgr.action.ClusterAction.configureServerForCluster(ClusterAction.java:72)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.configureCluster(ServerClusterConfigure.java:136)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.action(ServerClusterConfigure.java:59)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1290)
at sun.reflect.GeneratedMethodAccessor849.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:387)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:364)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:778)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)
at com.oracle.ovm.mgr.api.physical.ServerProxy.executeCurrentJobOperationAction(Unknown Source)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:240)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:332)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1644)
at sun.reflect.GeneratedMethodAccessor1608.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:387)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:364)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:111)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:92)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:86)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:701)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:464)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:529)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:64)
at java.lang.Thread.run(Thread.java:745)
>

On Oracle VM server, you can see the following in /var/log/ovs-agent.log:

[2018-03-08 14:57:39 21710] ERROR (service:97) catch_error: Cannot configure server for cluster in membership state Heartbeat_Configured
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/agent/lib/service.py", line 95, in wrapper
return func(*args)
File "/usr/lib64/python2.6/site-packages/agent/api/cluster/o2cb.py", line 84, in configure_server_for_cluster
raise Exception("Cannot configure server for cluster in membership state %s" % state)
Exception: Cannot configure server for cluster in membership state Heartbeat_Configured
[2018-03-08 14:57:39 21711] DEBUG (service:75) call start: leave_server_pool('0004fb00000200008d3eb824fa2e6f49',)
[2018-03-08 14:57:39 21711] ERROR (service:97) catch_error: Server must be in Pooled state
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/agent/lib/service.py", line 95, in wrapper
return func(*args)
File "/usr/lib64/python2.6/site-packages/agent/api/serverpool.py", line 168, in leave_server_pool
pool_remove_server(pool_uuid, final=False)
File "/usr/lib64/python2.6/site-packages/agent/api/serverpool.py", line 362, in pool_remove_server
raise Exception("Server must be in Pooled state")
Exception: Server must be in Pooled state

Changes

 No changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.