[ PCA ] Reprovisioning a Compute Node Fails at Stage initializing_stage_add_server_to_pool with "stderr: o2cb: Unknown code ____ 228 while storing the cluster configuration in file '/etc/ocfs2/cluster.conf'"

(Doc ID 2352348.1)

Last updated on JANUARY 25, 2018

Applies to:

Private Cloud Appliance - Version 2.0.5 and later
Linux x86-64

Symptoms

On Oracle Private Cloud Appliance, when attempting to provision or re-provision a compute node, the re-provision job fails at stage initializing_stage_add_server_to_pool with the following kind of error being reported in the /var/log/ovca.log of the master management node :

STDERR:
[2018-01-19 11:37:40 3756372] DEBUG (tenant_group:455) Failed to add ovcacn11r1 to Tenant Group Rack1_ServerPool: Looking for server pool [Rack1_ServerPool]
Looking for server [ovcacn11r1]
Adding ovcacn11r1 to the Tenant Group Rack1_ServerPool
Running [Add server to Tenant Group]
Successfully ran [Add server to Tenant Group]
Adding ovcacn11r1 to the cluster.
Failed to add ovcacn11r1 to the cluster
Beginning rollback to remove ovcacn11r1 from the serverpool
Running [Remove server from Tenant Group]
Successfully ran [Remove server from Tenant Group]

[2018-01-19 11:37:40 3756372] ERROR (utils:148) Exception running add server to pool. ILOM MAC: 00:10:e0:7f:e0:4d. Marking node as dead.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ovca/daemon/utils.py", line 139, in new_function
result = func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/ovca/daemon/nodestateserver.py", line 512, in add_server_to_pool
return ovmm_add_server_to_tenant_group(mac, nodeinfo)
File "/usr/lib/python2.6/site-packages/ovca/tenant_group.py", line 109, in ovmm_add_server_to_tenant_group
if not add_server_to_tenant_group(server_name, tenant_group):
File "/usr/lib/python2.6/site-packages/ovca/tenant_group.py", line 470, in add_server_to_tenant_group
(server_name, tenant_group))
Exception: The server ovcacn11r1 could not be added to the Rack1_ServerPool
[2018-01-19 11:37:40 3756372] ERROR (utils:105) Run # 43 after 1832 secs: Function add_server_to_pool.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ovca/daemon/utils.py", line 97, in new_function
% (func.__name__, max_time, max_trial))
JobTimeoutError: Polling add_server_to_pool exceeds 1800 seconds and max trial count 5. Breaking loop.
[2018-01-19 11:37:40 3756372] DEBUG (nodestateserver:1460) DEAD node: ilom 00:10:e0:7f:e0:4d. Last good state: initializing_stage_add_server_to_pool

Reviewing the /u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs/AdminServer.log :

####<Jan 19, 2018 12:47:22 PM UTC> <Info> <com.oracle.ovm.mgr.api.job.InternalJob> <ovcamn06r1> <AdminServer> <Odof Tcp Client Thread: /127.0.0.1:54321/35963> <> <> <0000M49KfuXEwGWFLzvH8A1QO9Wx00
0002> <1516366042571> <BEA-000000> <Job: add_server_to_cluster, Time: 1516366024307, Internal Error (Operation) OVMAPI_4010E Attempt to send command: dispatch to server: ovcacn12r1 failed. OVMAPI_4004E Serve
r Failed Command: dispatch https://?uname?:?pwd?@192.168.140.10:8899/api/3 update_cluster_configuration, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['/sbin/o2cb', '--config-f
ile=/etc/ocfs2/cluster.conf', 'add-cluster', '3f1cd7b6a11482f8'] failed (1): stderr: o2cb: Unknown code ____ 228 while storing the cluster configuration in file '/etc/ocfs2/cluster.conf'
stdout: [Fri Jan 19 12:47:22 UTC 2018] [Fri Jan 19 12:47:22 UTC 2018]
com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovcacn12r1 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@192.
168.140.10:8899/api/3 update_cluster_configuration, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['/sbin/o2cb', '--config-file=/etc/ocfs2/cluster.conf', 'add-cluster', '3f1cd7b
6a11482f8'] failed (1): stderr: o2cb: Unknown code ____ 228 while storing the cluster configuration in file '/etc/ocfs2/cluster.conf'
stdout: [Fri Jan 19 12:47:22 UTC 2018] [Fri Jan 19 12:47:22 UTC 2018]
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:517)
at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:454)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:384)
at com.oracle.ovm.mgr.action.ClusterAction.updateClusterConfiguration(ClusterAction.java:115)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigurationUpdate.reconfigureCluster(ServerClusterConfigurationUpdate.java:134)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigurationUpdate.action(ServerClusterConfigurationUpdate.java:53)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)
at sun.reflect.GeneratedMethodAccessor381.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:378)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:355)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:893)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)
at com.oracle.ovm.mgr.api.physical.ServerProxy.executeCurrentJobOperationAction(Unknown Source)
[...]
at java.lang.Thread.run(Thread.java:682)

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms