O3L 1.0: nova-compute service failed to be enabled after reboot compute node: AUTO: Failed to connect to libvirt

(Doc ID 2002139.1)

Last updated on NOVEMBER 23, 2017

Applies to:

Oracle OpenStack for Oracle Linux - Version OpenStack 1.0 to OpenStack 1.0 [Release ODOS1]
x86_64

Symptoms

nova-compute service failed to be enabled after reboot Compute Node:

[root@controlnode01 ~(keystone_admin)]# nova service-list
+------------------+---------------+----------+----------+-------+----------------------------+------------------------------------+
| Binary           | Host          | Zone     | Status   | State | Updated_at                 | Disabled Reason                    |
+------------------+---------------+----------+----------+-------+----------------------------+------------------------------------+
| nova-consoleauth | controlnode01 | internal | enabled  | up    | 2015-04-21T09:40:30.000000 | -                                  |
| nova-scheduler   | controlnode01 | internal | enabled  | up    | 2015-04-21T09:40:22.000000 | -                                  |
| nova-conductor   | controlnode01 | internal | enabled  | up    | 2015-04-21T09:40:30.000000 | -                                  |
| nova-cert        | controlnode01 | internal | enabled  | up    | 2015-04-21T09:40:22.000000 | -                                  |
| nova-compute     | computenode03 | nova     | disabled | down  | 2015-04-21T09:39:19.000000 | AUTO: Failed to connect to libvirt |
+------------------+---------------+----------+----------+-------+----------------------------+------------------------------------+

Check errors in /var/log/nova/nova-compute.log on Compute Node computenode03:

2015-04-21 17:38:33.395 3869 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2015-04-21 17:38:33.499 3869 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.7.2.125:5672
2015-04-21 17:38:33.551 3869 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.7.2.125:5672
2015-04-21 17:38:33.717 3869 AUDIT nova.service [-] Starting compute node (version 2014.1.3-4.0.1.el6)
2015-04-21 17:38:34.320 3869 ERROR nova.virt.libvirt.driver [-] Connection to libvirt failed: unable to connect to 'localhost:8000': Connection refused
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver Traceback (most recent call last):
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 788, in _connect
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver     libvirt.openAuth, uri, auth, flags)
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in proxy_call
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver     rv = execute(f,*args,**kwargs)
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver     rv = meth(*args,**kwargs)
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver libvirtError: unable to connect to 'localhost:8000': Connection refused
2015-04-21 17:38:34.320 3869 TRACE nova.virt.libvirt.driver
2015-04-21 17:38:34.623 3869 ERROR nova.openstack.common.threadgroup [-] Connection to the hypervisor is broken on host: computenode03
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     x.wait()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 486, in run_service
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     service.start()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/service.py", line 163, in start
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     self.manager.init_host()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1029, in init_host
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     self.driver.init_host(host=self.host)
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 657, in init_host
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     self._do_quality_warnings()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 640, in _do_quality_warnings
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     caps = self.get_host_capabilities()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2880, in get_host_capabilities
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     xmlstr = self._conn.getCapabilities()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 722, in _get_connection
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     wrapped_conn = self._get_new_connection()
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 675, in _get_new_connection
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     wrapped_conn = self._connect(self.uri(), self.read_only)
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 797, in _connect
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup     raise exception.HypervisorUnavailable(host=CONF.host)
2015-04-21 17:38:34.623 3869 TRACE nova.openstack.common.threadgroup HypervisorUnavailable: Connection to the hypervisor is broken on host: computenode03

 

Check errors in /var/log/libvirt/libvirtd.log on Compute Node computenode03:

2015-04-22 04:09:39.278+0000: 3701: error : do_connect:114 : internal error: failed to connect to xend
2015-04-22 04:09:39.279+0000: 3701: error : xenDaemonOpen_tcp:707 : unable to connect to 'localhost:8000': Connection refused
2015-04-22 04:09:39.663+0000: 3697: error : virNetSocketReadWire:1409 : End of file while reading data: Input/output error
2015-04-22 04:09:49.870+0000: 3700: error : do_connect:114 : internal error: failed to connect to xend

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms