My Oracle Support Banner

OLVM: The hosted-engine is unable to connect to storage domain (hyperconverged storage also known as glusterfs storage) (Doc ID 2848954.1)

Last updated on FEBRUARY 12, 2022

Applies to:

Linux OS - Version Oracle Linux 7.9 with Unbreakable Enterprise Kernel [5.4.17] and later
Linux x86-64
Information in this document applies to any platform.

Symptoms

On Oracle Linux Virtualization Manager(OLVM), the hosted-engine is unable to connect to storage domain (hyperconverged storage also known as glusterfs storage). Please refer to the details below.

# hosted-engine --vm-status
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
# service ovirt-ha-agent status
Redirecting to /bin/systemctl status ovirt-ha-agent.service
● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent
Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2022-02-11 10:04:10 EST; 7s ago
Process: 38095 ExecStart=/usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent (code=exited, status=157)
Main PID: 38095 (code=exited, status=157)
Tasks: 0
CGroup: /system.slice/ovirt-ha-agent.service
Feb 11 10:04:10 kvmhost01.example.com systemd[1]: Unit ovirt-ha-agent.service entered failed state.
Feb 11 10:04:10 kvmhost01.example.com systemd[1]: ovirt-ha-agent.service failed.

 

broker.log:
MainThread::WARNING::2022-01-26 11:56:44,322::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: Connection to storage server failed
MainThread::INFO::2022-01-26 11:56:44,820::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started
MainThread::INFO::2022-01-26 11:56:44,820::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors
MainThread::INFO::2022-01-26 11:56:44,820::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load
MainThread::INFO::2022-01-26 11:56:44,821::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine
MainThread::INFO::2022-01-26 11:56:44,822::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health
MainThread::INFO::2022-01-26 11:56:44,822::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free
MainThread::INFO::2022-01-26 11:56:44,822::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge
MainThread::INFO::2022-01-26 11:56:44,822::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network
MainThread::INFO::2022-01-26 11:56:44,822::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain

 

agent.log:
MainThread::ERROR::2022-01-26 12:12:29,904::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::INFO::2022-01-26 12:12:29,904::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down
MainThread::INFO::2022-01-26 12:12:40,317::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started
MainThread::INFO::2022-01-26 12:12:40,345::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: kvmhost01.example.com
MainThread::INFO::2022-01-26 12:12:40,404::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2022-01-26 12:12:40,405::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': xxx.xxx.xxx.xxx''}
MainThread::ERROR::2022-01-26 12:12:40,406::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors
MainThread::ERROR::2022-01-26 12:12:40,406::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent
return action(he)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper
return he.start_monitoring()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring
self._initialize_broker()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker
m.get('options', {}))
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor
).format(t=type, o=options, e=e)
RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': 'xxx.xxx.xxx.xxx'}]

 

engine.log:
2022-02-08 15:11:36,106-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/engine/engine' of volume '24267da7-920b-4bc4-8366-f63656bacbc7' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,107-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/engine/engine' of volume '24267da7-920b-4bc4-8366-f63656bacbc7' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,108-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/engine/engine' of volume '24267da7-920b-4bc4-8366-f63656bacbc7' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,109-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/data/data' of volume '18894c19-2bbe-4254-a629-c8cf5eee9676' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,110-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/data/data' of volume '18894c19-2bbe-4254-a629-c8cf5eee9676' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,111-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/data/data' of volume '18894c19-2bbe-4254-a629-c8cf5eee9676' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,111-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/vmstore/vmstore' of volume '4eed5468-b29d-45e3-8f50-2af3d62b6a7d' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,112-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6522eb6c] Could not associate brick 'xxx.xxx.xxx.xxx:/gluster_bricks/vmstore/vmstore' of volume '4eed5468-b29d-45e3-8f50-2af3d62b6a7d' with correct network as no gluster network found in cluster '4b9999f2-6b21-11ec-8d3b-00163e5dc843'
2022-02-08 15:11:36,113-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListRetur

Changes

 N/A

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.