OLVM: The self-hosted-engine host had entered maintenance mode without notice (Doc ID 2944293.1)

Last updated on APRIL 22, 2023

Applies to:

Symptoms

On Oracle Linux Virtualization Manager (OLVM), the cluster became GLOBAL MAINTENANCE mode unexpectedly on self-hosted-engine (SHE). See details below, for more information.

self-hosted-engine (SHE) host:/var/log/ovirt-engine/engine.log:
2023-02-09 16:07:08,563+09 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler10) [fddcda9] VDS error Command execution failed: rc=2 out='geo-replication command failed\n' err=''

olvm-kvm-host01.local.lab:/var/log/ovirt-hosted-engine-ha/agent.log:
MainThread::INFO::2023-02-09 15:44:13,041::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:44:13,145::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:44:23,161::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:44:23,261::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:44:32,277::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:44:32,379::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:44:42,395::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:44:42,497::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:44:52,514::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:44:52,611::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:02,628::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:02,729::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:12,746::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:12,847::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:22,864::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:22,972::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:32,989::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:33,092::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:43,108::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:43,157::ovf_store::151::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/2a8841a1-2469-4ae8-8ab5-04011574595a/7d3cbb0c-59c9-437e-b07e-f8e7ccaef99b/8effa917-83cf-4cd2-96e9-e7393f65ae10
MainThread::INFO::2023-02-09 15:45:43,209::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:43,209::hosted_engine::520::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Best remote host olvm-kvm-host02 (id: 2, score: 3400)
MainThread::INFO::2023-02-09 15:45:53,227::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected

olvm-kvm-host01.local.lab:/var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:
2023-02-09 15:03:40.107017] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:03:40.107070] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 20814804: WRITEV 4 (6f2a5023-70d5-4357-ae91-ba362538d6ef), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:04:39.286270] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 20814624: WRITEV 2 (21e36ed7-6f18-4592-9ead-1927e0ff45cf), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host02.local.lab:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:04:41.486439] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 87891: WRITEV 2 (c0f3860f-74d7-4bb8-adbe-ddc5c9e06596), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host03.local.lab:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
The message "E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]" repeated 2 times between [2023-02-09 15:03:40.107017] and [2023-02-09 15:04:41.486422]
<snip>
[2023-02-09 15:54:00.336625] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:54:00.336672] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 20831263: WRITEV 5 (c82860b4-74b5-4eb6-acda-84304e6a362c), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:54:59.521869] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:54:59.521924] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 20830999: WRITEV 2 (d4aa8d07-7e36-4cb0-a1f0-d624dd2ca163), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host02.local.lab:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:55:01.688336] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 104347: WRITEV 4 (d19dc611-913e-420e-b5fe-b78689ef33f2), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host03.local.lab:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:55:01.688319] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [[Invalid argument]
[2023-02-09 15:59:02.358225] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]

olvm-kvm-host02.local.lab:/var/log/ovirt-hosted-engine-ha/agent.log:
MainThread::INFO::2023-02-09 15:45:29,610::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:29,709::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:39,719::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:39,820::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:49,836::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:49,940::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:59,956::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:00,059::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:09,075::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:09,175::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:19,191::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:19,295::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:29,314::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:29,413::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:39,430::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:39,528::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:49,544::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:49,643::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:59,660::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:59,762::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:47:09,778::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:47:09,881::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:47:19,899::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:47:20,003::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:47:29,019::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:47:29,120::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:47:39,137::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:47:39,237::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:47:49,253::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:47:49,354::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:47:59,370::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': True}
MainThread::INFO::2023-02-09 15:47:59,370::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host olvm-kvm-host01.local.lab (id 1): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3801921 (Thu Feb 9 15:47:04 2023)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3801921 (Thu Feb 9 15:47:04 2023)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n', 'hostname': 'olvm-kvm-host01.local.lab', 'alive': True, 'host-id': 1, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'c2d75805', 'local_conf_timestamp': 3801921, 'host-ts': 3801921}
MainThread::INFO::2023-02-09 15:47:59,370::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host olvm-kvm-host02.local.lab (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3801928 (Thu Feb 9 15:47:43 2023)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=3801928 (Thu Feb 9 15:47:43 2023)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n', 'hostname': 'olvm-kvm-host03.local.lab', 'alive': True, 'host-id': 3, 'engine-status': {'health': 'good', 'vm': 'up', 'detail': 'Up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '51c132cf', 'local_conf_timestamp': 3801928, 'host-ts': 3801928}
MainThread::INFO::2023-02-09 15:47:59,370::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 2): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'network': 1.0, 'mem-free': 472552.0, 'maintenance': False, 'cpu-load': 0.0048, 'storage-domain': True}
MainThread::INFO::2023-02-09 15:47:59,370::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected

olvm-kvm-host02.local.lab:/var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:
[2023-02-09 15:49:40.362115] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:49:40.362165] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17144476: WRITEV 4 (a3298171-7c7c-4083-8e98-482a19f5656d), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:50:39.554611] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:50:39.554654] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17144407: WRITEV 3 (a8c6a2a3-7963-416e-a764-c6fd850e02b6), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host02.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:50:41.728842] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 84512: WRITEV 2 (2d3e4314-4e22-40f2-8d50-b4bbeea98d24), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host03.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:50:41.728826] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:54:42.402354] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:54:42.402400] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17145817: WRITEV 3 (c82860b4-74b5-4eb6-acda-84304e6a362c), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:55:41.587570] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17145784: WRITEV 3 (d4aa8d07-7e36-4cb0-a1f0-d624dd2ca163), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host02.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:55:43.754055] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 85887: WRITEV 4 (d19dc611-913e-420e-b5fe-b78689ef33f2), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host03.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
The message "E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [[Invalid argument]" repeated 2 times between [2023-02-09 15:54:42.402354] and [2023-02-09 15:55:43.754026]
[2023-02-09 15:59:44.423891] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:59:44.423943] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17147133: WRITEV 3 (bd97ab71-7fdd-4359-9e13-bd6f2c606d79), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
<snip>
[2023-02-09 15:49:40.362115] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:49:40.362165] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17144476: WRITEV 4 (a3298171-7c7c-4083-8e98-482a19f5656d), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host02.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:50:39.554611] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:50:39.554654] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17144407: WRITEV 3 (a8c6a2a3-7963-416e-a764-c6fd850e02b6), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host03.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:50:41.728842] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 84512: WRITEV 2 (2d3e4314-4e22-40f2-8d50-b4bbeea98d24), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host01.local.lab:E:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:50:41.728826] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:54:42.402354] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:54:42.402400] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17145817: WRITEV 3 (c82860b4-74b5-4eb6-acda-84304e6a362c), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host02.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:55:41.587570] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17145784: WRITEV 3 (d4aa8d07-7e36-4cb0-a1f0-d624dd2ca163), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host03.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:55:43.754055] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 85887: WRITEV 4 (d19dc611-913e-420e-b5fe-b78689ef33f2), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host01.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
The message "E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]" repeated 2 times between [2023-02-09 15:54:42.402354] and [2023-02-09 15:55:43.754026]
[2023-02-09 15:59:44.423891] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:59:44.423943] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 17147133: WRITEV 3 (bd97ab71-7fdd-4359-9e13-bd6f2c606d79), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host02.local.lab:engine-client-1-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]


olvm-kvm-host03.local.lab:/var/log/ovirt-hosted-engine-ha/agent.log:
MainThread::INFO::2023-02-09 15:45:22,985::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:23,087::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:33,104::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:33,207::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:43,223::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:43,326::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:45:53,343::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:45:53,447::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:03,463::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:03,569::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:13,586::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected
MainThread::INFO::2023-02-09 15:46:13,688::hosted_engine::512::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state GlobalMaintenance (score: 3400)
MainThread::INFO::2023-02-09 15:46:23,704::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': True}
MainThread::INFO::2023-02-09 15:46:23,704::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host svrHci11.web1.local (id 1): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3801820 (Thu Feb 9 15:45:22 2023)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3801820 (Thu Feb 9 15:45:22 2023)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n', 'hostname': 'olvm-kvm-host01.local.lab', 'alive': True, 'host-id': 1, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '395fe67f', 'local_conf_timestamp': 3801820, 'host-ts': 3801820}
MainThread::INFO::2023-02-09 15:46:23,704::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host svrHci12.web1.local (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3801827 (Thu Feb 9 15:46:09 2023)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3801827 (Thu Feb 9 15:46:09 2023)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n', 'hostname': 'olvm-kvm-host02.local.lab', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'cc341987', 'local_conf_timestamp': 3801827, 'host-ts': 3801827}
MainThread::INFO::2023-02-09 15:46:23,704::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 3): {'engine-health': {'health': 'good', 'vm': 'up', 'detail': 'Up'}, 'bridge': True, 'network': 1.0, 'mem-free': 493849.0, 'maintenance': False, 'cpu-load': 0.0048, 'storage-domain': True}
MainThread::INFO::2023-02-09 15:46:23,704::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected

olvm-kvm-host03.local.lab:/var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:
[2023-02-09 15:03:34.025138] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:03:34.025196] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 19005079: WRITEV 4 (6f2a5023-70d5-4357-ae91-ba362538d6ef), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-2-RECON_NO:-2, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:04:33.204312] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:04:33.204360] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 19004098: WRITEV 3 (21e36ed7-6f18-4592-9ead-1927e0ff45cf), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host02.local.labE:engine-client-2-RECON_NO:-2, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:04:35.404508] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 80204: WRITEV 2 (c0f3860f-74d7-4bb8-adbe-ddc5c9e06596), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host03.local.lab:engine-client-2-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:08:36.033987] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
<snip>
[2023-02-09 15:53:54.254769] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:53:54.254817] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 19020102: WRITEV 3 (c82860b4-74b5-4eb6-acda-84304e6a362c), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-2-RECON_NO:-2, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:54:53.440023] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:54:53.440064] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 19019040: WRITEV 2 (d4aa8d07-7e36-4cb0-a1f0-d624dd2ca163), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host02.local.lab:engine-client-2-RECON_NO:-2, error-xlator: engine-posix [Invalid argument]]
[2023-02-09 15:54:55.606466] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 95250: WRITEV 3 (d19dc611-913e-420e-b5fe-b78689ef33f2), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host03.local.lab:engine-client-2-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:54:55.606445] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:58:56.276373] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]
[2023-02-09 15:58:56.276418] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 19021567: WRITEV 2 (bd97ab71-7fdd-4359-9e13-bd6f2c606d79), client: CTX_ID:197e1f11-9d15-4019-bf11-21a235f04d67-GRAPH_ID:0-PID:11682-HOST:olvm-kvm-host01.local.lab:engine-client-2-RECON_NO:-2, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:59:55.462743] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 19020541: WRITEV 3 (590a1e96-a3f8-41ae-b85d-d50b8a09a1eb), client: CTX_ID:170011e3-d6cc-48ed-8f61-1ecd657f4f18-GRAPH_ID:0-PID:11503-HOST:olvm-kvm-host02.local.lab:engine-client-2-RECON_NO:-2, error-xlator: engine-posix [Invalid argument]
[2023-02-09 15:59:57.619977] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-engine-server: 96699: WRITEV 3 (b73083cf-159c-4af5-92ed-8d9fdaac0a28), client: CTX_ID:5bfaaff3-59f4-4b68-adfb-40f7234061ad-GRAPH_ID:0-PID:11326-HOST:olvm-kvm-host03.local.labE:engine-client-2-RECON_NO:-0, error-xlator: engine-posix [Invalid argument]
The message "E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument]" repeated 2 times between [2023-02-09 15:58:56.276373] and [2023-02-09 15:59:57.619950]

Changes

 N/A

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.