OLVM: SHE Storage Domain Became Invalid Leads to Self-Hosted Engine VM Shut Down (Doc ID 2812130.1)

Last updated on MAY 19, 2024

Applies to:

Symptoms

The Self-Hosted Engine VM automatically shuts down intermittently.

A warning was logged for Data Center status in engine log:
2021-08-23 02:47:29,516-10 WARN ... [1539658] EVENT_ID: SYSTEM_CHANGE_STORAGE_POOL_STATUS_PROBLEMATIC(980), Invalid status on Data Center <DC Name>. Setting status to Non Responsive.

Then SHE went down:
2021-08-23 02:47:34,989-10 INFO ... EVENT_ID: VM_DOWN(61), VM HostedEngine is down. Exit message: User shut down from within the guest

SPM vdsm.log indicates NFS based Storage Domain had connection timeout and became invalid:
2021-08-23 02:44:42,975-1000 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/<NFS Server>:<Storage Domain Export>/<UUID>/dom_md/metadata (monitor:499)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 497,
in _pathChecked delay = result.delay()
File "/usr/lib/python2.7/site-packages/vdsm/storage/check.py", line 391, in
delay raise exception.MiscFileReadException(self.path, self.rc, self.err)
MiscFileReadException: Internal file read failure:
(u'/rhev/data-center/mnt/<NFS Server>:<Storage Domain Export>/<UUID>/dom_md/metadata', 1, 'Read timeout')
2021-08-23 02:44:42,976-1000 INFO (check/loop) [storage.Monitor] Domain <UUID> became INVALID (monitor:470)
6753:2021-08-23 02:44:52,975-1000 WARN (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/<NFS Server>:<Storage Domain Export>/<UUID>/dom_md/metadata' is blocked for 20.00 seconds (check:282)

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.