My Oracle Support Banner

OLVM: SHE Storage Domain Became Invalid Leads to Self-Hosted Engine VM Shut Down (Doc ID 2812130.1)

Last updated on OCTOBER 06, 2021

Applies to:

Linux OS - Version Oracle Linux 7.9 with Unbreakable Enterprise Kernel [5.4.17] and later
Linux x86-64

Symptoms

The Self-Hosted Engine VM automatically shuts down intermittently.

A warning was logged for Data Center status in engine log:
2021-08-23 02:47:29,516-10 WARN ... [1539658] EVENT_ID: SYSTEM_CHANGE_STORAGE_POOL_STATUS_PROBLEMATIC(980), Invalid status on Data Center <DC Name>. Setting status to Non Responsive.

Then SHE went down:
2021-08-23 02:47:34,989-10 INFO ... EVENT_ID: VM_DOWN(61), VM HostedEngine is down. Exit message: User shut down from within the guest

SPM vdsm.log indicates NFS based Storage Domain had connection timeout and became invalid:
2021-08-23 02:44:42,975-1000 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/<NFS Server>:<Storage Domain Export>/<UUID>/dom_md/metadata (monitor:499)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 497,
in _pathChecked delay = result.delay()
File "/usr/lib/python2.7/site-packages/vdsm/storage/check.py", line 391, in
delay raise exception.MiscFileReadException(self.path, self.rc, self.err)
MiscFileReadException: Internal file read failure:
(u'/rhev/data-center/mnt/<NFS Server>:<Storage Domain Export>/<UUID>/dom_md/metadata', 1, 'Read timeout')
2021-08-23 02:44:42,976-1000 INFO (check/loop) [storage.Monitor] Domain <UUID> became INVALID (monitor:470)
6753:2021-08-23 02:44:52,975-1000 WARN (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/<NFS Server>:<Storage Domain Export>/<UUID>/dom_md/metadata' is blocked for 20.00 seconds (check:282)

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.