OLVM: VM Has Been Paused Due to Storage Space Error (Doc ID 2713907.1)

Last updated on APRIL 10, 2023

Applies to:

Symptoms

Symptom 1,  Below event can be seen in OLVM's main event table, 

"VM <VM name>has paused due to no Storage space error."

Symptom 2,  in engine log, VM state transitions from pause to up repeatedly, 

$ egrep "VM_PAUSED_ENOSPC|-->" engine.log | grep <Virtual Machine>
2020-09-11 05:21:53,503-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-66) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Up' --> 'Paused'
2020-09-11 05:21:53,552-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-66) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 05:22:23,583-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-96) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'Up'
2020-09-11 05:22:38,638-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Up' --> 'Paused'
2020-09-11 05:22:38,684-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 05:23:08,764-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'Up'
2020-09-11 05:23:23,824-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Up' --> 'Paused'
2020-09-11 05:23:23,838-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 05:23:53,882-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-42) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'Up'
2020-09-11 05:24:23,959-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Up' --> 'Paused'
2020-09-11 05:24:24,004-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 05:26:39,234-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'Up'
2020-09-11 08:26:06,383-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Up' --> 'Paused'
2020-09-11 08:26:06,427-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 08:26:51,477-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'Up'
2020-09-11 08:27:21,540-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Up' --> 'NotResponding'
2020-09-11 08:28:06,615-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'NotResponding' --> 'Paused'
2020-09-11 08:28:06,652-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 08:28:21,669-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'NotResponding'
2020-09-11 08:28:36,720-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'NotResponding' --> 'Paused'
2020-09-11 08:28:36,732-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 08:29:06,762-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-42) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'Up'
2020-09-11 08:30:06,864-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-59) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Up' --> 'Paused'
2020-09-11 08:30:06,902-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-59) [] EVENT_ID: VM_PAUSED_ENOSPC(138), VM <Virtual Machine> has been paused due to no Storage space error.
2020-09-11 08:32:07,040-07 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-50) [] VM '<Virtual Machine UUID>'(<Virtual Machine>) moved from 'Paused' --> 'Up'
$

Symptom 3, in vdsm-error.log, VM's virtual disk is extended multiple times in changing from "Paused" and "Up" state, 

$ grep 1cd894ef-2e12-4dd8-bbfc-13724eb78870  vdsm-error.log | egrep " Requesting extension|Extend volume|onIOError|onResume|threshold exceeded"  |grep -v START | grep -v FINISH | less

2020-09-11 08:26:01,495-0700 INFO  (periodic/111) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 99857989632, capacity: 2197949513728, allocated: 99865264128, physical: 99857989632 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:02,624-0700 INFO  (mailbox-hsm/0) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Extend volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 completed <Clock(total=3.14, extend-volume=2.72, refresh-volume=0.42)> (vm:1400)
2020-09-11 08:26:02,740-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU running: onResume (vm:6048)
2020-09-11 08:26:02,930-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') drive 'sdb' threshold exceeded (storage:874)
2020-09-11 08:26:03,172-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:03,222-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 100931731456, capacity: 2197949513728, allocated: 100931862528, physical: 100931731456 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:03,237-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:03,238-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 100931731456, capacity: 2197949513728, allocated: 100931862528, physical: 100931731456 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:03,244-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:03,245-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 100931731456, capacity: 2197949513728, allocated: 100931862528, physical: 100931731456 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:03,490-0700 INFO  (periodic/119) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 100931731456, capacity: 2197949513728, allocated: 100931862528, physical: 100931731456 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:05,500-0700 INFO  (periodic/111) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 100931731456, capacity: 2197949513728, allocated: 100931862528, physical: 100931731456 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:07,451-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') drive 'sdb' threshold exceeded (storage:874)
2020-09-11 08:26:07,488-0700 INFO  (periodic/119) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 102005473280, capacity: 2197949513728, allocated: 101643845632, physical: 102005473280 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:07,693-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:07,731-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 102005473280, capacity: 2197949513728, allocated: 102008815616, physical: 102005473280 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:07,738-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:07,740-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 102005473280, capacity: 2197949513728, allocated: 102008815616, physical: 102005473280 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:07,744-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:07,746-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 102005473280, capacity: 2197949513728, allocated: 102008815616, physical: 102005473280 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:07,751-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:07,753-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 102005473280, capacity: 2197949513728, allocated: 102008815616, physical: 102005473280 threshold_state: exceeded) (vm:1277)
...

2020-09-11 08:26:07,759-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 102005473280, capacity: 2197949513728, allocated: 102008815616, physical: 102005473280 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:07,764-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:07,765-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 102005473280, capacity: 2197949513728, allocated: 102008815616, physical: 102005473280 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:07,771-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:07,773-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain
...

2020-09-11 08:26:11,686-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 103079215104, capacity: 2197949513728, allocated: 103079936000, physical: 103079215104 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:13,500-0700 INFO  (periodic/118) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 103079215104, capacity: 2197949513728, allocated: 103079936000, physical: 103079215104 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:14,795-0700 INFO  (mailbox-hsm/2) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Extend volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 completed <Clock(total=3.30, extend-volume=2.93, refresh-volume=0.37)> (vm:1400)
2020-09-11 08:26:14,908-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU running: onResume (vm:6048)


2020-09-11 08:26:15,085-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') drive 'sdb' threshold exceeded (storage:874)
2020-09-11 08:26:15,521-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:26:15,545-0700 INFO  (periodic/111) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 104152956928, capacity: 2197949513728, allocated: 104156561408, physical: 104152956928 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:15,547-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 104152956928, capacity: 2197949513728, allocated: 104156561408, physical: 104152956928 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:15,555-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
...

2020-09-11 08:26:17,495-0700 INFO  (periodic/117) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 104152956928, capacity: 2197949513728, allocated: 104156561408, physical: 104152956928 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:18,918-0700 INFO  (mailbox-hsm/4) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Extend volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 completed <Clock(total=3.37, extend-volume=2.95, refresh-volume=0.42)> (vm:1400)
2020-09-11 08:26:19,090-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU running: onResume (vm:6048)


2020-09-11 08:26:19,390-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') drive 'sdb' threshold exceeded (storage:874)
2020-09-11 08:26:19,488-0700 INFO  (periodic/118) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 105226698752, capacity: 2197949513728, allocated: 104890433536, physical: 105226698752 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:19,678-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
...

2020-09-11 08:26:21,501-0700 INFO  (periodic/117) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 105226698752, capacity: 2197949513728, allocated: 105227091968, physical: 105226698752 threshold_state: exceeded) (vm:1277)
2020-09-11 08:26:23,011-0700 INFO  (mailbox-hsm/0) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Extend volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 completed <Clock(total=3.53, extend-volume=3.08, refresh-volume=0.44)> (vm:1400)
2020-09-11 08:26:23,175-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU running: onResume (vm:6048)
....

2020-09-11 08:31:54,006-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 161061273600, capacity: 2197949513728, allocated: 161061797888, physical: 161061273600 threshold_state: exceeded) (vm:1277)
2020-09-11 08:31:54,010-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU stopped: onIOError (vm:6048)
2020-09-11 08:31:54,012-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 161061273600, capacity: 2197949513728, allocated: 161061797888, physical: 161061273600 threshold_state: exceeded) (vm:1277)
2020-09-11 08:31:55,525-0700 INFO  (periodic/111) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Requesting extension for volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 on domain 312f8234-3cc3-4e17-b3fa-3b20f0896886 (apparent: 161061273600, capacity: 2197949513728, allocated: 161061797888, physical: 161061273600 threshold_state: exceeded) (vm:1277)
2020-09-11 08:31:59,381-0700 INFO  (mailbox-hsm/4) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') Extend volume 9b0b3b45-763d-4b27-87b6-4a83ea8fd3f9 completed <Clock(total=5.50, extend-volume=5.20, refresh-volume=0.29)> (vm:1400)
2020-09-11 08:31:59,550-0700 INFO  (libvirt/events) [virt.vm] (vmId='1cd894ef-2e12-4dd8-bbfc-13724eb78870') CPU running: onResume (vm:6048)
(END)

Symptom 4, the disk hitting ENOSPC error is a thin provisioned virtual disk on iSCSI or FC storage domain.

Symptom 5, The VM or the application in guest OS is writing to the same virtual disk. 

Changes

This issue could happen when VM is running normally. 

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.