Oracle OpenStack 4.0.1: Fail to Create Volume from Image via Cinder ZFSSA ISCSI Backend: Failed to copy image to volume: Volume device not found at

(Doc ID 2403740.1)

Last updated on JUNE 01, 2018

Applies to:

Oracle OpenStack for Oracle Linux - Version OpenStack 4.0.1 and later
Linux x86-64

Symptoms

Create a 15GB volume from image failed:
2018-05-25 14:19:05.335 32 WARNING os_brick.initiator.connectors.iscsi [req-2db06b87-5fdd-4169-92ea-e308ca85dfa3 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Couldn't find iscsi sessions because iscsiadm err: iscsiadm: No active sessions.
: VolumeDeviceNotFound: Volume device not found at .
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume [req-2db06b87-5fdd-4169-92ea-e308ca85dfa3 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Failed to copy image 848c3fd9-77f0-4f0f-a882-58517f1c0119 to volume: 2227cb63-4794-4926-b32a-3b0708f07f85: VolumeDeviceNotFound: Volume device not found at .
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 555, in _copy_image_to_v
olume
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/driver.py", line 791, in copy_image_to_volume
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id, encrypted=False)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/driver.py", line 811, in _copy_image_data_to_volume
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume attach_info, volume = self._attach_volume(context, volume, properties)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/driver.py", line 995, in _attach_volume
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume attach_info = self._connect_device(conn)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/driver.py", line 1066, in _connect_device
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume device = connector.connect_volume(conn['data'])
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 506, in connect_volume
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume self._cleanup_connection(connection_properties, force=True)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume self.force_reraise()
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume six.reraise(self.type_, self.value, self.tb)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 500, in connect_volume
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume return self._connect_single_volume(connection_properties)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume return r.call(f, *args, **kwargs)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 212, in call
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume raise attempt.get()
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 247, in get
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume six.reraise(self.value[0], self.value[1], self.value[2])
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 200, in call
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 575, in _connect_single_volume
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume raise exception.VolumeDeviceNotFound(device='')
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume VolumeDeviceNotFound: Volume device not found at .
2018-05-25 14:19:14.114 32 ERROR cinder.volume.flows.manager.create_volume
2018-05-25 14:19:14.453 32 WARNING cinder.volume.manager [req-2db06b87-5fdd-4169-92ea-e308ca85dfa3 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (861c8df3-cb41-405e-a2f0-6af6fb9a43e9) transitioned into state 'FAILURE' from state 'RUNNING'

Below error might also be seen:
2018-05-26 11:33:50.046 32 WARNING cinder.volume.manager [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e4cb7c80-e5c5-442c-afaa-fdf966044350) transitioned into state 'REVERTED' from state 'REVERTING'
2018-05-26 11:33:50.048 32 WARNING cinder.volume.manager [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ddd808e5-4c3d-446d-a79b-b11103d15b82) transitioned into state 'REVERTED' from state 'REVERTING'
2018-05-26 11:33:50.050 32 WARNING cinder.volume.manager [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a2d4fdf5-3707-4960-aea3-399f06310891) transitioned into state 'REVERTED' from state 'REVERTING'
2018-05-26 11:33:50.056 32 ERROR cinder.volume.flows.manager.create_volume [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Volume c9628842-8b13-4117-b79d-a8dbcdb5c14f: create failed
2018-05-26 11:33:50.057 32 WARNING cinder.volume.manager [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (116b15c6-0d14-44c2-b3d4-129921864e5a) transitioned into state 'REVERTED' from state 'REVERTING'
2018-05-26 11:33:50.058 32 WARNING cinder.volume.manager [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cc2a9137-fb61-461f-b3b6-a5b70b414e6d) transitioned into state 'REVERTED' from state 'REVERTING'
2018-05-26 11:33:50.059 32 WARNING cinder.volume.manager [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Flow 'volume_create_manager' (af3c56c0-32b3-4d79-940e-61769e87093a) transitioned into state 'REVERTED' from state 'RUNNING'
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server [req-7439faaa-c1b9-4a4f-9c16-16f9ae849302 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Exception during message handling: ImageCopyFailure: Failed to copy image to volume: Volume device not found at .
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "<decorator-gen-237>", line 2, in create_volume
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 651, in create_volume
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server _run_flow()
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 643, in _run_flow
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server flow_engine.run()
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server failures[0].reraise()
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 951, in execute
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server **volume_spec)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 898, in _create_from_image
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server image_service)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "<decorator-gen-236>", line 2, in _create_from_image_cache_or_download
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/coordination.py", line 176, in _synchronized
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server return f(*a, **k)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 797, in _create_from_image_cache_or_download
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server image_service
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 698, in _create_from_image_download
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server image_location, image_service)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 575, in _copy_image_to_volume
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server raise exception.ImageCopyFailure(reason=ex)
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server ImageCopyFailure: Failed to copy image to volume: Volume device not found at .
2018-05-26 11:33:50.064 32 ERROR oslo_messaging.rpc.server

15GB LUN was initiated on ZFSSA corresponding Project, however zero size.

Create an empty 1GB volume won't have same problem:
2018-05-26 11:47:37.891 32 INFO cinder.volume.flows.manager.create_volume [req-a941c0d3-c738-422b-912b-203f0986edf8 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Volume 86df297e-30d2-4dea-8795-3d74bb7bd2cf: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-86df297e-30d2-4dea-8795-3d74bb7bd2cf'}
2018-05-26 11:47:42.067 32 INFO cinder.volume.flows.manager.create_volume [req-a941c0d3-c738-422b-912b-203f0986edf8 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Volume volume-86df297e-30d2-4dea-8795-3d74bb7bd2cf (86df297e-30d2-4dea-8795-3d74bb7bd2cf): created successfully
2018-05-26 11:47:42.070 32 INFO cinder.volume.manager [req-a941c0d3-c738-422b-912b-203f0986edf8 1a08a0dfedbf4704b5af0788e06a327d a30e963736c747c694a9998060459c40 - default default] Created volume successfully.

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms