Oracle Openstack 4.0.1: Redeploy Might Fail with "no space left on device" Error on Certain Subvolumes under /var/lib/docker

(Doc ID 2356375.1)

Last updated on APRIL 18, 2018

Applies to:

Oracle OpenStack for Oracle Linux - Version OpenStack 4.0.1 and later
Linux x86-64

Symptoms

If there was Oracle Openstack 4.0.1 deployed, deploy again might have below error reported:


$ kollacli deploy
(snip)
PLAY RECAP *********************************************************************
o3lr2ctlr1.cn.oracle.com : ok=258 changed=135 unreachable=0 failed=0 ignored=0
o3lr2ovs1.cn.oracle.com : ok=55 changed=33 unreachable=0 failed=1 ignored=0
o3lr2kvm1.cn.oracle.com : ok=83 changed=48 unreachable=0 failed=0 ignored=0
o3lr2net1.cn.oracle.com : ok=342 changed=205 unreachable=0 failed=0 ignored=0
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/kollacli/commands/deploy.py", line 103, in take_action
.format(msg=job.get_error_message()))
CommandError: ERROR: Job failed:
Host: o3lr2ovs1.cn.oracle.com, Task: nova : Restart nova-compute container, Status: failed, Message: 'Traceback (most recent call last):\n File "/tmp/ansible_Yymhya/ansible_module_kolla_docker.py", line 799, in main\n result = bool(getattr(dw, module.params.get(\'action\'))())\n File "/tmp/ansible_Yymhya/ansible_module_kolla_docker.py", line 590, in recreate_or_restart_container\n self.start_container()\n File "/tmp/ansible_Yymhya/ansible_module_kolla_docker.py", line 617, in start_container\n self.dc.start(container=self.params.get(\'name\'))\n File "/usr/lib/python2.6/site-packages/docker/utils/decorators.py", line 21, in wrapped\n return f(self, resource_id, *args, **kwargs)\n File "/usr/lib/python2.6/site-packages/docker/api/container.py", line 366, in start\n self._raise_for_status(res)\n File "/usr/lib/python2.6/site-packages/docker/client.py", line 154, in _raise_for_status\n raise errors.APIError(e, response, explanation=explanation)\nAPIError: 500 Server Error: Internal Server Error ("{"message":"invalid header field value \\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lock/iscsi\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/btrfs/subvolumes/e39e0e2e57e60e008598348725698ef55bfcfd5d0b2f077d5e876838c1d412b1\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/btrfs/subvolumes/e39e0e2e57e60e008598348725698ef55bfcfd5d0b2f077d5e876838c1d412b1/run/lock/iscsi\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"no space left on device\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\"\\\\\\"\\\\n\\""}")\n'

 

Though we /var has plenty of disk space left atually:


[root@o3lr2ovs1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 50G 2.7G 44G 6% /
tmpfs 501M 0 501M 0% /dev/shm
/dev/sda1 477M 250M 198M 56% /boot
/dev/sda4 874G 5.6G 867G 1% /var
none 501M 40K 501M 1% /var/lib/xenstored

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms