My Oracle Support Banner

Oracle OpenStack 4.0.1: Re-deploy Fails with Error "no space left on device" on /var/lib/docker/overlay2 Subdirectories (Doc ID 2388306.1)

Last updated on AUGUST 04, 2018

Applies to:

Oracle OpenStack for Oracle Linux - Version OpenStack 3.0.1 and later
Linux x86-64

Symptoms

There was Oracle OpenStack 4.0.1 deployed.
Before re-deploy, environment was cleaned up with "# kollacli host destroy all --includedata --removeimages".
Re-deploy fails with below error:


$ kollacli deploy
(snip)
fatal: [srv2.oracle.com]: FAILED! => {"_ansible_parsed": true, "_ansible_no_log": false, "changed": true, "failed": true, "invocation": {"module_args": {"tls_key": null, "image": "oracle/ol-openstack-nova-compute:4.0", "labels": {}, "tls_verify": false, "tls_cacert": null, "auth_password": null, "graceful_timeout": 10, "environment": {"KOLLA_SERVICE_NAME": "nova-compute", "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "auth_registry": "", "volumes_from": null, "tls_cert": null, "privileged": true, "api_version": "auto", "remove_on_exit": true, "restart_retries": 10, "detach": true, "auth_username": null, "name": "nova_compute", "security_opt": [], "cap_add": [], "restart_policy": "unless-stopped", "auth_email": null, "ipc_mode": "host", "volumes": ["/etc/kolla//nova-compute/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/lib/modules:/lib/modules:ro", "/run:/run:shared", "/dev:/dev", "/boot:/boot", "/sys/fs/cgroup:/sys/fs/cgroup", "/tmp:/tmp", "/var/tmp:/var/tmp", "/var/lock/iscsi:/var/lock/iscsi", "kolla_logs:/var/log/kolla/", "/etc/iscsi:/etc/iscsi", "/var/lib/kolla/var/lib/libvirt:/var/lib/libvirt", "/var/lib/kolla/var/lib/nova:/var/lib/nova/"], "action": "recreate_or_restart_container"}}, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_Nqfl7I/ansible_module_kolla_docker.py\", line 799, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_Nqfl7I/ansible_module_kolla_docker.py\", line 590, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_Nqfl7I/ansible_module_kolla_docker.py\", line 617, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/lib/python2.6/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/lib/python2.6/site-packages/docker/api/container.py\", line 366, in start\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.6/site-packages/docker/client.py\", line 154, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"invalid header field value \\\\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lock/iscsi\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/overlay2/08bfb7e85ca5c8f0d5469a2c53a002216c2952b2c5934a6dad1bf9e6d74b7595/merged\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/overlay2/08bfb7e85ca5c8f0d5469a2c53a002216c2952b2c5934a6dad1bf9e6d74b7595/merged/run/lock/iscsi\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"no space left on device\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\"\\\\\\\\n\\\\\"\"}\")\\n'"}
(snip) 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.