Oracle Openstack 3.0.1: OpenStack Deployment failed at "Host: XXXXXXX, Task: mysqlcluster : Waiting for bootstrap container, Status: failed, Message: 1"

(Doc ID 2317690.1)

Last updated on NOVEMBER 13, 2017

Applies to:

Oracle OpenStack for Oracle Linux - Version OpenStack 3.0.1 to OpenStack 3.0.1 [Release ODOS3]
Linux x86-64

Symptoms

 Your multiple-host Oracle OpenStack 3.0.1 deployment failed with below errors in /var/log/kolla/kolla.log,

[2017-09-13 16:25:31,568] INFO kollacli.common.ansible.job TASK [mysqlcluster : Waiting for bootstrap container] **************************
[2017-09-13 17:26:03,552] INFO kollacli.common.ansible.job fatal: [openstackctr0]: FAILED! => {"cmd": ["docker", "wait", "bootstrap_mysqlcluster"], "end": "2017-09-13 17:26:03.270047", "_ansible_no_log": false, "stdout": "1", "changed": true, "rc": 0, "start": "2017-09-13 16:25:30.992815", "stderr": "", "delta": "1:00:32.277232", "invocation": {"module_name": "command", "module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "docker wait bootstrap_mysqlcluster", "removes": null, "creates": null, "chdir": null}}, "stdout_lines": ["1"], "failed_when_result": true, "warnings": [], "failed": true}
[2017-09-13 17:26:03,552] INFO kollacli.common.ansible.job
[2017-09-13 17:26:03,552] INFO kollacli.common.ansible.job PLAY RECAP *********************************************************************
[2017-09-13 17:26:03,552] INFO kollacli.common.ansible.job openstackcmp0 : ok=38 changed=14 unreachable=0 failed=0 ignored=0
[2017-09-13 17:26:03,552] INFO kollacli.common.ansible.job openstackctr0 : ok=116 changed=48 unreachable=0 failed=1 ignored=0
[2017-09-13 17:26:03,552] INFO kollacli.common.ansible.job openstackctr1 : ok=105 changed=45 unreachable=0 failed=0 ignored=0
[2017-09-13 17:26:03,578] ERROR kollacli Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/kollacli/commands/deploy.py", line 103, in take_action
.format(msg=job.get_error_message()))
CommandError: ERROR: Job failed:
Host: openstackctr0, Task: mysqlcluster : Waiting for bootstrap container, Status: failed, Message: 1

 

The mentioned bootstrap_mysql container's log looks like below

[root@openstackctr0 ansible]# docker logs -f bootstrap_mysqlcluster
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Copying /var/lib/kolla/config_files/mysqlcluster-api.my.cnf to /etc/my.cnf
INFO:__main__:Setting permissions for /etc/my.cnf
INFO:__main__:Writing out command to execute
+++ pre-start mysqld +++
+++ mysqld PID = 10 +++
+++ waiting for mysqld to start +++
2017-10-14 22:42:32 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-10-14 22:42:32 0 [Note] /usr/sbin/mysqld (mysqld 5.6.34-ndb-7.4.13-cluster-gpl) starting as process 10 ...
2017-10-14 22:42:32 10 [Note] Plugin 'FEDERATED' is disabled.
2017-10-14 22:42:32 10 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-10-14 22:42:32 10 [Note] InnoDB: The InnoDB memory heap is disabled
2017-10-14 22:42:32 10 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-10-14 22:42:32 10 [Note] InnoDB: Memory barrier is not used
2017-10-14 22:42:32 10 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-10-14 22:42:32 10 [Note] InnoDB: Using Linux native AIO
2017-10-14 22:42:32 10 [Note] InnoDB: Using CPU crc32 instructions
2017-10-14 22:42:32 10 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-10-14 22:42:32 10 [Note] InnoDB: Completed initialization of buffer pool
2017-10-14 22:42:32 10 [Note] InnoDB: Highest supported file format is Barracuda.
2017-10-14 22:42:32 10 [Note] InnoDB: 128 rollback segment(s) are active.
2017-10-14 22:42:32 10 [Note] InnoDB: Waiting for purge to start
2017-10-14 22:42:32 10 [Note] InnoDB: 5.6.34 started; log sequence number 1625987
2017-10-14 22:42:32 10 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
+++ waiting for mysqld to start +++
2017-10-14 22:43:03 10 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
2017-10-14 22:43:03 10 [Note] NDB Binlog: Starting...
2017-10-14 22:43:03 10 [Note] NDB Binlog: Started
2017-10-14 22:43:03 10 [Note] NDB Binlog: Setting up
2017-10-14 22:43:03 10 [Note] NDB Util: Starting...
2017-10-14 22:43:03 10 [Note] NDB Util: Wait for server start completed
2017-10-14 22:43:03 10 [Note] NDB Index Stat: Starting...
2017-10-14 22:43:03 10 [Note] NDB Index Stat: Wait for server start completed
2017-10-14 22:43:03 10 [Note] NDB Binlog: Created schema Ndb object, reference: 0x0, name: 'Ndb Binlog schema change monitoring'
2017-10-14 22:43:03 10 [Note] NDB Binlog: Created injector Ndb object, reference: 0x0, name: 'Ndb Binlog data change monitoring'
2017-10-14 22:43:03 10 [Note] NDB Binlog: Setup completed
2017-10-14 22:43:03 10 [Note] NDB Binlog: Wait for server start completed
2017-10-14 22:43:03 10 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 0a28182f-b131-11e7-a112-0021f6156c7d.
2017-10-14 22:43:04 10 [Note] Server hostname (bind-address): '192.168.252.101'; port: 3306
2017-10-14 22:43:04 10 [Note] - '192.168.252.101' resolves to '192.168.252.101';
2017-10-14 22:43:04 10 [Note] Server socket created on IP: '192.168.252.101'.
2017-10-14 22:43:04 10 [Note] Event Scheduler: Loaded 0 events
2017-10-14 22:43:04 10 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.34-ndb-7.4.13-cluster-gpl' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Cluster Community Server (GPL)
2017-10-14 22:43:04 10 [Note] NDB Util: Wait for cluster to start
2017-10-14 22:43:04 10 [Note] NDB Index Stat: Wait for cluster to start
2017-10-14 22:43:04 10 [Note] NDB Binlog: Check for incidents
2017-10-14 22:43:04 10 [Note] NDB Binlog: Wait for cluster to start
2017-10-14 23:43:05 10 [Warning] NDB : Tables not available after 3600 seconds. Consider increasing --ndb-wait-setup value

 

If you also check mysqlcluster_mgmt" container's log,  you will see below,

2017-10-14 22:56:07 [ndb_config] ERROR -- at line 47: At least one database node (ndbd) should be defined in config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Removing existing destination: /var/lib/mysql-cluster/config.ini
INFO:__main__:Copying /var/lib/kolla/config_files/mysqlcluster-mgmt.config.ini to /var/lib/mysql-cluster/config.ini
INFO:__main__:Setting permissions for /var/lib/mysql-cluster/config.ini
INFO:__main__:Writing out command to execute
2017-10-14 23:09:47 [ndb_config] ERROR -- at line 47: At least one database node (ndbd) should be defined in config file

 

Changes

You are  deploying your Oracle Openstack cluster in this way,


In host view 

  

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms