OCFS2 Fail If Cluster Heartbeat Is On iSCSI Device (Doc ID 2735866.1)

Last updated on JULY 03, 2023

Applies to:

Symptoms

In a system where OCFS2 cluster heartbeat use an iSCSI device the "o2cb" service might fail to start during boot.

The following messages are visible in the logs:

Dec 03 11:50:23 <HOST> kernel: ocfs2: Registered cluster interface o2cb
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Loading stack plugin "o2cb": OK
Dec 03 11:50:23 <HOST> kernel: OCFS2 DLMFS 1.8.0
Dec 03 11:50:23 <HOST> kernel: OCFS2 User DLM kernel interface loaded
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Loading filesystem "ocfs2_dlmfs": OK
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Mounting ocfs2_dlmfs filesystem at /dlm: OK
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Setting cluster stack "o2cb": OK
Dec 03 11:50:23 <HOST> kernel: o2hb: Heartbeat mode set to global
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Registering O2CB cluster "cluster01": OK
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Setting O2CB cluster timeouts : OK
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Starting global heartbeat for cluster "cluster01": Failed
Dec 03 11:50:23 <HOST> o2cb.init[1790]: o2cb: Heartbeat region could not be found 8D98CAF4A4BE4312BC3EADA2A556D710
Dec 03 11:50:23 <HOST> o2cb.init[1790]: Stopping global heartbeat on cluster "cluster01": OK
Dec 03 11:50:23 <HOST> systemd[1]: Started Load o2cb Modules.
Dec 03 11:50:23 <HOST> systemd[1]: Starting Load ocfs2 Modules...
Dec 03 11:50:23 <HOST> ocfs2.init[1933]: which: no fuser in (/sbin:/usr/sbin:/bin:/usr/bin)
Dec 03 11:50:23 <HOST> iscsid[1545]: iscsid: login response status 0000
Dec 03 11:50:23 <HOST> iscsid[1545]: iscsid: deleting a scheduled/waiting thread!
Dec 03 11:50:23 <HOST> iscsid[1545]: iscsid: Connection1:0 to [target: iqn.2015-12.com.oracleiaas:61201294-b36b-458d-b009-dffefcb7c13e, portal: 169.254.2.2,3260] through [iface: default] is operational now
Dec 03 11:50:23 <HOST> kernel: scsi 3:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
Dec 03 11:50:23 <HOST> systemd[1]: Started Load ocfs2 Modules.
Dec 03 11:50:23 <HOST> ocfs2.init[1933]: Starting Oracle Cluster File System (OCFS2) mount.ocfs2: Device name specified was not found while opening device /dev/sdb  
Dec 03 11:50:23 <HOST> ocfs2.init[1933]: [FAILED]
Dec 03 11:50:23 <HOST> kernel: scsi 3:0:0:0: Attached scsi generic sg1 type 12
Dec 03 11:50:23 <HOST> kernel: scsi 3:0:0:2: Direct-Access     ORACLE   BlockVolume      1.0  PQ: 0 ANSI: 6
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: Attached scsi generic sg2 type 0
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB)
Dec 03 11:50:23 <HOST> iscsiadm[1789]: Logging in to [iface: default, target: iqn.2015-12.com.oracleiaas:61201294-b36b-458d-b009-dffefcb7c13e, portal: 169.254.2.2,3260] (multiple)
Dec 03 11:50:23 <HOST> iscsiadm[1789]: Login to [iface: default, target: iqn.2015-12.com.oracleiaas:61201294-b36b-458d-b009-dffefcb7c13e, portal: 169.254.2.2,3260] successful.
Dec 03 11:50:23 <HOST> systemd[1]: Started Login and scanning of iSCSI devices.
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: [sdb] 4096-byte physical blocks
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: [sdb] Write Protect is off
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: [sdb] Mode Sense: 2b 00 10 08
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: [sdb] Optimal transfer size 1048576 bytes
Dec 03 11:50:23 <HOST> systemd[1]: Starting Availability of block devices...
Dec 03 11:50:23 <HOST> systemd[1]: Reached target Remote File Systems (Pre).
Dec 03 11:50:23 <HOST> systemd[1]: Started Availability of block devices.
Dec 03 11:50:23 <HOST> kernel: sd 3:0:0:2: [sdb] Attached SCSI disk
Dec 03 11:50:23 <HOST> systemd[1]: Found device BlockVolume ocfs2vol1.
Dec 03 11:50:23 <HOST> systemd[1]: Mounting /ocfs2...
Dec 03 11:50:23 <HOST> kernel: OCFS2 1.8.0
Dec 03 11:50:24 <HOST> kernel: o2cb: Global heartbeat has not been started.
Dec 03 11:50:24 <HOST> kernel: o2cb: Cluster check failed. Fix errors before retrying.
Dec 03 11:50:24 <HOST> kernel: (mount.ocfs2,1979,0):ocfs2_dlm_init:3348 ERROR: status = -22
Dec 03 11:50:24 <HOST> kernel: (mount.ocfs2,1979,0):ocfs2_mount_volume:1807 ERROR: status = -22
Dec 03 11:50:24 <HOST> kernel: (mount.ocfs2,1979,0):ocfs2_fill_super:1175 ERROR: status = -22
Dec 03 11:50:24 <HOST> kernel: ocfs2: Unmounting device (8,16) on (node 0)
Dec 03 11:50:24 <HOST> mount[1976]: mount.ocfs2: Invalid argument while mounting /dev/sdb on /ocfs2. Check 'dmesg' for more information on this error.
Dec 03 11:50:24 <HOST> systemd[1]: ocfs2.mount mount process exited, code=exited status=1
Dec 03 11:50:24 <HOST> systemd[1]: Failed to mount /ocfs2.
Dec 03 11:50:24 <HOST> systemd[1]: Dependency failed for Remote File Systems.
Dec 03 11:50:24 <HOST> systemd[1]: Job remote-fs.target/start failed with result 'dependency'.
Dec 03 11:50:24 <HOST> systemd[1]: Unit ocfs2.mount entered failed state.

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.