OVM - OCFS2 FS Corruption Cause The OVS Server To Hang And Get D State Processes

Last updated on JUNE 21, 2018

Applies to:

Symptoms

• A sosreport can not be collected due to activee D state processes.
• It is not possible to run/execute storage related commands such as: lsblk, df, fdisk because they end up hang on active ssh session.
• D state processes present in the system.

Example:

# ps -e -o pid,stat,comm,wchan=WIDE-WCHAN-COLUMN | grep D
PID STAT COMMAND WIDE-WCHAN-COLUMN
8648 D jbd2/dm-3-616 jbd2_journal_commit_transaction
8649 D ocfs2cmt call_rwsem_down_write_failed
9571 D< loop1 lock_page
10769 D< loop6 wait_transaction_locked
11874 D< loop11 wait_transaction_locked
12546 D< loop14 jbd2_log_wait_commit
15738 D< loop28 wait_transaction_locked
17434 D< loop33 wait_transaction_locked
19437 D 6.xvda-0 bt_get
25661 D df ocfs2_cluster_lock

• After fsck command execution it is validated the file-system of an OCFS2 repository isn't in good shape and presents level of file-system corruption:

# fsck.ocfs2 -fn /dev/mapper/<Repository-ID> 

Pass 1: Checking inodes and blocks
Duplicate clusters detected. Pass 1b will be run
[CLUSTER_ALLOC_BIT] Cluster 296723 is marked in the global cluster bitmap but it isn't in use. Clear its bit in the bitmap? n <<<<<<<<<<<<
.
<>----LOG-SNIP----<>
.
Pass 2: Checking directory entries
[DX_TREE_MISSING] Directory 514 is missing index. Rebuild? n <<<<<<<<<<<<

•   Local File-System is hung.
  
• VMs Presents performance issues.

Changes

 No recent changes.

Cause