Process waiting on event 'KSV Master Wait' 'ASM Metadata File Operation' with DBFS on the path (Doc ID 1921833.1)

Last updated on JUNE 28, 2023

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.1.0.2 [Release 11.2 to 12.1]
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Information in this document applies to any platform.

Symptoms

DBFS mount points
Databases reporting hang issues, users not able to connect to database.

Tue Aug 12 16:35:49 2014
DIA0 (ospid: 108661) waits for event 'ASM file metadata operation' for 333 secs.
Errors in file /u01/app/oracle/diag/rdbms/dbfs/dbfs1/trace/dbfs1_lmhb_108682.trc (incident=15746):
ORA-29770: global enqueue process DIA0 (OSID 108661) is hung for more than 300 seconds
Incident details in: /u01/app/oracle/diag/rdbms/dbfs/dbfs1/incident/incdir_15746/dbfs1_lmhb_108682_i15746.trc
DIA0 (ospid: 108661) is hanging without a blocker.
LMHB (ospid: 108682) kills DIA0 (ospid: 108661)

Errors related to dbfs reported on /var/log/messages file

Aug 12 17:02:57 db01 ntpd[11551]: synchronized to 10.24.64.1, stratum 2
Aug 12 17:05:56 db01 dbfs_client[33636]: OCI_ERROR 3113 - ORA-03113: end-of-file on communication channel Process ID: 42887 Session ID: 9 Serial number: 1
Aug 12 17:05:56 db01 dbfs_client[33636]: OCI_ERROR 3113 - ORA-03113: end-of-file on communication channel Process ID: 42887 Session ID: 9 Serial number: 1
Aug 12 17:05:56 db01 dbfs_client[33636]: OCI_ERROR 3113 - ORA-03113: end-of-file on communication channel Process ID: 42887 Session ID: 9 Serial number: 1
Aug 12 17:05:56 db01 dbfs_client[33636]: OCI_ERROR 3113 - ORA-03113: end-of-file on communication channel Process ID: 42887 Session ID: 9 Serial number: 1

Processes on the database waiting for event 'ASM File Metadata Operations'
Some processes on the database reporting ORA-600 error

ORA-00600: internal error code, arguments: [4005],

In file /var/log/messages across multiple nodes, some processes were in un-interrupted mode ("D") mode for more than 120 seconds.

Aug 12 16:30:10 db01 kernel: INFO: task oracle:28732 blocked for more than 120 seconds.
Aug 12 16:30:10 db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 12 16:30:10 db01 kernel: oracle D ffffffff8151d620 0 28732 1 0x00000080
Aug 12 16:30:10 db01 kernel: ffff88268a953c38 0000000000000086 00000000000001fb 0000000000012180
Aug 12 16:30:10 db01 kernel: ffff880fd8bee040 ffff883f0555a300 ffff88268a953c28 ffffffffa01ed88b
Aug 12 16:30:10 db01 kernel: 0000000000000202 ffff88407ffd9e00 0000000053ea78c2 00000000049e4bd8
Aug 12 16:30:10 db01 kernel: Call Trace:
Aug 12 16:30:10 db01 kernel: [<ffffffffa01ed88b>] ? fuse_dentry_revalidate+0x3b/0x200 [fuse]
Aug 12 16:30:10 db01 kernel: [<ffffffff811151ca>] ? prep_new_page+0x10a/0x150
Aug 12 16:30:10 db01 kernel: [<ffffffff815047f5>] schedule+0x45/0x60
Aug 12 16:30:10 db01 kernel: [<ffffffff81505466>] __mutex_lock_slowpath+0xd6/0x150
Aug 12 16:30:10 db01 kernel: [<ffffffff8150656e>] ? _raw_spin_lock+0xe/0x20
Aug 12 16:30:10 db01 kernel: [<ffffffff81187dae>] ? vfsmount_lock_local_unlock+0x1e/0x30
Aug 12 16:30:10 db01 kernel: [<ffffffff815050fb>] mutex_lock+0x2b/0x50
Aug 12 16:30:10 db01 kernel: [<ffffffff811788fc>] do_lookup+0x1bc/0x2a0
Aug 12 16:30:10 db01 kernel: [<ffffffff8117a56e>] path_lookupat+0xee/0x700
Aug 12 16:30:10 db01 kernel: [<ffffffff8150656e>] ? _raw_spin_lock+0xe/0x20
Aug 12 16:30:10 db01 kernel: [<ffffffff81175e55>] ? path_get+0x45/0x60
Aug 12 16:30:10 db01 kernel: [<ffffffff8117abb5>] do_path_lookup+0x35/0xd0
Aug 12 16:30:10 db01 kernel: [<ffffffff8117b6e9>] user_path_at_empty+0x59/0xa0
Aug 12 16:30:10 db01 kernel: [<ffffffff8117b741>] user_path_at+0x11/0x20
Aug 12 16:30:10 db01 kernel: [<ffffffff8116a3ef>] sys_faccessat+0xbf/0x1b0
Aug 12 16:30:10 db01 kernel: [<ffffffff8116a4f8>] sys_access+0x18/0x20
Aug 12 16:30:10 db01 kernel: [<ffffffff8150ed42>] system_call_fastpath+0x16/0x1b

Changes

None

Cause

	To view full details, sign in with your My Oracle Support account.
	Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.

Process waiting on event 'KSV Master Wait' 'ASM Metadata File Operation' with DBFS on the path (Doc ID 1921833.1)

Applies to:

Symptoms

Changes

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!