Two Node RAC Database Hang due the MMNL PROCESS WRITING a HUGE TRACE FILE and CONSUMING the FILE SYSTEM (Doc ID 2798273.1)

Last updated on JULY 20, 2024

Applies to:

Symptoms

On : 19.7.0.0.0 version, RDBMS

ACTUAL BEHAVIOR
---------------
Two node RAC database hang

We encountered some hang issue on our RAC database.
Just prior to this issue, it seemed that the diag filesystem for that database had filled up to 100%, mainly caused by a 13 GB mmnl trace file with the following content:
.
.
kqrhngc ph1: po 0xf7477710 req S from session 0x15c8e5120 lock 0xf25a9210
kqrhngc ph1: blocker #1 owner mode X session 0x160ffd4d0 lock 0x65432220
.
.

We purged this tracefile using adrci, but it got re-created immediately again with the same type of contents.

At that point, the database was completely unresponsive, no connection, even "sqlplus / as sysdba" was possible.

As a solution, we killed the pmon process on instance 2. This recovered the situation.


EXPECTED BEHAVIOR
-----------------------
No Database Hang


BUSINESS IMPACT
-----------------------
The issue has the following business impact:
Due to this issue, database hung.

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.