Huge Number Of Files Created Under /u01/app/grid/oradiag_grid/diag/clients/user_grid/host_XXXXX cause gi cannot startup (Doc ID 2295143.1)

Last updated on SEPTEMBER 15, 2017

Applies to:

Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

 Huge Number Of Files Created Under /u01/app/grid/oradiag_grid/diag/clients/user_grid/host_XXXXX on one node. (Usually the oakd master node)

It will cause tfa or other data collection tools nearly hang or the node run out of inode and showing out of space. And GI/ASM may cannot startup.

Here is one example GI cannot startup:

crsctl stat res -init  

showing:

NAME=ora.crsd
TYPE=ora.crs.type
TARGET=ONLINE
STATE=INTERMEDIATE on odax5rm2-base

 

Also ASM and storage showing online but 

ps -ef|grep smon

cannot find out ASM process.

Try to log into asm and hit below error:

$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Fri Sep 15 12:22:26 2017

Copyright (c) 1982, 2014, Oracle. All rights reserved.

ERROR:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925

 

$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 57060636 42932984 11222412 80% /
tmpfs 65879408 0 65879408 0% /dev/shm
/dev/xvda1 462820 67036 371464 16% /boot
/dev/xvdb1 95988492 71225684 19880120 79% /u01

showing /u01 did have enough space.

But:

$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda2 3637248 61310 3575938 2% /
tmpfs 16469852 1 16469851 1% /dev/shm
/dev/xvda1 121920 46 121874 1% /boot
/dev/xvdb1 6111232 6111232 0 100% /u01

 

--- So it is /u01 run out of inode which cause asm showing run out of space.

 

Clear the files at: /u01/app/grid/oradiag_grid/diag/clients/user_grid/host_XXXXX/trace/

[root@odax5rm2-base ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda2 3637248 61310 3575938 2% /
tmpfs 16469852 1 16469851 1% /dev/shm
/dev/xvda1 121920 46 121874 1% /boot
/dev/xvdb1 6111232 5343835 767397 88% /u01

 

After that GI can restart successfully.

 

 

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms