Oracle Linux 6 Server Experiences Slow Performance, High Load and Unable to Move Data to Directories on NAS Appliance (Doc ID 2644493.1)

Last updated on APRIL 13, 2020

Applies to:

Symptoms

An Oracle Linux 6 server experienced an increase in server load, and increasingly slow performance. Attempts to move or copy files to and from a NAS share also completely stalled.

Output from top showed a high load, but no individual process was using a large amount of CPU or RAM, and none were stuck in D (uninterruptable) state:

zzz ***Thu Feb 20 15:25:22 CST 2020
top - 15:20:24 up 149 days, 15:18, 9 users, load average: 32.16, 28.22, 22.08
Tasks: 608 total, 1 running, 607 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.4%us, 0.5%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.1%hi, 0.0%si, 0.0%st
Mem: 99047892k total, 70576896k used, 28470996k free, 92644k buffers
Swap: 4194300k total, 677872k used, 3516428k free, 35246472k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
48766 root 30 10 15424 1652 928 R 3.8 0.0 0:00.09 top
79881 <USER> 20 0 28.6g 3.0g 11m S 1.9 3.1 76:17.85 java
125766 root 20 0 15428 1708 944 S 1.9 0.0 0:49.45 top
6322 root 20 0 3233m 17m 5156 S 1.0 0.0 2074:08 klzagent

The process output from application processes showed that all the application processes were in "futex_s" (fast user lock) state and waiting to perform I/O to the NAS share, such as in the example below:

zzz ***Thu Feb 20 17:00:22 CST 2020
USER PID PPID PRI %CPU %MEM VSZ RSS WCHAN S STARTED TIME COMMAND
<USER> 79881 79845 19 2.3 3.1 29977120 3104760 futex_ S Feb 18 01:16:17 /<SHARE>/java/java-1.8.0-openjdk-1.8.0.111.x86_64/bin/java -Djavax.net.ssl.keyStore=<KEYSTORE_INFO> /<SHARE>/<USER>/ongoing-migration/ongoing-migration.jar --spring.profiles.active=prod --spring.config.location=/<SHARE>/<USER>/ongoing-migration/conf/

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.