My Oracle Support Banner

Oracle Linux: ksoftirqd Processes Take Up 99% of CPU Time and mpstat(1) Shows Very High %soft (Doc ID 2726240.1)

Last updated on APRIL 25, 2021

Applies to:

Oracle Cloud Infrastructure - Version N/A and later
Linux OS - Version Oracle Linux 7.0 and later
Information in this document applies to any platform.

Symptoms

During high workload periods, ksoftirqd processes can run very hot at 99% of CPU time. In this particular case, this is an ExaData virtual environment where DomUs/GuestVMs are running PDB remote cloning.

The main symptom is %soft for certain CPUs in mpstat(1) outputs would consistently take up more than 99% of CPU time.

Top.ExaWatcher
top - 02:21:04 up  7:42,  7 users,  load average: 1.77, 3.83, 9.21
top - 02:22:03 up  7:43,  7 users,  load average: 32.23, 11.83, 11.60
top - 02:23:02 up  7:44,  7 users,  load average: 42.85, 18.21, 13.79
top - 02:23:08 up  7:44,  7 users,  load average: 43.69, 19.20, 14.16
top - 02:24:01 up  7:45,  7 users,  load average: 44.71, 23.25, 15.80
top - 02:24:07 up  7:45,  7 users,  load average: 44.33, 23.53, 15.93
top - 02:25:06 up  7:46,  7 users,  load average: 48.31, 28.37, 18.07
top - 02:26:05 up  7:47,  4 users,  load average: 48.95, 32.27, 20.06
top - 02:27:04 up  7:48,  4 users,  load average: 51.14, 36.16, 22.17
top - 02:28:02 up  7:49,  4 users,  load average: 51.81, 38.76, 23.86
top - 02:28:09 up  7:49,  4 users,  load average: 50.24, 38.85, 24.05
top - 02:29:01 up  7:50,  2 users,  load average: 50.38, 40.58, 25.41
top - 02:29:08 up  7:50,  2 users,  load average: 50.59, 40.79, 25.56

Mpstat.ExaWatcher
02:21:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:22:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:22:05 AM    8    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:22:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:22:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:22:05 AM   18    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:22:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:22:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:23:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:23:05 AM    8    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:23:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:23:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:23:05 AM   18    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:23:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:23:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:24:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:24:05 AM    8    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:24:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:24:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:24:05 AM   18    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:24:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:24:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:25:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:25:05 AM    8    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:25:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:25:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00
02:25:05 AM   18    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00
02:25:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00
02:25:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:26:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:26:05 AM    8    0.00    0.00    0.20    0.00    0.00   99.40    0.40    0.00    0.00    0.00
02:26:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:26:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:26:05 AM   18    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:26:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:26:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:27:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:27:05 AM    8    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:27:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:27:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.20    0.80    0.00    0.00    0.00
02:27:05 AM   18    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:27:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:27:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:28:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:28:05 AM    8    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:28:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:28:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.40    0.60    0.00    0.00    0.00
02:28:05 AM   18    0.00    0.00    0.20    0.00    0.00   99.40    0.40    0.00    0.00    0.00
02:28:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:28:05 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:29:05 AM    0    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00
02:29:05 AM    8    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00
02:29:05 AM   16    0.00    0.00    0.00    0.00    0.00   99.60    0.40    0.00    0.00    0.00
02:29:05 AM   17    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00
02:29:05 AM   18    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00
02:29:05 AM   19    0.00    0.00    0.00    0.00    0.00   99.80    0.20    0.00    0.00    0.00

Other symptoms include but not limited to sporadic or high RDS ping latencies and ASM disk IO latencies.

RDSinfo.ExaWatcher
[2020-08-30-02:21:11] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 78 usec
[2020-08-30-02:21:11] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 1244 usec
[2020-08-30-02:21:12] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 92 usec
[2020-08-30-02:21:13] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 241 usec
[2020-08-30-02:21:14] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 65 usec
[2020-08-30-02:21:14] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 84 usec
[2020-08-30-02:21:14] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 275 usec
[2020-08-30-02:21:14] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 60 usec
[2020-08-30-02:23:17] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 9349 usec
[2020-08-30-02:23:17] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 114 usec
[2020-08-30-02:23:21] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 99428 usec
[2020-08-30-02:23:22] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 94 usec
[2020-08-30-02:23:26] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 111 usec
[2020-08-30-02:23:26] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 1576 usec
[2020-08-30-02:23:26] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 75240 usec
[2020-08-30-02:23:26] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 148 usec
[2020-08-30-02:25:30] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 129 usec
[2020-08-30-02:25:30] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 117 usec
[2020-08-30-02:25:35] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 23453 usec
[2020-08-30-02:25:35] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 228 usec
[2020-08-30-02:25:40] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 847 usec
[2020-08-30-02:25:40] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 114 usec
[2020-08-30-02:25:40] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 1078 usec
[2020-08-30-02:25:40] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 4500 usec
[2020-08-30-02:27:44] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 152 usec
[2020-08-30-02:27:45] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 1359 usec
[2020-08-30-02:27:49] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 104193 usec
[2020-08-30-02:27:50] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 142 usec
[2020-08-30-02:27:54] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 1922 usec
[2020-08-30-02:27:54] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 93 usec
[2020-08-30-02:27:54] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc:  1: 5669 usec
[2020-08-30-02:27:54] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd:  1: 207 usec
[2020-08-30-02:29:57] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc:  1: 582 usec
[2020-08-30-02:29:58] numactl ... rds-ping  -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd:  1: 71 usec

Changes

Running heavy network traffic or remote IOs.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.