Oracle Linux: ksoftirqd Processes Take Up 99% of CPU Time and mpstat(1) Shows Very High %soft
(Doc ID 2726240.1)
Last updated on MAY 15, 2023
Applies to:
Oracle Cloud Infrastructure - Version N/A and laterLinux OS - Version Oracle Linux 7.0 and later
Information in this document applies to any platform.
Symptoms
During high workload periods, ksoftirqd processes can run very hot at 99% of CPU time. In this particular case, this is an ExaData virtual environment where DomUs/GuestVMs are running PDB remote cloning.
The main symptom is %soft for certain CPUs in mpstat(1) outputs would consistently take up more than 99% of CPU time.
Top.ExaWatcher top - 02:21:04 up 7:42, 7 users, load average: 1.77, 3.83, 9.21 top - 02:22:03 up 7:43, 7 users, load average: 32.23, 11.83, 11.60 top - 02:23:02 up 7:44, 7 users, load average: 42.85, 18.21, 13.79 top - 02:23:08 up 7:44, 7 users, load average: 43.69, 19.20, 14.16 top - 02:24:01 up 7:45, 7 users, load average: 44.71, 23.25, 15.80 top - 02:24:07 up 7:45, 7 users, load average: 44.33, 23.53, 15.93 top - 02:25:06 up 7:46, 7 users, load average: 48.31, 28.37, 18.07 top - 02:26:05 up 7:47, 4 users, load average: 48.95, 32.27, 20.06 top - 02:27:04 up 7:48, 4 users, load average: 51.14, 36.16, 22.17 top - 02:28:02 up 7:49, 4 users, load average: 51.81, 38.76, 23.86 top - 02:28:09 up 7:49, 4 users, load average: 50.24, 38.85, 24.05 top - 02:29:01 up 7:50, 2 users, load average: 50.38, 40.58, 25.41 top - 02:29:08 up 7:50, 2 users, load average: 50.59, 40.79, 25.56 Mpstat.ExaWatcher 02:21:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:22:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:22:05 AM 8 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:22:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:22:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:22:05 AM 18 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:22:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:22:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:23:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:23:05 AM 8 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:23:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:23:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:23:05 AM 18 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:23:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:23:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:24:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:24:05 AM 8 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:24:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:24:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:24:05 AM 18 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:24:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:24:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:25:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:25:05 AM 8 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:25:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:25:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00 02:25:05 AM 18 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00 02:25:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00 02:25:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:26:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:26:05 AM 8 0.00 0.00 0.20 0.00 0.00 99.40 0.40 0.00 0.00 0.00 02:26:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:26:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:26:05 AM 18 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:26:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:26:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:27:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:27:05 AM 8 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:27:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:27:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.20 0.80 0.00 0.00 0.00 02:27:05 AM 18 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:27:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:27:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:28:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:28:05 AM 8 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:28:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:28:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.40 0.60 0.00 0.00 0.00 02:28:05 AM 18 0.00 0.00 0.20 0.00 0.00 99.40 0.40 0.00 0.00 0.00 02:28:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:28:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:29:05 AM 0 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00 02:29:05 AM 8 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00 02:29:05 AM 16 0.00 0.00 0.00 0.00 0.00 99.60 0.40 0.00 0.00 0.00 02:29:05 AM 17 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00 02:29:05 AM 18 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00 02:29:05 AM 19 0.00 0.00 0.00 0.00 0.00 99.80 0.20 0.00 0.00 0.00
Other symptoms include but not limited to sporadic or high RDS ping latencies and ASM disk IO latencies.
RDSinfo.ExaWatcher [2020-08-30-02:21:11] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 78 usec [2020-08-30-02:21:11] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 1244 usec [2020-08-30-02:21:12] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 92 usec [2020-08-30-02:21:13] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 241 usec [2020-08-30-02:21:14] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 65 usec [2020-08-30-02:21:14] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 84 usec [2020-08-30-02:21:14] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 275 usec [2020-08-30-02:21:14] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 60 usec [2020-08-30-02:23:17] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 9349 usec [2020-08-30-02:23:17] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 114 usec [2020-08-30-02:23:21] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 99428 usec [2020-08-30-02:23:22] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 94 usec [2020-08-30-02:23:26] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 111 usec [2020-08-30-02:23:26] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 1576 usec [2020-08-30-02:23:26] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 75240 usec [2020-08-30-02:23:26] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 148 usec [2020-08-30-02:25:30] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 129 usec [2020-08-30-02:25:30] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 117 usec [2020-08-30-02:25:35] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 23453 usec [2020-08-30-02:25:35] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 228 usec [2020-08-30-02:25:40] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 847 usec [2020-08-30-02:25:40] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 114 usec [2020-08-30-02:25:40] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 1078 usec [2020-08-30-02:25:40] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 4500 usec [2020-08-30-02:27:44] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 152 usec [2020-08-30-02:27:45] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 1359 usec [2020-08-30-02:27:49] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 104193 usec [2020-08-30-02:27:50] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 142 usec [2020-08-30-02:27:54] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 1922 usec [2020-08-30-02:27:54] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 93 usec [2020-08-30-02:27:54] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.cc: 1: 5669 usec [2020-08-30-02:27:54] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.bbb 192.168.xxx.dd: 1: 207 usec [2020-08-30-02:29:57] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.cc: 1: 582 usec [2020-08-30-02:29:58] numactl ... rds-ping -c 1 -i 5 -Q 0 -I 192.168.xxx.aaa 192.168.xxx.dd: 1: 71 usec
Changes
Running heavy network traffic or remote IOs.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Changes |
Cause |
Solution |
References |