RDS send slows down when running massively parallel workload (Doc ID 1903432.1)

Last updated on JUNE 20, 2017

Applies to:

Linux OS - Version Oracle Linux 6.5 with Unbreakable Enterprise Kernel [3.8.13] to Oracle Linux 6.5 with Unbreakable Enterprise Kernel [3.8.13] [Release OL6U5]
Linux OS - Version Oracle Linux 5.9 with Unbreakable Enterprise Kernel [2.6.39] to Oracle Linux 5.9 with Unbreakable Enterprise Kernel [2.6.39] [Release OL5U9]
Linux x86-64
Linux x86

Symptoms

RDS send slows down when running massively parallel workload which might lead to a crash.
Below message could be seen in system logs if rds-info script is running:

Nov 17 22:18:00 oracle kernel: ------------[ cut here ]------------
Nov 17 22:18:00 oracle kernel: WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x24a/0x260()
Nov 17 22:18:00 oracle kernel: Hardware name: Sun Fire X4800 M2
Nov 17 22:18:00 oracle kernel: NETDEV WATCHDOG: ib0 (mlx4_core): transmit queue 0 timed out
Nov 17 22:18:00 oracle kernel: Modules linked in:
Nov 17 22:18:00 oracle kernel: Pid: 104563, comm: ora_p21k_gehc4 Tainted: P 2.6.39-400.126.1.el5uek #1
Nov 17 22:18:00 oracle kernel: Call Trace:
Nov 17 22:18:00 oracle kernel: <IRQ> [<ffffffff8145774a>] ? dev_watchdog+0x24a/0x260
Nov 17 22:18:00 oracle kernel: [<ffffffff8106f030>] warn_slowpath_common+0x90/0xc0
Nov 17 22:18:00 oracle kernel: [<ffffffff8106f15e>] warn_slowpath_fmt+0x6e/0x70
Nov 17 22:18:00 oracle kernel: [<ffffffff81089c8c>] ? wake_up_worker+0x1c/0x30
Nov 17 22:18:00 oracle kernel: [<ffffffff81089d87>] ? insert_work+0x57/0x70
Nov 17 22:18:00 oracle kernel: [<ffffffff815065b4>] ? _raw_spin_lock_irqsave+0x34/0x50
Nov 17 22:18:00 oracle kernel: [<ffffffff8108ba1b>] ? __queue_work+0xeb/0x290
Nov 17 22:18:00 oracle kernel: [<ffffffff8150656e>] ? _raw_spin_lock+0xe/0x20
Nov 17 22:18:00 oracle kernel: [<ffffffff8145774a>] dev_watchdog+0x24a/0x260
Nov 17 22:18:00 oracle kernel: [<ffffffff81457500>] ? dev_deactivate+0x50/0x50
Nov 17 22:18:00 oracle kernel: [<ffffffff8107dcfa>] call_timer_fn+0x4a/0x110
Nov 17 22:18:00 oracle kernel: [<ffffffff81457500>] ? dev_deactivate+0x50/0x50
Nov 17 22:18:00 oracle kernel: [<ffffffff8107f97a>] run_timer_softirq+0x13a/0x220
Nov 17 22:18:00 oracle kernel: [<ffffffff810395b5>] ? native_apic_msr_write+0x35/0x40
Nov 17 22:18:00 oracle kernel: [<ffffffff81033f2d>] ? lapic_next_event+0x1d/0x30
Nov 17 22:18:00 oracle kernel: [<ffffffff810a2a33>] ? tick_dev_program_event+0x43/0xc0
Nov 17 22:18:00 oracle kernel: [<ffffffff81075c59>] __do_softirq+0xb9/0x1d0
Nov 17 22:18:03 oracle kernel: [<ffffffff81095e09>] ? hrtimer_interrupt+0x129/0x240
Nov 17 22:18:03 oracle kernel: [<ffffffff8150ff7c>] call_softirq+0x1c/0x30
Nov 17 22:18:03 oracle kernel: [<ffffffff810172e5>] do_softirq+0x65/0xa0
Nov 17 22:18:03 oracle kernel: [<ffffffff8107658b>] irq_exit+0xab/0xc0
Nov 17 22:18:03 oracle kernel: [<ffffffff815108aa>] smp_apic_timer_interrupt+0x4a/0x5a
Nov 17 22:18:03 oracle kernel: [<ffffffff8150f733>] apic_timer_interrupt+0x13/0x20
Nov 17 22:18:03 oracle kernel: <EOI> [<ffffffff8150ee74>] ? sysret_audit+0x16/0x20
Nov 17 22:18:03 oracle kernel: ---[ end trace 31fb1caef2cfc878 ]---

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms