My Oracle Support Banner

Exadata - RDS/IB Communication Issue RDS-ping Failures And Node Eviction (Doc ID 2517594.1)

Last updated on MAY 26, 2020

Applies to:

Linux OS - Version Oracle Linux 6.8 with Unbreakable Enterprise Kernel [2.6.39] to Oracle Linux 6.9 with Unbreakable Enterprise Kernel [4.1.12] [Release OL6U8 to OL6U9]
Linux x86-64

Symptoms

RDS/IB communication issue with RDS-ping failures being observed 

Following messages being logged in /var/log/messages


Sep 3 07:10:06 <hostname> kernel: RDS/IB: connection <x.x.x.x,y.y.y.y,0> dropped due to 'send completion error'
Sep 3 07:10:06 <hostname> kernel: RDS/IB: connection <x.x.x.x,y.y.y.y,5> dropped due to 'send completion error'
Sep 3 07:10:06 <hostname> kernel: RDS/IB: send completion <x.x.x.x,y.y.y.y,0> status 12 vendor_err 0x81, disconnecting and reconnecting
Sep 3 07:10:06 <hostname> kernel: RDS/IB: send completion <x.x.x.x,y.y.y.y,5> status 12 vendor_err 0x81, disconnecting and reconnecting
Sep 3 07:10:08 <hostname> kernel: RDS/IB: connection <x.x.x.x,z.z.z.z,3> dropped due to 'send completion error'
Sep 3 07:10:08 <hostname> kernel: RDS/IB: send completion <x.x.x.x,z.z.z.z,3> status 12 vendor_err 0x81, disconnecting and reconnecting
Sep 3 07:10:21 <hostname> kernel: INFO: task perfquery:374328 blocked for more than 120 seconds.
Sep 3 07:10:21 <hostname> kernel: INFO: task iblinkinfo:375100 blocked for more than 120 seconds.

and rds-ping  failing

[INFO:2017-09-17-02:09:56] numactl --cpunodebind=0 --membind=0 rds-ping -c 1 -i 5 -Q 4 -I x.x.x.x y.y.y.y: failed
[INFO:2017-09-17-02:10:02] numactl --cpunodebind=0 --membind=0 rds-ping -c 1 -i 5 -Q 5 -I x.x.x.x y.y.y.y: failed
[INFO:2017-09-17-02:10:07] numactl --cpunodebind=0 --membind=0 rds-ping -c 1 -i 5 -Q 0 -I x.x.x.x y.y.y.y: failed
[INFO:2017-09-17-02:10:12] numactl --cpunodebind=0 --membind=0 rds-ping -c 1 -i 5 -Q 4 -I x.x.x.x y.y.y.y: failed

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.