Troubleshooting "nf_conntrack: table full, dropping packet." message on a BDA Node (Doc ID 1961802.1)

Last updated on DECEMBER 03, 2019

Applies to:

Purpose

"nf_conntrack: table full, dropping packet." messages look like:

and can be observed:

1. In the ILOM.
2. In /var/log/messages.
3. In the sos report/var/log/messages.

The message: "nf_conntrack: table full, dropping packet." indicates that the connection tracking table, nf_conntrack (i.e. the table that iptables uses to store persistent connections) is full on the Node issuing the messaging.  This can happen if a Node is flooded with excessive NAT connections. This is typically the result of heavy traffic passing by the server.  On the BDA a likely root cause could be an application or user unexpectedly spawning a number of connections exceeding what the table can hold as set by the nf_conntrack kernel maximum number assigned.

Note: The nf_conntrack table will not be impacted by iptables missing the Table filter. See iptables Missing "Table: filter" on one Node of a BDA Cluster (Doc ID 1961763.1)
because that table does not track connections.

The result of this event can be Cloudera Manager (CM) reporting "Bad" health for some services due to an inability to access the node with the excessive traffic. Basically nf_conntrack gets full and drops subsequent connection attempts via NAT.

The example here uses Node 4.  In this case CM reports "Bad" health for hdfs/hive services.

The purpose of this document is to evaluate the connections on the Node with the messaging as compared to the rest of the cluster to help diagnose the cause of the problem.

Troubleshooting Steps

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.