My Oracle Support Banner

Cloudera Manager Reports Hive/HDFS Services in Bad Health with "nf_conntrack: table full, dropping packet." Messaging on One Node. (Doc ID 1961806.1)

Last updated on JANUARY 17, 2015

Applies to:

Big Data Appliance Integrated Software - Version 3.0.1 and later
Linux x86-64

Symptoms

After the moving a BDA rack (see Changes section) and rebooting the system:

1. Cloudera Manager (CM) reports the hdfs and hive services to be in "Bad" health.  Drilling down the problem is found to be with a inaccessibility of a particular Node, for example in this case Node 4.

2. The connection tracking table, nf_conntrack is table is observed to be full on that node, causing additional connections to be dropped.

3. Running bdacheckcluster reports a problem like:

a) From bdacheckcluster:

# time bdacheckcluster
  
INFO: Logging results to /tmp/bdacheckcluster_1420477645/
Enter CM admin password to enable check for CM services and hosts
Press ENTER twice to skip CM services and hosts checks
Enter password:
Enter password again:
SUCCESS: Mammoth configuration file is valid.
Error : No output returned
Failed. Full error log in : /tmp/bdacheckcluster_1420477645/clusters_<cluster-name>_services_hdfs_view_full_1420477752.err
/tmp/bdacheckcluster_1420477645/hadoophealthsummary.out could not be opened at /opt/oracle/bda/bin/bdacheckcluster line 476, <STDIN> line 2.


b) The referred to file /tmp/bdacheckcluster_1420477645/clusters_<cluster-name>_services_hdfs_view_full_1420477752.err, reports:

curl: (7) couldn't connect to host


4. From the ILOM on Node 4 the following messaging is repeated over and over showing that the connection tracking table is full. The messaging is also seen in /var/log/messages and in the sos report /var/log/messages from the node.

...
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
net_ratelimit: 26950 callbacks suppressed
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
...
net_ratelimit: 27129 callbacks suppressed
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
....


5. The bdachecknet output shows the network and all network services are working properly.  The bdacheckvnics script output (/opt/oracle/bda/network/bdacheckvnics) shows the status of both the client network bondeth0 and its slaves eth8 and eth9 as well as the private network bondib0 and its slaves ib0 and ib1 to be working properly as well.

Changes

The BDA rack was moved a few meters in the same Data Center.  The reported errors started after the move.
 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.