Cloudera Manager Reports Hive/HDFS Services in Bad Health with "nf_conntrack: table full, dropping packet." Messaging on One Node. (Doc ID 1961806.1)

Last updated on JANUARY 17, 2015

Applies to:

Big Data Appliance Integrated Software - Version 3.0.1 and later
Linux x86-64

Symptoms

After the moving a BDA rack (see Changes section) and rebooting the system:

1. Cloudera Manager (CM) reports the hdfs and hive services to be in "Bad" health.  Drilling down the problem is found to be with a inaccessibility of a particular Node, for example in this case Node 4.

2. The connection tracking table, nf_conntrack is table is observed to be full on that node, causing additional connections to be dropped.

3. Running bdacheckcluster reports a problem like:

a) From bdacheckcluster:


5. The bdachecknet output shows the network and all network services are working properly.  The bdacheckvnics script output (/opt/oracle/bda/network/bdacheckvnics) shows the status of both the client network bondeth0 and its slaves eth8 and eth9 as well as the private network bondib0 and its slaves ib0 and ib1 to be working properly as well.

Changes

The BDA rack was moved a few meters in the same Data Center.  The reported errors started after the move.
 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms