Cloudera Manager Reports Hive/HDFS Services in Bad Health with "nf_conntrack: table full, dropping packet." Messaging on One Node.
(Doc ID 1961806.1)
Last updated on AUGUST 02, 2021
Applies to:
Big Data Appliance Integrated Software - Version 3.0.1 and laterLinux x86-64
Symptoms
After the moving a BDA rack (see Changes section) and rebooting the system:
1. Cloudera Manager (CM) reports the hdfs and hive services to be in "Bad" health. Drilling down the problem is found to be with a inaccessibility of a particular Node, for example in this case Node 4.
2. The connection tracking table, nf_conntrack is table is observed to be full on that node, causing additional connections to be dropped.
3. Running bdacheckcluster reports a problem like:
a) From bdacheckcluster:
5. The bdachecknet output shows the network and all network services are working properly. The bdacheckvnics script output (/opt/oracle/bda/network/bdacheckvnics) shows the status of both the client network bondeth0 and its interface's eth8 and eth9 as well as the private network bondib0 and its interfaces's ib0 and ib1 to be working properly as well.
Changes
The BDA rack was moved a few meters in the same Data Center. The reported errors started after the move.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |