HBase Master and Region Servers Are Down with "Unexpected Exception: java.nio.channels.CancelledKeyException"

(Doc ID 2330500.1)

Last updated on NOVEMBER 20, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.9.0 and later
Linux x86-64

Symptoms

HBase Master and Region Servers fail to come up. 

1. While coming, they fail to establish a successful connection to Zookeeper.  The log files show output like:

2017-11-20 13:50:06,213 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
2017-11-20 13:50:06,214 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2449)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:138)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2459)

2. The main error is in Zookeeper.  The output is like:

2017-11-20 14:49:55,184 ERROR org.apache.zookeeper.server.NIOServerCnxn: Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1075)
at org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:907)
at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:365)
at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:202)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:236)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms