Spark Jobs Throw "Error [Shut down address: akka.tcp://driverPropsFetcher@<node>:41534]" (Doc ID 2199185.1)

Last updated on NOVEMBER 20, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.4.0 to 4.4.0 [Release 4.4]
Linux x86-64

Symptoms

On BDA V4.4 / CDH 5.5.1 the following error is raised when running spark shell jobs:

Error [Shut down address: akka.tcp://driverPropsFetcher@<node>:41534] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@<node.example.com>:41534
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
16/09/28 12:41:19 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@<node.example.com>:34898/user/Executor#-935920980]) with ID 29
16/09/28 12:41:19 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 2086, <node.example.com>, partition 0,NODE_LOCAL, 2181 bytes)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms