Cloudera Manager BDR Replication Throws Exception "java.lang.OutOfMemoryError: GC overhead limit exceeded" (Doc ID 1906373.1)

Last updated on JULY 01, 2017

Applies to:

Big Data Appliance Integrated Software - Version 2.4.0 and later
Linux x86-64

Symptoms

Facing problems with the HDFS replication through Cloudera Manager, although the manual distcp command is working properly. Configured 6 HDFS folder replication between two different clusters, and 5 of them are working fine except the bigger one.
The following tests:

1. Folder size: 24.1T Number of files: 14 Number of blocks: 98560 Result: Success
2. Folder size: 24.1T Number of files: 15 Number of blocks: 98660 Result: FAIL
3. Folder size: 803.6G Number of files: 112 Number of blocks: 3230 Result: Success
4. (Changing block size from 256M to 10M)
Folder size: 105.3G Number of files: 1 Number of blocks: 107848 Result: FAIL

The replication of a folder with more than 98600 blocks (approximately) fails with the following error:

Full error stack:
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
  at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
  at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:777)
  at org.apache.hadoop.io.Text.encode(Text.java:432)
  at org.apache.hadoop.io.Text.set(Text.java:196)
  at org.apache.hadoop.io.Text.<init>(Text.java:88)
  at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:570)
  at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:556)
  at org.apache.hadoop.hdfs.protocolPB.PBHelper.convertLocatedBlock(PBHelper.java:939)
  at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:957)
  at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1075)
  at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1135)
  at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1144)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:444)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
  at com.sun.proxy.$Proxy10.getListing(Unknown Source)
  at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1526)
  at org.apache.hadoop.hdfs.DistributedFileSystem$1.<init>(DistributedFileSystem.java:466)
  at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:457)
  at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1760)
  at com.cloudera.enterprise.distcp.SimpleCopyListing.recordEntries(SimpleCopyListing.java:184)
  at com.cloudera.enterprise.distcp.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:127)
  at com.cloudera.enterprise.distcp.CopyListing.buildListing(CopyListing.java:91)
  at com.cloudera.enterprise.distcp.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:98)
  at com.cloudera.enterprise.distcp.CopyListing.buildListing(CopyListing.java:91)
  at com.cloudera.enterprise.distcp.CopyListing.buildListing(CopyListing.java:102)
  at com.cloudera.enterprise.distcp.DistCp.createInputFileListing(DistCp.java:657)
  at com.cloudera.enterprise.distcp.DistCp.execute(DistCp.java:264)

  

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms