Error Transferring Compressed Files using Flume-NG to Hadoop Distributed File System(HDFS) (Doc ID 1501692.1)

Last updated on OCTOBER 11, 2016

Applies to:

Big Data Appliance Integrated Software - Version 1.0.3 and later
Linux x86-64

Symptoms

Using Flume NG 1.2 to transfer compressed data/files to HDFS on Oracle Big Data Appliance (BDA) using File Channel and HDFS Sink
 
Configuration on the HDFS sink

  hdfs.fileType = SequenceFile
  hdfs.codeC = gzip


Fails with below error ...

2012-10-17 18:17:45,606 INFO hdfs.BucketWriter: Creating hdfs://scajbda.us.oracle.com:8020/user/flume-events/TFlume/TestData.1350497865016.gz.tmp
2012-10-17 18:17:45,705 WARN hdfs.HDFSEventSink: HDFS IO error
java.io.IOException: java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
    at org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202)
    at org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48)
    at org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155)
    at org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:152)
    at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:125)
    at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:152)
    at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:307)
    at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:716)
    at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:713)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:386)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:329)
    at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:65)
    at org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:195)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms