[HADOOP-13138] Unable to append to a SequenceFile with Compression.NONE.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Vinayakumar B
• Reporter: Gervais Mickaël
• Created: 12/May/16 08:29
• Updated: 02/Jan/19 22:15
• Resolved: 18/May/16 16:42

Description

Hi,

I'm trying to use the append functionnality to an existing SequenceFile.

If I set Compression.NONE, it works when the file is created, but when the file already exists I've a NullPointerException, by the way it works if I specify a compression with a codec.

Failing code

Option compression = compression(CompressionType.NONE);
Option keyClass = keyClass(LongWritable.class);
Option valueClass = valueClass(BytesWritable.class);
Option out = file(dfs);
Option append = appendIfExists(true);

writer = createWriter(conf,
             out,
             append,
             compression,
             keyClass,
             valueClass);

The following exeception is thrown when the file exists because compression option is checked:

Exception in thread "main" java.lang.NullPointerException
	at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1119)
	at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)

This is due to the codec which is null:

SequenceFile.java

 if (readerCompressionOption.value != compressionTypeOption.value
                || !readerCompressionOption.codec.getClass().getName()
                    .equals(compressionTypeOption.codec.getClass().getName())) {
              throw new IllegalArgumentException(
                  "Compression option provided does not match the file");
            }

Thansk

Mickaël

Issue Links

• duplicates: HADOOP-13216 Append SequenceFile with compressionType(NONE,RECORD,BLOCK) throws NullPointerException
• is duplicated by: HADOOP-16021 SequenceFile.createWriter appendIfExists codec cause NullPointerException
• relates to: HADOOP-7139 Allow appending to existing SequenceFiles