Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13138

Unable to append to a SequenceFile with Compression.NONE.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hi,

      I'm trying to use the append functionnality to an existing SequenceFile.

      If I set Compression.NONE, it works when the file is created, but when the file already exists I've a NullPointerException, by the way it works if I specify a compression with a codec.

      Failing code
      Option compression = compression(CompressionType.NONE);
      Option keyClass = keyClass(LongWritable.class);
      Option valueClass = valueClass(BytesWritable.class);
      Option out = file(dfs);
      Option append = appendIfExists(true);
      
      writer = createWriter(conf,
                   out,
                   append,
                   compression,
                   keyClass,
                   valueClass);
      

      The following exeception is thrown when the file exists because compression option is checked:

      Exception in thread "main" java.lang.NullPointerException
      	at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1119)
      	at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
      

      This is due to the codec which is null:

      SequenceFile.java
       if (readerCompressionOption.value != compressionTypeOption.value
                      || !readerCompressionOption.codec.getClass().getName()
                          .equals(compressionTypeOption.codec.getClass().getName())) {
                    throw new IllegalArgumentException(
                        "Compression option provided does not match the file");
                  }
      

      Thansk

      Mickaël

        Attachments

        1. HADOOP-13138-01.patch
          5 kB
          Vinayakumar B
        2. HADOOP-13138-02.patch
          5 kB
          Vinayakumar B

          Issue Links

            Activity

              People

              • Assignee:
                vinayrpet Vinayakumar B
                Reporter:
                mgervais Gervais Mickaël
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: