Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8540

Compression with non-default codec's fail on Windows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I was trying to run the compression test case with GZipCodec. However the test fails with NPE. It is running fine with DefaultCodec.

      With text input
      c:\Workspace\winhadoopqe>c:\hdp\branch-1-win\bin\hadoop jar c:\hdp\branch-1-win\
      build\hadoop-examples-1.1.0-SNAPSHOT.jar sort "-Dmapred.compress.map.output=true
      " "-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
      " "-Dmapred.output.compress=true" "-Dmapred.output.compression.type=NONE" "-Dm
      apred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
      org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text Compression/texti
      nput Compression/textoutput-123

      Running on 1 nodes to sort from hdfs://localhost:8020/user/Administrator/Compres
      sion/textinput into hdfs://localhost:8020/user/Administrator/Compression/textout
      put-123 with 1 reduces.
      Job started: Thu Jun 28 10:02:18 PDT 2012
      12/06/28 10:02:18 INFO mapred.FileInputFormat: Total input paths to process : 1
      12/06/28 10:02:18 INFO mapred.JobClient: Running job: job_201206271409_0045
      12/06/28 10:02:19 INFO mapred.JobClient: map 0% reduce 0%
      12/06/28 10:02:36 INFO mapred.JobClient: Task Id : attempt_201206271409_0045_m_0
      00000_0, Status : FAILED
      java.lang.NullPointerException
      at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:102)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask
      .java:1407)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1
      298)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
      tion.java:1103)
      at org.apache.hadoop.mapred.Child.main(Child.java:265)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tdhavle Trupti Dhavle
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: