[HADOOP-8540] Compression with non-default codec's fail on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Trupti Dhavle
• Created: 28/Jun/12 18:14
• Updated: 28/Mar/14 18:27
• Resolved: 24/Oct/12 01:08

Description

I was trying to run the compression test case with GZipCodec. However the test fails with NPE. It is running fine with DefaultCodec.

With text input
c:\Workspace\winhadoopqe>c:\hdp\branch-1-win\bin\hadoop jar c:\hdp\branch-1-win\
build\hadoop-examples-1.1.0-SNAPSHOT.jar sort "-Dmapred.compress.map.output=true
" "-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
" "-Dmapred.output.compress=true" "-Dmapred.output.compression.type=NONE" "-Dm
apred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text Compression/texti
nput Compression/textoutput-123

Running on 1 nodes to sort from hdfs://localhost:8020/user/Administrator/Compres
sion/textinput into hdfs://localhost:8020/user/Administrator/Compression/textout
put-123 with 1 reduces.
Job started: Thu Jun 28 10:02:18 PDT 2012
12/06/28 10:02:18 INFO mapred.FileInputFormat: Total input paths to process : 1
12/06/28 10:02:18 INFO mapred.JobClient: Running job: job_201206271409_0045
12/06/28 10:02:19 INFO mapred.JobClient: map 0% reduce 0%
12/06/28 10:02:36 INFO mapred.JobClient: Task Id : attempt_201206271409_0045_m_0
00000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:102)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask
.java:1407)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1
298)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1103)
at org.apache.hadoop.mapred.Child.main(Child.java:265)

Issue Links

• is part of: HADOOP-8645 Stabilize branch-1-win
• is related to: HADOOP-10450 Build zlib native code bindings in hadoop.dll for Windows.
• relates to: HADOOP-8564 Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue