Issue Details (XML | Word | Printable)

Key: HADOOP-2943
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Chris Douglas
Reporter: Chris Douglas
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

Compression for intermediate map output is broken

Created: 06/Mar/08 02:30 AM   Updated: 08/Jul/09 04:52 PM
Return to search
Component/s: None
Affects Version/s: 0.17.0
Fix Version/s: 0.17.0

Time Tracking:
Not Specified

File Attachments:
  File Name Date Attached Attached By Size
Text File Licensed for inclusion in ASF works 2943.patch 2008-03-07 06:57 PM Chris Douglas 6 kB
Text File Licensed for inclusion in ASF works 2943.patch 2008-03-06 10:02 PM Chris Douglas 6 kB
Text File Licensed for inclusion in ASF works 2943.patch 2008-03-06 02:45 AM Chris Douglas 4 kB
Text File Licensed for inclusion in ASF works 2943.patch 2008-03-06 02:34 AM Chris Douglas 3 kB
Issue Links:
Blocker
 

Resolution Date: 09/Mar/08 04:05 AM


 Description  « Hide
It looks like SequenceFile::RecordCompressWriter and SequenceFile::BlockCompressWriter weren't updated to use the new serialization added in HADOOP-1986. This causes failures in the merge when mapred.compress.map.output is true and mapred.map.output.compression.type=BLOCK: 
java.io.IOException: File is corrupt!
        at org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:1656)
        at org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java:1969)
        at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:2985)
        at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2785)
        at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2494)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:654)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:740)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2077)

mapred.map.output.compression.type=RECORD works for Writables, but should be updated.

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Chris Douglas made changes - 06/Mar/08 02:34 AM
Field Original Value New Value
Attachment 2943.patch [ 12377218 ]
Chris Douglas made changes - 06/Mar/08 02:34 AM
Assignee Chris Douglas [ chris.douglas ]
Status Open [ 1 ] Patch Available [ 10002 ]
Chris Douglas made changes - 06/Mar/08 02:45 AM
Attachment 2943.patch [ 12377219 ]
Chris Douglas made changes - 06/Mar/08 02:45 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Chris Douglas made changes - 06/Mar/08 02:45 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Chris Douglas made changes - 06/Mar/08 10:02 PM
Attachment 2943.patch [ 12377288 ]
Chris Douglas made changes - 06/Mar/08 10:03 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Chris Douglas made changes - 06/Mar/08 10:03 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Chris Douglas made changes - 06/Mar/08 10:11 PM
Link This issue blocks HADOOP-2919 [ HADOOP-2919 ]
Arun C Murthy made changes - 07/Mar/08 06:49 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Arun C Murthy made changes - 07/Mar/08 06:50 AM
Affects Version/s 0.17.0 [ 12312913 ]
Fix Version/s 0.17.0 [ 12312913 ]
Chris Douglas made changes - 07/Mar/08 06:57 PM
Attachment 2943.patch [ 12377380 ]
Chris Douglas made changes - 07/Mar/08 06:58 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Chris Douglas made changes - 07/Mar/08 09:26 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Chris Douglas made changes - 07/Mar/08 09:26 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Chris Douglas made changes - 07/Mar/08 10:42 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Chris Douglas made changes - 07/Mar/08 10:43 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Chris Douglas made changes - 08/Mar/08 01:33 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Chris Douglas made changes - 08/Mar/08 01:34 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Chris Douglas made changes - 09/Mar/08 04:05 AM
Resolution Fixed [ 1 ]
Status Patch Available [ 10002 ] Resolved [ 5 ]
Nigel Daley made changes - 21/May/08 08:05 PM
Status Resolved [ 5 ] Closed [ 6 ]
Owen O'Malley made changes - 08/Jul/09 04:52 PM
Component/s mapred [ 12310690 ]

Powered by a free Atlassian JIRA open source license for Apache Software Foundation. Try JIRA - bug tracking software for your team.
Atlassian JIRA the Professional Issue Tracker. (Enterprise Edition, Version: 3.13.2-#335) - Bug/feature request - Atlassian news - Contact Administrators