[HADOOP-3131] enabling BLOCK compression for map outputs breaks the reduce progress counters

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17.0, 0.17.1, 0.17.2, 0.18.0, 0.19.0
• Fix Version/s: 0.19.0
• Assignee: Matei Zaharia
• Reporter: Colin Evans
• Created: 29/Mar/08 01:21
• Updated: 08/Jul/09 16:53
• Resolved: 31/Jul/08 00:06

Description

Enabling map output compression and setting the compression type to BLOCK causes the progress counters during the reduce to go crazy and report progress counts over 100%.

This is problematic for speculative execution because it thinks the tasks are doing fine.