[MAPREDUCE-4100] Sometimes gridmix emulates data larger much larger then acutal counter for map only jobs

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 2.1.0-beta
• Component/s: contrib/gridmix
• Hadoop Flags: Reviewed
• Release Note: Bug fixed in compression emulation feature for map only jobs.
• Tags: gridmix compression-emulation map-only-jobs
• Assignee: Amar Kamat
• Reporter: Karam Singh
• Created: 03/Apr/12 15:06
• Updated: 03/Sep/14 22:45
• Resolved: 19/Apr/12 04:28

Description

While running 1400+ jobs trace I encountered this issue.
For map-only jobs, observed that some Maps generating data of around 9 GB (From HDFS_BYTES_WRITTEN) whereas actual value is around 5GB in trace.
This can sometimes also cause jobs to fail intermittently.

Other GridMix version coming be Hadoop-1.1.X and above might also effected