[PIG-1829] "0" value seen in PigStat's map/reduce runtime, even when the job is successful

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Hadoop Flags: Reviewed
• Assignee: Richard Ding
• Reporter: Thejas Nair
• Created: 27/Jan/11 20:48
• Updated: 20/Dec/11 19:58
• Resolved: 01/Mar/11 22:55

Description

Pig runtime calls JobClient.getMapTaskReports(jobId) and JobClient.getReduceTaskReports(jobId) to get statistics about numbers of maps/reducers, as well as max/min/avg time of these tasks. But from time to time, these calls return empty lists. When that happens pig is reports 0 values for the stats.

The jobtracker keeps the stats information only for a limited duration based on the configuration parameters mapred.jobtracker.completeuserjobs.maximum and mapred.job.tracker.retiredjobs.cache.size. Since pig collects the stats after jobs have finished running, it is possible that the stats for the initial jobs are no longer available. To have better chances of getting the stats, it should be collected as soon as the job is over.

Issue Links

• is related to: PIG-2429 Better error message in pig_*.log
• requires: MAPREDUCE-1941 Need a servlet in JobTracker to stream contents of the job history file