[PIG-1102] Collect number of spills per job

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Assignee: Sriranjan Manjunath
• Reporter: Olga Natkovich
• Created: 23/Nov/09 20:43
• Updated: 14/May/10 06:46
• Resolved: 24/Dec/09 22:21

Description

Memory shortage is one of the main performance issues in Pig. Knowing when we spill do the disk is useful for understanding query performance and also to see how certain changes in Pig effect that.

Other interesting stats to collect would be average CPU usage and max mem usage but I am not sure if this information is easily retrievable.

Using Hadoop counters for this would make sense.