[PIG-5385] Skip calling extra gc() before spilling large bag when unnecessary

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 0.18.0
• Hadoop Flags: Reviewed
• Assignee: Koji Noguchi
• Reporter: Koji Noguchi
• Created: 21/Mar/19 21:39
• Updated: 03/Jul/19 17:53
• Resolved: 03/Jul/19 17:53

Description

In order to determine when to start spilling bags, Pig uses MemoryNotification for both MEMORY_THRESHOLD_EXCEEDED and MEMORY_COLLECTION_THRESHOLD_EXCEEDED.
https://docs.oracle.com/javase/8/docs/api/java/lang/management/MemoryNotificationInfo.html

Since spilling a large bag is expensive, Pig explicitly call System.gc() when the expected size is huge. I think we can skip this step when notification is based on MEMORY_COLLECTION_THRESHOLD_EXCEEDED since this means jvm has called the gc already.