[PIG-4449] Optimize the case of Order by + Limit in nested foreach

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.18.0
• Labels: Performance
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 05/Mar/15 20:12
• Updated: 30/Jul/19 18:02

Description

This is one of the very frequently used patterns

grouped_data_set = group data_set by id;

capped_data_set = foreach grouped_data_set
{
  ordered = order joined_data_set by timestamp desc;
  capped = limit ordered $num;
 generate flatten(capped);
};

But this performs very poorly when there are millions of rows for a key in the groupby with lot of spills. This can be easily optimized by pushing the limit into the InternalSortedBag and maintain only $num records any time and avoid memory pressure.

Issue Links

• is related to: PIG-5211 Optimize Nested Limited Sort
• is required by: PIG-4536 LIMIT and DISTINCT inside nested foreach should have combiner optimization
• relates to: PIG-419 Combiner optimizations extended to nested foreach statements as well