[PIG-4542] OutputConsumerIterator should flush buffered records

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Pending Closed
• Affects Version/s: spark-branch
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Mohit Sabharwal
• Reporter: Mohit Sabharwal
• Created: 08/May/15 22:15
• Updated: 13/May/15 03:56
• Resolved: 13/May/15 03:56

Description

Certain operators may buffer the output. We need to flush the last set of records from such operators, when we encounter the last input record, before calling getNextTuple() for the last time.

Currently, to flush the last set of records, we compute RDD.count() and compare the count with a running counter to determine if we have reached the last record. This is an unnecessary and inefficient.

Issue Links

• links to: review board