[TEZ-3769] Unordered: Fix wrong stats being sent out in the last event, when final merge is disabled

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 22/Jun/17 11:46
• Updated: 22/Aug/17 00:03
• Resolved: 28/Jun/17 08:07

Description

When final merge is disabled (without pipelining), wrong stats was sent out in the last event.

It was based on numRecordsPerPartition which contains the overall partition data. It should be ideally be based on the spill result and its buffers.

Also, finalSpill was unncessarily sending events when no data was present (i.e, when currentBuffer didn't have any data). This can be optimized to reduce the number of events being sent across.