[CARBONDATA-2895] [Batch-sort]Query result mismatch with Batch-sort in save to disk (sort temp files) scenario.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Ajantha Bhat
• Reporter: Ajantha Bhat
• Created: 27/Aug/18 15:23
• Updated: 05/Sep/18 15:02
• Resolved: 05/Sep/18 15:02

Description

probelm: Query result mismatch with Batch-sort in save to disk (sort temp files) scenario.

scenario:
a) Configure batchsort but give batch size more than UnsafeMemoryManager.INSTANCE.getUsableMemory().
b) Load data that is greater than batch size. Observe that unsafeMemoryManager save to disk happened as it cannot process one batch.  
c) so load happens in 2 batch.
d) When query the results. There result data rows is more than expected data rows.

root cause:

For each batch, createSortDataRows() will be called.
Files saved to disk during sorting of previous batch was considered for this batch.

solution:
Files saved to disk during sorting of previous batch ,should not be considered for this batch.
Hence use batchID as rangeID field of sorttempfiles.
So getFilesToMergeSort() will select files of only this batch.

Issue Links

• links to: GitHub Pull Request #2664