[CARBONDATA-3267] Data loading is failing with OOM using range sort

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.2
• Assignee: Kumar Vishal
• Reporter: Kumar Vishal
• Created: 23/Jan/19 16:16
• Updated: 24/Jan/19 09:13
• Resolved: 24/Jan/19 09:13

Description

Problem:

Range sort is failing with OOM.

Root cause:

This is because UnsafeSortStorageMemory is not able to control the off heap memory because of this when huge data is loaded it OOM exception is coming fron UnsafeMemoryAllocator.allocate.

Solution:

Control Sort Storage memory. After sorting the rows if memory is available then only add sorted records to sort storage memory otherwise write to disk

Issue Links

• links to: GitHub Pull Request #3096