[CARBONDATA-2927] Multiple issue fixes for varchar column and complex columns that grows more than 2MB

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Assignee: Ajantha Bhat
• Reporter: Ajantha Bhat
• Created: 11/Sep/18 15:59
• Updated: 19/Sep/18 12:33
• Resolved: 19/Sep/18 12:33

Description

Fixed:
1. varchar data length is more than 2MB, buffer overflow exception (thread local row buffer)

root casue*: thread* loaclbuffer was hardcoded with 2MB.

solution: grow dynamically based on the row size.

2. read data from carbon file having one row of varchar data with 150 MB length is very slow.

root casue:  At UnsafeDMStore, ensure memory is just incresing by 8KB each time and lot of time malloc and free happens before reaching 150MB. hence very slow performance.

solution: directly check and allocate the required size.

3. Jvm crash when data size is more than 128 MB in unsafe sort step.

root cause: unsafeCarbonRowPage is of 128MB, so if data is more than 128MB  for one row, we access block beyond allocated, leading to JVM crash.

solution: validate the size before access and prompt user to increase unsafe memory. (by carbon property)

Issue Links

• links to: GitHub Pull Request #2706