[CARBONDATA-117] BlockLet distribution for optimum resource usage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0-incubating, 0.1.1-incubating
• Component/s: data-query
• Assignee: Mohammad Shahid Khan
• Reporter: Mohammad Shahid Khan
• Created: 28/Jul/16 11:13
• Updated: 24/Sep/16 10:41
• Resolved: 23/Aug/16 15:08

Description

*Problem: *
In present scenario during querying data, the data scan can be done minimum at the block level.
Suppose if the system can execute 10 parallel task and number of blocks to be scanned is 3, then only 3 cores could be utilised remaining 7 will be un-utilized.
Solution:
To optimize the resouce usage the blocks can be further divided into multiple blocks.
A block consist of multiple blocklets, these blocklet could be distributed into multiple logical blocks by assigning the startBlockletNumber and no of BlockLetsTobeScanned.

Issue Links

• links to: GitHub Pull Request #56