[TAJO-1995] Improve range partitioning using histogram

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.13.0
• Component/s: QueryMaster
• Assignee: Jihoon Son
• Reporter: Jihoon Son
• Created: 27/Nov/15 06:25
• Updated: 02/Dec/17 17:33

Description

Currently implemented range repartition algorithm has two major problems as follows:

• It assumes that data distribution is uniform, so is fragile for skewed data distribution.
• Given floating point values, it ignores the numbers to the right to the decimal point, so is difficult to guess the proper partition number.

One of the solutions for this problem is to use the histogram. With a histogram, we can figure out data distribution and provide a proper handling of floating point values.

Issue Links

• is related to: TAJO-1950 Query master uses too much memory during range shuffle

Sub-Tasks

• Implement addition and subtraction methods for tuple: Open