[TEZ-1993] Implement a pluggable InputSizeEstimator for grouping fairly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.7.0
• Target Version/s: 0.7.0
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 22/Jan/15 21:52
• Updated: 30/Jun/15 04:52
• Resolved: 06/Apr/15 21:01

Description

Split grouping is currently done using a file size measurement which is the exact size of the split as it stays at rest on HDFS.

This is not valid for columnar formats and especially suffers from highly compressible data skews.

Issue Links

• is related to: HIVE-7428 OrcSplit fails to account for columnar projections in its size estimates