[KYLIN-2518] Improve the sampling performance of FactDistinctColumns step

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v2.0.0
• Assignee: XIE FAN
• Reporter: XIE FAN
• Created: 27/Mar/17 06:47
• Updated: 30/Apr/17 12:21
• Resolved: 10/Apr/17 14:59

Description

The method putRowKeyToHLL() in FactDistinctColumnsMapper can be very slow when sampling rate is high. After carefully profiling, we believe that it's performance can be improved by modifying it's hash method. At the same time, we also found an algorithm that can estimate the row nums of each cuboid accurately with a lower sampling rate. I will share more test results and details of the algorithm once after this issue is done.