[KYLIN-3452] Optimize spark cubing memory footprint

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v2.5.0
• Component/s: Spark Engine
• Assignee: Shao Feng Shi
• Reporter: Shao Feng Shi
• Created: 15/Jul/18 09:56
• Updated: 17/Sep/18 00:59
• Resolved: 02/Aug/18 06:13

Description

Noticed that there are some steps can be optimized:

1) Cuboid.findForMandatory() will create new object each time, which this can be cached;

2) Executor repeatedly load dictionary, not hit the cache;

3) NDCuboidBuilder and RowkeyEncoder, can optimize for less array copy;