[KYLIN-1094] improve performance of spark cubing

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: v1.4.0
• Fix Version/s: Backlog
• Component/s: Spark Engine
• Assignee: Dong Li
• Reporter: Qianhao Zhou
• Created: 25/Oct/15 16:54
• Updated: 29/Nov/17 07:26
• Resolved: 29/Nov/17 07:26

Description

POC result of spark cubing shows that, on a dataset of 150 million records, MR is about 100% faster than Spark, however we believe that Spark could be at least at same speed as MR, so optimization is needed here.
We are asking Spark community for help now.

the cluster info:
vm: 8 nodes * (128G mem + 64 core)
hadoop cluster: hdp 2.2.6
spark running mode: yarn-client
spark version: 1.5.1