[KYLIN-4925] Use Spark 3 as build and query engine for Kylin 4

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v4.0.0
• Component/s: Job Engine, Query Engine
• Assignee: Congling Xia
• Reporter: Congling Xia
• Created: 04/Mar/21 16:18
• Updated: 29/Jul/21 05:57
• Resolved: 29/Jul/21 05:57

Description

Spark 3 provides much more developed AQE framework, DPP and the increased support for GPUs and Kubernetes. The perspectives of performance increase are very promising. 

Currently, we try to run Kylin 4.0 with Spark 3.0 as build & query engine. We'd like to share the code changes here.

Sub-Tasks

• Make Spark3.1 compatible with hadoop2.7 and hive1.2 in standalone docker image: Open