[KYLIN-4213] The new build engine with Spark-SQL

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v4.0.0-alpha
• Component/s: Job Engine
• Assignee: yiming.xu
• Reporter: yiming.xu
• Created: 29/Oct/19 13:47
• Updated: 13/Sep/20 12:17
• Resolved: 03/Sep/20 09:21

Description

1. Use Spark-SQL to compute cuboid, build cuboid A, B, C , Sum(D) is sql "select A B C Sum(D) from table group by A, B, C".
2. To void many memory errors or other exceptions, we can auto set spark conf with build job.E.g use adaptive execution.
3. The snapshot table will save a table with parquet format.

Issue Links

• Blocked: KYLIN-4188 Parquet as Cube storage V2
• is related to: KYLIN-4659 Prepare a technical preview version for Parquet Storage

Sub-Tasks

• Add new engine type: Open
• New task SparkExecutable: Open
• New build engine: Open
• Auto spark conf parameter tuning: Open
• Add new interface to extract metadata to build: Open
• Add new storage interface to support write to any format: Open
• Create flat table wich spark sql: Closed