[KYLIN-679] Adding Spark Support to Apache Kylin

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v2.0.0
• Component/s: Spark Engine
• Assignee: Unassigned
• Reporter: Luke Han
• Created: 13/Apr/15 13:28
• Updated: 02/Feb/18 02:22
• Resolved: 02/Feb/18 02:22

Description

Challenges in current architecture:

High latency when reading data from Hive
--Several hours to fetch data when join big tables
--Route to SQL-on-Hadoop turned off due to performance issue

Time-to-Market of data latency
--Huge IO & Network traffic with MR jobs

Streaming
--Streaming process and pre-calculate cubes

Where Spark could bring benefits to Kylin:

Integrating with Spark SQL:
--Option I: Read data from SparkSQL instead of Hive
--Option II: Route unsupported queries to SparkSQL
--Option III: Kylin to be OLAP source of SparkSQL

Spark Cube Build Engine
--Efficiency cube generate engine with Spark

Spark Streaming
--Leverage SparkStreaming for StreamingOLAP

HBase?
--Any idea?

Issue Links

• incorporates: KYLIN-741 Read data from SparkSQL
• incorporates: KYLIN-742 Route unsupported queries to Hive (on Spark)
• incorporates: KYLIN-743 Kylin to be OLAP source of SparkSQL
• incorporates: KYLIN-744 Spark Cube Build Engine