[KYLIN-3679] Fetch Kafka topic with Spark streaming

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark Engine
• Assignee: weibin0516
• Reporter: Shao Feng Shi
• Created: 11/Nov/18 11:22
• Updated: 28/Feb/20 14:03

Description

Now Kylin uses a MR job to fetch Kafka messages in parallel and then persist to HDFS for subsequent processing. If user selects to use Spark engine, we can use Spark streaming API to do this. Spark streaming can read the Kafka message in a given offset range as a RDD, then it would be easy to process;

https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html 

With Spark streaming, Kylin can also easily connect with other data source like Kinesis, Flume, etc.