[SPARK-6051] Add an option for DirectKafkaInputDStream to commit the offsets into ZK

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.3.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Saisai Shao
• Created: 27/Feb/15 04:51
• Updated: 09/Dec/15 16:12
• Resolved: 10/Jul/15 07:39

Description

Currently in DirectKafkaInputDStream, offset is managed by Spark Streaming itself without ZK or Kafka involved, which will make several third-party offset monitoring tools fail to monitor the status of Kafka consumer. So here as a option to commit the offset to ZK when each job is finished, the process is implemented as a asynchronized way, so the main processing flow will not be blocked, already tested with KafkaOffsetMonitor tools.

Issue Links

• links to: [Github] Pull Request #4805 (jerryshao)