[SPARK-21836] [STREAMING] Retry when kafka broker is down in kafka-streaming-0-8

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.3, 2.1.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: yue long
• Created: 25/Aug/17 08:16
• Updated: 25/Aug/17 08:43
• Resolved: 25/Aug/17 08:38

Description

When using the package spark-streaming-kafka-0-8 for accessing kafka in spark dstream, many user will face the "could not find leader" exception if some of kafka brokers are down. This will cause the whole streaming fail, like SPARK-18983 said. The failed kafka brokers may also cause other problems when creating Dstream or creating the batch job.

Even though the down of kafka broker is not the bug of spark streaming, we can avoid this failure in spark streaming. Especially for the reason that kakfa cluster is not always stable in the real production.
Actually, our streaming may take a few minutes to re-submit it but the kafka cluster will only take a few seconds to replace the failed broker by an alive one!

Does anyoner think we should add some retry logic when kakfa broker is down? I have implement this function in spark 1.6.3 and spark 2.1.0, and test them. If we implement this function, it will reduce the failure number of kafka-streaming which may help streaming users.

Issue Links

• duplicates: SPARK-18983 Couldn't find leader offsets exception when the one of kafka cluster brokers is down
• is related to: SPARK-18983 Couldn't find leader offsets exception when the one of kafka cluster brokers is down
• is related to: SPARK-6431 Couldn't find leader offsets exception when creating KafkaDirectStream