[SPARK-22606] There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access"

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.2.0
• Fix Version/s: 2.4.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: eaton
• Created: 25/Nov/17 09:04
• Updated: 12/Dec/18 08:46
• Resolved: 12/Dec/18 08:40

Description

If the value of param 'spark.streaming.concurrentJobs' is more than one, and the value of param 'spark.executor.cores' is more than one, there may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access"

Issue Links

• duplicates: SPARK-19185 ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing
• links to: [Github] Pull Request #19819 (eatoncys)