Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Not A Problem
-
2.4.1, 2.4.3
-
None
-
None
Description
If a DirectKafkaInputDStream is started in one thread and is being stopped in another thread (e.g. by shutdown hook) a java.util.ConcurrentModificationException (KafkaConsumer is not safe for multi-threaded access) is thrown.
This happens even if "spark.streaming.kafka.consumer.cache.enabled" is set to "false".
MWE: https://gist.github.com/ov7a/fc783315ea252a03d51804ce326a13b1
Attachments
Issue Links
- is related to
-
SPARK-19185 ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing
- Closed