Description
Streaming consider the receiver as a common spark job, thus if an error occurs in the receiver's logical(after 4 times(default) retries ), streaming will no longer get any data but the streaming job is still running.
A general scenario is that: we config the `spark.streaming.receiver.writeAheadLog.enable` as true to use the `ReliableKafkaReceiver` but do not set the checkpoint dir. Then the receiver will soon be shut down but the streaming is alive.
Attachments
Issue Links
- is superceded by
-
SPARK-8882 A New Receiver Scheduling Mechanism to solve unbalanced receivers
-
- Resolved
-