[SPARK-17606] New batches are not created when there are 1000 created after restarting streaming from checkpoint.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.1
• Component/s: DStreams
• Labels: bulk-closed
• External issue URL: http://stackoverflow.com/questions/39578588/spark-what-happened-when-restart-streaming-from-checkpoint
• Assignee: Unassigned
• Reporter: etienne
• Created: 20/Sep/16 09:17
• Updated: 21/May/19 04:32
• Resolved: 21/May/19 04:32

Description

When spark restarts from a checkpoint after being down for a while.
It recreates missing batch since the down time.

When there are few missing batches, spark creates new incoming batch every batchTime, but when there is enough missing time to create 1000 batches no new batch is created.

So when all these batch are completed the stream is idle ...

I think there is a rigid limit set somewhere.

I was expecting that spark continue to recreate missed batches, maybe not all at once ( because it's look like it's cause driver memory problem ), and then recreate batches each batchTime.

Another solution would be to not create missing batches but still restart the direct input.

Right know for me the only solution to restart a stream after a long break it to remove the checkpoint to allow the creation of a new stream. But losing all my states.

ps : I'm speaking about direct Kafka input because it's the source I'm currently using, I don't know what happens with other sources.