[SPARK-17921] checkpointLocation being set in memory streams fail after restart. Should fail fast

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.0.0, 2.0.1
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Burak Yavuz
• Created: 13/Oct/16 20:58
• Updated: 06/Dec/16 20:29
• Resolved: 06/Dec/16 20:29

Description

The checkpointLocation option in memory streams in StructuredStreaming is not used during recovery. However, it can use this location if it is being set. However, during recovery, if this location was set, we get an exception saying that we will not use this location for recovery, please delete it.

It's better to just fail before you start the stream in the first place

Issue Links

• is cloned by: SPARK-18749 CLONE - checkpointLocation being set in memory streams fail after restart. Should fail fast
• relates to: SPARK-18337 Memory Sink should be able to recover from checkpoints in Complete OutputMode
• links to: [Github] Pull Request #15470 (brkyvz)