[SPARK-21667] ConsoleSink should not fail streaming query with checkpointLocation option

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Structured Streaming
• Assignee: Rekha Joshi
• Reporter: Jacek Laskowski
• Created: 08/Aug/17 09:35
• Updated: 10/Nov/17 23:19
• Resolved: 10/Nov/17 23:19

Description

As agreed on the Spark users mailing list in the thread "[SS] Console sink not supporting recovering from checkpoint location? Why?" in which marmbrus said:

I think there is really no good reason for this limitation.

Using ConsoleSink should therefore not fail a streaming query when used with checkpointLocation option.

// today's build from the master
scala> spark.version
res8: String = 2.3.0-SNAPSHOT

scala> val q = records.
     |   writeStream.
     |   format("console").
     |   option("truncate", false).
     |   option("checkpointLocation", "/tmp/checkpoint"). // <--
checkpoint directory
     |   trigger(Trigger.ProcessingTime(10.seconds)).
     |   outputMode(OutputMode.Update).
     |   start
org.apache.spark.sql.AnalysisException: This query does not support recovering from checkpoint location. Delete /tmp/checkpoint/offsets to start over.;
  at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:222)
  at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:278)
  at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:284)
  ... 61 elided

The "trigger" is SPARK-16116 and this line in particular.

This also relates to SPARK-19768 that was resolved as not a bug.

Issue Links

• links to: [Github] Pull Request #19407 (rekhajoshm)