Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21667

ConsoleSink should not fail streaming query with checkpointLocation option

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.2.1, 2.3.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      As agreed on the Spark users mailing list in the thread "[SS] Console sink not supporting recovering from checkpoint location? Why?" in which Michael Armbrust said:

      I think there is really no good reason for this limitation.

      Using ConsoleSink should therefore not fail a streaming query when used with checkpointLocation option.

      // today's build from the master
      scala> spark.version
      res8: String = 2.3.0-SNAPSHOT
      
      scala> val q = records.
           |   writeStream.
           |   format("console").
           |   option("truncate", false).
           |   option("checkpointLocation", "/tmp/checkpoint"). // <--
      checkpoint directory
           |   trigger(Trigger.ProcessingTime(10.seconds)).
           |   outputMode(OutputMode.Update).
           |   start
      org.apache.spark.sql.AnalysisException: This query does not support recovering from checkpoint location. Delete /tmp/checkpoint/offsets to start over.;
        at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:222)
        at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:278)
        at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:284)
        ... 61 elided
      

      The "trigger" is SPARK-16116 and this line in particular.

      This also relates to SPARK-19768 that was resolved as not a bug.

        Attachments

          Activity

            People

            • Assignee:
              rekhajoshm Rekha Joshi
              Reporter:
              jlaskowski Jacek Laskowski
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: