Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21667

ConsoleSink should not fail streaming query with checkpointLocation option

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.3.0
    • 2.2.1, 2.3.0
    • Structured Streaming
    • None

    Description

      As agreed on the Spark users mailing list in the thread "[SS] Console sink not supporting recovering from checkpoint location? Why?" in which marmbrus said:

      I think there is really no good reason for this limitation.

      Using ConsoleSink should therefore not fail a streaming query when used with checkpointLocation option.

      // today's build from the master
      scala> spark.version
      res8: String = 2.3.0-SNAPSHOT
      
      scala> val q = records.
           |   writeStream.
           |   format("console").
           |   option("truncate", false).
           |   option("checkpointLocation", "/tmp/checkpoint"). // <--
      checkpoint directory
           |   trigger(Trigger.ProcessingTime(10.seconds)).
           |   outputMode(OutputMode.Update).
           |   start
      org.apache.spark.sql.AnalysisException: This query does not support recovering from checkpoint location. Delete /tmp/checkpoint/offsets to start over.;
        at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:222)
        at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:278)
        at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:284)
        ... 61 elided
      

      The "trigger" is SPARK-16116 and this line in particular.

      This also relates to SPARK-19768 that was resolved as not a bug.

      Attachments

        Activity

          People

            rekhajoshm Rekha Joshi
            jlaskowski Jacek Laskowski
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: