Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18749

CLONE - checkpointLocation being set in memory streams fail after restart. Should fail fast

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Invalid
    • 2.0.0, 2.0.1
    • None
    • Structured Streaming
    • None

    Description

      The checkpointLocation option in memory streams in StructuredStreaming is not used during recovery. However, it can use this location if it is being set. However, during recovery, if this location was set, we get an exception saying that we will not use this location for recovery, please delete it.

      It's better to just fail before you start the stream in the first place

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              brkyvz Burak Yavuz
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: