Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18337

Memory Sink should be able to recover from checkpoints in Complete OutputMode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.3, 2.1.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      Memory sinks are not meant to be fault tolerant, but there are certain cases, where it would be nice that it can recover from checkpoints. In cases where you may use a scalable StateStore in StructuredStreaming (when you have an aggregation), and you add a filter based on a key or value in your state, it's nice to be able to continue from where you left off after failures.

      For correctness reasons, the output will ONLY be correct in Complete mode, so we could support that.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brkyvz Burak Yavuz
                Reporter:
                brkyvz Burak Yavuz
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: