Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11324

Flag to close Write Ahead Log after writing

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: DStreams
    • Labels:
      None
    • Target Version/s:

      Description

      Currently the Write Ahead Log in Spark Streaming flushes data as writes need to be made. S3 does not support flushing of data, data is written once the stream is actually closed.

      In case of failure, the data for the last minute (default rolling interval) will not be properly written. Therefore we need a flag to close the stream after the write, so that we achieve read after write consistency.

        Attachments

          Activity

            People

            • Assignee:
              brkyvz Burak Yavuz
              Reporter:
              brkyvz Burak Yavuz
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: