Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1647

Prevent data loss when Streaming driver goes down

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • None
    • None
    • DStreams
    • None

    Description

      Currently when the driver goes down, any uncheckpointed data is lost from within spark. If the system from which messages are pulled can replay messages, the data may be available - but for some systems, like Flume this is not the case.

      Also, all windowing information is lost for windowing functions.

      We must persist raw data somehow, and be able to replay this data if required. We also must persist windowing information with the data itself.

      This will likely require quite a bit of work to complete and probably will have to be split into several sub-jiras.

      Attachments

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. SPARK-3129 Prevent data loss in Spark Streaming on driver failure using Write Ahead Logs

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-1645 Improve Spark Streaming compatibility with Flume

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              hshreedharan Hari Shreedharan
              hshreedharan Hari Shreedharan
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: