Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6599

Improve usability and reliability of Kinesis stream

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: DStreams
    • Labels:
      None
    • Target Version/s:

      Description

      Usability improvements:
      API improvements, AWS SDK upgrades, etc.

      Reliability improvements:
      Currently, the KinesisReceiver can loose some data in the case of certain failures (receiver and driver failures). Using the write ahead logs can mitigate some of the problem, but it is not ideal because WALs dont work with S3 (eventually consistency, etc.) which is the most likely file system to be used in the EC2 environment. Hence, we have to take a different approach to improving reliability for Kinesis. See https://issues.apache.org/jira/browse/SPARK-9215 for more details.

        Issue Links

          Activity

          Hide
          rtshadow Przemyslaw Pastuszka added a comment -

          Is there any work being done on this? Can I help somehow?

          Show
          rtshadow Przemyslaw Pastuszka added a comment - Is there any work being done on this? Can I help somehow?
          Hide
          tdas Tathagata Das added a comment -

          I am already working on this and have made the necessary changes, will
          produce a PR shortly.

          On Monday, June 29, 2015, Przemyslaw Pastuszka (JIRA) <jira@apache.org>


          Sent from Gmail Mobile

          Show
          tdas Tathagata Das added a comment - I am already working on this and have made the necessary changes, will produce a PR shortly. On Monday, June 29, 2015, Przemyslaw Pastuszka (JIRA) <jira@apache.org> – Sent from Gmail Mobile
          Hide
          smartnut007 Arun Ramakrishnan added a comment -

          Tathagata Das Curious about the design docs for this.

          Show
          smartnut007 Arun Ramakrishnan added a comment - Tathagata Das Curious about the design docs for this.
          Hide
          tdas Tathagata Das added a comment -

          Arun Ramakrishnan If you are looking for the design docs for the reliability improvements, it is in this JIRA - https://issues.apache.org/jira/browse/SPARK-9215

          Show
          tdas Tathagata Das added a comment - Arun Ramakrishnan If you are looking for the design docs for the reliability improvements, it is in this JIRA - https://issues.apache.org/jira/browse/SPARK-9215
          Hide
          tdas Tathagata Das added a comment -

          Apologies to all who received mails because they were watching. I did a clean up of this JIRA and all the related JIRAs.

          Show
          tdas Tathagata Das added a comment - Apologies to all who received mails because they were watching. I did a clean up of this JIRA and all the related JIRAs.

            People

            • Assignee:
              tdas Tathagata Das
              Reporter:
              tdas Tathagata Das
            • Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development