Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6599

Improve usability and reliability of Kinesis stream

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: DStreams
    • Labels:
      None
    • Target Version/s:

      Description

      Usability improvements:
      API improvements, AWS SDK upgrades, etc.

      Reliability improvements:
      Currently, the KinesisReceiver can loose some data in the case of certain failures (receiver and driver failures). Using the write ahead logs can mitigate some of the problem, but it is not ideal because WALs dont work with S3 (eventually consistency, etc.) which is the most likely file system to be used in the EC2 environment. Hence, we have to take a different approach to improving reliability for Kinesis. See https://issues.apache.org/jira/browse/SPARK-9215 for more details.

        Attachments

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              tdas Tathagata Das Assign to me
              Reporter:
              tdas Tathagata Das

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment