Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2447

Add common solution for sending upsert actions to HBase (put, deletes, and increment)

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: DStreams, Spark Core
    • Labels:
      None

      Description

      Going to review the design with Tdas today.

      But first thoughts is to have an extension of VoidFunction that handles the connection to HBase and allows for options such as turning auto flush off for higher through put.

      Need to answer the following questions first.

      • Can it be written in Java or should it be written in Scala?
      • What is the best way to add the HBase dependency? (will review how Flume does this as the first option)
      • What is the best way to do testing? (will review how Flume does this as the first option)
      • How to support python? (python may be a different Jira it is unknown at this time)

      Goals:

      • Simple to use
      • Stable
      • Supports high load
      • Documented (May be in a separate Jira need to ask Tdas)
      • Supports Java, Scala, and hopefully Python
      • Supports Streaming and normal Spark

        Attachments

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              ted.m Theodore michael Malaska Assign to me
              Reporter:
              ted.m Theodore michael Malaska

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment