HBase
  1. HBase
  2. HBASE-8691

High-Throughput Streaming Scan API

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.95.0
    • Fix Version/s: None
    • Component/s: Scanners
    • Labels:

      Description

      I've done some working testing various ways to refactor and optimize Scans in HBase, and have found that performance can be dramatically increased by the addition of a streaming scan API. The attached code constitutes a proof of concept that shows performance increases of almost 4x in some workloads.

      I'd appreciate testing, replication, and comments. If the approach seems viable, I think such an API should be built into some future version of HBase.

      1. StreamServletDirect.java
        3 kB
        Sandy Pratt
      2. ScannerTest.java
        5 kB
        Sandy Pratt
      3. StreamReceiverDirect.java
        4 kB
        Sandy Pratt
      4. StreamHRegionServer.java
        1 kB
        Sandy Pratt
      5. RecordReceiver.java
        0.2 kB
        Sandy Pratt
      6. README.txt
        8 kB
        Sandy Pratt
      7. HRegionServlet.java
        0.3 kB
        Sandy Pratt

        Issue Links

          Activity

          Sandy Pratt created issue -
          Sandy Pratt made changes -
          Field Original Value New Value
          Attachment HRegionServlet.java [ 12586266 ]
          Attachment README.txt [ 12586267 ]
          Attachment RecordReceiver.java [ 12586268 ]
          Attachment ScannerTest.java [ 12586269 ]
          Attachment StreamHRegionServer.java [ 12586270 ]
          Attachment StreamReceiverDirect.java [ 12586271 ]
          Attachment StreamServletDirect.java [ 12586272 ]
          Sandy Pratt made changes -
          Summary High-Throuhput Streaming Scan API High-Throughput Streaming Scan API
          Sandy Pratt made changes -
          Labels patch perfomance scan perfomance scan
          Sandy Pratt made changes -
          Attachment ScannerTest.java [ 12586274 ]
          Sandy Pratt made changes -
          Attachment ScannerTest.java [ 12586269 ]
          Sandy Pratt made changes -
          Attachment StreamServletDirect.java [ 12586275 ]
          Sandy Pratt made changes -
          Attachment StreamServletDirect.java [ 12586272 ]
          Ted Yu made changes -
          Assignee Sandy Pratt [ prattrs ]
          Andrew Purtell made changes -
          Link This issue duplicates HBASE-13071 [ HBASE-13071 ]

            People

            • Assignee:
              Sandy Pratt
              Reporter:
              Sandy Pratt
            • Votes:
              0 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

              • Created:
                Updated:

                Development