HBase
  1. HBase
  2. HBASE-6136

Optimize the multi get to be executed in parallel.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Optimize the multi get to be executed in parallel using multi thread in hbase client

      Description

      Currently the multiAction will be executed by one IPC handler thread in the server side. It would be more efficient to optimize the multi get particularly to be executed in parallel since the multiget doesn't need to write to the WAL.

        Activity

        Hide
        Lars Hofhansl added a comment -

        That still seems like a bad idea. You'll hammer the region servers with individual get requests.

        Show
        Lars Hofhansl added a comment - That still seems like a bad idea. You'll hammer the region servers with individual get requests.
        Hide
        Liu Shaohui added a comment -

        if we group the Get requests by RegionServer or Region, the multi gets in same region will be executed sequentially.
        What we want is that multi gets in same region can be executed in parallel to reduce the latency. Usually the number of get in this kind of multi gets is between 10 and 50.

        Show
        Liu Shaohui added a comment - if we group the Get requests by RegionServer or Region, the multi gets in same region will be executed sequentially. What we want is that multi gets in same region can be executed in parallel to reduce the latency. Usually the number of get in this kind of multi gets is between 10 and 50.
        Hide
        Lars Hofhansl added a comment -

        I think we can do better than that. This will fire individual Get requests at great cost to the region server.
        We should group by RegionServer and break the RegionServer portions into sub chunks.
        Or we could group by RegionServer and Region.

        See HConnectionManager.HConnectionImplementation.processBatchCallback(...).

        Show
        Lars Hofhansl added a comment - I think we can do better than that. This will fire individual Get requests at great cost to the region server. We should group by RegionServer and break the RegionServer portions into sub chunks. Or we could group by RegionServer and Region. See HConnectionManager.HConnectionImplementation.processBatchCallback(...).
        Hide
        Liu Shaohui added a comment -

        Any suggestion about this patch?
        Now it's not easy to impl multi-thread multiAction in the server side.
        The mulit-thread gets in client side will decrease the latency of multi gets.

        Show
        Liu Shaohui added a comment - Any suggestion about this patch? Now it's not easy to impl multi-thread multiAction in the server side. The mulit-thread gets in client side will decrease the latency of multi gets.
        Hide
        Lars Hofhansl added a comment -

        Also see HBASE-8362 as another possible optimization.

        Show
        Lars Hofhansl added a comment - Also see HBASE-8362 as another possible optimization.
        Hide
        Liu Shaohui added a comment -

        Optimize the multi get to be executed in parallel using multi threads in hbase client

        Show
        Liu Shaohui added a comment - Optimize the multi get to be executed in parallel using multi threads in hbase client
        Hide
        Kannan Muthukkaruppan added a comment -

        +1, as a configurable option.

        Show
        Kannan Muthukkaruppan added a comment - +1, as a configurable option.

          People

          • Assignee:
            Unassigned
            Reporter:
            Liyin Tang
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development