Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: mapreduce, scaling
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Added couple of API's to Admin.java:

      Returns region load map of all regions hosted on a region server
      Map<byte[], RegionLoad> getRegionLoad(ServerName sn) throws IOException;

      Returns region load map of all regions of a table hosted on a region server
      Map<byte[], RegionLoad> getRegionLoad(ServerName sn, TableName tableName) throws IOException

      Added an API to region server:

      public GetRegionLoadResponse getRegionLoad(RpcController controller,
          GetRegionLoadRequest request) throws ServiceException;

      Primary intention is to use this API for RegionSizeCalculator and not rely on Master for ClusterStatus. On large clusters, ClusterStatus() can take a long time. IfMaster is down/busy, then some of the jobs timeout/fail. Other possible uses:
      1. If there is a lighter version of GetClusterStatus API (i.e without the ServerLoad for each RS), then custom maintenance tools can be better. In current world ClusterStatus is heavy. With the new APIs, each API's payload is smaller and distributed. So custom tools can call getRegionLoad() when needed, it will be more accurate. This helps with large clusters. For tools that don't need RegionLoad, the lighter version of API is fine enough.
      2. Another use case is a tool like RSTop - since we can see selective metrics at RegionLevel (possibly even deltas between each RPC to the server).
      Show
      Added couple of API's to Admin.java: Returns region load map of all regions hosted on a region server Map<byte[], RegionLoad> getRegionLoad(ServerName sn) throws IOException; Returns region load map of all regions of a table hosted on a region server Map<byte[], RegionLoad> getRegionLoad(ServerName sn, TableName tableName) throws IOException Added an API to region server: public GetRegionLoadResponse getRegionLoad(RpcController controller,     GetRegionLoadRequest request) throws ServiceException; Primary intention is to use this API for RegionSizeCalculator and not rely on Master for ClusterStatus. On large clusters, ClusterStatus() can take a long time. IfMaster is down/busy, then some of the jobs timeout/fail. Other possible uses: 1. If there is a lighter version of GetClusterStatus API (i.e without the ServerLoad for each RS), then custom maintenance tools can be better. In current world ClusterStatus is heavy. With the new APIs, each API's payload is smaller and distributed. So custom tools can call getRegionLoad() when needed, it will be more accurate. This helps with large clusters. For tools that don't need RegionLoad, the lighter version of API is fine enough. 2. Another use case is a tool like RSTop - since we can see selective metrics at RegionLevel (possibly even deltas between each RPC to the server).

      Description

      RegionSizeCalculator is needed for better split generation of MR jobs. This requires RegionLoad which can be obtained via ClusterStatus, i.e. accessing Master. We don't want master to be in this path.

      The proposal is to add an API to the RegionServer that gets RegionLoad of all regions hosted on it or those of a table if specified. RegionSizeCalculator can use the latter.

        Attachments

        1. HBASE-16169.master.000.patch
          152 kB
          Thiruvel Thirumoolan
        2. HBASE-16169.master.001.patch
          153 kB
          Thiruvel Thirumoolan
        3. HBASE-16169.master.002.patch
          153 kB
          Thiruvel Thirumoolan
        4. HBASE-16169.master.003.patch
          153 kB
          Thiruvel Thirumoolan
        5. HBASE-16169.master.004.patch
          153 kB
          Thiruvel Thirumoolan
        6. HBASE-16169.master.005.patch
          121 kB
          Thiruvel Thirumoolan
        7. HBASE-16169.master.006.patch
          121 kB
          Thiruvel Thirumoolan
        8. HBASE-16169.master.007.patch
          121 kB
          Thiruvel Thirumoolan
        9. HBASE-16169.master.007.patch
          121 kB
          Michael Stack
        10. HBASE-16169.master.008.patch
          121 kB
          Thiruvel Thirumoolan

          Issue Links

            Activity

              People

              • Assignee:
                thiruvel Thiruvel Thirumoolan
                Reporter:
                thiruvel Thiruvel Thirumoolan
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: