HBase
  1. HBase
  2. HBASE-4755

HBase based block placement in DFS

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.94.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The feature as is only useful for HBase clusters that care about data locality on regionservers, but this feature can also enable a lot of nice features down the road.

      The basic idea is as follows: instead of letting HDFS determine where to replicate data (r=3) by place blocks on various regions, it is better to let HBase do so by providing hints to HDFS through the DFS client. That way instead of replicating data at a blocks level, we can replicate data at a per-region level (each region owned by a promary, a secondary and a tertiary regionserver). This is better for 2 things:

      • Can make region failover faster on clusters which benefit from data affinity
      • On large clusters with random block placement policy, this helps reduce the probability of data loss

      The algo is as follows:

      • Each region in META will have 3 columns which are the preferred regionservers for that region (primary, secondary and tertiary)
      • Preferred assignment can be controlled by a config knob
      • Upon cluster start, HMaster will enter a mapping from each region to 3 regionservers (random hash, could use current locality, etc)
      • The load balancer would assign out regions preferring region assignments to primary over secondary over tertiary over any other node
      • Periodically (say weekly, configurable) the HMaster would run a locality checked and make sure the map it has for region to regionservers is optimal.

      Down the road, this can be enhanced to control region placement in the following cases:

      • Mixed hardware SKU where some regionservers can hold fewer regions
      • Load balancing across tables where we dont want multiple regions of a table to get assigned to the same regionservers
      • Multi-tenancy, where we can restrict the assignment of the regions of some table to a subset of regionservers, so an abusive app cannot take down the whole HBase cluster.
      1. hbase-4755-notes.txt
        3 kB
        Devaraj Das
      2. 4755-wip-1.patch
        131 kB
        Devaraj Das

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Karthik Ranganathan created issue -
          Hide
          Todd Lipcon added a comment -

          This will need appropriate HDFS-side support to provide location hints to the NN when allocating blocks. Is there an HDFS JIRA already filed for it?

          Show
          Todd Lipcon added a comment - This will need appropriate HDFS-side support to provide location hints to the NN when allocating blocks. Is there an HDFS JIRA already filed for it?
          Hide
          Karthik Ranganathan added a comment -

          Yes, totally. Not sure if there is a JIRA, but Pritam already has a patch for adding favored nodes on an addBlock call.

          Pritam - is there an HDFS JIRA for this?

          Show
          Karthik Ranganathan added a comment - Yes, totally. Not sure if there is a JIRA, but Pritam already has a patch for adding favored nodes on an addBlock call. Pritam - is there an HDFS JIRA for this?
          Christopher Gist made changes -
          Field Original Value New Value
          Assignee Christopher Gist [ cgist ]
          Hide
          Andrew Purtell added a comment -

          +1

          We could also build upon this to locate read slaves (should that be separately pursued) on the secondary and tertiary node options.

          Show
          Andrew Purtell added a comment - +1 We could also build upon this to locate read slaves (should that be separately pursued) on the secondary and tertiary node options.
          Hide
          Andrew Purtell added a comment -

          Proposing for 0.94.

          Show
          Andrew Purtell added a comment - Proposing for 0.94.
          Andrew Purtell made changes -
          Affects Version/s 0.94.0 [ 12316419 ]
          Hide
          Devaraj Das added a comment -

          HDFS-2576 is the hdfs jira that should be addressed. I have started digging in in this area with an aim to get this feature in hbase trunk (with the corresponding hdfs changes).

          Show
          Devaraj Das added a comment - HDFS-2576 is the hdfs jira that should be addressed. I have started digging in in this area with an aim to get this feature in hbase trunk (with the corresponding hdfs changes).
          Hide
          stack added a comment -

          Devaraj Das Sweet!

          Show
          stack added a comment - Devaraj Das Sweet!
          Andrew Purtell made changes -
          Link This issue is related to HDFS-2576 [ HDFS-2576 ]
          Nicolas Liochon made changes -
          Link This issue is required by HBASE-5843 [ HBASE-5843 ]
          Hide
          Devaraj Das added a comment -

          This is the first stab at this issue. The patch is a huge WIP patch (and has a lot of classes from the 0.89 branch). I think I am going to break this up into multiple jiras.

          1) In this patch, what I have done is only handled the createTable flow to honor region placements. It's really random assignment, but has the plumbing for updating meta with the region locations, etc.

          2) The next step is to have the creation of store files honor this region placement.

          3) The third step is to be able to run tools against meta to ensure the placement looks optimal.

          There could be more steps involved but the above are the high level ones for now, and probably each could be a subtask.

          Show
          Devaraj Das added a comment - This is the first stab at this issue. The patch is a huge WIP patch (and has a lot of classes from the 0.89 branch). I think I am going to break this up into multiple jiras. 1) In this patch, what I have done is only handled the createTable flow to honor region placements. It's really random assignment, but has the plumbing for updating meta with the region locations, etc. 2) The next step is to have the creation of store files honor this region placement. 3) The third step is to be able to run tools against meta to ensure the placement looks optimal. There could be more steps involved but the above are the high level ones for now, and probably each could be a subtask.
          Devaraj Das made changes -
          Attachment 4755-wip-1.patch [ 12569594 ]
          Hide
          Enis Soztutar added a comment -

          2) The next step is to have the creation of store files honor this region placement.

          Is this patch useful without giving hints to DFSClient about block placement. I though we still don't have the pluming in hadoop yet.

          Show
          Enis Soztutar added a comment - 2) The next step is to have the creation of store files honor this region placement. Is this patch useful without giving hints to DFSClient about block placement. I though we still don't have the pluming in hadoop yet.
          Hide
          Devaraj Das added a comment -

          Is this patch useful without giving hints to DFSClient about block placement.

          The stuff in Hadoop is tracked in HDFS-2576. I am thinking that we can use the reflection to figure out whether the underlying Hadoop supports the API to do with block placements or not.

          Show
          Devaraj Das added a comment - Is this patch useful without giving hints to DFSClient about block placement. The stuff in Hadoop is tracked in HDFS-2576 . I am thinking that we can use the reflection to figure out whether the underlying Hadoop supports the API to do with block placements or not.
          Devaraj Das made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          Devaraj Das added a comment -

          Added two subtasks for now - HBASE-7932, HBASE-7942

          Show
          Devaraj Das added a comment - Added two subtasks for now - HBASE-7932 , HBASE-7942
          Hide
          Jonathan Hsieh added a comment -

          Some design level questions:

          I'm curious about where the block location policy (the load balancer) lives or is intended to live. Is it in HDFS or in HBase? (and why). If hbase's region balancer and hdfs block balancer are in conflict who wins? How should they work together?

          Are there alternatives you've considered?

          What is the story for cases where node membership changes (DN's and RS's)? (what if the hinted rs/dn is not there?)

          Show
          Jonathan Hsieh added a comment - Some design level questions: I'm curious about where the block location policy (the load balancer) lives or is intended to live. Is it in HDFS or in HBase? (and why). If hbase's region balancer and hdfs block balancer are in conflict who wins? How should they work together? Are there alternatives you've considered? What is the story for cases where node membership changes (DN's and RS's)? (what if the hinted rs/dn is not there?)
          Hide
          Devaraj Das added a comment -

          I'm curious about where the block location policy (the load balancer) lives or is intended to live. Is it in HDFS or in HBase? (and why). If hbase's region balancer and hdfs block balancer are in conflict who wins? How should they work together?

          The block placement policy is there in HBase presently. This is because HBase specifies the location of the blocks on a per file basis (as per the current approach). The HDFS side treats these as hints.
          The balancers run independently, and if the HDFS rebalancer damages the block placements, it'd be repaired at the time compactions in hbase happen.

          What is the story for cases where node membership changes (DN's and RS's)? (what if the hinted rs/dn is not there?)

          HBase should be able to handle these cases (and the new patches will handle these). In addition, there would be a tool (that would be a subtask of this jira) that should be run periodically that would look at the block placements and update region maps (region -> favored nodes) in the meta table in HBase to keep the mapping more optimal in terms of locality of data.

          Show
          Devaraj Das added a comment - I'm curious about where the block location policy (the load balancer) lives or is intended to live. Is it in HDFS or in HBase? (and why). If hbase's region balancer and hdfs block balancer are in conflict who wins? How should they work together? The block placement policy is there in HBase presently. This is because HBase specifies the location of the blocks on a per file basis (as per the current approach). The HDFS side treats these as hints. The balancers run independently, and if the HDFS rebalancer damages the block placements, it'd be repaired at the time compactions in hbase happen. What is the story for cases where node membership changes (DN's and RS's)? (what if the hinted rs/dn is not there?) HBase should be able to handle these cases (and the new patches will handle these). In addition, there would be a tool (that would be a subtask of this jira) that should be run periodically that would look at the block placements and update region maps (region -> favored nodes) in the meta table in HBase to keep the mapping more optimal in terms of locality of data.
          Hide
          Andrew Purtell added a comment -

          If block device type affinity hints are available, then this could be an additional factor for a future block placement algorithm. See HDFS-2832 and HBASE-6572 for relevant discussion. If device affinity hints might have a path component (e.g. xattr in the namespace that associates a path with an affinity hint), then this could be related to HDFS-2576 also.

          Show
          Andrew Purtell added a comment - If block device type affinity hints are available, then this could be an additional factor for a future block placement algorithm. See HDFS-2832 and HBASE-6572 for relevant discussion. If device affinity hints might have a path component (e.g. xattr in the namespace that associates a path with an affinity hint), then this could be related to HDFS-2576 also.
          Andrew Purtell made changes -
          Link This issue relates to HBASE-6572 [ HBASE-6572 ]
          Hide
          Jonathan Hsieh added a comment -

          I know work as already started here, but I want to make sure we consider multiple designs, have thought about where they may end up and the tradeoffs.

          I posted a long post
          over on the HDFS-2576 jira, probably should have posted here instead.

          I'm assuming that HBase provides the list of DN's, likely selected at region creation time?

          The balancers run independently, and if the HDFS rebalancer damages the block placements, it'd be repaired at the time compactions in hbase happen.

          So they will potentially "fight" – but we'll just have a perf penalty upon recovery. Does the HDFS balancer by default run automatically (or is only triggered manually)?

          HBase should be able to handle these cases (and the new patches will handle these). In addition, there would be a tool (that would be a subtask of this jira) that should be run periodically that would look at the block placements and update region maps (region -> favored nodes) in the meta table in HBase to keep the mapping more optimal in terms of locality of data.

          Ok, so so when we attempt to write when we compact or flush maybe we'd check to see all N hdfs replica targets are alive? Would the tool be the only process/thread that select data node targets? Is this the only mechanism we have to force block replcias to different datanodes?

          My spidey sense feels that putting hdfs specific info in hbase means we need to go all in or push it hdfs stuff into hdfs an interact with a policy.

          Show
          Jonathan Hsieh added a comment - I know work as already started here, but I want to make sure we consider multiple designs, have thought about where they may end up and the tradeoffs. I posted a long post over on the HDFS-2576 jira, probably should have posted here instead. I'm assuming that HBase provides the list of DN's, likely selected at region creation time? The balancers run independently, and if the HDFS rebalancer damages the block placements, it'd be repaired at the time compactions in hbase happen. So they will potentially "fight" – but we'll just have a perf penalty upon recovery. Does the HDFS balancer by default run automatically (or is only triggered manually)? HBase should be able to handle these cases (and the new patches will handle these). In addition, there would be a tool (that would be a subtask of this jira) that should be run periodically that would look at the block placements and update region maps (region -> favored nodes) in the meta table in HBase to keep the mapping more optimal in terms of locality of data. Ok, so so when we attempt to write when we compact or flush maybe we'd check to see all N hdfs replica targets are alive? Would the tool be the only process/thread that select data node targets? Is this the only mechanism we have to force block replcias to different datanodes? My spidey sense feels that putting hdfs specific info in hbase means we need to go all in or push it hdfs stuff into hdfs an interact with a policy.
          Hide
          Jonathan Hsieh added a comment -

          If block device type affinity hints are available, then this could be an additional factor for a future block placement algorithm.

          hm.. this could be really interesting for potentially forcing the hlog files to particular specialized device.

          It would be really cool to have different files from the same region on the same machine but on different drives. This could reduce seek latencies by making them concurrent when dealing with multiple hfiles/colfams. Or we could force compaction/flush writes to drives that aren't involved.

          Show
          Jonathan Hsieh added a comment - If block device type affinity hints are available, then this could be an additional factor for a future block placement algorithm. hm.. this could be really interesting for potentially forcing the hlog files to particular specialized device. It would be really cool to have different files from the same region on the same machine but on different drives. This could reduce seek latencies by making them concurrent when dealing with multiple hfiles/colfams. Or we could force compaction/flush writes to drives that aren't involved.
          Hide
          Andrew Purtell added a comment -

          This could reduce seek latencies by making them concurrent when dealing with multiple hfiles/colfams

          Or by servicing families with read-mostly access patterns from a low latency high-IOPS device in conjunction with HDFS-347.

          Show
          Andrew Purtell added a comment - This could reduce seek latencies by making them concurrent when dealing with multiple hfiles/colfams Or by servicing families with read-mostly access patterns from a low latency high-IOPS device in conjunction with HDFS-347 .
          Hide
          Devaraj Das added a comment -

          Thanks Jonathan Hsieh for spending time on the issue. I think I should write a spec up and we can continue the discussion on that spec as the base. Answers to some questions/comments:

          I'm assuming that HBase provides the list of DN's, likely selected at region creation time?

          Yes (in particular, I am currently only considering pre-split tables, and the createTable call in the master allocates the regions to particular datanodes). The regionservers would use this information (as the favored nodes) for creating any new file on the hdfs.

          So they will potentially "fight" – but we'll just have a perf penalty upon recovery. Does the HDFS balancer by default run automatically (or is only triggered manually)?

          They could fight, but the hdfs balancer could in theory be tweaked to not move blocks for certain paths. The hdfs balancer needs to be run manually.

          Ok, so so when we attempt to write when we compact or flush maybe we'd check to see all N hdfs replica targets are alive? Would the tool be the only process/thread that select data node targets? Is this the only mechanism we have to force block replcias to different datanodes?

          The tool would look at the regions, their locality information, and try to make sure the map from regions to favored nodes is optimal. It might reassign regions in this process (i.e., update the meta table with the new information, that would then be propagated to the regionservers). The regionservers, like before, would use this information (as the favored nodes) for creating any new file on the hdfs.

          Show
          Devaraj Das added a comment - Thanks Jonathan Hsieh for spending time on the issue. I think I should write a spec up and we can continue the discussion on that spec as the base. Answers to some questions/comments: I'm assuming that HBase provides the list of DN's, likely selected at region creation time? Yes (in particular, I am currently only considering pre-split tables, and the createTable call in the master allocates the regions to particular datanodes). The regionservers would use this information (as the favored nodes) for creating any new file on the hdfs. So they will potentially "fight" – but we'll just have a perf penalty upon recovery. Does the HDFS balancer by default run automatically (or is only triggered manually)? They could fight, but the hdfs balancer could in theory be tweaked to not move blocks for certain paths. The hdfs balancer needs to be run manually. Ok, so so when we attempt to write when we compact or flush maybe we'd check to see all N hdfs replica targets are alive? Would the tool be the only process/thread that select data node targets? Is this the only mechanism we have to force block replcias to different datanodes? The tool would look at the regions, their locality information, and try to make sure the map from regions to favored nodes is optimal. It might reassign regions in this process (i.e., update the meta table with the new information, that would then be propagated to the regionservers). The regionservers, like before, would use this information (as the favored nodes) for creating any new file on the hdfs.
          Hide
          Jonathan Hsieh added a comment -

          I think I should write a spec up and we can continue the discussion on that spec as the base.

          +1. Especially since this feature touches multiple components (hdfs, region creation, table creation, region balancer).

          Yes (in particular, I am currently only considering pre-split tables, and the createTable call in the master allocates the regions to particular datanodes). The regionservers would use this information (as the favored nodes) for creating any new file on the hdfs.

          I'm assuming the first cut would use "random" assignment for 2nd and 3rd replicas?

          Handling "natural" splits would be follow on jira? (a simple policy for splits is for the daughters to choose the same dns as the parent, something slightly smarter would have the parent as one of the replicas but pick 2 new nodes with higher priority).

          They could fight, but the hdfs balancer could in theory be tweaked to not move blocks for certain paths. The hdfs balancer needs to be run manually.

          Make sense, but sounds like an argument for adding some persistent attribute state in hdfs. (or have hdfs consult hbase.. yuck).

          The tool would look at the regions, their locality information, and try to make sure the map from regions to favored nodes is optimal. It might reassign regions in this process (i.e., update the meta table with the new information, that would then be propagated to the regionservers). The regionservers, like before, would use this information (as the favored nodes) for creating any new file on the hdfs.

          So this tool sounds like a new balancer that might fight with the built in hbase balancer. This makes me want to make the hbase balancer external from the master.

          Show
          Jonathan Hsieh added a comment - I think I should write a spec up and we can continue the discussion on that spec as the base. +1. Especially since this feature touches multiple components (hdfs, region creation, table creation, region balancer). Yes (in particular, I am currently only considering pre-split tables, and the createTable call in the master allocates the regions to particular datanodes). The regionservers would use this information (as the favored nodes) for creating any new file on the hdfs. I'm assuming the first cut would use "random" assignment for 2nd and 3rd replicas? Handling "natural" splits would be follow on jira? (a simple policy for splits is for the daughters to choose the same dns as the parent, something slightly smarter would have the parent as one of the replicas but pick 2 new nodes with higher priority). They could fight, but the hdfs balancer could in theory be tweaked to not move blocks for certain paths. The hdfs balancer needs to be run manually. Make sense, but sounds like an argument for adding some persistent attribute state in hdfs. (or have hdfs consult hbase.. yuck). The tool would look at the regions, their locality information, and try to make sure the map from regions to favored nodes is optimal. It might reassign regions in this process (i.e., update the meta table with the new information, that would then be propagated to the regionservers). The regionservers, like before, would use this information (as the favored nodes) for creating any new file on the hdfs. So this tool sounds like a new balancer that might fight with the built in hbase balancer. This makes me want to make the hbase balancer external from the master.
          Hide
          Devaraj Das added a comment -

          Attaching the notes. Thanks all, for the active conversation on this topic. Let me know if you need more details on anything. I'll continue with the patches on the subtasks for now.

          Show
          Devaraj Das added a comment - Attaching the notes. Thanks all, for the active conversation on this topic. Let me know if you need more details on anything. I'll continue with the patches on the subtasks for now.
          Devaraj Das made changes -
          Attachment hbase-4755-notes.txt [ 12571553 ]
          Hide
          Devaraj Das added a comment -

          The notes as a comment on the jira (for easy access )

          At a high level this has a couple of parts to it:

          Creation of table flow (assuming pre-split table)
          ---------------------------------------------------------------------
          0. HBase layer has policies based on which it decides where to place the region files. AssignmentDomain is defined when a new table is created (today it is all the nodes in the cluster).
          1. The HMaster chooses the primary RS on a round-robin basis, and the secondary/tertiary RSs are chosen on different racks (best effort, and best effort to place secondary/tertiary on the same rack).
          2. The meta table is updated with the information about the location mapping for the added regions.
          3. The RegionServers are then asked to open the regions and they get the favored nodes information as well. The mapping information from regions to favorednodes is cached in the regionservers.
          4. This information is then passed to the filesystem (HDFS-2576) when the regionservers create new files on the filesystem. For now, a create API has been added in HDFS to accept a favorednodes list as an additional argument.

          Failure recovery
          ---------------------------------------------------------------------
          When the primary RS dies, ideally, we should assign the regions on that RS to their respective secondaries (maybe whichever has less load or fewer primaries among the remaining two). At some point the maintenance tool should run and set the mapping in meta right (three RS locations, etc.)

          Maintenance of the metadata & region locations
          ---------------------------------------------------------------------
          Over a period of time, nodes may fail, and/or hdfs-balancer may run that might potentially have a bad impact on the locality set up in above steps.

          Periodically, a tool would be run that would inspect the meta table, and see if the mapping is still optimal. The tool (as per the code in facebook's branch) takes a couple of options it can optimize for - maximum-locality, minimum-region-reassign, munkres algorithm for assigning secondary/tertiary RS for regions. There is a chore that periodically checks for updates in meta (based on timestamps) for the region locations and updates assignment-plans.
          In 0.89-fb and in prior versions of HBase, the hbase balancer is run upon regionserver reporting heartbeats, and the balancer basically ensures that the assignment-plans that have been precomputed are met (and regions might get unassigned from their current regionservers, etc.).

          I think it makes sense to have the above tool be part of the locality-aware loadbalancer itself since the loadbalancer today runs asynchronously and it could do a lot more work without impacting heartbeat latencies, etc. It will also address the issue to do with the conflicts that Jonathan Hsieh raised in his previous comment. I'll look at this aspect some more.

          TODO:
          Handle non pre-split tables

          Show
          Devaraj Das added a comment - The notes as a comment on the jira (for easy access ) At a high level this has a couple of parts to it: Creation of table flow (assuming pre-split table) --------------------------------------------------------------------- 0. HBase layer has policies based on which it decides where to place the region files. AssignmentDomain is defined when a new table is created (today it is all the nodes in the cluster). 1. The HMaster chooses the primary RS on a round-robin basis, and the secondary/tertiary RSs are chosen on different racks (best effort, and best effort to place secondary/tertiary on the same rack). 2. The meta table is updated with the information about the location mapping for the added regions. 3. The RegionServers are then asked to open the regions and they get the favored nodes information as well. The mapping information from regions to favorednodes is cached in the regionservers. 4. This information is then passed to the filesystem ( HDFS-2576 ) when the regionservers create new files on the filesystem. For now, a create API has been added in HDFS to accept a favorednodes list as an additional argument. Failure recovery --------------------------------------------------------------------- When the primary RS dies, ideally, we should assign the regions on that RS to their respective secondaries (maybe whichever has less load or fewer primaries among the remaining two). At some point the maintenance tool should run and set the mapping in meta right (three RS locations, etc.) Maintenance of the metadata & region locations --------------------------------------------------------------------- Over a period of time, nodes may fail, and/or hdfs-balancer may run that might potentially have a bad impact on the locality set up in above steps. Periodically, a tool would be run that would inspect the meta table, and see if the mapping is still optimal. The tool (as per the code in facebook's branch) takes a couple of options it can optimize for - maximum-locality, minimum-region-reassign, munkres algorithm for assigning secondary/tertiary RS for regions. There is a chore that periodically checks for updates in meta (based on timestamps) for the region locations and updates assignment-plans. In 0.89-fb and in prior versions of HBase, the hbase balancer is run upon regionserver reporting heartbeats, and the balancer basically ensures that the assignment-plans that have been precomputed are met (and regions might get unassigned from their current regionservers, etc.). I think it makes sense to have the above tool be part of the locality-aware loadbalancer itself since the loadbalancer today runs asynchronously and it could do a lot more work without impacting heartbeat latencies, etc. It will also address the issue to do with the conflicts that Jonathan Hsieh raised in his previous comment. I'll look at this aspect some more. TODO: Handle non pre-split tables
          Hide
          Jonathan Hsieh added a comment - - edited

          Devaraj Das thanks! This level of description is exactly what I was hoping for.

          Questions (some are just clarifying):

          • An assignmentDomain is not what I've been calling an affinity group over in HDFS-2576 – it is the set of possible DN's to assign to, yeah? Are assignment domains persisted or just queried when creating tables (how does this info come from hdfs)?
          • Table creation was recently changed with the inclusion of HBASE-7365. Probably need figure out how to thread the AssignmentDomain stuff in, has interesting follow-on work for snapshot clones. (snapshots should likely ignore on the first cut)
          • Step 3 – this is done in assignment manager?
          • recovery done by assignment manager?
          • What does the maintenance tool interact with and need to see – the new meta table cols, the master, the assignment domain? is this maintenance tool the only place other than the creation time that changes the preferred dn's? Should there be commands to manually override the dn choices? What would the be roughly for surgery purposes?
          • The optimization policies are nice.
          • For natural splits – for the first implementation – what is the story? (no preferred dn's specified? copies parent's preferred DN's?) If there are no dn's specified or if the specified dn's are invalid we "fall back" to the old randome policies, yeah?
          Show
          Jonathan Hsieh added a comment - - edited Devaraj Das thanks! This level of description is exactly what I was hoping for. Questions (some are just clarifying): An assignmentDomain is not what I've been calling an affinity group over in HDFS-2576 – it is the set of possible DN's to assign to, yeah? Are assignment domains persisted or just queried when creating tables (how does this info come from hdfs)? Table creation was recently changed with the inclusion of HBASE-7365 . Probably need figure out how to thread the AssignmentDomain stuff in, has interesting follow-on work for snapshot clones. (snapshots should likely ignore on the first cut) Step 3 – this is done in assignment manager? recovery done by assignment manager? What does the maintenance tool interact with and need to see – the new meta table cols, the master, the assignment domain? is this maintenance tool the only place other than the creation time that changes the preferred dn's? Should there be commands to manually override the dn choices? What would the be roughly for surgery purposes? The optimization policies are nice. For natural splits – for the first implementation – what is the story? (no preferred dn's specified? copies parent's preferred DN's?) If there are no dn's specified or if the specified dn's are invalid we "fall back" to the old randome policies, yeah?
          Hide
          Devaraj Das added a comment -

          Thanks Jonathan Hsieh for the review. Responses to your comments/questions below:

          An assignmentDomain is not what I've been calling an affinity group over in HDFS-2576 – it is the set of possible DN's to assign to, yeah? Are assignment domains persisted or just queried when creating tables (how does this info come from hdfs)?

          Yes, AssignmentDomain is a set of nodes you carve out based on some policies. Currently it is the whole cluster. I think work needs to be done to make it a useful abstraction for multi-tenancy or load-balancing, etc. and no they are not persisted as of now.

          Table creation was recently changed with the inclusion of HBASE-7365. Probably need figure out how to thread the AssignmentDomain stuff in, has interesting follow-on work for snapshot clones. (snapshots should likely ignore on the first cut)

          I don't think this matters in the short term. We are mostly talking about the real data in the context of AssignmentDomain as opposed to metadata (about file paths and so on).

          Step 3 – this is done in assignment manager?

          Yes. An assign() method that takes a Map<HRegionInfo, List<ServerName>> has been introduced. The method internally assigns the region to the 0th element (the primary RS) in the list for every region in the map. This flow works well for (pre-split) tables created newly. I need to see how the flow of "enableTable -> (after sometime)disableTable -> (after sometime)enableTable" works in this context.

          recovery done by assignment manager?

          Yes, AM and SSH I'd think. I haven't thought much about this part yet.

          What does the maintenance tool interact with and need to see – the new meta table cols, the master, the assignment domain? is this maintenance tool the only place other than the creation time that changes the preferred dn's? Should there be commands to manually override the dn choices? What would the be roughly for surgery purposes?

          The tool would interact with the metatable cols and the AssignmentDomain (although in the first implementation, the assignmentdomain is the whole cluster). Yes this is the only place other than the table creation that would change the preferred DNs. Agree with manual overrides (that's what you meant by surgery?).

          For natural splits – for the first implementation – what is the story? (no preferred dn's specified? copies parent's preferred DN's?) If there are no dn's specified or if the specified dn's are invalid we "fall back" to the old randome policies, yeah?

          I think we can do better than random policies, and copying parent's preferred DNs seems fine as a start as well. I'll address this at some point.

          Show
          Devaraj Das added a comment - Thanks Jonathan Hsieh for the review. Responses to your comments/questions below: An assignmentDomain is not what I've been calling an affinity group over in HDFS-2576 – it is the set of possible DN's to assign to, yeah? Are assignment domains persisted or just queried when creating tables (how does this info come from hdfs)? Yes, AssignmentDomain is a set of nodes you carve out based on some policies. Currently it is the whole cluster. I think work needs to be done to make it a useful abstraction for multi-tenancy or load-balancing, etc. and no they are not persisted as of now. Table creation was recently changed with the inclusion of HBASE-7365 . Probably need figure out how to thread the AssignmentDomain stuff in, has interesting follow-on work for snapshot clones. (snapshots should likely ignore on the first cut) I don't think this matters in the short term. We are mostly talking about the real data in the context of AssignmentDomain as opposed to metadata (about file paths and so on). Step 3 – this is done in assignment manager? Yes. An assign() method that takes a Map<HRegionInfo, List<ServerName>> has been introduced. The method internally assigns the region to the 0th element (the primary RS) in the list for every region in the map. This flow works well for (pre-split) tables created newly. I need to see how the flow of "enableTable -> (after sometime)disableTable -> (after sometime)enableTable" works in this context. recovery done by assignment manager? Yes, AM and SSH I'd think. I haven't thought much about this part yet. What does the maintenance tool interact with and need to see – the new meta table cols, the master, the assignment domain? is this maintenance tool the only place other than the creation time that changes the preferred dn's? Should there be commands to manually override the dn choices? What would the be roughly for surgery purposes? The tool would interact with the metatable cols and the AssignmentDomain (although in the first implementation, the assignmentdomain is the whole cluster). Yes this is the only place other than the table creation that would change the preferred DNs. Agree with manual overrides (that's what you meant by surgery?). For natural splits – for the first implementation – what is the story? (no preferred dn's specified? copies parent's preferred DN's?) If there are no dn's specified or if the specified dn's are invalid we "fall back" to the old randome policies, yeah? I think we can do better than random policies, and copying parent's preferred DNs seems fine as a start as well. I'll address this at some point.
          Hide
          Jonathan Hsieh added a comment -

          Good stuff. Thanks for bearing with me – I like to understand things at this level to set my expectations before I dive into code reviews.

          The tool would interact with the metatable cols and the AssignmentDomain (although in the first implementation, the assignmentdomain is the whole cluster). Yes this is the only place other than the table creation that would change the preferred DNs. Agree with manual overrides (that's what you meant by surgery?).

          sorry about the bad english. surgery == hbck-related things like fixups in case the tools or nodes make mistakes to unstuck a cluster.

          I think it would be good to spell out the different planned pieces for the tools, placment policy framework/hooks, etc as subtasks.

          Show
          Jonathan Hsieh added a comment - Good stuff. Thanks for bearing with me – I like to understand things at this level to set my expectations before I dive into code reviews. The tool would interact with the metatable cols and the AssignmentDomain (although in the first implementation, the assignmentdomain is the whole cluster). Yes this is the only place other than the table creation that would change the preferred DNs. Agree with manual overrides (that's what you meant by surgery?). sorry about the bad english. surgery == hbck-related things like fixups in case the tools or nodes make mistakes to unstuck a cluster. I think it would be good to spell out the different planned pieces for the tools, placment policy framework/hooks, etc as subtasks.
          Hide
          Devaraj Das added a comment -

          sorry about the bad english. surgery == hbck-related things like fixups in case the tools or nodes make mistakes to unstuck a cluster.

          To me, one issue that might happen is that the assignmentdomain is small and when failures happen, you run out of nodes to choose from for assigning regions. Other than that, I don't think we should ever get to a situation where we end up in a situation where a cluster is stuck because of the placements. In the worst case, the cluster should operate as it does today (with region file blocks on random nodes). Do you have any other thought in mind?

          I think it would be good to spell out the different planned pieces for the tools, placment policy framework/hooks, etc as subtasks.

          Okay will do.

          Show
          Devaraj Das added a comment - sorry about the bad english. surgery == hbck-related things like fixups in case the tools or nodes make mistakes to unstuck a cluster. To me, one issue that might happen is that the assignmentdomain is small and when failures happen, you run out of nodes to choose from for assigning regions. Other than that, I don't think we should ever get to a situation where we end up in a situation where a cluster is stuck because of the placements. In the worst case, the cluster should operate as it does today (with region file blocks on random nodes). Do you have any other thought in mind? I think it would be good to spell out the different planned pieces for the tools, placment policy framework/hooks, etc as subtasks. Okay will do.
          Hide
          Nicolas Liochon added a comment -

          I think it's quite good imho. I don't have much issues or warning, I think it's just gonna work.

          Some comments:

          Creation of table flow (assuming pre-split table)

          All this is great imho. Taking into account the racks is quite important. AssignmentDomain seems good.

          2. The meta table is updated with the information about the location mapping for the added regions.

          I understand this as 'meta holds the favored nodes information'. It's fine for me.

          Failure recovery

          The difficult point is to choose the third RS now: we've got one missing. Some comments:
          -> We now have 2 RS on the same rack. So the config will be primary & secondary on the same rask and tertiary on another (not ideal).
          -> We can imagine a situation where the first RS will come back to life soon (rolling restart for example).

          TODO: Handle non pre-split tables

          From a locality point of view, it's already an issue today: after the split, it seems better to move them to different servers, but we just created files locally. I would tend to think that it's the job a the next major compaction to clean all this.

          We may have a first step in which we just go to the same servers for WAL & newly created HFiles. Rationale:

          • today the locality is achieved by triggering a major compaction, so this would remain ok.
          • from a WAL point of view, with a 100 nodes cluster and 8 files per WAL, you will not recover from a 5 nodes loss 5% of the time. If we go on the same servers we divide this probability by 8.

          And a second step could be to use this in the balance.

          • if we lose a RS will have locality as well on the new node.
          Show
          Nicolas Liochon added a comment - I think it's quite good imho. I don't have much issues or warning, I think it's just gonna work. Some comments: Creation of table flow (assuming pre-split table) All this is great imho. Taking into account the racks is quite important. AssignmentDomain seems good. 2. The meta table is updated with the information about the location mapping for the added regions. I understand this as 'meta holds the favored nodes information'. It's fine for me. Failure recovery The difficult point is to choose the third RS now: we've got one missing. Some comments: -> We now have 2 RS on the same rack. So the config will be primary & secondary on the same rask and tertiary on another (not ideal). -> We can imagine a situation where the first RS will come back to life soon (rolling restart for example). TODO: Handle non pre-split tables From a locality point of view, it's already an issue today: after the split, it seems better to move them to different servers, but we just created files locally. I would tend to think that it's the job a the next major compaction to clean all this. We may have a first step in which we just go to the same servers for WAL & newly created HFiles. Rationale: today the locality is achieved by triggering a major compaction, so this would remain ok. from a WAL point of view, with a 100 nodes cluster and 8 files per WAL, you will not recover from a 5 nodes loss 5% of the time. If we go on the same servers we divide this probability by 8. And a second step could be to use this in the balance. if we lose a RS will have locality as well on the new node.
          Hide
          Devaraj Das added a comment -

          The difficult point is to choose the third RS now: we've got one missing. Some comments:
          -> We now have 2 RS on the same rack. So the config will be primary & secondary on the same rask and tertiary on another (not ideal).
          -> We can imagine a situation where the first RS will come back to life soon (rolling restart for example).

          Hmm.. We should designate an RS in a different rack (all new store files would go to that node, and all existing data would eventually get to that node via compactions). For the rolling restart case, it should be fine since the meta assignments wouldn't change and when the primary comes back to life, the regions (probably currently assigned to the secondary) would be reassigned. But yeah, I see that the loadbalancer would probably have to be aware of the rolling restart situation so that it doesn't prematurely assume certain RSs are really "down" and take (wasteful) corrective actions.

          We may have a first step in which we just go to the same servers for WAL & newly created HFiles.

          Hmm.. good point. Will tackle WALs as a subtask of this jira.

          The patch in HBASE-7932 does a major part of the work for getting the location information to the meta table, and then send it down to the RS. I need to use the API (HDFS-2576) in HBASE-7942 to really create the files on specific nodes. The balancer work would be separate subtask.

          Show
          Devaraj Das added a comment - The difficult point is to choose the third RS now: we've got one missing. Some comments: -> We now have 2 RS on the same rack. So the config will be primary & secondary on the same rask and tertiary on another (not ideal). -> We can imagine a situation where the first RS will come back to life soon (rolling restart for example). Hmm.. We should designate an RS in a different rack (all new store files would go to that node, and all existing data would eventually get to that node via compactions). For the rolling restart case, it should be fine since the meta assignments wouldn't change and when the primary comes back to life, the regions (probably currently assigned to the secondary) would be reassigned. But yeah, I see that the loadbalancer would probably have to be aware of the rolling restart situation so that it doesn't prematurely assume certain RSs are really "down" and take (wasteful) corrective actions. We may have a first step in which we just go to the same servers for WAL & newly created HFiles. Hmm.. good point. Will tackle WALs as a subtask of this jira. The patch in HBASE-7932 does a major part of the work for getting the location information to the meta table, and then send it down to the RS. I need to use the API ( HDFS-2576 ) in HBASE-7942 to really create the files on specific nodes. The balancer work would be separate subtask.
          Hide
          Devaraj Das added a comment -

          I was discussing with Sanjay Radia on this topic since there is HDFS dependency.. He came up with another idea on the HDFS side (and he plans to implement it soon). Seemed good to me. When a RS failure happens, a random RS picks some region up (as usual). Now when the region is being served by that RS, HDFS also transparently replicates the associated blocks onto that. Eventually, the remote blocks become local. To start with, one simple API could be exposed by the HDFS - makeBlocksLocal(Path). When some client tries to access the region, the RS serves the region but also makes this API call for the HFile paths that the region is comprised of. The copying of the blocks happen in the background.

          The pros is that it is simple to think about, and doesn't intrude much into HBase. In the current approach (via HDFS-2576), on a failure, the new RS would have all the blocks local. But it requires HBase to periodically go and make sure the locality maps (in the meta table) are optimal (since nodes go down, etc.), and maybe reassign regions based on degree of locality w.r.t. the datanodes, etc.
          In this approach, we won't wait for a compaction to happen to rewrite the hfile data locally in the new RS.

          The cons is that when a failure happens, there might be significant network communication when the blocks of the accessed regions are getting localized into the new RSs for the regions (but maybe many of them are rack local to the new RSs). This will not be worse than what would happen today though.

          What do people think about the above?

          Show
          Devaraj Das added a comment - I was discussing with Sanjay Radia on this topic since there is HDFS dependency.. He came up with another idea on the HDFS side (and he plans to implement it soon). Seemed good to me. When a RS failure happens, a random RS picks some region up (as usual). Now when the region is being served by that RS, HDFS also transparently replicates the associated blocks onto that. Eventually, the remote blocks become local. To start with, one simple API could be exposed by the HDFS - makeBlocksLocal(Path). When some client tries to access the region, the RS serves the region but also makes this API call for the HFile paths that the region is comprised of. The copying of the blocks happen in the background. The pros is that it is simple to think about, and doesn't intrude much into HBase. In the current approach (via HDFS-2576 ), on a failure, the new RS would have all the blocks local. But it requires HBase to periodically go and make sure the locality maps (in the meta table) are optimal (since nodes go down, etc.), and maybe reassign regions based on degree of locality w.r.t. the datanodes, etc. In this approach, we won't wait for a compaction to happen to rewrite the hfile data locally in the new RS. The cons is that when a failure happens, there might be significant network communication when the blocks of the accessed regions are getting localized into the new RSs for the regions (but maybe many of them are rack local to the new RSs). This will not be worse than what would happen today though. What do people think about the above?
          Hide
          Ted Yu added a comment -

          Interesting idea.

          one simple API could be exposed by the HDFS - makeBlocksLocal(Path)

          Should the parameter for the new method be of type List<Path> or Path[] ? There may be more than one HFile involved with the region.
          Does region server have to issue this call more than once to the Namenode ? Ideally there should be a call back informing the progress of data movement for the locality request.

          Show
          Ted Yu added a comment - Interesting idea. one simple API could be exposed by the HDFS - makeBlocksLocal(Path) Should the parameter for the new method be of type List<Path> or Path[] ? There may be more than one HFile involved with the region. Does region server have to issue this call more than once to the Namenode ? Ideally there should be a call back informing the progress of data movement for the locality request.
          Hide
          Enis Soztutar added a comment -

          I like Sanjay's idea in that it is more generic, and can help other users apart from hbase. It can also be used further to get rid of distributed cache, which is a giant hack, and have true distributed cache with very high replication from within hdfs.
          Assuming we have that, it is not clear to me how much we can gain by doing HDFS-2576 on top of this. I think Devaraj lists the pros and cons quite neatly.
          I would say, depending on the plan to implement it, we might as well push for it, and reevaluate the plans for HDFS-2576.

          Show
          Enis Soztutar added a comment - I like Sanjay's idea in that it is more generic, and can help other users apart from hbase. It can also be used further to get rid of distributed cache, which is a giant hack, and have true distributed cache with very high replication from within hdfs. Assuming we have that, it is not clear to me how much we can gain by doing HDFS-2576 on top of this. I think Devaraj lists the pros and cons quite neatly. I would say, depending on the plan to implement it, we might as well push for it, and reevaluate the plans for HDFS-2576 .
          Hide
          Gary Helmling added a comment -

          Sanjay's idea seems like a useful capability in itself, but I don't think it solves the whole problem. For certain use cases, being able to fail a region over to a host that already has data locality for the store files is going to be critical. Adding the capability to pull blocks local on demand would shorten the period of high latency due to remote HDFS reads, but it doesn't eliminate it. So I think that having HDFS-2576 in place would still be extremely valuable.

          Show
          Gary Helmling added a comment - Sanjay's idea seems like a useful capability in itself, but I don't think it solves the whole problem. For certain use cases, being able to fail a region over to a host that already has data locality for the store files is going to be critical. Adding the capability to pull blocks local on demand would shorten the period of high latency due to remote HDFS reads, but it doesn't eliminate it. So I think that having HDFS-2576 in place would still be extremely valuable.
          Hide
          Andrew Purtell added a comment -

          For certain use cases, being able to fail a region over to a host that already has data locality for the store files is going to be critical

          +1

          Online serving cases in general will notice the increased latency of remote reads. Especially if local reads are short circuit, and maybe even on devices that can handle high read IOPS. It would be great if we can avoid remote reads as much as possible.

          Show
          Andrew Purtell added a comment - For certain use cases, being able to fail a region over to a host that already has data locality for the store files is going to be critical +1 Online serving cases in general will notice the increased latency of remote reads. Especially if local reads are short circuit, and maybe even on devices that can handle high read IOPS. It would be great if we can avoid remote reads as much as possible.
          Hide
          Devaraj Das added a comment -

          Quick comments -

          Ted Yu, agree that the api should take List<Path> or Path[]..

          Gary Helmling, so I'd say that Sanjay's idea would improve things without too much intrusion into HBase's internals. In the cases, where the RS failures are not very commonplace, it'd benefit a lot..

          Show
          Devaraj Das added a comment - Quick comments - Ted Yu , agree that the api should take List<Path> or Path[].. Gary Helmling , so I'd say that Sanjay's idea would improve things without too much intrusion into HBase's internals. In the cases, where the RS failures are not very commonplace, it'd benefit a lot..
          Hide
          Enis Soztutar added a comment -

          Online serving cases in general will notice the increased latency of remote reads

          On failover, we are paying at least for the following:

          • replaying the log. Historically this has been the biggest problem, but should be getting better. And with Jeffrey's HBASE-7835, I would expect this becomes even better. We are not doing local reads to the log files here, and Sanjay's proposal does not help in this case. But the data is supposed to be one-two orders of magnitude smaller than in hfiles.
          • filling up the block cache: I think this is the biggest cost we are paying for online serve. HDFS improvements wont help us here.
          • remote reads to fill up the block cache: without HDFS-2576, we will get eventual local reads. On failover, we can trigger all the blocks of all files to be copied local. I think we have to quantify how eventual this would be, and what we are paying for.
          Show
          Enis Soztutar added a comment - Online serving cases in general will notice the increased latency of remote reads On failover, we are paying at least for the following: replaying the log. Historically this has been the biggest problem, but should be getting better. And with Jeffrey's HBASE-7835 , I would expect this becomes even better. We are not doing local reads to the log files here, and Sanjay's proposal does not help in this case. But the data is supposed to be one-two orders of magnitude smaller than in hfiles. filling up the block cache: I think this is the biggest cost we are paying for online serve. HDFS improvements wont help us here. remote reads to fill up the block cache: without HDFS-2576 , we will get eventual local reads. On failover, we can trigger all the blocks of all files to be copied local. I think we have to quantify how eventual this would be, and what we are paying for.
          Hide
          Enis Soztutar added a comment -

          To be clear, I think we can do both of these. It is just that the more generic approach can be integrated quickly, and will benefit other hdfs clients, and might be a 80-20% situation. We can still pursue HDFS-2576 in parallel if there is enough justification for the complexity it brings. I am just saying that it would be good to quantify the benefits, because managing block locations from HBase opens another can of worms.

          Show
          Enis Soztutar added a comment - To be clear, I think we can do both of these. It is just that the more generic approach can be integrated quickly, and will benefit other hdfs clients, and might be a 80-20% situation. We can still pursue HDFS-2576 in parallel if there is enough justification for the complexity it brings. I am just saying that it would be good to quantify the benefits, because managing block locations from HBase opens another can of worms.
          Devaraj Das made changes -
          Link This issue is related to HDFS-4606 [ HDFS-4606 ]
          Hide
          Devaraj Das added a comment -

          Sanjay's new jira. If HDFS-4606 is addressed, it'd make sense to use it to get some immediate benefits. HDFS-2576 is still relevant and it would definitely bring benefits, and the point I am trying to get my heads around is what is the cost/benefit of the two approaches.

          Show
          Devaraj Das added a comment - Sanjay's new jira. If HDFS-4606 is addressed, it'd make sense to use it to get some immediate benefits. HDFS-2576 is still relevant and it would definitely bring benefits, and the point I am trying to get my heads around is what is the cost/benefit of the two approaches.
          Hide
          Gary Helmling added a comment -

          replaying the log. Historically this has been the biggest problem, but should be getting better. And with Jeffrey's HBASE-7835, I would expect this becomes even better. We are not doing local reads to the log files here, and Sanjay's proposal does not help in this case. But the data is supposed to be one-two orders of magnitude smaller than in hfiles.

          This could also be handled by HDFS-2576 if we allowed a WAL per region, an idea with it's own pros and cons that we probably shouldn't get into here. So this is not intractable.

          filling up the block cache: I think this is the biggest cost we are paying for online serve. HDFS improvements wont help us here.

          This assumes a cachable working set, which is not always the case. Of course the block cache will help mask the cost of remote reads when you have good cache affinity.

          I also agree that Sanjay's API is worth pursuing. I think it was something that was actually proposed way back with the idea of in-process lucene indexing with HBASE-3529, but was unfortunately shot down. I also agree that this would be an improvement over the current situation, but the time to pull blocks local can still be a painful cost, especially in the case of larger multi-node failures.

          I do agree that HDFS-2576 comes at a cost of complexity, especially in terms of interactions with the HDFS balancer and normal decommissioning or under-replication handling. But I think closing that final gap for immediate locality will make HBase more attractive for certain scenarios.

          Show
          Gary Helmling added a comment - replaying the log. Historically this has been the biggest problem, but should be getting better. And with Jeffrey's HBASE-7835 , I would expect this becomes even better. We are not doing local reads to the log files here, and Sanjay's proposal does not help in this case. But the data is supposed to be one-two orders of magnitude smaller than in hfiles. This could also be handled by HDFS-2576 if we allowed a WAL per region, an idea with it's own pros and cons that we probably shouldn't get into here. So this is not intractable. filling up the block cache: I think this is the biggest cost we are paying for online serve. HDFS improvements wont help us here. This assumes a cachable working set, which is not always the case. Of course the block cache will help mask the cost of remote reads when you have good cache affinity. I also agree that Sanjay's API is worth pursuing. I think it was something that was actually proposed way back with the idea of in-process lucene indexing with HBASE-3529 , but was unfortunately shot down. I also agree that this would be an improvement over the current situation, but the time to pull blocks local can still be a painful cost, especially in the case of larger multi-node failures. I do agree that HDFS-2576 comes at a cost of complexity, especially in terms of interactions with the HDFS balancer and normal decommissioning or under-replication handling. But I think closing that final gap for immediate locality will make HBase more attractive for certain scenarios.
          Sanjay Radia made changes -
          Link This issue relates to HDFS-2121 [ HDFS-2121 ]
          Sanjay Radia made changes -
          Link This issue relates to HDFS-2121 [ HDFS-2121 ]
          Sanjay Radia made changes -
          Link This issue relates to HDFS-4606 [ HDFS-4606 ]
          Sanjay Radia made changes -
          Link This issue relates to HDFS-4606 [ HDFS-4606 ]
          Hide
          Bing Jiang added a comment -

          if hdfs makes block movement because of DN crashes or other reasons, would hbase need to track the changes of these blocks? Or hdfs new feature can support movement based on their previous hints?

          Show
          Bing Jiang added a comment - if hdfs makes block movement because of DN crashes or other reasons, would hbase need to track the changes of these blocks? Or hdfs new feature can support movement based on their previous hints?
          Hide
          Devaraj Das added a comment -

          Bing Jiang, yes HBase would need to periodically refresh the mappings, and also when compactions happen, the data would be rewritten in the three current nodes. I need to implement the balancer in FavoredNodeLoadBalancer (balanceCluster method). I should have something shortly.

          Show
          Devaraj Das added a comment - Bing Jiang , yes HBase would need to periodically refresh the mappings, and also when compactions happen, the data would be rewritten in the three current nodes. I need to implement the balancer in FavoredNodeLoadBalancer (balanceCluster method). I should have something shortly.
          Devaraj Das made changes -
          Link This issue requires HBASE-8549 [ HBASE-8549 ]
          Devaraj Das made changes -
          Link This issue requires HBASE-9116 [ HBASE-9116 ]

            People

            • Assignee:
              Christopher Gist
              Reporter:
              Karthik Ranganathan
            • Votes:
              1 Vote for this issue
              Watchers:
              40 Start watching this issue

              Dates

              • Created:
                Updated:

                Development