HBase
  1. HBase
  2. HBASE-57

[hbase] Master should allocate regions to regionservers based upon data locality and rack awareness

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.2.0
    • Fix Version/s: None
    • Component/s: master
    • Labels:

      Description

      Currently, regions are assigned regionservers based off a basic loading attribute. A factor to include in the assignment calcuation is the location of the region in hdfs; i.e. servers hosting region replicas. If the cluster is such that regionservers are being run on the same nodes as those running hdfs, then ideally the regionserver for a particular region should be running on the same server as hosts a region replica.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Bryan Duxbury added a comment -

          This seems like it should be an important issue, since it should significantly improve the performance of the cluster (faster, less network traffic). Elevating to Major.

          Show
          Bryan Duxbury added a comment - This seems like it should be an important issue, since it should significantly improve the performance of the cluster (faster, less network traffic). Elevating to Major.
          Hide
          stack added a comment -

          This might actually already be happening. Any time I've seen a regionserver reading from hdfs, it seems to be getting its data from the local datanode. Verify.

          Show
          stack added a comment - This might actually already be happening. Any time I've seen a regionserver reading from hdfs, it seems to be getting its data from the local datanode. Verify.
          Hide
          Billy Pearson added a comment -

          From what I understand about writing files to data nodes I do not thank is is a major problem. If the regions are not located on the host serving server on the first compaction they would be stored local first and stay local from there on out unless the current servers hard drive is full and it has to store it on a different server. Over all the region servers would self fix this problem by default over time.

          So only time this would be an issue is on a restart or after a failed region server.

          Show
          Billy Pearson added a comment - From what I understand about writing files to data nodes I do not thank is is a major problem. If the regions are not located on the host serving server on the first compaction they would be stored local first and stay local from there on out unless the current servers hard drive is full and it has to store it on a different server. Over all the region servers would self fix this problem by default over time. So only time this would be an issue is on a restart or after a failed region server.
          Hide
          Jim Kellerman added a comment -

          Downgrading priority because we should leverage Hadoop's rack awareness where possible, and there is a lot of work left to do (in Hadoop) before we can

          Show
          Jim Kellerman added a comment - Downgrading priority because we should leverage Hadoop's rack awareness where possible, and there is a lot of work left to do (in Hadoop) before we can
          Hide
          Sijie Guo added a comment -

          Hi hbasers,
          I'd like to work on this issue as my GSOC project "Exploit locality when assigning regions in HBase".

          After talking with Stack in emails, I have got some initial thoughts on this issue. I'd like to share them with you and welcome for your comments.

          Before designing a suitable mechanism to using the region's locality, we need to know how blocks are allocated in a hbase cluster and the data-blocks distribution of a specified region over its lifetime in hbase. so that we can find out how the region locality effect the performance. It is difficult to capture all these information in a real cluster. An alternative way to study the locality phenomeon may be simulating the data-block placement procedure in HDFS(local node, local rack, and remote rack) and the regions-allocation mechanism of a hbase cluster in a single machine. And a approximate detail report from simulation can be used for analysis and development.

          Although I haven't got any detail information about the locality phenomeon, I try to give an initial proposal first. The initial proposal is to schedule the regions to the datanodes(regionservers) that contains most data-blocks of the specified region. The most challenge thing is to know the data-blocks layout(we can query namenode in HDFS to get these information) of a region in master. And an initial method is to record these layout information of regions in .META. table.
          Some background threads may be run on the master scanning the .META. table to pick up the candidate nodes for region-allocation(these nodes may be sorted by the number of blocks they contain). The detail allocation mechanism will be discussed below.
          (1) A blank region created when the table is first created. As we haven't got any data in it, we can allocate it according to the current loads of the cluster. It is an easy way. And after the region grows up and were flushed back to HDFS, we get the blocks' locations information and records them to .META. table for next-allocation.
          (2) A region is created by splitting its parent region. We can use parent-region's blocks' location information to make an allocation decision. And after we finish the splitting procedure, we can simply copy the parent-region's blocks' location information to each sub-region's .META. table information.
          (3) A region is re-allocated after the regionserver crash. The logfiles' blocks information will be considered into allocation so that we may accelerate the recovery of a failed-region.

          Show
          Sijie Guo added a comment - Hi hbasers, I'd like to work on this issue as my GSOC project "Exploit locality when assigning regions in HBase". After talking with Stack in emails, I have got some initial thoughts on this issue. I'd like to share them with you and welcome for your comments. Before designing a suitable mechanism to using the region's locality, we need to know how blocks are allocated in a hbase cluster and the data-blocks distribution of a specified region over its lifetime in hbase. so that we can find out how the region locality effect the performance. It is difficult to capture all these information in a real cluster. An alternative way to study the locality phenomeon may be simulating the data-block placement procedure in HDFS(local node, local rack, and remote rack) and the regions-allocation mechanism of a hbase cluster in a single machine. And a approximate detail report from simulation can be used for analysis and development. Although I haven't got any detail information about the locality phenomeon, I try to give an initial proposal first. The initial proposal is to schedule the regions to the datanodes(regionservers) that contains most data-blocks of the specified region. The most challenge thing is to know the data-blocks layout(we can query namenode in HDFS to get these information) of a region in master. And an initial method is to record these layout information of regions in .META. table. Some background threads may be run on the master scanning the .META. table to pick up the candidate nodes for region-allocation(these nodes may be sorted by the number of blocks they contain). The detail allocation mechanism will be discussed below. (1) A blank region created when the table is first created. As we haven't got any data in it, we can allocate it according to the current loads of the cluster. It is an easy way. And after the region grows up and were flushed back to HDFS, we get the blocks' locations information and records them to .META. table for next-allocation. (2) A region is created by splitting its parent region. We can use parent-region's blocks' location information to make an allocation decision. And after we finish the splitting procedure, we can simply copy the parent-region's blocks' location information to each sub-region's .META. table information. (3) A region is re-allocated after the regionserver crash. The logfiles' blocks information will be considered into allocation so that we may accelerate the recovery of a failed-region.
          Hide
          Jim Kellerman added a comment -

          > Samuel Guo added a comment - 26/Mar/09 06:48 AM
          > Hi hbasers,
          > I'd like to work on this issue as my GSOC project "Exploit locality when assigning regions in HBase".
          >
          > After talking with Stack in emails, I have got some initial thoughts on this issue. I'd like to share them with you and
          > welcome for your comments.
          >
          > Before designing a suitable mechanism to using the region's locality, we need to know how blocks are allocated in
          > a hbase cluster and the data-blocks distribution of a specified region over its lifetime in hbase. so that we can find
          > out how the region locality effect the performance. It is difficult to capture all these information in a real cluster. An
          > alternative way to study the locality phenomeon may be simulating the data-block placement procedure in
          > HDFS(local node, local rack, and remote rack) and the regions-allocation mechanism of a hbase cluster in a single
          > machine. And a approximate detail report from simulation can be used for analysis and development.

          Although the JobTracker in Hadoop attempts to assign tasks to machines that are hosting the data, currently
          HDFS clients still must go through the data node to access the blocks. I believe there is a Jira open for Hadoop
          to go directly to local disk to get the blocks (removing communication with the datanode) if the blocks are on
          the local machine.

          I think that direct disk access for local blocks would be the biggest payoff.

          It is unclear if there is any advantage for locality (other than limiting network access) if direct disk access is not
          available. Benchmarking needs to be done to determine the relative latency of accessing a block from another
          server is much faster than accessing a block (via the datanode) if the block resides on the local machine. It may
          not provide much advantage, unless the block is being served from a server on the other side of a switch or
          router (and of course the speed of the connection).

          Solid performance data evaluating the cost of:
          1) network access to a block in a different rack
          2) network access to a block in the same rack but on a different server
          3) network access to a block on the same server
          4) direct disk access to a block on the same server

          would be highly useful. If there is little difference between 1, 2, 3 (access to a block through a datanode) then
          locality may not be useful. On the other hand, if there is a significant difference between 1, 2, 3 then we should
          try to exploit locality if we can.

          I would expect that direct disk access would be much faster than access through a datanode, but there is no
          hard data on that available. If data were available, then that would indicate if direct disk access (bypassing
          the datanode) is useful or not. If so, that argues in favor of implementing direct disk access as has been
          proposed, and if direct disk access were available, that would argue in favor of locality based region assignment.

          As you point out, blocks migrate over time (especially if you are using the HDFS balancer), and that would
          greatly complicate assignment of regions to region servers.

          Suppose there was one 'hot' datanode that hosted blocks from many regions. Using locality might end up in
          overloading the region server on that node, resulting in poorer performance.

          There is a lot of performance evaluation that needs to be done before we actually take the step of using
          locality-based region assignment. If doing that performance evaluation sounds interesting to you, I think
          that would be a great GSOC project.

          Before we try locality-based assignment, we need to have this analysis to see if the idea is worth pursuing.

          Show
          Jim Kellerman added a comment - > Samuel Guo added a comment - 26/Mar/09 06:48 AM > Hi hbasers, > I'd like to work on this issue as my GSOC project "Exploit locality when assigning regions in HBase". > > After talking with Stack in emails, I have got some initial thoughts on this issue. I'd like to share them with you and > welcome for your comments. > > Before designing a suitable mechanism to using the region's locality, we need to know how blocks are allocated in > a hbase cluster and the data-blocks distribution of a specified region over its lifetime in hbase. so that we can find > out how the region locality effect the performance. It is difficult to capture all these information in a real cluster. An > alternative way to study the locality phenomeon may be simulating the data-block placement procedure in > HDFS(local node, local rack, and remote rack) and the regions-allocation mechanism of a hbase cluster in a single > machine. And a approximate detail report from simulation can be used for analysis and development. Although the JobTracker in Hadoop attempts to assign tasks to machines that are hosting the data, currently HDFS clients still must go through the data node to access the blocks. I believe there is a Jira open for Hadoop to go directly to local disk to get the blocks (removing communication with the datanode) if the blocks are on the local machine. I think that direct disk access for local blocks would be the biggest payoff. It is unclear if there is any advantage for locality (other than limiting network access) if direct disk access is not available. Benchmarking needs to be done to determine the relative latency of accessing a block from another server is much faster than accessing a block (via the datanode) if the block resides on the local machine. It may not provide much advantage, unless the block is being served from a server on the other side of a switch or router (and of course the speed of the connection). Solid performance data evaluating the cost of: 1) network access to a block in a different rack 2) network access to a block in the same rack but on a different server 3) network access to a block on the same server 4) direct disk access to a block on the same server would be highly useful. If there is little difference between 1, 2, 3 (access to a block through a datanode) then locality may not be useful. On the other hand, if there is a significant difference between 1, 2, 3 then we should try to exploit locality if we can. I would expect that direct disk access would be much faster than access through a datanode, but there is no hard data on that available. If data were available, then that would indicate if direct disk access (bypassing the datanode) is useful or not. If so, that argues in favor of implementing direct disk access as has been proposed, and if direct disk access were available, that would argue in favor of locality based region assignment. As you point out, blocks migrate over time (especially if you are using the HDFS balancer), and that would greatly complicate assignment of regions to region servers. Suppose there was one 'hot' datanode that hosted blocks from many regions. Using locality might end up in overloading the region server on that node, resulting in poorer performance. There is a lot of performance evaluation that needs to be done before we actually take the step of using locality-based region assignment. If doing that performance evaluation sounds interesting to you, I think that would be a great GSOC project. Before we try locality-based assignment, we need to have this analysis to see if the idea is worth pursuing.
          Hide
          stack added a comment -

          The going direct to local blocks reading is HADOOP-4801. In summary, the payoff short-circuiting the datanode is small, and yet to be seen – at least to date – and it seems doubtful that a second route to the data will be opened because of security concerns, etc. Thats my take on the issue (It could change of course).

          I think that if we only made savings in network traffic, that'd be reason enough to implement locality algorithms. JK makes an interesting point above that we could manufacture hot datanodes if we blindly serve regions from a datanode that hosts all the data but this can happen now since we operate blindly and its only smart use of the locality info that will help damp hot spots.

          Samuel, if still interested, have you made petition to become a GSOC student using this issue as your project? (Add in some of JKs notes on need to research what happens in a running cluster so know best what to implement).

          Show
          stack added a comment - The going direct to local blocks reading is HADOOP-4801 . In summary, the payoff short-circuiting the datanode is small, and yet to be seen – at least to date – and it seems doubtful that a second route to the data will be opened because of security concerns, etc. Thats my take on the issue (It could change of course). I think that if we only made savings in network traffic, that'd be reason enough to implement locality algorithms. JK makes an interesting point above that we could manufacture hot datanodes if we blindly serve regions from a datanode that hosts all the data but this can happen now since we operate blindly and its only smart use of the locality info that will help damp hot spots. Samuel, if still interested, have you made petition to become a GSOC student using this issue as your project? (Add in some of JKs notes on need to research what happens in a running cluster so know best what to implement).
          Hide
          Sijie Guo added a comment -

          Thanks for your comments, Jim.

          > Solid performance data evaluating the cost of:
          > 1) network access to a block in a different rack
          > 2) network access to a block in the same rack but on a different server
          > 3) network access to a block on the same server
          > 4) direct disk access to a block on the same server
          > would be highly useful. If there is little difference between 1, 2, 3 (access to a block through a datanode) then
          > locality may not be useful. On the other hand, if there is a significant difference between 1, 2, 3 then we should
          > try to exploit locality if we can.

          > There is a lot of performance evaluation that needs to be done before we actually take the step of using
          > locality-based region assignment. If doing that performance evaluation sounds interesting to you, I think
          > that would be a great GSOC project.

          Yes, I agree with you. We need to do a detail analysis of most behaviors of HDFS and HBase before we try locality-based assignment. And the analysis work will be the main part of my GSOC project.

          > Suppose there was one 'hot' datanode that hosted blocks from many regions. Using locality might end up in
          > overloading the region server on that node, resulting in poorer performance.

          Yes, Locality should be taken carefully not to overload the region server or the data node. An ideal region assignment can assign regions close to its data to reduce network traffic while balancing the loads between region servers, datanodes and avoiding disk competition on the same datanode. As what you suggested, we need to know the following things clearly before making it.
          1) what is the difference we access data from different locations(local, local by-pass, remote, remote rack)?
          2) In regions' life time, what is the data-blocks' distribution? And how many bytes that the region reads data from local node? how many from remote? from remote rack?
          3) After a balance operation happened in HDFS, how 2) changes?
          4) After some region servers failed, how 2) changes?

          I am not so clear now about how to analysis it. but I think I can take them one by one to make things clearly.

          Show
          Sijie Guo added a comment - Thanks for your comments, Jim. > Solid performance data evaluating the cost of: > 1) network access to a block in a different rack > 2) network access to a block in the same rack but on a different server > 3) network access to a block on the same server > 4) direct disk access to a block on the same server > would be highly useful. If there is little difference between 1, 2, 3 (access to a block through a datanode) then > locality may not be useful. On the other hand, if there is a significant difference between 1, 2, 3 then we should > try to exploit locality if we can. > There is a lot of performance evaluation that needs to be done before we actually take the step of using > locality-based region assignment. If doing that performance evaluation sounds interesting to you, I think > that would be a great GSOC project. Yes, I agree with you. We need to do a detail analysis of most behaviors of HDFS and HBase before we try locality-based assignment. And the analysis work will be the main part of my GSOC project. > Suppose there was one 'hot' datanode that hosted blocks from many regions. Using locality might end up in > overloading the region server on that node, resulting in poorer performance. Yes, Locality should be taken carefully not to overload the region server or the data node. An ideal region assignment can assign regions close to its data to reduce network traffic while balancing the loads between region servers, datanodes and avoiding disk competition on the same datanode. As what you suggested, we need to know the following things clearly before making it. 1) what is the difference we access data from different locations(local, local by-pass, remote, remote rack)? 2) In regions' life time, what is the data-blocks' distribution? And how many bytes that the region reads data from local node? how many from remote? from remote rack? 3) After a balance operation happened in HDFS, how 2) changes? 4) After some region servers failed, how 2) changes? I am not so clear now about how to analysis it. but I think I can take them one by one to make things clearly.
          Hide
          Sijie Guo added a comment -

          Thanks stack.

          > Samuel, if still interested, have you made petition to become a GSOC student using this issue as your project? (Add in some of JKs notes on need
          > to research what happens in a running cluster so know best what to implement).

          Yes. I will add Jim's notes on my proposal.

          Show
          Sijie Guo added a comment - Thanks stack. > Samuel, if still interested, have you made petition to become a GSOC student using this issue as your project? (Add in some of JKs notes on need > to research what happens in a running cluster so know best what to implement). Yes. I will add Jim's notes on my proposal.
          Hide
          stack added a comment -

          If this makes it in time for 0.20.0, good, but for now moving it out.

          Show
          stack added a comment - If this makes it in time for 0.20.0, good, but for now moving it out.
          Hide
          stack added a comment -

          This thread, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/9366, has another description of what needs fixing here.

          Is there a call you can make to the NN that will give you metadata on a file – the blocks it comprises of and where they are located?

          Chatting w/ a possible GSOC candidate about doing this one.

          Show
          stack added a comment - This thread, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/9366 , has another description of what needs fixing here. Is there a call you can make to the NN that will give you metadata on a file – the blocks it comprises of and where they are located? Chatting w/ a possible GSOC candidate about doing this one.
          Hide
          Karthik K added a comment -

          Is there a call you can make to the NN that will give you metadata on a file - the blocks it comprises of and where they are located?

          BlockLocation[] DistributedFileSystem#getBlockLocations(String src, long start, long length);

          KFS also has an implementation of FileSystem, appropriately overridden. Is that the one being referred here ?

          Show
          Karthik K added a comment - Is there a call you can make to the NN that will give you metadata on a file - the blocks it comprises of and where they are located? BlockLocation[] DistributedFileSystem#getBlockLocations(String src, long start, long length); KFS also has an implementation of FileSystem, appropriately overridden. Is that the one being referred here ?
          Hide
          stack added a comment -

          That might do. When it came time to assign a region, you'd get a listing of all the files its comprised of (Fairly cheap interrogation of NN). You'd then per file, use the above to figure blocks and their hosts per file. You'd then take the whole mess and figure which RS had the most blocks local to it and assign the region there (with some damping to compensate for case where blocks are unevenly distributed).

          Show
          stack added a comment - That might do. When it came time to assign a region, you'd get a listing of all the files its comprised of (Fairly cheap interrogation of NN). You'd then per file, use the above to figure blocks and their hosts per file. You'd then take the whole mess and figure which RS had the most blocks local to it and assign the region there (with some damping to compensate for case where blocks are unevenly distributed).
          Hide
          Karthik K added a comment -
          figure which RS had the most blocks local to it and assign the region there

          if dfs.replication > 1 (say 3 , as advised) with N blocks , that would be finding the RS with the most blocks among the (3 ^ N permutations) + damping to take care of 'hot datanodes' as you had pointed out. It should be interesting !

          Show
          Karthik K added a comment - figure which RS had the most blocks local to it and assign the region there if dfs.replication > 1 (say 3 , as advised) with N blocks , that would be finding the RS with the most blocks among the (3 ^ N permutations) + damping to take care of 'hot datanodes' as you had pointed out. It should be interesting !
          Hide
          Jan Lukavsky added a comment -

          Hi,
          I suspect this issue is causing us trouble during Map/Reduce having HBase as data source. TableInputFormat tells JobTracker that regions are data-local to RegionServer, which serves them. This IMO causes serious imbalance of load on small clusters (our has about 10 nodes), because the RegionServer may (and probably will) contact DataNode on different machine. Thus, in extreme case, single DataNode may (in some time) be handling reads from all the Mappers.

          If regions were assigned to RegionServer which holds the most blocks, I suppose this imbalance will be minimized. Stack's proposed solution seems fairly appropriate to me.

          Show
          Jan Lukavsky added a comment - Hi, I suspect this issue is causing us trouble during Map/Reduce having HBase as data source. TableInputFormat tells JobTracker that regions are data-local to RegionServer, which serves them. This IMO causes serious imbalance of load on small clusters (our has about 10 nodes), because the RegionServer may (and probably will) contact DataNode on different machine. Thus, in extreme case, single DataNode may (in some time) be handling reads from all the Mappers. If regions were assigned to RegionServer which holds the most blocks, I suppose this imbalance will be minimized. Stack's proposed solution seems fairly appropriate to me.
          Hide
          stack added a comment -

          @Jan Do you see this: ",,,in extreme case, single DataNode may (in some time) be handling reads from all the Mappers"? On a long-running cluster, blocks will migrate to be local to the regoinserver that is serving them. Is this not happening in your case? Are you seeing RSs mostly reaching across the net to remote DNs?

          Show
          stack added a comment - @Jan Do you see this: ",,,in extreme case, single DataNode may (in some time) be handling reads from all the Mappers"? On a long-running cluster, blocks will migrate to be local to the regoinserver that is serving them. Is this not happening in your case? Are you seeing RSs mostly reaching across the net to remote DNs?
          Hide
          Jan Lukavsky added a comment -

          @stack
          Our cluster is not "long running", we have written approx. 1TB of data, which we are trying to analyze using M/R. After the data was written, we do not modify them. In this case, I'm afraid dislocated blocks will stay put, until major compaction (which might therefore solve this problem for us). The blocks may get dislocated for example after some RS becomes temporarily unavailable. Therefore I suppose assigning regions based on their real location is somewhat 'cleaner', so that after the RS becomes available again it regains its regions.

          Show
          Jan Lukavsky added a comment - @stack Our cluster is not "long running", we have written approx. 1TB of data, which we are trying to analyze using M/R. After the data was written, we do not modify them. In this case, I'm afraid dislocated blocks will stay put, until major compaction (which might therefore solve this problem for us). The blocks may get dislocated for example after some RS becomes temporarily unavailable. Therefore I suppose assigning regions based on their real location is somewhat 'cleaner', so that after the RS becomes available again it regains its regions.
          Hide
          stack added a comment -

          Assigning to our successful GSOC applicant, Li Chongxin (Wahoo!!)

          Show
          stack added a comment - Assigning to our successful GSOC applicant, Li Chongxin (Wahoo!!)
          Hide
          Jonathan Gray added a comment -

          I'm going to work on this as part of a load balancer rewrite I'm doing that is part of HBASE-2485. Initially my scope is just to add this to cluster startup and not part of normal load balancing but will look at that as well. I think that doing it as part of normal cluster balancing is less important than introducing read/write load information but at startup we almost guarantee horrible locality

          Show
          Jonathan Gray added a comment - I'm going to work on this as part of a load balancer rewrite I'm doing that is part of HBASE-2485 . Initially my scope is just to add this to cluster startup and not part of normal load balancing but will look at that as well. I think that doing it as part of normal cluster balancing is less important than introducing read/write load information but at startup we almost guarantee horrible locality
          Hide
          Jacques added a comment -

          I'm curious whether there was a discussion around doing this another, less complete way. Why can't we persist info about the allocation of regions across cluster shutdown in zk. We could then use this as a starting point for which region servers serve which regions. Then assign the remaining regions randomly/however it is being done now. Wouldn't this solve 80% of the problem in a simple way?

          Show
          Jacques added a comment - I'm curious whether there was a discussion around doing this another, less complete way. Why can't we persist info about the allocation of regions across cluster shutdown in zk. We could then use this as a starting point for which region servers serve which regions. Then assign the remaining regions randomly/however it is being done now. Wouldn't this solve 80% of the problem in a simple way?
          Hide
          Jonathan Gray added a comment -

          @Jacques, yeah that approach is being considered. once the master rewrite gets merged into trunk (HBASE-2692) it should be possible to retain assignment information across a shutdown. you wouldn't even need zk because you could just leave it in META. it would actually be a fairly simple change once the other stuff is in.

          And yeah, this solves 80% of the problem. But I've also written some partial code to do stuff by locality; it's not trivial but not too bad. Perhaps this simple solution first and then we can start looking at block locations when we extend our notion of "load" for the next version of load balancing.

          Show
          Jonathan Gray added a comment - @Jacques, yeah that approach is being considered. once the master rewrite gets merged into trunk ( HBASE-2692 ) it should be possible to retain assignment information across a shutdown. you wouldn't even need zk because you could just leave it in META. it would actually be a fairly simple change once the other stuff is in. And yeah, this solves 80% of the problem. But I've also written some partial code to do stuff by locality; it's not trivial but not too bad. Perhaps this simple solution first and then we can start looking at block locations when we extend our notion of "load" for the next version of load balancing.
          Hide
          Jonathan Gray added a comment -

          Filed HBASE-2896 for the simpler solution

          Show
          Jonathan Gray added a comment - Filed HBASE-2896 for the simpler solution
          Hide
          stack added a comment -

          Fixed by the stochastic balancer in trunk/0.95

          Show
          stack added a comment - Fixed by the stochastic balancer in trunk/0.95

            People

            • Assignee:
              Li Chongxin
              Reporter:
              stack
            • Votes:
              3 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development