Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Not A Problem
-
None
-
None
-
None
Description
When frozen, a region would not flush new files or delete old files.
Context:
Was talking to Dhruba about the possibility of a fast intra-cluster copy, and he says it is totally do-able. The idea is this: if there is a HBase table that is very large (multiple terabytes) then distcp takes a long time. The major bottleneck is the cross rack network transfer time. We can easily avoid this by going to each of the block replicas and replicating them on the same machine from one disk to another. For our infrastructure setup, this could make hours of copy time seconds.
If we are able to implement the above, then copying an entire HBase table will only take seconds. Now we just have to ensure that the region files do not "change" during this period - so we need to disable compactions and flushes. Hence the need for the freeze and thaw of regions in a region server.
The way this would work is to flush a region, freeze it, snapshot it, then thaw it and let normal operations proceed. The snapshot would proceed at disk read/write speed so it should take only seconds per region.