[HBASE-13639] SyncTable - rsync for HBase tables

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.98.14, 1.2.0, 2.0.0
• Component/s: mapreduce, Operability, tooling
• Labels: tooling
• Hadoop Flags: Reviewed

Release Note

Tool to sync two tables that tries to send the differences only like rsync.

Adds two new MapReduce jobs, SyncTable and HashTable. See usage for these jobs on how to use. See design doc for generally overview: https://docs.google.com/document/d/1-2c9kJEWNrXf5V4q_wBcoIXfdchN7Pxvxv1IO6PW0-U/edit

From comments below, "It can be challenging to run against a table getting live writes, if those writes are updates/overwrites. In general, you can run it against a time range to ignore new writes, but if those writes update existing cells, then the time range scan may or may not see older versions of those cells depending on whether major compaction has happened, which may be different in remote clusters."

• Assignee: Dave Latham
• Reporter: Dave Latham
• Created: 07/May/15 01:16
• Updated: 24/Jun/22 18:44
• Resolved: 03/Jul/15 17:44

Description

Given HBase tables in remote clusters with similar but not identical data, efficiently update a target table such that the data in question is identical to a source table. Efficiency in this context means using far less network traffic than would be required to ship all the data from one cluster to the other. Takes inspiration from rsync.

Design doc: https://docs.google.com/document/d/1-2c9kJEWNrXf5V4q_wBcoIXfdchN7Pxvxv1IO6PW0-U/