[HBASE-787] Postgresql to HBase table replication example

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.2.0
• Fix Version/s: 0.18.0
• Assignee: Michael Stack
• Reporter: Tim Sell
• Created: 30/Jul/08 15:46
• Updated: 13/Sep/09 22:33
• Resolved: 12/Aug/08 18:44

Description

It is useful to have an easy way to replicate data from Postgresql tables to a HBase tables.
I made a simple python tool which does this, called hbrep.

hbrep is a tool for replicating data from postgresql tables to hbase tables.

Dependancies:

• python 2.4
• hbase 0.2.0
• skytools 2.1.7
• postgresql

It has two main functions.

• bootstrap, which bootstraps all the data from specified columns of a table
• play, which processes incoming insert, update and delete events and applies them to hbase.

Example usage:
install triggers:
./hbrep.py hbrep.ini install schema1.table1 schema2.table2
now that future updates are queuing, bootstrap the tables.
./hbrep.py hbrep.ini bootstrap schema1.table1 schema2.table2
start pgq ticker (this is part of skytools, it manages event queues and sends the events to registered consumers).
pgqadm.py pgq.ini ticker
play our queue consumer to replicate events
./hbrep.py hbrep.ini play schema1.table1 schema2.table2

more details in the readme.
feedback and improvements appreciated.