[TEPHRA-74] Replication of transaction state

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: phoenix
• Assignee: Gary Helmling
• Reporter: Gary Helmling
• Created: 10/Mar/15 01:13
• Updated: 10/Mar/15 01:13

Description

This is an issue to capture discussion around how Tephra might work with HBase replication, and the key considerations involved.

HBase supports eventually consistent replication between clusters, but defining how Tephra should work with this has been tricky. Where HBase replication is used for DR scenarios, the transaction metadata managed by Tephra must also some how be replicated in order to ensure consistency as the data is read. However this presents two problems:

1. how to replicate the transaction metadata (currently stored in files in HDFS)
2. how to reconcile the eventually consistent semantics of HBase replication with transactional semantics of Tephra – what guarantees can still be provided in this case?

For replicating transaction metadata, we seem to have three options:

• Use existing HBase replication mechanisms to replicate the metadata. This would mean backing the transaction metadata by an HBase table and scanning the table in order to replay the logs. This would allow the transaction state changes to propagate over normal HBase mechanisms with no new requirements, but could mean that state transitions arrive out of order on the target cluster. We would need some constraints around this solution so that we avoid out of order transitions, possibly limiting replay to some replication lag window.
• Extend HBase replication to provide new guarantees. Again transaction metadata would need to be backed by an HBase table and piggy back on HBase replication, but in this case, we would want better replication guarantees to avoid out of order delivery. For example, would it be possible to provide timeline consistent replication for specific categories in HBase, using dedicated WALs in order to avoid out of order deliveries?
• Use an alternate state replication mechanism. This could be as simple as a frequent distcp process or as complicated as a consensus protocol.

It would be ideal if we could avoid out of order delivery of transaction metadata. This would at least allow consistent handling of data records where the transaction metadata has replicated. Any data records with newer transaction IDs (where the metadata has not yet replicated) could be ignored until the transaction metadata does replicate, providing no worse guarantees than HBase replication, without sacrificing consistency of the data that is available.