[PHOENIX-5442] Change Data Capture

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Geoffrey Jacoby
• Reporter: Geoffrey Jacoby
• Created: 13/Aug/19 21:18
• Updated: 11/Oct/19 20:57

Description

Create a change detection capture framework for Phoenix to allow for changes to tables to be streamed to a message bus such as Kafka. This is an umbrella JIRA, and will require several subcomponents, such as

• Phoenix schema information annotated into HBase WALs
• The ability to generate Avro (or similar) schemas for Phoenix objects using the above schema information
• An HBase replication endpoint that tails HBase WALs, formats the data changes as Avro using the schema information, and then persists to a message bus. Parts of the implementation might work better as part of the connectors repo, or some other repo, to avoid importing too many new dependencies into Phoenix proper.

Code will depend on interfaces wherever possible so that different technologies can be used for the message format and message bus.

Sub-Tasks

• Annotate HBase WALs with Phoenix Metadata: Resolved
• API to Generate Avro Schema of Phoenix Object: Open
• Create generic Avro schema for Phoenix Mutations: Resolved
• ReplicationSink Message Producer: Open
• Schema-aware Message Consumer: Open
• CDC Avro Serialization: Open
• Option for DDL changes to export to external schema repository: Resolved
• Annotate HBase WALs with Schema Id: Resolved
• Add metrics for schema registry export: Resolved
• Change Data Capture - Populate Table / Topic Mappings: Resolved
• CDC REST API for Stream Management and Consumption: Open
• Change Data Capture - Transform Compatibility: Open