The HBase connector source implementation should be updated in accordance with FLIP-27: Refactor Source Interface.
One source should map to one table in HBase. Users can specify which column[families] to watch; each change in one of the columns triggers a record with change type, table, column family, column, value, and timestamp.
The new Flink HBase Source makes use of the internal replication mechanism of HBase. The Source is registering at the HBase cluster and will receive all WAL edits written in HBase. From those WAL edits the Source can create the DataStream.
We're still not 100% sure which information a Split should contain. We have the following possibilities:
- There is only one Split per Source and the Split contains all the necessary information to connect with HBase. The SourceReader which processes the Split will receive all WAL edits for all tables and filters the relevant edits.
- There are multiple Splits per Source, each Split representing one HBase Region to read from. This assumes that it is possible to only receive WAL edits from a specific HBase Region and not receive all WAL edits. This would be preferable as it allows parallel processing of multiple regions, but we still need to figure out how this is possible.
In both cases the Split will contain information about the HBase instance and table.
Depending on which Split we'll decide on, the split enumerator will connect to HBase and get all relevant regions or just create one Split.