Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20955

Refactor HBase Source in accordance with FLIP-27

    XMLWordPrintableJSON

    Details

      Description

      The HBase connector source implementation should be updated in accordance with FLIP-27: Refactor Source Interface.

      One source should map to one table in HBase. Users can specify which column[families] to watch; each change in one of the columns triggers a record with change type, table, column family, column, value, and timestamp.

      Idea

      The new Flink HBase Source makes use of the internal replication mechanism of HBase. The Source is registering at the HBase cluster and will receive all WAL edits written in HBase. From those WAL edits the Source can create the DataStream. 

      Split

      We're still not 100% sure which information a Split should contain. We have the following possibilities: 

      1. There is only one Split per Source and the Split contains all the necessary information to connect with HBase. The SourceReader which processes the Split will receive all WAL edits for all tables and filters the relevant edits. 
      2. There are multiple Splits per Source, each Split representing one HBase Region to read from. This assumes that it is possible to only receive WAL edits from a specific HBase Region and not receive all WAL edits. This would be preferable as it allows parallel processing of multiple regions, but we still need to figure out how this is possible.

      In both cases the Split will contain information about the HBase instance and table. 

      Split Enumerator

      Depending on which Split we'll decide on, the split enumerator will connect to HBase and get all relevant regions or just create one Split.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                moritz.manner Moritz Manner
                Reporter:
                moritz.manner Moritz Manner
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: