[FLINK-24533] Flink SQL Upsert To Hbase Appear data loss

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.12.0
• Component/s: Connectors / HBase

Environment

Flink 1.12.0 on Yarn

HBase1.4

• External issue URL: https://issues.apache.org/jira/browse/HBASE-8626?focusedCommentId=13669455&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13669455
• External issue ID: HBASE-8626

Docs Text

I found that HBASE-8626 has a similar situation and uses discussion programs. Finally, the verification data can be aligned after the source code。

Language

Java

• Assignee: Unassigned
• Reporter: licp
• Created: 13/Oct/21 18:13
• Updated: 08/Dec/23 02:39

Description

Data flow direction is described below：

      Source：Mysql 

      Sink：HBase

      Parallelism：1

------------------Code Example One  Using Left Join ------------------
– Mysql Source ,Total Records:4829
create table user(
       user_id string,
       user_name string,
       primary key(user_id) not enforced
)with(
'connector' = 'mysql-cdc',
'hostname' = 'localhost',
'port' = '3308',
'username' = 'user_name',
'password' = '******',
'database-name' = 'database_name',
'table-name' = 'table_name',
'debezium.event.processing.failure.handling.mode' = 'warn',
'debezium.snapshot.locking.mode' = 'none'
);

create table user_profile(
         user_id string,
         age int,
         primary key(user_id) not enforced
)with(
'connector' = 'mysql-cdc',
'hostname' = 'localhost',
'port' = '3308',
'username' = 'user_name',
'password' = '******',
'database-name' = 'database_name',
'table-name' = 'table_name',
'debezium.event.processing.failure.handling.mode' = 'warn',
'debezium.snapshot.locking.mode' = 'none'
);

– HBase sink ;Total Record:4826
create table real_dwd_user_info_to_hbase(
     rowkey string,
    f ROW(user_name string,age int)
)with(
'connector' = 'hbase-1.4',
'table-name' = 'table_name',
'zookeeper.quorum' = 'zk',
'zookeeper.znode.parent' = '/hbase'
);

insert into real_dwd_user_info_to_hbase
select
     u.user_id,
     row(u.user_name,up.age) as
from user u
left join user_profile up
on u.user_id = up.user_id

where u.user_id<1000000

;

---------Code Example Two  Using Left Join  And  Specify some key--------

insert into real_dwd_user_info_to_hbase
select
     u.user_id,
     row(u.user_name,up.age) as
from user u
left join user_profile up
on u.user_id = up.user_id

where u.user_id=0
-------------------------------------------------------------------------------

I printed the same code logic results, I can find that the specified key has three results of " + i -d + i", but in HBase is still not find the key