[HUDI-6460] Fix Hbase Index for deletes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: index
• Assignee: Prashant Wason
• Reporter: sivabalan narayanan
• Created: 29/Jun/23 20:18
• Updated: 30/Jun/23 17:06

Description

With  adding delete support for RLI, https://github.com/apache/hudi/pull/9058/files 

Hbase index needs some fixes. 

Test that is failing is:

TestSparkHoodieHBaseIndex.

testTagLocationAndPartitionPathUpdateWithExplicitRollback

 

Root cause:

when update partition path is set to true, within same batch we have a deleted record and a new insert record. So, to hbase we are sending both the records and for some inserts take precedence, while for others deletes take precedence. 

 

we need to fix SparkHoodieHbaseIndex.

updateLocation

to do one pass overWriteStatus and ensure we de-dup if we have two records where one of them is deleted and another is inserted. 

but there are chances that only deletes are present, so in such cases, we need to ensure deletes are routed to hbase.