[IMPALA-12758] Event Processor is ignoring the prev_id while reloading the existing partitions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.5.0
• Component/s: Catalog
• Labels: catalog-2024
• Epic Link: event-processor-completeness
• Epic Color: ghx-label-11
• Assignee: Sai Hemanth Gantasala
• Reporter: Sai Hemanth Gantasala
• Created: 25/Jan/24 20:12
• Updated: 2 days ago 17:32
• Resolved: 2 days ago 17:32

Description

Partitioned table Insert events consumed by the event processor reloads the partitions to update file metadata. Currently while reloading the filemetadata, 'prev_id' of the old partitions is being ignored in the partition builder

HdfsPartition oldPartition = entry.getValue();
HdfsPartition.Builder partBuilder = createPartitionBuilder(
   hmsPartition.getSd(), hmsPartition, permissionCache); 

As a result 'prev_id' of the partBuilder will always be -1, so when the catalogDelta is sent from the state store to impala demons, because prev_id is not valid, impalads will not know whether to invalidate the current partition and then request the new partition information. 

This might lead to data correctness issues.