[HUDI-199] Fix inconsistency between File creation and HBase Index mapping when spark DAG is re-computed

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.5.0
• Component/s: index
• Assignee: Venkatesh Rudraraju
• Reporter: Venkatesh Rudraraju
• Created: 04/Aug/19 03:57
• Updated: 28/Aug/19 17:21

Description

• When WriteStatus RDD lineage is re-triggered, index is not updated with new File ID created.
• This is happening because updateLocation() is returning back input WriteStatus RDD, which does not have the index update operation chained in it.
• While addressing this inconsistency issue, we also need to cache the WriteStatus RDD before updating index, so that there are less chances of the RDD being re-triggered.

Issue Links

• links to: Fix inconsistent mapping between File creation and HBase index