[HUDI-4631] Enhance retries for failed writes w/ write conflicts in a multi writer scenarios

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0-beta1
• Component/s: multi-writer
• Labels: pull-request-available
• Story Points: 3
• Epic Link: Multi-writer & OCC
• Assignee: Sagar Sumit
• Reporter: sivabalan narayanan
• Created: 16/Aug/22 20:29
• Updated: 28/Aug/23 11:18
• Resolved: 28/Aug/23 11:18

Description

lets say there are two writers from t0 to t5. so hudi fails w2 and succeeds w1. and user restarts w2 and for next 5 mins, lets say there are no other overlapping writers. So the same write from w2 will now succeed. so, whenever there is a write conflict and pipeline fails, all user needs to do is, just restart the pipeline or retry to ingest the same batch.

 

Ask: can we add retries within hudi during such failures. Anyways, in most cases, users just restart the pipeline in such cases. 

 

Issue Links

• is duplicated by: HUDI-5028 Handle conficting writes for multi-writers with OCC enabled
• links to: GitHub Pull Request #6854