[HUDI-648] Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.14.0
• Component/s: deltastreamer, writer-core
• Labels: pull-request-available sev:normal user-support-issues
• Assignee: liujinhui
• Reporter: Vinoth Chandar
• Created: 28/Feb/20 17:10
• Updated: 29/Mar/24 01:43

Description

We would like a way to hand the erroring records from writing or compaction back to the users, in a separate table or log. This needs to work generically across all the different writer paths.

Issue Links

• depends upon: HUDI-944 Support more complete concurrency control when writing data
• is a parent of: HUDI-1505 Allow pluggable option to write error records to side table, queue
• is a parent of: HUDI-5207 Add error handling logic flow for WriteStatus in spark sql writer
• links to: GitHub Pull Request #2710
• links to: GitHub Pull Request #3308
• links to: GitHub Pull Request #3309
• links to: GitHub Pull Request #3312
• mentioned in: Page Loading...