[HUDI-204] Make MOR rollback idempotent and disable using rolling stats for small file selection

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: writer-core
• Labels: pull-request-available
• Assignee: Balaji Varadarajan
• Reporter: Balaji Varadarajan
• Created: 12/Aug/19 21:37
• Updated: 28/Aug/19 17:21
• Resolved: 22/Aug/19 04:54

Description

1. Rollback assumes all fileIds that are affected by rollback to be always present. This breaks when rollback fails intermittently and is retried

2. Sometime Rolling stats wrongly tracks file sizes to be 0. We had earlier discussed to completely revisit rolling stats. We will disable using rolling stats to prevent wrong small file selection

Issue Links

• links to: GitHub Pull Request #833