[HUDI-1098] Marker file finalizing may block on a data file that was never written

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: writer-core
• Labels: pull-request-available
• Assignee: sivabalan narayanan
• Reporter: Vinoth Chandar
• Created: 15/Jul/20 04:54
• Updated: 24/Aug/20 18:17
• Resolved: 24/Aug/20 18:17

Description

// Ensure all files in delete list is actually present. This is mandatory for an eventually consistent FS. // Otherwise, we may miss deleting such files. If files are not found even after retries, fail the commit 
if (consistencyCheckEnabled) { 
  // This will either ensure all files to be deleted are present.     waitForAllFiles(jsc, groupByPartition, FileVisibility.APPEAR); 
}

We need to handle the case where marker file was created, but we crashed before the data file was created.

Issue Links

• links to: GitHub Pull Request #1912