[HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.13.1, 0.12.3
• Component/s: timeline-server, writer-core
• Labels: pull-request-available
• Story Points: 6
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: Ethan Guo (this is the old account; please use "yihua")
• Created: 28/Feb/23 19:59
• Updated: 30/May/24 19:09
• Resolved: 07/Mar/23 03:18

Description

We observe that for MOR table, occasionally (<10% for large tables with frequent updates and compactions), the deltacommit after the compaction commit may add a new log file to the old file slice, not the latest file slice, in the corresponding file group.  This happens when both the metadata table and timeline server are enabled.  If either is disabled, the problem does not show up.

Deeper analysis of the code surfaces that the file system view at the timeline server may serve the stale view, causing the issue.  This is because the sync of HoodieMetadataFileSystemView is not atomic when the metadata table is enabled:

 

AbstractTableFileSystemView:

@Override
public void sync() {
  HoodieTimeline oldTimeline = getTimeline();
  HoodieTimeline newTimeline = metaClient.reloadActiveTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
  try {
    writeLock.lock();
    runSync(oldTimeline, newTimeline);
  } finally {
    writeLock.unlock();
  }
}

HoodieMetadataFileSystemView:

@Override
  public void sync() {
    super.sync();
    tableMetadata.reset();
  }

The
tableMetadata.reset()
happens outside the write lock, causing the issue during race conditions (reading the inconsistent state while the table metadata is being reset).

 

 

Issue Links

• links to: GitHub Pull Request #8079