[HUDI-3068] Add support to sync all partitions in hive sync tool

• Type: New Feature
• Status: In Progress
• Priority: Blocker
• Resolution: Unresolved
• Component/s: meta-sync
• Labels: pull-request-available sev:critical
• Assignee: Harshal Patil
• Reporter: sivabalan narayanan
• Created: 20/Dec/21 02:53
• Updated: 21/Dec/23 15:33

Description

If a user runs hive sync occationally and if archival kicked in and trimmed some commits and if there were partitions added during those commits which was never updated later, hive sync will miss out those partitions. 

  LOG.info("Last commit time synced is " + lastCommitTimeSynced.get() + ", Getting commits since then");
  return TimelineUtils.getPartitionsWritten(metaClient.getActiveTimeline().getCommitsTimeline()
      .findInstantsAfter(lastCommitTimeSynced.get(), Integer.MAX_VALUE));
} 

bcoz, we for recurrent syncs, we always fetch new commits from timeline after the last synced instant and fetch commit metadata and go on to fetch the partitions added as part of it. 

 

We can add a new config to hive sync tool to override this behavior. 

--sync-all-partitions 

when this config is set to true, we should ignore last synced instant and should go the below route which is done when syncing for the first time. 

 

if (!lastCommitTimeSynced.isPresent()) {
  LOG.info("Last commit time synced is not known, listing all partitions in " + basePath + ",FS :" + fs);
  HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(metaClient.getHadoopConf());
  return FSUtils.getAllPartitionPaths(engineContext, basePath, useFileListingFromMetadata, assumeDatePartitioning);
} 

 

 

Ref issue: 

https://github.com/apache/hudi/issues/3890

Issue Links

• links to: GitHub Pull Request #4405