[HUDI-7478] Fix max delta commits guard check w/ MDT

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0, 1.0.0
• Component/s: metadata
• Labels: pull-request-available
• Assignee: Vova Kolmakov Assign to me
• Reporter: sivabalan narayanan
• Created: 04/Mar/24 16:21
• Updated: 09/Jun/24 23:07
• Resolved: 15/Mar/24 00:28

Description

protected static void checkNumDeltaCommits(HoodieTableMetaClient metaClient, int maxNumDeltaCommitsWhenPending) {
final HoodieActiveTimeline activeTimeline = metaClient.reloadActiveTimeline();
Option<HoodieInstant> lastCompaction = activeTimeline.filterCompletedInstants()
.filter(s -> s.getAction().equals(COMPACTION_ACTION)).lastInstant();
int numDeltaCommits = lastCompaction.isPresent()
? activeTimeline.getDeltaCommitTimeline().findInstantsAfter(lastCompaction.get().getTimestamp()).countInstants()
: activeTimeline.getDeltaCommitTimeline().countInstants();
if (numDeltaCommits > maxNumDeltaCommitsWhenPending)

{ throw new HoodieMetadataException(String.format("Metadata table's deltacommits exceeded %d: " + "this is likely caused by a pending instant in the data table. Resolve the pending instant " + "or adjust `%s`, then restart the pipeline.", maxNumDeltaCommitsWhenPending, HoodieMetadataConfig.METADATA_MAX_NUM_DELTACOMMITS_WHEN_PENDING.key())); }

}

Here we account for action type "compaction. But compaction completed instant will have "commit" as action. So, we need to fix it. 

 

 

 

Issue Links

• links to: GitHub Pull Request #10820