[HUDI-6659] Remove the validateRollback restriction for Spark/Flink MDT rollback

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: metadata
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Askwang
• Created: 07/Aug/23 03:25
• Updated: 28/Mar/24 22:39

Description

Spark offline compaction rollbacks failed inflight instant throw this exception info when mdt enabled.

org.apache.hudi.exception.HoodieMetadataException: Commit being rolled back 20230802174236306 is earlier than the latest compaction 20230803201423881001. There are 3 deltacommits after this compaction: [[20230803201545303__deltacommit__COMPLETED__20230803201721395], [20230803201729007__deltacommit__COMPLETED__20230803201848687], [20230803201852499__deltacommit__COMPLETED__20230803202010862]]
    at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.validateRollback(HoodieBackedTableMetadataWriter.java:1034)
    at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1002)
    at org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
    at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
    at org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
    at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:256)
    at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:118)
    at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:139)
    at org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:218)
    at org.apache.hudi.table.HoodieTable.rollbackInflightInstant(HoodieTable.java:650)
    at org.apache.hudi.table.HoodieTable.rollbackInflightCompaction(HoodieTable.java:623)
    at org.apache.hudi.client.SparkRDDTableServiceClient.compact(SparkRDDTableServiceClient.java:80)
    at org.apache.hudi.client.SparkRDDWriteClient.compact(SparkRDDWriteClient.java:307)
    at org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1034)
    at org.apache.hudi.utilities.HoodieCompactor.doCompact(HoodieCompactor.java:306) 

Issue Links

• links to: GitHub Pull Request #9378