[HUDI-6748] Enhance TransactionManager to support multi-table transaction

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Epic Link: hudi-multi-txn
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit
• Created: 25/Aug/23 02:56
• Updated: 25/Mar/24 11:30

Description

Transaction Manager should be database timeline-aware and return a transaction instance when a transaction is started, which can be used with dataframe writes as below.

val transaction = HoodieTransactionManager.beginTransaction()
df1.write.format("hudi").withTransaction(transaction).options(opts1).save(basePath1)
df2.write.format("hudi").withTransaction(transaction).options(opts2).save(basePath2) 
// All success, go ahead and commit or finish rollback (ROLLED_BACK state on timeline is written in that case)
transaction.commit()