[HUDI-6864] Auto-tune dedup parallelism without affecting write parallelism

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Assignee: Unassigned
• Reporter: Ethan Guo
• Created: 15/Sep/23 05:13
• Updated: 21/Dec/23 15:28

Description

See HUDI-6863 and https://github.com/apache/hudi/pull/6802#issuecomment-1455802492

I think we need to make sure that the dedup parallelism is only applied to the dedup stage, not affecting subsequent stages, which may require better parallelism control by repartitioning with right parallelism before workload profiling.