Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
This causes some tests like testToWriteWithoutParametersIncludedInHoodieTableConfig taking longer time to finish than expected. For this particular one, reducing the upsert shuffle parallelism from default 200 to 2 makes the test finish in 15s instead of 90s locally.
The actual root cause of the problem is that the dedup parallelism is taken directly from the Hudi write shuffle parallelism, without auto-tuning.
Attachments
Issue Links
- links to