[HUDI-5326] Incorrect cluster grouping in SparkSizeBasedClusteringPlanStrategy

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: clustering
• Labels: pull-request-available
• Assignee: Xinyu Zou
• Reporter: Xinyu Zou
• Created: 03/Dec/22 09:26
• Updated: 16/Feb/23 07:04
• Resolved: 16/Feb/23 07:04

Description

Currently, the size of the clusting group building in `SparkSizeBasedClusteringPlanStrategy` will be greater than `hoodie.clustering.plan.strategy.max.bytes.per.group`, which will cause the merged file size to be inconsistent with`hoodie.clustering.plan.strategy.target.file.max.bytes`

E.g:

we set max.bytes.per.group=20, target.file.max.bytes=10

if we have small files, which sizes are 4, 4, 4, 4, 3, 3, 3

4, 4, 4, 4, 3, 3 will be divided into one clusting groups, they will be merged into 22/10 = 3 files, which is inconsistent with `max.bytes.per.group`/`target.file.max.bytes`=2

and their average size will be 7, which has a big gap with 10.

Issue Links

• links to: GitHub Pull Request #7372