[TEZ-4407] Misleading split info in TezSplitGrouper logs when adjusting small splits

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.1
• Fix Version/s: 0.10.5
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 20/Apr/22 16:42
• Updated: 23/Dec/24 08:12

Description

The log message in [TezSplitGrouper.getGroupedSplits|
https://github.com/apache/tez/blob/627f33077480afdcefc0611fbde87d6be0010176/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/grouper/TezSplitGrouper.java#L272] is misleading and quite often it may show that the "Desired splits" is the same with "New desired splits" which does not make much sense.

2022-04-19 01:59:05,064 [INFO] [App Shared Pool - #18] |grouper.TezSplitGrouper|: Desired splits: 4 too large.  Desired splitLength: 7589213 Min splitLength: 268435456 New desired splits: 4 Final desired splits: 4 All splits have localhost: false Total length: 1047311531 Original splits: 18

Due to the above it is difficult/impossible to see what was the initial desired splits without reading the code.

This was caused by TEZ-3291.

Issue Links

• is caused by: TEZ-3291 Optimize splits grouping when locality information is not available
• links to: GitHub Pull Request #202