[SPARK-22540] HighlyCompressedMapStatus's avgSize is incorrect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Spark Core
• Target Version/s: 2.2.1, 2.3.0
• Assignee: yucai
• Reporter: yucai
• Created: 16/Nov/17 08:13
• Updated: 24/Nov/17 21:16
• Resolved: 17/Nov/17 13:55

Description

The calculation of HighlyCompressedMapStatus's avgSize is incorrect.
Currently, it looks like "sum of small blocks / count of all non empty blocks", the count of all non empty blocks not only contains small blocks, which contains huge blocks number also, but we need the count of small blocks only.

Issue Links

• links to: [Github] Pull Request #19765 (yucai)