[SPARK-20801] Store accurate size of blocks in MapStatus when it's above threshold.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Jin Xing
• Reporter: Jin Xing
• Created: 18/May/17 15:45
• Updated: 22/May/17 14:13
• Resolved: 22/May/17 14:12

Description

Currently, when number of reduces is above 2000, HighlyCompressedMapStatus is used to store size of blocks. in HighlyCompressedMapStatus, only average size is stored for non empty blocks. Which is not good for memory control when we shuffle blocks. It makes sense to store the accurate size of block when it's above threshold.

Issue Links

• links to: [Github] Pull Request #18031 (jinxing64)