[SPARK-20084] Remove internal.metrics.updatedBlockStatuses accumulator from history files

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.2, 2.2.0
• Component/s: Spark Core, Web UI
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 24/Mar/17 19:33
• Updated: 27/Apr/17 22:20
• Resolved: 31/Mar/17 16:43

Description

History files for large jobs can be hundreds of GB. These history files take too much space and create a backlog on the history server.

Most of the size is from Accumulables in SparkListenerTaskEnd. The largest accumulable is internal.metrics.updatedBlockStatuses, which has a small update (the blocks that were changed) but a huge value (all known blocks). Nothing currently uses the accumulator value or update, so it is safe to remove it. Information for any block updated during a task is also recorded under Task Metrics / Updated Blocks.

Issue Links

• is duplicated by: SPARK-16333 Excessive Spark history event/json data size (5GB each)
• links to: [Github] Pull Request #17412 (rdblue)