[SPARK-42204] Remove redundant logging of TaskMetrics internal accumulators in JsonProtocol event logs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 27/Jan/23 00:25
• Updated: 06/Sep/24 22:58
• Resolved: 06/Sep/24 22:58

Description

Spark's JsonProtocol event logs (used by the history server) contain redundancy in how TaskMetrics are represented in SparkListenerTaskEnd events:

• The "Task Metrics" field is a map from metric names to values.
• Under the hood, each metric is implemented using an accumulator and those accumulator values are redundantly stored in the `Task Info`.`Accumulables` field. These Accumulable entries contain the metric value from the task, plus the cumulative "sum-so-far" from the completed tasks in that stage.

The Spark History Server doesn't rely on the redundant information in the Accumulables field.

I believe that this redundancy was introduced back in SPARK-10620 when Spark 1.x's separate TaskMetrics implementation was replaced by the current accumulator-based version.

I think that we should eliminate this redundancy by skipping JsonProtocol logging of the TaskMetric accumulators. Although I think it's somewhat unlikely that third-party code is relying on the presence of that redundant information, I think we should hedge by adding an internal configuration flag to re-enable the redundant logging if needed.

Issue Links

• links to: [Github] Pull Request #39763 (JoshRosen)
• links to: GitHub Pull Request #39763