[SPARK-26139] Support passing shuffle metrics to exchange operator

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Target Version/s: 3.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 21/Nov/18 14:49
• Updated: 10/Dec/18 21:58
• Resolved: 10/Dec/18 21:58

Description

Due to the way Spark's architected (SQL is defined on top of the RDD API), there are two separate metrics system used in core vs SQL. Ideally, we'd want to be able to get the shuffle metrics for each of the exchange operator independently, e.g. blocks read, number of records.

 

Issue Links

• Is contained by: SPARK-26221 Improve Spark SQL instrumentation and metrics

Sub-Tasks

• Enable custom shuffle metrics implementation in shuffle reader: Resolved
• Enable custom shuffle metrics implementation in shuffle write: Resolved
• Implement shuffle read metrics in SQL: Resolved
• Implement shuffle write metrics in SQL: Resolved