Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.4.0
-
None
-
None
Description
This is a followup to SPARK-36620:
When push-based shuffle is disabled (the default), I think that we should skip the logging of the new push-based shuffle read metrics. Because these metrics are logged for every task, they will add significant additional size to Spark event logs. It would be great to avoid this cost in cases where it's not necessary.
Attachments
Issue Links
- is caused by
-
SPARK-36620 Client side related push-based shuffle metrics
- Resolved