[SPARK-15569] Executors spending significant time in DiskObjectWriter.updateBytesWritten function

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 2.0.0
• Assignee: Sital Kedia
• Reporter: Sital Kedia
• Created: 26/May/16 17:48
• Updated: 17/May/20 18:31
• Resolved: 27/May/16 18:23

Description

Profiling a Spark job spilling large amount of intermediate data we found that significant portion of time is being spent in DiskObjectWriter.updateBytesWritten function. Looking at the code (https://github.com/sitalkedia/spark/blob/master/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala#L206), we see that the function is being called too frequently to update the number of bytes written to disk. We should reduce the frequency to avoid this.

Issue Links

• links to: [Github] Pull Request #13332 (sitalkedia)