[SPARK-30458] The Executor Computing Time in Time Line of Stage Page is Wrong

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Web UI
• Assignee: SongXun
• Reporter: SongXun
• Created: 08/Jan/20 11:24
• Updated: 12/Jan/20 04:11
• Resolved: 12/Jan/20 04:09

Description

The Executor Computing Time in Time Line of Stage Page is Wrong. It includes the Scheduler Delay Time, while the Proportion excludes the Scheduler Delay

val executorOverhead = serializationTime + deserializationTime
val executorRunTime = if (taskInfo.duration.isDefined)

{ totalExecutionTime - executorOverhead - gettingResultTime }

else

{ metricsOpt.map(_.executorRunTime).getOrElse( totalExecutionTime - executorOverhead - gettingResultTime) }

val executorComputingTime = executorRunTime - shuffleReadTime - shuffleWriteTime
val executorComputingTimeProportion =
math.max(100 - schedulerDelayProportion - shuffleReadTimeProportion -
shuffleWriteTimeProportion - serializationTimeProportion -
deserializationTimeProportion - gettingResultTimeProportion, 0)

Issue Links

• links to: GitHub Pull Request #27135