Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.2.0
-
None
Description
Hit this on 2.2, but probably has been there forever. This is similar in spirit to SPARK-20205.
Event is sent here, around L1154:
listenerBus.post(SparkListenerTaskEnd( stageId, task.stageAttemptId, taskType, event.reason, event.taskInfo, taskMetrics))
Accumulators are updated later, around L1173:
val stage = stageIdToStage(task.stageId) event.reason match { case Success => task match { case rt: ResultTask[_, _] => // Cast to ResultStage here because it's part of the ResultTask // TODO Refactor this out to a function that accepts a ResultStage val resultStage = stage.asInstanceOf[ResultStage] resultStage.activeJob match { case Some(job) => if (!job.finished(rt.outputId)) { updateAccumulators(event)
Same thing applies here; UI shows correct info because it's pointing at the mutable TaskInfo structure. But the event log, for example, may record the wrong information.
Attachments
Issue Links
- blocks
-
SPARK-18085 SPIP: Better History Server scalability for many / large applications
-
- Resolved
-
- is duplicated by
-
SPARK-21009 SparkListenerTaskEnd.taskInfo.accumulables might not be accurate
-
- Resolved
-
- links to