[SPARK-24539] HistoryServer does not display metrics from tasks that complete after stage failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.1
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: Imran Rashid
• Created: 13/Jun/18 03:17
• Updated: 05/Sep/18 16:56
• Resolved: 05/Sep/18 16:56

Description

I noticed that task metrics for completed tasks with a stage failure do not show up in the new history server. I have a feeling this is because all of the tasks succeeded after the stage had been failed (so they were completions from a "zombie" taskset). The task metrics (eg. the shuffle read size & shuffle write size) do not show up at all, either in the task table, the executor table, or the overall stage summary metrics. (they might not show up in the job summary page either, but in the event logs I have, there is another successful stage attempt after this one, and that is the only thing which shows up in the jobs page.) If you get task details from the api endpoint (eg. http://[host]:[port]/api/v1/applications/[app-id]/stages/[stage-id]/[stage-attempt]) then you can see the successful tasks and all the metrics

Unfortunately the event logs I have are huge and I don't have a small repro handy, but I hope that description is enough to go on.

I loaded the event logs I have in the SHS from spark 2.2 and they appear fine.

Issue Links

• duplicates: SPARK-24415 Stage page aggregated executor metrics wrong when failures