[SPARK-8593] History Server doesn't show complete application when one attempt inprogress

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.2, 1.5.0
• Component/s: YARN
• Assignee: Rekha Joshi
• Reporter: Thomas Graves
• Created: 24/Jun/15 16:01
• Updated: 17/Jul/15 21:47
• Resolved: 17/Jul/15 21:47

Description

The Spark history server doesn't show an application if the first attempt of the application is still inprogress.

Here are the files in hdfs:
rwxrwx-- 3 tgraves hdfs 234 2015-06-24 15:49 sparkhistory/application_1433751980223_18926_1.inprogress
rwxrwx-- 3 tgraves hdfs 9609450 2015-06-24 15:51 sparkhistory/application_1433751980223_18926_2

The UI shows them if I set the showIncomplete=true.

Removing the inprogress file allows it to show up when showIncomplete is false.

It should be smart enough to atleast show the second successful attempt.

Issue Links

• is duplicated by: SPARK-8594 History Server doesn't show complete application when one attempt inprogress
• links to: [Github] Pull Request #7253 (rekhajoshm)