[SPARK-4168] Completed Stages Number are misleading webUI when stages are more than 1000

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.3.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Zhang, Liye
• Created: 31/Oct/14 09:28
• Updated: 12/Dec/14 12:42
• Resolved: 12/Dec/14 12:42

Description

The number of completed stages and failed stages showed on webUI will always be less than 1000. This is really misleading when there are already thousands of stages completed or failed. The number should be correct even when only partial of all stages listed on the webUI (stage info will be removed if the number is too large).

Issue Links

• is duplicated by: SPARK-3719 Spark UI: "complete/failed stages" is better to show the total number of stages
• links to: [Github] Pull Request #3035 (liyezhang556520)