[SPARK-30092] Number of active tasks is negative in Live UI Executors page

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.1
• Component/s: Web UI
• Labels: bulk-closed

Environment

Hadoop version: 2.7.3

ResourceManager version: 2.7.3

• Assignee: Unassigned
• Reporter: Yu Zhong
• Created: 02/Dec/19 02:09
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:44

Description

The number of active tasks is negative in Live UI Executors page when there is executor lost and task failure. I am using spark on yarn which built on AWS spot instances. When yarn work lost, there is a large probability to become negative active tasks in Spark Live UI.  

I saw related tickets below and resolved in earlier version of Spark. But Same things happened again in Spark 2.4.1. See attachment.

https://issues.apache.org/jira/browse/SPARK-8560

https://issues.apache.org/jira/browse/SPARK-10141

https://issues.apache.org/jira/browse/SPARK-19356