[SPARK-40617] Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.2.1, 3.3.0, 3.2.2
• Fix Version/s: 3.2.0, 3.3.1, 3.4.0
• Component/s: Spark Core
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 29/Sep/22 19:42
• Updated: 06/May/23 16:32
• Resolved: 03/Oct/22 13:31

Description

Spurious failures because of the assert:

22/09/29 09:46:24 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 3063.0 in stage 1997.0 (TID 677249),5,main]
java.lang.AssertionError: assertion failed: task count shouldn't below 0
	at scala.Predef$.assert(Predef.scala:223)
	at org.apache.spark.executor.ExecutorMetricsPoller.decrementCount$1(ExecutorMetricsPoller.scala:130)
	at org.apache.spark.executor.ExecutorMetricsPoller.$anonfun$onTaskCompletion$3(ExecutorMetricsPoller.scala:135)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1822)
	at org.apache.spark.executor.ExecutorMetricsPoller.onTaskCompletion(ExecutorMetricsPoller.scala:135)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:737)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
22/09/29 09:46:24 INFO MemoryStore: MemoryStore cleared
22/09/29 09:46:24 INFO BlockManager: BlockManager stopped
22/09/29 09:46:24 INFO ShutdownHookManager: Shutdown hook called
22/09/29 09:46:24 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1664443624160_0001/spark-93efc2d4-84de-494b-a3b7-2cb1c3a45426

Issue Links

• is related to: SPARK-34779 ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs
• links to: [Github] Pull Request #38056 (attilapiros)
• links to: [Github] Pull Request #38056 (attilapiros)
• links to: [Github] Pull Request #38083 (attilapiros)
• links to: [Github] Pull Request #38083 (attilapiros)