[SPARK-19146] Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 10/Jan/17 07:08
• Updated: 06/Apr/17 14:18
• Resolved: 23/Jan/17 11:02

Description

The performance of the drop function is bad.

• Log the time consuming:

        if (stageData.taskData.size > retainedTasks) {
          val start = System.currentTimeMillis()
          stageData.taskData = stageData.taskData.drop(stageData.taskData.size - retainedTasks)
          logInfo(s"Time consuming: ${System.currentTimeMillis() - start}")
        }
  

• Time consuming:

  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 156
  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 145
  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 148
  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 159
  

My opinion is drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop.

Issue Links

• is duplicated by: SPARK-19068 Large number of executors causing a ton of ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(41,WrappedArray())
• links to: [Github] Pull Request #16527 (wangyum)