Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

The performance of the drop function is bad.

• Log the time consuming:

        if (stageData.taskData.size > retainedTasks) {
          val start = System.currentTimeMillis()
          stageData.taskData = stageData.taskData.drop(stageData.taskData.size - retainedTasks)
          logInfo(s"Time consuming: ${System.currentTimeMillis() - start}")
        }
  

• Time consuming:

  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 156
  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 145
  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 148
  17/01/10 14:04:05 INFO JobProgressListener: Time consuming: 159
  

My opinion is drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop.