[SPARK-9026] SimpleFutureAction.onComplete should not tie up a separate thread for each callback

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 1.6.0
• Assignee: Richard W. Eggert II
• Reporter: Josh Rosen
• Created: 14/Jul/15 02:40
• Updated: 16/Dec/15 03:49
• Resolved: 16/Dec/15 02:22

Description

As zsxwing points out at https://github.com/apache/spark/pull/7276#issuecomment-121097747, SimpleFutureAction currently blocks a separate execution context thread for each callback registered via onComplete:

  override def onComplete[U](func: (Try[T]) => U)(implicit executor: ExecutionContext) {
    executor.execute(new Runnable {
      override def run() {
        func(awaitResult())
      }
    })
  }

We should fix this so that callbacks do not steal threads.

Issue Links

• is duplicated by: SPARK-11296 FutureAction.onComplete consumes a thread while waiting for job
• links to: [Github] Pull Request #7385 (JoshRosen)
• links to: [Github] Pull Request #9264 (reggert)