[SPARK-2677] BasicBlockFetchIterator#next can wait forever

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.2, 1.0.0, 1.0.1
• Fix Version/s: 1.1.0
• Component/s: Spark Core
• Target Version/s: 1.0.3, 1.1.0
• Assignee: Kousuke Saruta
• Reporter: Kousuke Saruta
• Created: 24/Jul/14 21:58
• Updated: 12/May/16 10:21
• Resolved: 16/Aug/14 21:17

Description

In BasicBlockFetchIterator#next, it waits fetch result on result.take.

    override def next(): (BlockId, Option[Iterator[Any]]) = {
      resultsGotten += 1
      val startFetchWait = System.currentTimeMillis()
      val result = results.take()
      val stopFetchWait = System.currentTimeMillis()
      _fetchWaitTime += (stopFetchWait - startFetchWait)
      if (! result.failed) bytesInFlight -= result.size
      while (!fetchRequests.isEmpty &&
        (bytesInFlight == 0 || bytesInFlight + fetchRequests.front.size <= maxBytesInFlight)) {
        sendRequest(fetchRequests.dequeue())
      }
      (result.blockId, if (result.failed) None else Some(result.deserialize()))
    }

But, results is implemented as LinkedBlockingQueue so if remote executor hang up, fetching Executor waits forever.

Issue Links

• is duplicated by: SPARK-2681 Spark can hang when fetching shuffle blocks
• relates to: SPARK-2717 BasicBlockFetchIterator#next should log when it gets stuck
• links to: [Github] Pull Request #1619 (witgo)
• links to: [Github] Pull Request #1632 (sarutak)