Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2468 Netty-based block server / client module
  3. SPARK-2940

Support fetching multiple blocks in a single request in Netty network module

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      ShuffleCopier.getBlock gets one block at a time (in one request).

        def getBlock(host: String, port: Int, blockId: BlockId,
            resultCollectCallback: (BlockId, Long, ByteBuf) => Unit) {
      
          val handler = new ShuffleCopier.ShuffleClientHandler(resultCollectCallback)
          val connectTimeout = conf.getInt("spark.shuffle.netty.connect.timeout", 60000)
          val fc = new FileClient(handler, connectTimeout)
      
          try {
            fc.init()
            fc.connect(host, port)
            fc.sendRequest(blockId.name)
            fc.waitForClose()
            fc.close()
          } catch {
            // Handle any socket-related exceptions in FileClient
            case e: Exception => {
              logError("Shuffle copy of block " + blockId + " from " + host + ":" + port + " failed", e)
              handler.handleError(blockId)
            }
          }
        }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rxin Reynold Xin
                Reporter:
                rxin Reynold Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: