Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1912

Datanode should support block replacement

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.14.1
    • 0.16.0
    • None
    • None

    Description

      This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy. The block placement is carried in the following steps:
      1. A block copy command is sent to datanode PS in the format of "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
      2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>".
      3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
      4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.

      In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed.

      Attachments

        1. replace6.patch
          38 kB
          Hairong Kuang
        2. replace5.patch
          38 kB
          Hairong Kuang
        3. replace4.patch
          38 kB
          Hairong Kuang
        4. replace3.patch
          37 kB
          Hairong Kuang
        5. replace2.patch
          35 kB
          Hairong Kuang
        6. replace1.patch
          28 kB
          Hairong Kuang
        7. replace.patch
          66 kB
          Hairong Kuang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hairong Hairong Kuang
            hairong Hairong Kuang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment