Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2559

DFS should place one replica per rack

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.17.0
    • None
    • None
    • Change DFS block placement to allocate the first replica locally, the second off-rack, and the third intra-rack from the second.

    Description

      Currently, when writing out a block, dfs will place one copy to a local data node, one copy to a rack local node
      and another one to a remote node. This leads to a number of undesired properties:

      1. The block will be rack-local to two tacks instead of three, reducing the advantage of rack locality based scheduling by 1/3.

      2. The Blocks of a file (especiallya large file) are unevenly distributed over the nodes: One third will be on the local node, and two thirds on the nodes on the same rack. This may make some nodes full much faster than others,
      increasing the need of rebalancing. Furthermore, this also make some nodes become "hot spots" if those big
      files are popular and accessed by many applications.

      Attachments

        1. HADOOP-2559-1.patch
          8 kB
          Lohit Vijaya Renu
        2. HADOOP-2559-2.patch
          11 kB
          Lohit Vijaya Renu
        3. Trunk_Block_Report.png
          30 kB
          Lohit Vijaya Renu
        4. Patch1_Block_Report.png.jpg
          52 kB
          Lohit Vijaya Renu
        5. Patch2 Block Report.jpg
          56 kB
          Lohit Vijaya Renu
        6. Trunk_Rack_Node_Mapping.jpg
          33 kB
          Lohit Vijaya Renu
        7. Patch1_Rack_Node_Mapping.jpg
          34 kB
          Lohit Vijaya Renu
        8. Patch2_Rack_Node_Mapping.jpg
          35 kB
          Lohit Vijaya Renu
        9. HADOOP-2559-1.patch
          8 kB
          Lohit Vijaya Renu
        10. HADOOP-2559-1-2.patch
          7 kB
          Lohit Vijaya Renu
        11. HADOOP-2559-1-3.patch
          7 kB
          Lohit Vijaya Renu
        12. HADOOP-2559-1-4.patch
          7 kB
          Lohit Vijaya Renu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            lohit Lohit Vijaya Renu
            runping Runping Qi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment